One way organizations can improve their security and operational ability is to collect logs in a central location. Centralized logging allows engineers across the entire organization to have a "common view" of the system under load, and can provide vital shared context when things go wrong.
Over the last few months, we at Threat Stack have been reworking how we handle all aspects of our logging system. This project encompasses everything, from the content of our log data to the infrastructure that collects it. In this post you'll learn about how our internal applications send log data, where they send it to, and the trade offs we considered in making our collection system reliable.