[Recap] Detection and Response: Unveiling the Azazel Rootkit Webinar

Yesterday we hosted a webinar on “Detection and Response: Unveiling the Azazel Rootkit”. It was our first tech talk webinar and we had an amazing turn out! Thanks to those of you who joined us. For anyone who was unable to make it, we have the slide deck (above) and recording (below) for you so you won’t miss a beat.

Here’s what we covered:

  • Our co-founder, Dustin Webber, and CEO, Doug Cahill, led a tech talk on what it means for a security platform to be designed for the cloud.

  • They then dove into the cloud threat landscape and the increased attack surface area based on some recent examples.

  • Dustin explained why legacy on-premise security solutions don’t meet the requirements of cloud environments.

  • Last, Dustin demonstrated how Cloud Sight was able to detect the Azazel rootkit through a live case study.

Tune into the entire webinar recording here: http://vimeo.com/89736605

Interested in joining the Threat Stack beta program? We’re opening the doors to more beta users this week. You can request an invite here.

Stay tuned for many more updates coming soon!


[Webinar] Beyond Detection and Response: Unveiling the Azazel Rootkit

Now that the dust has finally settled after RSA and we’ve had a chance to come up for a breath of fresh air, we’re excited to announce that we’re hosting our first webinar!

On Wednesday, March 19th at 1pm (one week from today) we will be presenting a webinar on “Detection, Response and the Azazel Compromise”. It will be, first and foremost, a cloud security tech talk with our CEO, Doug Cahill, and one of our founders, Dustin Webber, on the state of threat detection and response for cloud-based servers.

You’ll also get to see Cloud Sight in action, detecting and responding to attacks in the cloud in real-time. Specifically, we’ll be investigating a cloud asset compromised by the open source Azazel rootkit (https://github.com/chokepoint/azazel). The Azazel rootkit has received a lot of notoriety recently due to its heavy focus on anti-debugging, anti-detection and PCAP hooking capabilities. We will trace the compromise from start to finish and highlight the collection abilities of our Cloud Sight sensor.

And of course, we’ll be having a Q&A at the end if you have any questions at all for Doug or Dustin!

Please click here to sign up! We look forward to talking to you then.


RSA Grows with the Attack Surface Area

Wow -- what a busy (and awesome) time at RSA this year! I missed the conference last year and was surprised (but impressed) to see the event grow as much as it has in such a short period of time; they literally doubled the exhibitor space, filling both the North and South Halls. While on one hand surprising, the growth makes complete sense -- not just because security is a hot space, but because of the massive increase in the attack surface area.

Mobility and the multi-device user, along with the rapid adoption of all things cloud -- from SaaS applications to IaaS and PaaS -- has a multiplicative effect of increasing both the vectors and targets for attacks.

Let’s review two recent compromises to highlight the changing attack surface that RSA proved true:

MongoHQ, the database-as-a-service company, was recently compromised. As a result, this jeopardized customer data. SaaS companies have a growing challenge because they are the single attack surface to not just one company but every company that leverages their offering.

This is the Holy Grail to most attackers because the cost involved to infiltrate is overwhelming justified by payout.

Next up is Github, a very popular source code hosting company that recently started a public security bug bounty program. One of the recent submissions demonstrated how an attacker could achieve remote code execution by leveraging a configuration issue with how code is pushed to their backend.

It is completely unknown if this was actively used in the wild before it was reported. Additionally, it’s impossible to know or prove this -- unless companies established a way to gain visibility into their infrastructure and view historical data. Much like MongoHQ, this was not only a security concern for Github's infrastructure, but a concern for the data of all their customers.

All that said, and as proven at RSA, the cloud is not inherently more secure but it’s not less secure either. The truth of the matter is it’s exactly the same as any other type of infrastructure -- and the security challenges are the same. SaaS, however, does make you more of a target because of the amount data diversity an attacker will gain from compromising assets. This was a massive takeaway both for our team and the general audience at RSA -- and one that is sure to cause waves of change in 2014.

The current status of the cloud security industry, which was made abundantly clear at RSA, just sets us even more vigorously on our mission to protect cloud-based workloads and explain why having a cloud-ready by design implementation is the right way to gain the visibility and control that companies actually need.

Looking forward, we will be leading many exciting discussions in the market around what it means to protect elastic infrastructure with an elastic solution and how DevOps can proactively take command of their part of the shared responsibility security model that comes with leveraging the tremendous benefits of the cloud.

(And stay tuned -- we also have several exciting announcements for you very soon!)

Exacom Uses Threat Stack For Network Detection and Protection at the Host Level


Screen Shot 2014-02-21 at 11.53.49 AM.png

Located in Central Europe, Exacom provides virtualization, hosting and managed services and connectivity that enables customers to manage and enhance their cloud infrastructure.

As developers, system and network engineers and architects themselves, they understand the importance of providing safe, secure and high quality infrastructure services by proactively monitoring the security of their customers’ cloud environments.


Exacom takes security seriously -- especially when it comes to their customers’ data. The company already provides enterprise-grade security to its customers, but also believes strongly in having a second level of defense. With their own Tier 4 datacenter they have implemented high-level perimeter hardening, but lacked the necessary insights to instantly detect and track unusual server activity.

Lefteris Lertas, CTO of Exacom, needed peace of mind from a monitoring perspective that if any level of intrusion occurs -- especially at the host level -- he would be alerted immediately. In parallel, more and more Exacom customers were asking for even deeper levels of security, leading Lertas to seek out the right cloud security monitoring solution.


The opportunity for Lertas to define a better way to manage the security of their service, coupled with his existing use of Snorby, the widely deployed front end management tool for popular open source intrusion detection systems such as Snort and Suricata, brought him to Threat Stack.  Snorby was developed by Dustin Webber, Co-Founder of Threat Stack, and continues to be maintained by the company. Because of Lertas’ immense respect for Snorby, he was eager to speak with Threat Stack when the flagship product, Cloud Sight, was launched.

After speaking with the Threat Stack team and implementing Cloud Sight on several of their servers, Lertas quickly realized that Cloud Sight was exactly what they needed to add that second level of defense and receive real-time activity alerts.

Exacom’s existing perimeter hardening was strong, but it was only rules based. Since Cloud Sight lives on their servers, the team was instantly able to gain server-related insights to augment their network IDS. This now helps them truly harden their perimeter as Cloud Sight provides analysis of where processes are being run from.

After gaining these immediate insights into the security of their networks, it became obvious for Lertas to implement Cloud Sight on many more of their cloud servers. Using Cloud Sight, the Exacom team is now able to monitor, detect and respond to any potentially malicious activity on their cloud servers.


Since installing Cloud Sight by Threat Stack, Exacom is now able to:

  • Gain real-time visibility at the host level

  • Truly harden their perimeter defenses

  • Understand exactly who is doing what on Exacom’s servers

  • Give customers a powerful new level of security

  • Receive both instant alerts and daily reports to understand exactly what is happening on their servers each day

Unlike other legacy security solutions, Cloud Sight is built for dynamic cloud environments and installs in under a minute, alleviating resources so that Exacom can instead focus on improving their core products for their growing number of European customers.

If you are interested in deploying Cloud Sight’s security monitoring solution for your business, visit http://threatstack.com or contact us today at sales@threatstack.com.


Threat Stack January Update

At Threat Stack, we’ve started the new year off with a bang by getting straight to work on many new exciting features that we can’t wait to share with you! No more eggnog, traveling to the in-laws’ house or singing carols -- our heads are down making our flagship product Cloud Sight better than ever.

Performance & Speed

This past month we worked tirelessly to make the task of tracking, auditing, and displaying forensic information for an organization even easier and faster than before.

To see an overview of how this now works, take a peek at this screencast demonstrating a vulnerable cloud deployment. As you saw (you watched the screencast, right?!) because of poor user and password management, multiple systems were compromised. Just like Superman, Cloud Sight swept in and reconstructed a full timeline of events -- from the introduction of a new user right down to the nitty-gritty details of the rootkit installed after the compromise.

Cutting-Edge Dashboard Visualization

We’re also excited to announce that the Cloud Sight analytics dashboard is now available!  This feature provides an overview of activity across your organization for any given time period (by default, the window of time is 24 hours, however, this is configurable).


One-Click Access to Login Activity

Want a quick overview of what your users are doing and where they are logging in from?  With our new analytics dashboard and login tracking, you can now get a bird’s-eye view of account activity across your infrastructure.

Seamless View of User Sessions

Each login entry is now linked to an audit history of commands that were run by a user. But we didn’t stop there -- you can also follow a user’s session through any account transition. For example, if a user changes credentials, (e.g., `sudo su`) we will display the history of commands run under the new identity within a single window.

From here, you can click on individual commands to view detailed information on that specific process, such as network connections, file events (if applicable), and more.

Pretty sweet, right?!  (By the way, you need to first  upgrade the agent to v1.0.8 to take advantage of these features -- it’s quick, we promise).

Unified Process & Network History

We worked several long nights (and chugged lots of caffeine-related products) to also give you the ability to visualize all of your new process and network activity across your entire organization -- right from the dashboard:


Effortless File Integrity Monitoring

We’ve made it possible for you to now determine (in seconds) the files a process may have modified, created, or accessed.

With Cloud Sight’s new file integrity monitoring capabilities, we associate for you the file activity and process activity -- right in the “Process Detail” page.

Screen Shot 2014-01-29 at 6.59.12 PM.png

Improved Search For Processes

Using Cloud Sight’s search interface, you can query for a specific filename (below shows filename like ‘authorized_keys’). From the search results, click on the ‘View Process Details’ eyeball which then displays detailed information about the process that accessed, created, deleted, or modified the file. It’s like a surveillance camera on all of your servers!


Meaningful Visualizations For Statistical Information

Many of the Cloud Sight widgets now include an option to display statistical information visually (your manager will love this!).


Compact Agents and Alerts


Too many agents? Too many alerts? Try our new compact view options. Under “View Options”, enable this feature by clicking “List View”.

Screen Shot 2014-01-29 at 7.16.10 PM.png

Other Changes:

  • Performance optimizations and extended filtering & logging for the Threat Stack audit and file monitoring services.

  • Fixes for issues where process arguments were not being tracked with the process.

  • Disabled email alerts by default. We quickly realized that this was too much of an annoyance -- so we wiped them away.

  • Removed potentially noisy alert rules which were also enabled by default.

  • Agent install or upgrade will enable pam_loginuid if not already enabled

Interested? Join our beta today

Cloud server forensics take center stage

At Threat Stack, we’re constantly exploring ways to advance cloud server forensics. We’re especially attentive to this as it’s an area of cloud security that’s becoming more critical since the attack vector of cloud is growing.

Forensic logs can lay out the scope of an attack that’s occurred on your servers, but getting to the bottom of what’s been done is usually much easier said than done. In fact, you can easily find yourself paying up to $600/hr for a security consultant to do this exact work if you don’t have the right tools in the first place. But what does it mean to have the right tools?

Do existing methods work?

You can assume that your prevention methods are so mature that you won’t ever need to do forensics, but that’s a big risk. Even if you think your traditional threat prevention methods really are solid, it’s wise to assume that a breach will -- not might -- happen.

We found that companies using the cloud have attempted to use traditional forensics solutions to solve their cloud forensics needs. Digging deeper, we discovered that these solutions failed them in the end because their systems do not consider the transient nature of data in the cloud (e.g., what happens if the server is destroyed before you have been able to collect your evidence?) Unless you are savvy enough to save your disk images before you deprovision your cloud boxes (AND assuming your IaaS provider supports that -- many of them do not), you will not be able to successfully determine what happened on a box using traditional forensic disk acquisition and analysis tools like Encase. Moreover, these methods are time-consuming and require the skills of an expert.

If you’re a typical cloud user, you may not have these experts on-hand -- this is where you shell out lots of money for consulting services.

Detecting all of the things (even internally)

You know your business needs a dedicated solution for understanding exactly what happened when a breach occurred -- and ideally one that requires little to no human intervention so that you can continue focusing on other security priorities.

Let’s say you have a rogue employee running malicious code on your own servers right this moment. How would you go about discovering that this is happening? Well, it could either cost you $600/hr for a security consultant to come in after-the-fact to discover this compromise (and likely be late to perform effective remediation before valuable data is stolen), or you could be proactive and receive alerts as soon as abnormal activity occurs.

Let’s use ‘Bob’ as an example. Bob is a disgruntled employee and has decided to maliciously install a backdoor on your system. Using traditional methods, your sysadmin may never know he’s doing this.

Using Cloud Sight, your sysadmin will get notified of this strange activity through an alert:

Screen Shot 2014-01-15 at 11.18.35 AM.png

They can quickly gather more information about this strange activity by clicking on the ‘Process Details’:

Screen Shot 2014-01-15 at 11.22.57 AM.png

Let the forensic investigation begin.

The details shows that nc was launched from ‘/sbin/service1’:

Screen Shot 2014-01-15 at 11.21.03 AM.png

A search in Cloud Sight for ‘service1’ shows activity from Bob around /sbin/service1:

Screen Shot 2014-01-15 at 11.21.50 AM.png

Pivoting in ‘Process Details’ again, your sysadmin will see that Bob was creating this backdoor and attempting to cover his tracks by deleting evidence of his activity in local system logs (by removing ~/.bash_history). But since Cloud Sight forensics are stored off-box, everything is recorded!

Screen Shot 2014-01-15 at 11.22.25 AM.png

What can we learn from this?

Well, first, never discount the fact that attacks can originate from inside your own company by someone with good credentials. On top of that, having a forensic bread crumb trail to detect both external and internal activity can seriously reduce -- or even eliminate -- expensive expert consultants to identify what happened. Since we record and archive all processes and network activity, you won’t have to fork up thousands (and even hundreds of thousands) of dollars for an outside source to investigate your logs.

On top of that, instantaneous detection of a compromise, along with complete forensics history in one-click, will lead to rapid remediation of vulnerable systems and data. A little cloud security monitoring and proactive evidence gathering can go a long way towards saving yourself from a lot of hassle.

“Yt? Seeing something odd in the logs…”

A tale of a suspicious Linux process (with a dash of dog food thrown in)

The other day my coworker informs me, ‘hey, there’s a weird process making network connections on your box.’ A dreaded string of words if there ever were any for the security-conscious developer.


This is what he pastes to me:

/usr/bin/curl -G —location —max-time 60 —silent —write-out %{http_code} —data since=2009-12-31%2023%3A0%3A0 —data program=prof —data count=1 —output /dev/null;


Now, I fully expected this to be some normal activity observed and misinterpreted, but as I saw this I began to grow distressed. Curl POSTing to some random IP address and popcon.php?

The computer in question was not a production box, but my development VM environment. This didn’t do much to ease my worries, though. Developers are often the weakest links in the security chain. They have access to all of the source code, the ability to push to repos and often direct access to production boxes. If my dev environment was hacked, I would not be having a fun time of it.

Now, you’re probably asking yourself: how did he know this was running? Great question!

At Threat Stack, Inc, we kind of make it our business to know what exactly is running on your boxes, and alerting you of strange activity. New network activity from a process, well, that falls in that category. Like any good company, we dogfood our product on our dev and production boxes. My coworker saw the email alert about the strange curl process, gave me the heads up, and I proceeded right away to investigating it by diving into our user interface:

curl command in question

Right away, I noticed a few weird things: it’s new activity (which I observed by viewing the lonely bar in the executable history diagram), and the user that ran it was ‘root’, which was even more alarming. Looking at the process tree, I observed that its parent process was a shell script.

In Linux, all new processes are created by fork() and execve() from a parent process. We designed our UI so you can walk up the process tree to the parent process all the way up to init (pid=1), just by clicking on the parent process in the tree. So, I did exactly that. What launched this shell script? Was it some insidious piece of malware running active in memory? A new init script?

/usr/bin/perl launches my curl command

It turns out it was a perl script, launched by cron, located at /usr/share/pp-popularity-contest/cron.daily:

Hmm. On my box, I took a look at the contents of /usr/share/pp-popularity-contest/cron.daily, and I confirmed execution of the curl command in question:

I looked at the rest of the perl script, which searches for a bin/usr/bin/pp_popcon_cnt. The script appeared to post the results of this program up to using curl.

The script didn’t look particularly malicious, although I had no idea what/usr/bin/pp_popcon_cnt was. A quick google showed me that it was part of a valid package, ‘pp-popularity-contest’:

Further investigation indicated I installed this and a bunch of other random packages earlier while performing some file modification testing! Doh!

So, despite the questionable perl script, and despite the fact I was participating in a ‘popularity contest’ I didn’t know I was participating in, it turned out this was really something benign. And I was able to determine that with < 2 minutes of investigation, which was pretty cool.

What have we learned here?

Well, for one, that my coworkers and myself may be unhealthily paranoid. ;) Certainly, 99.9% of the processes making network activity on your machines are benign, but how can you make that judgement unless you even know what is happening?

At my previous job, we observed companies with enormous security budgets who still didn’t know they were hacked for months and months. Even once they were aware of it, they still had a really tough time figuring out what boxes the attackers actually accessed, what credentials they stole, and what malware they installed, etc.

The thing is, you can save yourself a lot of headache down the line if you assume that this kind of breach can happen and prepare for it proactively.No one wants to think their access controls will fail. But at the same time, you don’t want to hear a merchant saying, ‘sorry, we did everything you can to protect your credit card numbers, but it looks like someone got them anyway and was using them for months and months, without anyone knowing.’ A little logging and a little security monitoring go a long way.

Insider Threats: Your Biggest Risk

While many companies have become exceptional at protecting against external threats, is there ever worry for attacks happening internally? Enterprises are focused on stabilizing perimeter defenses against outside hackers, but according to a recent report by Forrester, internal breaches have become the top source of breaches in 2013, with 36% of breaches originating from employees.  

Faulty logic: External threats are more pervasive than internal

Insider threats are typically dismissed as a secondary risk to the security of a company -- but this is counter intuitive. Enterprises are realizing that they have a lack of system controls and little to no insight on internal network activity. On top of that, given the dynamic capabilities of the cloud, users can quickly and easily spin up new machines on AWS, for example, without explicit consent or security protocols to follow. Consequently, enterprises are becoming more and more aware that internal users who have the right permissions can access sensitive data and cause destruction from within.

Image courtesy of Toby Talbot/AP

Image courtesy of Toby Talbot/AP

Mitigation of these (very real) risks involves actively managing and monitoring user access levels and activity to lessen -- and hopefully prevent -- any potential damage. Combine the growing complexity of networks with the abundance of new applications and security products, and it’s becoming harder for IT teams to manage their security force -- but it doesn’t need to be complicated.

Internal defenses: Detection and auditing

Enterprises must know who is doing what on their cloud servers and when, in order to reduce and cease internal threats. This involves gaining clear visibility into systems and networks to detect and audit user behaviors.

What they need is the ability to audit what a user does once they connect to a network. This includes a full list of commands a user executes when they connect and a profile of normal user behavior to be able to easily detect abnormal activity and stop an attack in an instant. If an employee’s normal behaviors start to change, it’s easy to detect that they could be attempting an attack. Spotting these behaviors consistently and early goes a long way to understanding when an internal threat will occur.

Intrusion detection from within

Enterprises must be strategic at a high level when it comes to security -- for both external and internal protection. Ideally, a multi-layered approach should be implemented. Since your users are already inside your network, you need to create deep security layers to prevent them from executing malicious commands when logged in. Always start with the understanding that an attacker is already on the inside, and work backwards from there.

If your organization does not already have these controls in place to restrict privileged access from within, you should be implementing a strong detection and auditing system which logs, monitors, records and alerts you on all session activities. An effective detection and auditing system will manage, monitor and secure your cloud infrastructure against internal (and external) attackers. You’ll also benefit from the control and accountability that is critical for meeting compliance regulations and audits.

At Threat Stack, for example, we’ve spared no effort to ensure our cloud security monitoring solution solves internal vulnerabilities and attacks. Our Cloud Sight product is equally adept at watching for internal anomalies as it is for external activities. Our logs dive deep into every command a user has run to give you a full understanding of what they did before, during and after a malicious command was executed.

We’ve found that this is far more effective at not only recognizing abnormal activity and stopping it, but helping organizations understand how attacks happen so they can take measure to prevent them from happening in the future.

The scope of security monitoring is changing, especially with the exponential growth of cloud and social applications, and enterprises need to be prepared for an entirely new realm of vulnerabilities and attacks -- ones which can easily be identified and stopped with the right continuous security monitoring systems in place.

Interested in how Cloud Sight by Threat Stack can help you monitor internal and external activity at the deepest level?

Join our beta today and begin truly protecting your cloud infrastructure.