One of the concerns that has been raised about the Heartbleed vulnerability is that it was introduced into the OpenSSL code base several years ago, and it’s possible that some attackers were aware of it and launching attacks before it was publicly disclosed this week. Unfortunately, the attack, when performed properly, does not create any suspicious web server logs, so it’s difficult for server administrators to go back in time and determine whether or not their web sites may have been attacked in the past.
One web server administrator who kept detailed packet captures of traffic to his server dating back many months has reported that he believes his server was hit by the attack in November 2013 from two different source IP addresses (188.8.131.52 and 184.108.40.206). While most sites aren’t able to keep months’ worth of packet captures, many sites store long-term logs and can check for activity coming from those specific addresses. The blog post linked here provides detailed instructions on how to search the archive of NetFlow in Lancope’s StealthWatch System for traffic emanating from those IP addresses.
However, it is also possible to identify flows associated with Heartbleed attacks based on their unique traffic characteristics. Note the screenshot below, which shows NetFlows that were generated by Heartbleed attacks in our lab. Pay particular attention to the Client Ratio and Duration. As Lancope customers often store months of NetFlow, they may be able to use that NetFlow to identify attacks against their web servers that occurred earlier this week or even before public disclosure of the vulnerability. (Click image to enlarge.)
Attacks that target the Heartbleed vulnerability involve repeatedly issuing a specific request that produces a response that has the same size every time. Hence, the resulting flows will have a particular client/server byte ratio. Our tests produced results that were consistently close to 4.8% client bytes. It’s possible for the attacker to manipulate this ratio by changing the amount of data requested from the server each time. In our tests, we requested the maximum amount, which is the most likely attack scenario.
Attacks that target Heartbleed are also likely to produce long flows. The reason that long flows come into play is that every Heartbleed transaction reveals to the attacker a new piece of data from the server's memory. On a busy web server that is handling lots of transactions, the piece of memory that gets returned from a Heartbleed attack is likely to have different contents every time. Therefore, it makes sense for attackers to run the attack over and over and over again in order to collect more and more data from the victim server. It is realistic for attackers to run this attack for hours on end against a particular server. This behavior will produce long flows.
In the StealthWatch System, there is a Security Event that provides for real-time detection of long flows (called Suspect Long Flow). By default, Suspect Long Flow identifies flows that have lasted longer than 9 hours, but some Lancope customers have Suspect Long Flow tuned as low as 30 minutes after whitelisting popular web services that produce legitimate long flows.
In the latest release of the StealthWatch System (Version 6.5), it’s also possible to create a custom alarm that fires on long duration flows headed to the web server host group from outside hosts. The blog post linked here provides some more information about configuring custom alarms in Version 6.5 of the StealthWatch System.
Logs of NetFlow can also be searched to identify past flows that fit these characteristics. The StealthWatch System can be searched for every flow between outside hosts and the web server host group. The resulting flow table can be sorted based on flow duration or client ratio, providing results to investigate further. It’s also possible to pull these flows out of the StealthWatch System for analysis using our web services API.
One should be particularly concerned about flows that occurred since the Heartbleed bug was disclosed. If you've seen an increase in long flows fitting these characteristics since the disclosure, your server may have been targeted. Flows fitting these characteristics that occurred before public disclosure are also worth investigating further. Of course, there are a variety of explanations for why a flow might have these characteristics, but on the other hand, if attacks were happening before public disclosure they were likely launched by sophisticated threat actors and should be taken very seriously.
It’s important for Lancope customers to understand that while this approach may detect suspicious flows that are worthy of further investigation, it cannot do the opposite – it cannot be used to rule out the possibility that an attack took place against a particular server. The attacker can vary the client ratio that is seen in the attack, and they can also generate short duration attacks that still extract valuable information from a server. Therefore, even if you do not find flows in your environment that fit these characteristics, you cannot assume that your servers were never targeted.
Heartbleed is a very serious security issue – the tools for exploiting this vulnerability are publicly available and attacks are happening right now. If you have a vulnerable server, then you have data that is exposed to the Internet. The best thing for you to do is patch that server immediately. The capabilities offered by the StealthWatch System can help you investigate attacks against your network, but they should not be seen as a substitute for patching vulnerable servers.
Thanks to Charles Herring, Brandon Tansey and Gary Owen for their help with this research.
TAGS netflow, network security, stealthwatch, lancope, network visibility, forensics, vulnerability, heartbleed, openssl