Narus Blog

by Supranamaya Ranjan

In my last blog post, I had provided a preliminary analysis showing how the DNS resolvers had caused an uptick in the amount of DNS traffic they were sending towards a set of DNS servers that NarusInsight Secure Suite (NSS) monitors. Since that time, the DNS exploit was released in the wild and hence I got curious to see how fast are organizations patching their systems.

Figure 1: Aggregate volume of anomalous traffic patterns seen on DNS traffic

The figure above shows the aggregate volume of anomalous traffic patterns that NSS saw on DNS traffic. We define an anomalous activity as one where either the volume of traffic going towards an ip-address increases suddenly or the behavior of traffic (e.g., in terms of number of sources contacting a destination ip-address, etc) changes suddenly. Thus, since most of the resolvers had patches applied in a co-ordinated manner, they sent traffic towards destination DNS servers at the same time- and NSS classifies them as anomalous traffic patterns as compared to the historical past baselines, the traffic volume as well as behavior to these DNS servers had changed.

From the above figure, it seems that the biggest amount of patches were applied in the June 9 - 10 timeframe and then sporadically on June 18, 21, 23 and 26. So the biggest amount of cache resets definitely occurred at the very beginning of the announcement and hence from our view via NSS, we believe the vast majority of patches were applied at the earliest announcement on June 9th.

Interestingly, the total amount of DNS traffic (as measured on a per-day basis) passing the NSS probes didn’t exhibit any significant changes (see the figure below). Though the total amount of DNS traffic on a day-by-day basis didn’t change, there were surges that occurred on a minute-level granularity (which is what the first figure refers to). In other words, the patched resolvers which had in co-ordination sent query traffic towards certain DNS servers, didn’t need to send more traffic afterwards as the responses were already cached.

by Supranamaya Ranjan

Two days ago, on July 8th 2008, security researcher Dan Kaminsky of IO Active disclosed to the public a flaw in the DNS implementations for both client-side resolvers and servers. This flaw would allow a clever adversary to launch a cache poisoning attack and hence take over the mapping that a user’s DNS cache could have for say, BankXYZ.com to point to an ip-address that the adversary controls. In short this could lead to someone impersonating your bank or your ISP and hence be able to eavesdrop on all your traffic. A smarter adversary could even set up a look-alike site for your bank and hence lure victims via a smart phishing attack.

US CERT released a note for this vulnerability on July 8th at 02:46:15 PM (most likely EDT). And the ISPs started deploying the fixes that were being released by Microsoft, Cisco and other vendors. There was widespread concern that malicious entities may try to take advantage of this newly published vulnerability to trap ISPs or users that may still be un-patched. Exact details of the attack methodology haven’t yet been disclosed by Dan Kaminsky in order to allow ISPs and users valuable time to patch their systems.

I verified two interesting hypotheses using Narus Insight Secure Suite (NSS): First, whether someone has already begun using this exploit to launch attacks against the DNS infrastructure. Fortunately, I didn’t find any attacks yet. Second, what kinds of changes in the Internet traffic did this global patching cause. The answer for this turns out to be quite interesting as the amount of DNS traffic was definitely found to be increased in our “periscope” to the Internet. NSS found a sudden increase in the number of anomalies in DNS traffic going towards the ISPs being protected.

Look at the figure below, which shows the aggregate volume (in Mbits/hour) over time for the DNS anomalies seen between July 7th and 11th. Clearly, before the CERT announcement and release of the patches, there were no anomalies. But after the announcement on July 8th, NSS saw a 1000x increase in aggregate volume of anomalous DNS traffic. NSS defines a traffic event as an anomaly if the amount or behavior of traffic heading to an ip-address exhibits sudden changes. A further analysis of the sources of these queries shows that they were being originated from open DNS proxies on the Internet and from DNS clients from well-reputed institutions from around the world. The reputation of the anomaly sources leads to the conclusion that these anomalies were not really attacks, but a side-effect of the synchronized patching.

Evidently, the DNS servers within the monitored networks were receiving a significantly larger than normal amount of queries after July 8th. These queries were being sent from all over the world by the DNS clients and resolvers whose DNS cache (which contains a mapping of domain names and ip-addresses) had been cleared out as a result of the patching. Since the clients don’t know which mappings were already “poisoned”, this complete drop of the cache would make sense. Thus, the patched DNS clients and resolvers had to send queries to almost every domain name to start re-building their cache from scratch. And since the patching itself was co-ordinated, we saw clients from all over the world trying in a synchronized manner to resolve the domains within the networks that NSS protects. Infact, in one anomaly, we saw 3500+ clients sending queries to a DNS server within an hour when historically this server used to be contacted by 100 unique clients on average.