On April 2nd, one of our customers was hit by multiple waves of application layer DDoS attacks. Unlike the large multi-Gbps attacks we commonly see on the news, an application layer (L7) DDoS is designed to overwhelm a system through massive flood of HTTP requests specifically crafted to target URLs involving back-end processes. In many dynamic applications, like API services for product pricing & availability, the network is usually not the bottleneck, but rather back-end services such as databases. A targeted, smaller scale L7 DDoS can slip past the typical network layer protections (including CDNs), to bring down a customer’s site.
Layer 7 DDoS Attack
This particular incident hit our customer with multiple waves of L7 DDoS traffic exceeding 5 million requests/min over a period of 18 hours. The highest peak reached around 8.9 million requests/min, or ~150 thousand requests/second (see Figure 1). To put this in perspective, this customer has daily peaks of ~2 thousand requests/sec across all their properties hosted on our CDN, and a max of 100 requests/second to the endpoints that were attacked. So this attack represented a spike 1500 times larger than the normal load on the customers origin infrastructure.
L7 DDoS Attack Profile for over 18 hours.
The sources of the attack primarily originated from Europe with France, Spain, and Italy rounding out the top three countries, including only several hundred unique IP addresses.
L7 DDoS Attack Source.
Further investigation revealed that all of the requests were HTTP POSTs to the customer’s quoting & pricing summary API endpoints, which could involve a lot of back-end database queries (see Figure 2). While we don’t completely understand the motives of the attack, the load was real. Moreover, without proper protection in place, their website and API endpoints would have most likely suffered an outage.
HTTP method & URL targets
Edgecast Rate Limiting Mitigates Layer 7 DDoS Attack
The good news is the customer had been utilizing our Rate Limiting product (a.k.a. L7 DDoS protection) which automatically enforced policies to drop all of the attack traffic. Our Rate Limiting product dropped more than 730 million attack-related requests for this customer over the 24-hour period of the attack and potentially prevented more than 8 hours of customer downtime.
To highlight the capacity of our CDN (see Figure 3) and emphasize the scale of our edge-based Web Application Firewall (WAF) solution, a top-of-the-line WAF hardware appliance with >$200K price tag can only handle roughly 1/2 the amount of request/second that we mitigated at a small fraction of the cost.
Oh and by the way, did we mention that 89% of the attacks were mitigated by only 1 point of Presence (PoP)? (see Figure 4)
Distribution of Attack Across PoPs.
Incidents like these demonstrate that in addition to proxying your website on our CDN, which inherently comes with network layer (L3) DDoS protection, it’s also crucially important to complement that protection with a Layer 7 DDoS protection such as our Rate Limiting product. Notably, given the pervasiveness of Layer 3 mitigation, Layer 7 attacks could be poised to rise since they require fewer compromised machines to be effective.