«

Bees with Machine Guns (Attack of the Bees)!

Posted by Raymond Mintz

A quick and efficient way for load testing from hundreds of Amazon Web Service (AWS) instances to generate a large amount of traffic from different regions around the world to a single or range of urls.

Ever been stung by a bee?

It’s not that bad and they are not all that scary. Have you ever been attacked by a swarm of bees? I hope not!

So why the random analogy? Well, it’s not so random. Load testing from a single node or IP is one thing, but load testing from many nodes and seperate IP’s can be just as scary as… well, a swarm of bees!

Is your web application ready for production? _Is your web application ready for production?_

Bees With Machine Guns (BWMG) is a neat project created by developers at Chicago Tribune. The main ingredients are: AWS EC2 which runs Linux virtual machines, and the boto and paramiko Python libraries.

Boto allows us to talk to AWS to spin-up (and later tear-down) the EC2 instances and paramiko allows us to talk to our instances in parallel via SSH.

A typical run of BWMG follows these steps:

  1. Bring up a number of bees

  2. Load test a url

  3. Get a report status

  4. Bring bees down cleanly


Off to a Good Start!

When BWMG was first put out there for the public, it could only be used from a single AWS region (e.g. US-West-1b) which is 1 of 14 different regions Amazon has to offer (at the time of writing). From the chosen region, a user could start up and control as many bees as was allowed by that user’s account limitations. They would also have to have a pre-existing AMI with ApacheBench load testing tool to spin up their instances from.

In this state, BWMG was already an amazing resource for anyone who needed to easily and quickly load test their web application.


Bees Evolving

We’ve made a couple contributions thus far:

Adding support for multiple regions has a few neat advantages:

We chose to integrate hurl with BWMG to provide an alternative to ApacheBench, partly to validate hurl’s performance (who doesn’t want to have the most performant load tester), and partly because it is in active development, and we are constantly striving to improve it’s performance, features and reporting richness.

Based on our testing, hurl is capable of a sustained 10,000 complete (full request and response parsing) GET requests per second from a t2.micro AWS EC2 instance.

So what do we use this for?

Only for good, we promise. So far we’ve used BWMG to validate the responsiveness and performance of our HTTP Rate Limiting Layer-7 Anti-DDoS solution, as well as to verify the correctness of the geographical blocking capability of our WAF.

Our customers depend on us to ensure that these services remain reliable, and BWMG allows us to bring real-world attack scenarios against our platform in a controlled and safe fashion. Fat pipes, planning and amazing architectures are great, but there is nothing quick like real-world validation of a product.


In Action

Here’s an example of the output bringing up a few bees (2 bees in 3 different regions):

     $ ./bees up -s 2 -k bees -g bees2 -l ubuntu -i ami-9342c0e0,ami-fd489d
     9e,ami-e8c93e88 -z eu-west-1b,ap-southeast-1b,us-west-2b
     Connecting to the hive.
     GroupId found: bees2
     Placement: eu-west-1b
     Attempting to call up 2 bees.
     Connecting to the hive.
     GroupId found: bees2
     Placement: ap-southeast-1b
     Attempting to call up 2 bees.
     Connecting to the hive.
     GroupId found: bees2
     Placement: us-west-2b
     Attempting to call up 2 bees.
     Waiting for bees to load their machine guns...
     Waiting for bees to load their machine guns...
     .
     Waiting for bees to load their machine guns...
     .
     .
     .
     .
     Bee i-5568c1d9 is ready for the attack.
     Bee i-5668c1da is ready for the attack.
     Bee i-2cf8aba2 is ready for the attack.
     The swarm has assembled 2 bees.
     Bee i-2bf8aba5 is ready for the attack.
     The swarm has assembled 2 bees.
     Bee i-d05a6c08 is ready for the attack.
     Bee i-d15a6c09 is ready for the attack.
     The swarm has assembled 2 bees.

Getting a quick status report on the swarm:

     $ ./bees report
     Read 2 bees from the roster: eu-west-1b
     Bee i-5568c1d9: running @ 54.194.192.20
     Bee i-5668c1da: running @ 54.194.197.233
     Read 2 bees from the roster: ap-southeast-1b
     Bee i-2cf8aba2: running @ 52.77.228.132
     Bee i-2bf8aba5: running @ 52.221.250.224
     Read 2 bees from the roster: us-west-2b
     Bee i-d05a6c08: running @ 54.187.100.142
     Bee i-d15a6c09: running @ 54.201.177.125

And running a small load test of 1000 req/sec against a test url:

     $ ./bees attack --hurl -u $testurl -S20 -M1000 -H "Accept : text/html" --long_output
     eu-west-1b
     Read 2 bees from the roster: eu-west-1b
     Connecting to the hive.
     Assembling bees.
     ap-southeast-1b
     Read 2 bees from the roster: ap-southeast-1b
     Connecting to the hive.
     Assembling bees.
     us-west-2b
     Read 2 bees from the roster: us-west-2b
     Connecting to the hive.
     Assembling bees.
     Each of 2 bees will fire 500 rounds, 50 at a time.
     Stinging URL so it will be cached for the attack.
     Organizing the swarm.
     Bee 0 is joining the swarm.
     Bee 1 is joining the swarm.
     Each of 2 bees will fire 500 rounds, 50 at a time.
     Stinging URL so it will be cached for the attack.
     Organizing the swarm.
     Bee 0 is joining the swarm.
     Bee 1 is joining the swarm.
     Each of 2 bees will fire 500 rounds, 50 at a time.
     Stinging URL so it will be cached for the attack.
     Organizing the swarm.
     Bee 0 is joining the swarm.
     Bee 1 is joining the swarm.
     Bee 0 is firing her machine gun. Bang bang!
     Bee 1 is firing her machine gun. Bang bang!
     Bee 1 is firing her machine gun. Bang bang!
     Bee 0 is firing her machine gun. Bang bang!
     Bee 0 is firing her machine gun. Bang bang!
     Bee 1 is firing her machine gun. Bang bang!
     hurl http://$testurl/ -p 50  -H "Accept : text/html" -H "Content-Type : text/plain" -o /tmp/tmp.aAojMFs3ob -l 20 -A 1000 -j

     i-fb457323
     ec2-54-186-204-52.us-west-2.compute.amazonaws.com
     Running 1 threads 50 parallel connections per thread with 1 reqests per connection
     +-----------/-----------+-----------+-----------+--------------+-----------+-------------+-----------+
     |    Cmpltd /     Total |    IdlKil |    Errors | kBytes Recvd |   Elapsed |       Req/s |      MB/s |
     +-----------/-----------+-----------+-----------+--------------+-----------+-------------+-----------+
     |         0 /         0 |         0 |         0 |         0.00 |     0.50s |       0.00s |     0.00s |
     |       505 /       505 |         0 |         0 |       313.02 |     1.00s |    1007.98s |     0.61s |
     |       970 /       970 |         0 |         0 |       633.81 |     1.50s |     930.00s |     0.63s |
     |      1442 /      1442 |         0 |         0 |       961.08 |     2.00s |     944.00s |     0.64s |
     |      1900 /      1900 |         0 |         0 |      1276.81 |     2.50s |     916.00s |     0.62s |
     <---SNIP---->
     |     16855 /     16855 |         0 |         0 |     11612.32 |    19.00s |     938.00s |     0.64s |
     |     17320 /     17320 |         0 |         0 |     11932.69 |    19.50s |     930.00s |     0.63s |
     |     17794 /     17794 |         0 |         0 |     12261.34 |    20.00s |     948.00s |     0.64s |

     Bee: i-fb457323
     max-parallel:           50
     1st-resp-ms-min:        10.036
     response-codes
     200:            184
     403:            17560
     seconds:                20.006
     connect-ms-max:         8.729
     1st-resp-ms-max:        130.893
     bytes:                  16541472.0
     1st-resp-ms-mean:       40.5336982642
     end2end-ms-mean:        40.5487644838
     fetches-per-sec:        889.433170049
     connect-ms-min:         0.0
     fetches:                17794
     bytes-per-sec:          826825.552334
     end2end-ms-min:         10.053
     end2end-ms-max:         130.907
     mean-bytes-per-conn:    929.609531303
     connect-ms-mean:        0.0224927299369

Spike chart

This image shows down-sampled requests (100:1) made both from a single hurl instance initially (the blue spike), and then from a swarm of bees in multiple regions. Interestingly you can see some regions struggled to reach the target request rate of 1000/sec, while others had no issues.


Contributions

Thou shalt contribute..

Wanna help make BWMG better? Pop over to the repo and show it some love!

Also feel free to checkout our fork where you can find more examples of bees in action.

Share