This guide describes a technique to perform on-demand, any-scale load testing using The Grinder 3 on Amazon Web Services in a few clicks.
It provides a simple-to-use AWS CloudFormation (CFN) template that implements this technique. The template deploys The Grinder 3 in a distributed configuration, and uses EC2 Auto Scaling to reach gradually the desired load without manual intervention.
1) Edit the following scripts with the URL of your service and upload them to Amazon S3:
2) Click here to launch the CloudFormation stack, choose the default values and point to your S3 bucket above
3) At the end of the test, don’t forget to delete your stack
AWS CloudFormation (CFN) provides an easy way to describe and launch a collection of related AWS resources. You can deploy and update a template and its associated collection of resources (called a stack) by using AWS Management Console, AWS Command Line Interface, or APIs.
CloudFormation is available at no additional charge, and you pay only for the AWS resources needed to run your applications. To learn more please visit AWS CloudFormation homepage.
This guide provides a simple-to-use CFN template that alleviates the need to set up, configure and maintain your load-testing environment. It takes care of the heavy lifting – launching the right amount of Amazon EC2 instances (virtual servers), installing the required software, configuring and running The Grinder, distributing and running your tests, etc.
You simply pass the location of your test files and the desired load in terms of Tests per Seconds (TPS) as parameters when you launch the CloudFormation Stack.
Load testing may require massive compute resources, and is often done periodically. With traditional approaches, this may lead to expensive equipment sitting idle and underutilized during a significant portion of the time, or to insufficient resources to test your desired load.
The CloudFormation (CFN) template provided with this guide creates a complete load-testing environment on AWS that automatically scales up to accommodate your desired load. Once you’re done testing, simply delete your CFN Stack, release all elastic resources and stop paying.
With load testing it is often hard to predict the amount of machines one should use to generate a desired load. It generally depends on the type of tests being run and the desired load level. Some load tests may suffice with a single machine while other might require tens, hundreds or even thousands of machines to generate.
With the provided CloudFormation template, testers specify the test they wish to run and their desired load in terms of Tests per Second (TPS). Behind the scenes Amazon EC2 instances are launched incrementally until the desired load is reached.
With The Grinder 3, test scripts are written using a dynamic scripting language, and specify the tests to run. The default script language is Jython, a Java implementation of the Python language. The Grinder 3 allows any Java, Jython, or Clojure code to be encapsulated as a test.
If you're new to writing tests for The Grinder, you may want to visit The Grinder’s Script Gallery for sample test scripts.
Once you're done writing your test scripts and defined your test properties (*.properties files) place them on Amazon S3.
Amazon S3 is storage for the Internet. It is designed to make web-scale computing easier for developers. It provides a simple web-services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web.
When launching a Stack from this CloudFormation template you indicate where the test files are located on Amazon S3. The Grinder Console instance launched by this template will then download these tests from S3 and distribute them amongst the Grinder Agent instances that participate in executing the tests.
If you are not familiar with how to use Amazon S3 you may find the Amazon S3 Getting Started Guide helpful.
This step is optional.
To SSH into the EC2 instances launched by the CFN template you will need an EC2 Key Pair. You may create a new Key Pair or use an existing one. If you do not wish to SSH into your instances you may skip this step.
When you launch the CFN Stack, input the name of your Key Pair into the KeyPairName parameter, or alternatively leave it empty.
Information on how to connect to your EC2 instances using your private key can be found in the EC2 User Guide.
To create a Key Pair you may follow these steps:
1. Click here to enter the Key Pairs page of the Amazon EC2 Console.
2. Click the Create Key Pair button at the top of the page.
3. Enter a name for this Key Pair and click Create.
4. Your browser will download a *.pem file with your private key. Keep it in order to be able to connect to your instances.
This guide provides a CloudFormation (CFN) template that provisions the required resources, launches the Grinder and initiates the load-test.
Make sure to select the region in which you wish to deploy your Grinder environment. You may inspect and select your desired region on the top-right corner of AWS Management Console as illustrated below.
Click the Next button to see the parameters for this template.
You define how quickly you wish your load to ramp up, using the DesiredIncreasePercentage and DesiredIntervalBetweenIncreases parameters.
You specify the load you wish to reach in the DesiredLoad parameter. The load is measured in terms of Tests per Second (TPS).
The DesiredIncreasePercentage indicates the percentage by which the fleet of Grinder Agents should increment in every step. The increment applied is the product of the current fleet size and the percentage specified, rounded down to the closest integer. If the result is 0, it is adjusted to 1.
For example, a CFN Stack launched with the following parameters:
· DesiredIncreasePercentage = 50
· DesiredIntervalBetweenIncreases = 150
· DesiredLoad = 1000
The load generating agents would scale until a load of 1000 Tests per Second (TPS) is reached. The number of machines used to generate this load would remain and sustain the load until the environment is terminated (see step 8).
Initially a single Grinder Agent would be launched and begin generating load. After 150 seconds (2.5 minutes), if the load did not reach 1000 TPS, our fleet of Grinder Agents would increment by 50%. As it is not possible to launch a portion of an instance, a single instance will be launched, bringing to fleet size to 2.
Consequently, if after 150 seconds a load of 1000 TPS is not reached, an additional single Grinder instance will be launched bringing the fleet size to 3. The fleet would then scale from 3 to 4, from 4 to 6, from 6 to 9, and so on until the desired load of 1000 TPS is reached.
The CFN template uses two parameters to determine which load tests to run.
The TestDirectory parameter points to the location on Amazon S3 where the test scripts reside, using the S3 URI notation (s3://<bucket-name>/<object-key>).
The TestProperties parameter points to the test properties file you wish to execute, relative to the provided TestDirectory.
For example, a CFN Stack launched with the following parameters:
· TestDirectory = s3://mytestsbucket/mytestscripts
· TestProperties = TestHTTPEndpoint.properties
The single Grinder Console instance would initially copy the content of the directory specified by the TestDirectory parameter to a local disk. In this example, it would copy all the Objects stored in mytestbucket Amazon S3 bucket, whose key begins with mytestscripts/.
The CFN Stack expects to find a file named TestHTTPEndpoint.properties under the copied directory.
Each Grinder Agent that subsequently launched would initially register with the Grinder Console. The Console would then distribute the entire test directory to the registered Grinder Agent and would request it to run the test specified in the copied TestProperties file – in our example - TestHTTPEndpoint.properties.
The CFN Stack launches a single Grinder Console instance and a fleet of Grinder Agents, managed by Auto Scaling group.
The InstanceTypeConsole and InstanceTypeAgent parameters control the EC2 instance types of the Grinder Console and Grinder Agents respectively.
The default instance type used for the Grinder Console is “m3.large”. It is recommended to use the M3 instance family for the Grinder Console instance, as it is optimized for general-purpose workloads. For deployments with a large expected number of load-generating Grinder Agents (100+ agents), it is recommended to use a larger instance type (e.g. m3.xlarge) for the Grinder Console.
The optimal instance type for the Grinder Agents depends on the nature of the tests to be performed. For CPU intensive test workload, it is recommended to use the C3 instance family. For memory intensive test workload it is recommended to use R3 family. For general-purpose workload it is recommended to use the M3 family. The default instance type used by the CFN template is m3.medium, which is suitable for most workloads.
The full list of instance types offered by Amazon EC2 can be found here.
Some applications cannot scale beyond a certain threshold, no matter how many Grinder Agents generate the load.
The MaxAgents parameter controls the maximal number of Grinder Agents that will be launched to generate the desired load. This assists in restricting the resources used for generating the load, and hence assists in controlling the cost of running the test.
The default value for MaxAgents is 20. You may alter the default value when launching the CFN Stack. You may also alter the MaxAgents value of an existing CFN stack, allowing you better control on your scale and cost.
To adjust the MaxAgents value of an existing stack, you may follow these steps:
1. Enter the CloudFormation Management Console
2. Select your Stack (TheGrinderLoadTesting) from the list of running stacks
3. Click the “Update Stack” button at the top left corner of the page
4. Under the “Template” section, pick “Specify an Amazon S3 template URL” and enter https://s3-eu-west-1.amazonaws.com/www.aws-il.com/load-testing/grinder/grinder-load-testing.template
5. Click “Next”
6. Modify the value of the MaxAgents parameter
7. Click “Next” again, as we do not need to override the Stack Policy
8. Review your change and click “Update”
You might want to retain root access to the EC2 instances launched by the CFN template.
Similarly to accessing any Amazon EC2 instance, you will need to use a private key file (.pem) that corresponds to a Key Pair you previously generated. To allow for root access to the EC2 instances launched by this template you will need to provide the name of your Key Pair as input to the KeyPairName parameter.
You could then connect to these instances using your private key file (.pem) of the specified Key Pair, and use the ec2-user to connect.
After entering the CFN stack parameters, click Next.
The CFN template takes care of tagging the resources it generates with meaningful Name tags. When prompted for additional tags, click Next as there is no immediate need for additional tagging.
You see a review of the stack. Click Create to launch the stack.
Select the stack named TheGrinderLoadTesting in the stack list and then click the Events tab to track the progress of the creation of the farm. If you do not see the stack, click the Refresh button at the top right corner of the page.
You can click Refresh to see new events. When the stack status changes to CREATE_COMPLETE, this step is complete. This process usually takes 7-10 minutes.
The provided CloudFormation template leverages Amazon CloudWatch to monitor the load test performance. It also uses an Amazon CloudWatch Alarm to scale the load generating Grinder Agents until the desired load is reached.
Amazon CloudWatch provides a reliable, scalable, and flexible monitoring solution for AWS cloud resources and the applications customers run on AWS. Developers and system administrators can use it to collect and track metrics, gain insight, and react immediately to keep their applications and businesses running smoothly.
The CloudFormation (CFN) stack provides useful information about the load-testing environment it provisioned. This information is provided by the CFN Stack’s outputs. To view the CFN Stack outputs, select the stack named TheGrinderLoadTesting in the stack list and then click the Outputs tab.
The following outputs are available:
· CloudWatchMetrics – A link to the CloudWatch custom GRINDER metrics. These metrics provide valuable test statistics over time.
· CloudWatchAlarm – A link to the CloudWatch Alarm in charge of scaling out the Agent fleet.
· ConsoleIP – The public IP address of the Grinder Console. This IP may be used for SSH access to the Grinder Console host.
· GrinderConsoleProperties – A link to the Grinder Console properties REST API
· GrinderConsoleStats - A link to the Grinder Console collected statistics REST API
· GrinderConsoleAgents - A link to the Grinder Console agents’ status REST API
The Grinder Console instance created by the provided CloudFormation template puts custom metrics to Amazon CloudWatch every 30 seconds under the “GRINDER” namespace. These metrics can be used to inspect the load testing statistics over time, and reflect the data collected by the Grinder Console.
One of these metrics (TotalPeakTps) is also used as the source metric by which the CloudWatch Alarm is configured to trigger the scale-out policy of the Grinder Agents fleet.
In the Stack’s Outputs tab mentioned above, you’d find the CloudWatchMertics URL. This URL will list all the custom “GRINDER” metrics.
The following custom “GRINDER” metrics will be available:
· TPS – Tests Per Second, as measured in the proceeding 30 seconds.
· TotalTests – The total number of tests performed since testing begun.
· TotalErrors – The total number of test errors since testing begun.
· TotalTps – The average Tests Per Second since testing begun.
· TotalPeakTps – The peak Tests Per Second recorded since testing begun.
· TotalMeanTestTimeMs – The mean time in milliseconds of all tests performed since testing begun.
· TotalTestTimeStandardDeviationMs – The test time standard deviation in milliseconds since testing begun.
Depending on the type of tests run by the stack (as specified in the TestProperties stack parameter), you may find additional metrics, as provided by The Grinder’s collected statistics.
Your load test would continue to run even when having reached the desired load. To end the test and clean up the resources, you will need to delete your CloudFormation Stack.
AWS CloudFormation makes it easy to provision, as well as dispose of an entire stack of resources. When a CFN Stack is deleted, the entire array of resources that were provisioned by the stack is deleted as well.
You can use the AWS CloudFormation console to delete the stack:
1. In the CloudFormation Management Console, select the TheGrinderLoadTesting stack
2. Click the Delete button
3. To confirm, click Yes Delete
4. The Stack status changes to DELETE_IN_PROGRESS
5. Within 2-5 minutes the Stack status should change to DELETE_COMPETE and disappear from the list of active stacks.
The CloudFormation template creates the following resources:
1. Amazon Virtual Private Cloud (VPC) with CIDR 10.0.0.0/16
2. A single Subnet with CIDR 10.0.0.0/24 with Network Access Control List
3. Internet Gateway that is attached to out VPC.
4. Routing table with public route to the Internet Gateway, associated with our Subnet
5. AWS Identity and Access Management (IAM) role for the Grinder Console instance, with full S3 and CloudWatch permissions
6. Security Groups for the Grinder Console and for the Grinder Agents
7. Auto Scaling launch configuration for the Grinder Agents
8. Auto Scaling Group using this launch configuration
9. Auto Scaling scale-up policy
10. Amazon CloudWatch alarm when the peak Test per Seconds (TPS) is below the desired load, and have that alarm trigger the scale-up policy associated with the Auto Scaling group.
11. EC2 instance on which The Grinder is installed and run in headless Console mode. This instance is launched with the role created in step 5.
For simplicity, the template installs the same software stack, create the same scripts, and configure both the console and agent instances in the same way. The difference between the two types is merely which scripts are being run upon launch.
The CFN template uses Amazon Linux AMIs and configures the instances as follows:
· Install The Grinder v3.11
· Install jq. We use jq to parse the statistics returned by the Grinder Console REST API, and put that datum onto Amazon CloudWatch
· Create grinder Linux user
· Create four scripts and place them under /home/grinder:
Upon instance launch the following actions take place:
· The instance is configured as described above.
· The tests are downloaded from Amazon S3 into /home/grinder/tests using the aws s3 sync command
· It starts a headless Grinder console by calling /home/grinder/console_headless_start.sh
· It adds cron jobs that:
o Run /home/grinder/console_distribute_and_start_workers.sh every minute
o Run /home/grinder/ console_put_metrics_to_cloudwatch.sh every 30 seconds
Putting custom CloudWatch metrics for the test statistics provides better visibility into the test progress. It also allows the fleet of Grinder Agents to simply scale out until the desired load is reached using Amazon CloudWatch Alarm.
Upon instance launch the following actions take place:
· The instance is configured as described above.
· A headless Grinder console is started by calling our /home/grinder/agent_start.sh and passing it the internal IP address of the Console’s instance.
Scaling policies can be configured to scale in or out to an exact capacity (e.g. use exactly 5 instances), by changing an existing capacity (e.g. add or remove 5 instances from the ASG), or by changing an existing capacity by percentage (e.g. add or remove 50% of the current number of ASG instances).
Scaling policies also specify a cooldown period. A cooldown indicates the time period after a scaling activity (instance launch or instance terminate) ends and before another scaling activity can start. A cooldown period gives the system time to perform and adjust to the most recent scaling activities invoked by CloudWatch alarm.
The scale-out policy created by this template is set to scale by the percentage specified in the DesiredIncreasePercentage parameter, and have a cooldown period as specified by the DesiredIntervalBetweenIncreases parameter.
Once a Grinder Agent instance is launched, it registers with the Grinder Console, receives the tests to perform, and begins execution.
The Console instance periodically reports the test statistics as CloudWatch custom metrics.
The Amazon CloudWatch Alarm triggers the Grinder Agents ASG scale-out policy as long as the peak Tests per Second (TotalPeakTps) is below the desired load, as specified by the DesiredLoad parameter.
Once the desired load has been reached, the Alarm status changes to OK and the Agents fleet remains at steady capacity until the CloudFormation stack is deleted.
The Grinder is a Java™ load-testing framework that makes it easy to run a distributed test using many load injector machines. It is freely available under a BSD-style open-source license. To learn more please visit The Grinder website.
For the purpose of this guide, load-testing frameworks were benchmarked by the following criteria:
· Be free and open-source
· Be commonly used and familiar to the community
· Allow spawning new “slaves” during a test run to gradually increase the load
· Provide online test metrics by which the environment could scale automatically
It probably narrows down to Apache JMeter and The Grinder. Both are free and open-source load-testing frameworks. Both are widely adopted. Each takes its own approach along with its pros and cons. There is no right or wrong when coming to choose between these two, but rather which fits the use case better.
Proceeding to evaluate the last two requirements. Connection between the master node and slave nodes is done differently with these two frameworks. Apache JMeter remote testing require that the JMeter Client (master) be aware of all its JMeter Servers (slaves) at launch time. With The Grinder, Agents (slaves) register dynamically with the Console (master). This behavior is more suitable for this use case.
Furthermore, The Grinder’s Console offers REST API to initiate and stop tests across its Agents, as well as to retrieve test statistics. This API came in handy in automating load testing.