Load and Performance tests are a key component of a thorough testing strategy. When I first wrote about load and performance tests I said that you shouldn’t need either of these types of tests until well into the life of an application. And that’s still true. The interesting question is to figure out when is it a good time to bring them into the fold.

Before we can figure that out, let’s look over what each type of test brings to the table and how they relate to each other.

Load Testing is the process of observing the behaviour of a system under load. In the case of an API, this means sending a high volume of HTTP requests to an endpoint that is likely to be heavily solicited in production. For a Worker, such as a Windows Service or Serverless Function, this means sending messages to its trigger. The trigger for a Worker is typically a queue or notification mechanism but can also be HTTP or storage-based.

Here are some behaviours that should be observed during the execution of load tests:

• Which resources are being used at a higher rate than normal?
  • Are there obvious memory leaks?
  • Is the database limiting the throughput?
  • Are there sufficient concurrrent connections?
  • Is a downstream service causing a slow down of the application?
• When does the application stop working entirely? What was its state before it fell over?

Performance tests, on the other hand, measure the response time of the server at a given level of load. They can be considered a subset of load testing since they need to run in conjuction with a load test. For an API, this means measuring the response time of the method. For a worker, it’s measuring the rate at which the service processes requests from the queue.

When Do You Need Them?

A good rule of thumb to start writing load and performance tests is once you start considering increasing the size or number of machines that are hosting your application because of an increase in traffic.

It’s a waste of time to write load or performance tests when a service is not being used very much. The information gathered by load and performance tests is especially valuable for scalability planning. Say you run a load test and see that past 5000 requests per minute the service slows to a crawl. It means that the service should scale up before reaching that threshold so that it continues to respond in an acceptable time.

One of the first services that is likely to need these tests are authentication endpoints. They are likely candidates for high throughput since they tend to be called on almost every request to validate tokens. After authentication, the most used feature of your application, whatever it happens to be, is likely to need some load and performance tuning.

Start by running load and performance tests manually, outside of your CI server. They are a lot simpler to set up and monitor by hand than via automation. This gives you the chance to identity obvious problem areas in your application and establish some benchmarks for expected results.

You can add the tests to your CI server once you’ve determined those benchmarks. They’re likely to be different per environment since a development configuration is often less powerful than a production configuration. The tests should be integrated to every build to quickly identify changes that cause a performance degration.

Resources and Tooling

Web apps and APIs

The name everyone knows and loves for web testing is JMeter. It’s customizable to just about any need and is also extensible if there is something it can’t do. The other great thing about it is that you can use it without any costs since it’s open-sourced under the Apache license. There are some great starter tutorials available but the best resource is the JMeter user manual which goes into the nitty-gritty details of how to turn every knob.

Another open-source option is Netling. It’s not as feature-rich as JMeter but for simple test cases it may be more practical to use.

Load and performance tests should be representative of real production traffic. The easiest way to achieve that for an IIS-hosted ASP.NET application is to replay the logs. Maarten Balliauw has written an excellent post on how to do just that.

Replicating production traffic on APIs that run on serverless or Platform-as-a-Service technology is a bit more complex. The simplest approach would be to log all incoming requests in their rawest form and then use those logs as input to JMeter. Luckily, load testing is less important for serverless and PaaS because of their auto-scaling abilities.

Worker applications

There aren’t many specialized tools for load and performance testing workers. Workers are usually fed by a queue mechanism that allow the work to be completed asynchronously without waiting for the result to be available. JMeter could be used to add messages to the queue but it wouldn’t be able to measure the speed at which the work is being processed.

It’s fairly easy to come up with your own custom solution to the problem. Logging SQS messages on AWS is a simple matter of enabling CloudTrail and saving the logs to S3. The same thing can be accomplished on Azure using Service Bus diagnostics. Once you have the logs, you’ll need a custom utility that can replay the messages on your test environment.

Profiling .NET Applications

There’s a few tools that can help you pinpoint problem areas in your code, whether its for a Web API, a Console app or anything else in between.

One of the best profiling tools is DotTrace by JetBrains. It’s easy to use and can be run locally as well as on a remote machine. JetBrains also recently launched DotMemory, which takes the built-in memory analyzer that is in Visual Studio to the next level. The two together make for a killer profiling solution.

For applications and services that are hosted on Azure, by far the best option is to integrate Application Insights to monitor, detect and diagnose performance issues. Read this article for a great primer on what you can do with Application Insights.