Testing Cloud Applications

Every cloud application needs to have a proper testing strategy in place to ensure it adheres to its quality goals. Those goals can include being defect-free, easily deployable, scalable, secure, and more, all without introducing unwanted breaking changes.

The traditional test pyramid, as introduced by Mike Cohn, has three layers. The largest layer is reserved for unit tests, which most developers are very familiar with. The next layer up are the service tests which “test the services of an application separately from its user interface.” And finally, the top of the pyramid is for the UI test layer.

The test pyramid that I like to use is based on Mike’s original but explodes the service layer into five separate layers in an effort to make the UI tests layer as small as possible.

Unit Tests

The goal of unit testing is to take the smallest possible amount of code, usually a single function, mock out its dependencies, and verify that is behaves as expected.

How to write them?

There are many ways to write unit tests:

• The Original Way. Write some code and then write the tests for that code.
• The TDD Way: Write the test first, then write the code to make the test pass.
• The BDD Way: Write your tests so that they test the behaviour of a method, not its implementation details.

An approach that combines the three methods above will ensure that your tests are written in a reusable way while covering most test cases.

When should they run?

Unit tests only need to run once in a CI/CD pipeline since their behaviour is predictable and repeatable from one run to the next.

More Resources

If you’d like to know more about unit testing and TDD right now, I highly recommend Uncle Bob’s Agile Practices, Patterns and Practices book and his series on TDD at Clean Coders.

UPDATE: I’ve written an in-depth article focusing on how to structure unit tests within a .NET Core project.

Acceptance Tests

Acceptance tests have three purposes that are achieved with the same suite of tests:

• Verify that the requirements of the feature are satisfied. This can be called integration testing but the term acceptance testing fits better with Agile methodologies usage of Acceptance Criteria.
• Ensure that the application is deployed correctly to a given environment. Without these tests you have no way of knowing if the code that was deployed is working as expected, making them especially critical when implementing a Continuous Delivery pipeline.
• Ensure that no breaking changes are introduced to the customer-facing contract.

How to write them?

Let’s say you’re working on an API to add friends to a list of favorites. The acceptance criteria state that any existing user can be added to the list. Any request for an invalid user should be refused. For this fictional scenario, we’d need three acceptance tests:

• When adding an existing user to the favorites list
  • returns an okay status code.
  • the user can be retrieved from the favourites list
• When adding a non-existing user to the favourites return a bad request status code.

The tests should be checking the shape of the contract returned by every method in your service. Using the previous example, a successful request to retrieve the friends list would ensure that an array of users is returned, and that each element of the array has a properly formed user object.

When should they run?

Acceptance tests should run every time the service is deployed to a new environment since they ensure that your service is up and running correctly.

These tests should be written once and never touched again. You can add tests to the suite but you should never remove or change the existing ones.

UPDATE: I’ve written an in-depth article that details how to go about creating acceptance tests in .NET Core projects.

External Contract Tests

Most cloud apps rely on external dependencies. Whether that dependency is inside or outside your organization, you should have tests that use that dependency in a similar way to your application. The goal is that you’re aware of any unexpected change that occurs to that dependency’s contract as early as possible.

How to write them?

These tests are very similar to writing Acceptance Tests since they have the same goal in mind.

When should they run?

Once for every build in your CI/CD pipeline.

UPDATE: I’ve written an article that explains how to write proper external contract tests.

Security Tests

Data breaches are happening on a daily basis these days, making these tests a requirement for any serious development. They are also among the hardest to write since they require lots of thinking outside the box.

How to write them?

There are a large number of tests that can be automated by invoking a deployed version of your service. Some low hanging fruit include, but are not limited to:

• Validating that inputs are sanitized and rejected appropriately when invalid
• Attempting to send large amounts of data as “data bombs” to an endpoint
• Checking for valid, non self-signed certificates
• Ensuring that all endpoints require authentication and enforce proper authorization.

UPDATE: I’ve written an in-depth article that shows how to write some of the security tests mentioned in this section.

When should they run?

Because you can never know when or where an attack will come from, it’s best to run them against each environment the application is deployed to.

Load & Performance Tests

Load tests measure how much traffic your service can handle before failing to respond. Performance tests measure the response time of your system at a given load level.

Generally speaking, a service doesn’t need load or performance tests until well into its life. They are mentioned here because any service that achieves a certain level of popularity will eventually need them.

How to write them?

There are tools and frameworks available that will do a lot of the heavy lifting for you (JMeter being a popular one). It’s then just a matter of integrating it to your Continuous Integration/Continuous Deployment (CI/CD) pipeline.

When should they run?

They can be run off-peak, on an environment configured identically to production, or on production itself if you’re feeling adventurous.

UPDATE: I’ve written an article dedicated entirely to load and performance testing.

End to End Tests

Out of all the layers, this is the trickiest of the bunch. Because it integrates a front end with a back-end, they are notoriously hard to get right and are often more trouble than they’re worth. Luckily, with all the service-level testing we’ve done before getting to them, we should only need to test the happiest of paths.

How to write them?

There are many frameworks to automate UI steps, the most popular of which is Selenium as of the time of writing.

When should they run?

They should be part of every build, especially since these are the most brittle tests that you can write. As soon as one fails, you need to be ready to jump on it to fix it.

Putting It All Together

You’ve probably noticed there’s been no mention of integration tests within this article. That’s because all of the layers within the pyramid, excluding the unit tests, should be considered integration tests. They all test the cloud application in its deployed and integrated form.

There are a lot of layers in this pyramid but it’s not necessary to have all of them from day one. Starting with Unit and Acceptance Tests will set a project up nicely for the future. Adding on additional layers as the need arises is a sensible approach to testing a modern, cloud-based application.