On testing software
I’ve taught some college courses and have always been asked to provide a teaching philosophy as a guide to both students, but also the…
On testing software
I’ve taught some college courses and have always been asked to provide a teaching philosophy as a guide to both students, but also the institution. Your philosophy hopefully puts the students first, and explains the expectations and setting that you strive for. This same idea applies to leadership philosophies. The best leaders that I have had the pleasure of knowing always set expectations with a leadership philosophy. These were usually spoken but sometimes written. Which brings me to the question — As software engineering leaders why don’t we take the same approach when thinking about testing and observability?
Tom Gamon wrote an interesting post based on a paper about the differences between a winner’s game and a loser’s game and cites tennis as an example where professional players win by getting a higher score and amateurs playing the same game win by not beating themselves by succumbing to mistakes. Gammon reasons that software development is a loser’s game as most development is undertaken by amateurs and the key to winning is to deliver a solution that is as error-free and scalable as possible. The key to achieving this goal is testing and observability!
We all know that time and money are finite resources so the goal of testing should never be to achieve perfection (if you want to ship code) but to find the set of tests that you can employ that provides you with the most confidence in the release. This article will explain how to employ a layered approach to testing that ensures the most coverage, adding additional testing capabilities at known areas of concern such as the database layer or contracts with other services.
The intent is to inform and open a discussion versus prescribe one method over the other as ultimately the risk/reward or cost/benefit can only be determined by the owner of the system and the business value or risks associated with varied fault tolerance levels. Let’s start with a standard microservice architecture that uses a data access layer to speak to a SQL database as well as a cache. This application also connects to an external service and is built with an HTTP client and API gateway (perhaps grpc?).
A sample application
Another great reference on testing is Testing Microservices with Mountebank by Brandon Byars. Byars uses the pyramid of testing to explain the various types of testing levels and how and they address the coverage versus time and money question.
Credit should go to Testing Microservices with Mountebank by Brandon Byars Published by Manning Publications, 2018
Unit Tests
Unit tests should be written to test code at the function level. If you are using a language such as go which has an amazing built-in testing suite then you can expect a unit test If you take a look at the drawing above you see that unit tests fill the bottom of the pyramid as they are faster to implement, often used in test-driven development, and are often and if you are operating in a continuous integration / continuous delivery environment these tests are run with every commit. Unit tests are not a silver bullet as many of the more difficult bugs that teams struggle with come from areas outside of the purview of unit tests.
Service Tests
I’m using the term service test in the manner that Byars described as a black box test run in a virtual environment where the dependencies are virtualized meaning these tests are deterministic, they run in an environment where you are in control, such as a lab. We can use this step in the pyramid for functional and non-functional testing. Sure, that sounds great but what does that mean? I’ll go over the three differences below.
First, since we are talking about black box testing we are talking about tests that don’t test the internal code but test the input and expected output of the application. If you want to be assured that services can be released independently or even in a monolithic deployment still want to have the confidence that code is tested and functional at every step of the development process (between releases) you will need to virtualize dependencies. Mountebank is a tool that can be used to automate this by inserting imposters which can record request response tuples and allow developers to create service tests against these imposters masquerading as third-party APIs of other services in your monolith.
Second, since functional testing focuses on the user story and acceptance criteria if we are working under your typical agile framework — and service tests test inputs and expected outputs you can model user scenarios and determine if the service meets the acceptance criteria. This is a far better approach than the product owner demo as the tests can be written before the code is complete and used to gauge how close or far you are from the expected functionality of the feature.
Finally, load testing and performance testing can be done on a service-by-service basis using various commercially available tools, such as selenium for front-end testing, Ixia for testing various types of network traffic at various rates, or in-house solutions such as a custom test harness. It’s important to know the subtle differences between the types of tests.
Load testing aims to determine the maximum load usually in the number of concurrent users that the system will support. During a load test, you are looking for bottlenecks that prevent your system from meeting its service level agreement given x number of concurrent users. Some of the causes of issues that I have witnessed working on systems with millions of concurrent requests per second are bottlenecks at the database layer that involve open connections or latency in the requests being sent to third parties as they experience a surge in traffic. These types of errors are unlikely to be caught at the unit test level and the latter requires that impostors running as part of the service test harness introduce a latency, which still is not representative of real-world conditions. There are subsets of load testing such as stress testing which aims to find and document the breaking point of a piece of software and spike testing which looks at the ability to recover from an unexpected surge in users versus a gradual increase. For a real-world analogy just take a look at the tires that your car uses, load testing can be denoted by the type of tire, commercial, light-duty, or passenger, and indicates that the tire has been tested for use with various loads.
Performance testing aims to determine if the system can handle the load and is more focused on the hardware. The key is to determine if the changes made to the service will work on your existing infrastructure. We’ve become accustomed to running applications in the cloud which have made things like autoscaling and sharded database systems seem like magic, however, there are still consequences to code that is not as performant as it could be in terms of cost, and on metal, performance becomes even more critical as all of the processes running share the same resources and spiking CPU in one service can have an impact on other services. I’ve seen this manifest itself as dropped packets when enabling rule sets in Suricata that do not allow the engine to have any idle time.
Contract Test
Contract testing mirrors service tests, however, this time the dependencies are real so we are effectively removing the lab. This can be done by standing up a stack that points to the same third-party APIs and running tests against this stack. In most cases, exploratory tests using manual testers are run against this stack. This step of testing is focused on validating the assumptions in the service tests above. If we are dealing with a monolith then this can be viewed as a soak test where you are looking for stability over x number of weeks or days under a normal load.
Exploratory Tests
Exploratory tests include manual testing and aim to root out the trickiest problems or unexpected behaviors by using real users to identify issues that are technically sound but functionally take away from the user experience.
Observability
If you have read this far congratulations! This may be the most important point that this article can make — testing will not solve all of your problems, the software is complex and the pursuit of perfection will leave you paralyzed as an organization, thus the most important thing to understand is that a comprehensive test plan is a series of tradeoffs that mitigate risks to the business without becoming too expensive or time-consuming and thus affecting the business it is aiming to protect. Every industry and application will have different levels of tolerance. With that in mind, you must build in observability from the start to enable your team to detect errors and respond to them before the customer. I’ve found that solutions such as New Relic that allow for out-of-the-box CUSUM type calculations on traffic patterns and availability provide a good place to start. Of course, an alert that is raised and never heard is the same as an alert never raised so you should focus on integrating the alerts into a notification system like slack, pager duty, or email.
By Joshua McDonald on April 19, 2021.
Exported from Medium on August 26, 2026.
Reader discussion