Sam Owens joins us to tell us all about our approach to testing at Form3. He gives us an overview of our testing strategy, the different types of tests we run and explains how to use Pact for testing your services. Finally, he tells us why he prefers BDD style tests.
Sam Owens is Head of Architecture at Form3. He guides the technical decisions that our engineers make across the platform. Generally, his role involves coordinating decision making. At Form3, we take a collaborative approach to system design and Sam is in charge of facilitating decision discussions.
We spend a lot of time focusing on automation, so we don't have dedicated QA teams. Instead, it is the responsibility of the teams that build the software to ensure that it is well tested and production ready. We run our teams using the DevOps model.
We really believe that focusing on end-to-end tests is the key to that. We put a lot of onus on allowing people to write tests that cover as much of their software in a black box fashion, without relying on unit tests that mock out different behaviour. This allows us to have a very high quality test suite at the end of it.
We have a variety of functional tests:
On the non-functional testing side, we run load tests on our platforms. We also run continuous load on our development environments to ensure that our systems are able to scale correctly.
In general, we don't have many unit tests of our codebases, as we have found low level unit tests (for example ones around a single function) are too tied to the implementation details of the code. Re-implementing services then requires refactoring the tests (sometimes even deleting them!), which can be cumbersome, time consuming and also means it is hard to be confident in the new implementation when you have to change the test too. This allows us to swap underlying services and dependencies without having to rewrite a whole bunch of unit tests.
For us at Form3, end to end tests using Pact for contract testing hits the sweet spot between speed of delivery and test coverage.
Because they are in charge of the full development lifecycle, our engineers have more ownership of our services, which has proven a very successful approach for us. In the 5 years that we've been running our platform, we've not had major incidents caused by defects in production.
Writing the majority of our tests as end-to-end tests can be more complex, especially as our architecture involves asynchronous processing. Brittle or flaky tests are frustrating for engineers.
Fixing timing issues with sleeps and waits or depending on log messages to synchronise your tests are common traps. These quick fixes should be avoided.
Pact is an implementation of consumer contract testing. During test suite definition, we specify the interaction between the consumer and the service. The test suite will specify the URL, parameter and expected responses. These are put in a contract that specifies all of these things together.
Pact then provides a mock server that verifies requests and responses together. The test runner is able to ensure that the server behaviour is as expected both on the consumer and server side. This allows us to test both sides of this contract.
Even though our services are written in Go, we write all of our tests in BDD style, as opposed to table testing.
The choice to write our tests in BDD gives us higher test readability, which is important for our engineers as maintainers.
blogs · 7 min
Daniel Teixeira, Lead of Offensive Security at Form3 discusses exploiting Distroless images, covering the topics of:
September 22, 2022
blogs · 5 min
Natan Yellin joins us to his insights on the challenges of running software at scale, which now involves maintaining more complex system architecture than ever. Then, he walks us through the open-source tool Robusta Dev and how it can make running systems on Kubernetes easier!
September 15, 2022
blogs · 4 min
Evelina Vrabie joins us to share her insights into measuring the success of engineering teams. She tells us about the role of an engineering manager as well as the four types of success. Then, she walks us through how to measure productivity and high performance through research-based frameworks.
August 16, 2022