This Is How To Make Continuous Testing

8 min readSep 3, 2021

Continuous Testing is the way to get continuous feedback on our software delivery process. It aims to regularly assess the quality of what we are building versus our expectations as soon as possible. This is where testing automation makes all sense.

Done right, Continuous Testing accelerates the delivery and improves the overall quality attributes of the software being delivered. Teams are able to correctly with more ease an issue detected early.

Continuous Testing starts from the needs expression up to the software operations, where our user experience takes place. Our test automation must therefore be able to be quickly deployed and leveraged by the software actors at every stage.

The actors let us come to the additional value of transparency and improved collaboration with Continuous Testing. The actors have to share, align and co-build the testing processes to get end-to-end, early and valuable feedback; avoiding siloed optimizations.

In this article, we share how you can implement Continuous Testing using Cerberus Testing, the 100% open-source test automation framework. We cover the following steps of Continuous Testing with:

Test Planning
Test Engineering
Test Deployment
Test Operations
Test Measurement

Let’s start by the test planning.

Follow Cerberus Testing for more open source test automation.

First things first, Continuous Test Planning

It all starts with an application. An opportunity to provide value to specific users translates into the need to build a software application. From that point, you have to define the what and the how of the automated tests.

An application referential is the first building block of a continuous testing framework. You have to know the overall applications available, their state, number of bugs, environments, etc. The role of an application repository is therefore to keep the big picture up-to-date, mastering the IT assets for the next stages of Continuous Testing.

Figure 1: The Test Repository of Cerberus Testing allowing the assets visibility

The second element is the test repository containing all test cases per application. It should contain the typology of tests, their execution status and detailed steps. You can that way access existing tests defined that can be very useful to get information on the application structure. Additionally, you can leverage part of the existing tests when replacing an existing application.

The last element of Continuous Test Planning is defining how the tests will be performed. A framework that provides native integrations and automation for the various test typologies save you time. It is better to have the big picture with a centralized vision, the work can then be decentralized more easily.

Once we know the right test to implement, we can move the implementing the test right.

Fast implementation with Continuous Test Engineering

Test Engineering follows the similar structure of DevOps for software applied to testing. In Continuous Test Engineering, you have to implement fast feedback loops of designing the planned tests, implementing them and then deploying them into production.

Test design requires to quickly compose the tests from business requirements into an effective test automation implementation. The composition of modules with both business and technical elements is therefore essential to avoid switching between the two worlds, losing valuable time.

Figure 2: The Cerberus Testing test structures facilitate collaboration with BDD

The frameworks of ATDD (Acceptance Test Driven Development) and BDD (Behavior Driven Development) are very useful to that effect. Combined with a keyword-driven test framework, you completely bridge the gap removing the need for specific mapping or translations.

When you add the capability of composing tests with reusable modules, you can quickly create and adapt existing tests based on new requirements. This framework capability is modular testing. This is where you can be x10 times faster in both implementation and maintenance. To complete your feedback loop, you need to verify the implementation with rapid execution.

Figure 3: You can quickly compose tests with reusable steps in Cerberus Testing

A test automation framework must enable fast test execution. It means that optimized network traffic, parallel execution and flexible integration with local or remote farms makes the difference. Cerberus Testing supports all these various requirements for an effective test execution. You can literally execute testing loops lasting a few seconds using the “Rerun” functionalities.

Leveraging a framework, you concentrate on what is more valuable, deploying valuable tests.

Get end-to-end feedbacks with Continuous Test Deployment

The deployment of automated tests requires a high standard in Continuous Testing. We don’t limit ourselves to a manual launch in the testing environment; we have to ensure the automated execution of test suites at every stage of the deployment process.

The first step is to actually compose the various test suites with ease, speed and flexibility. You can achieve that in Cerberus Testing creating test campaigns with different filters available: per test case, application, system, environment, tags. You have the power to compose them in the user interface easily. You can additionally define the parameters of scheduling, notifications and quality gates scores.

Figure 4: Cerberus Testing natively provides test campaign management

You can then address the deployment of our test suites. Our goal is to execute the test campaigns as early as possible for the identified stages. In a traditional CI/CD environment, you want to execute non-regression tests quickly after a deployment of the application in any environment. You can leverage the native connectors with Jenkins, Azure DevOps, among others to achieve that with Cerberus Testing.

The added value of a continuous testing framework is to natively provide you the whole set of necessary integrations. You have the application repository to know all the applications, environments, and connection information. You then have all the test definitions, associated with application, tags and features. You then join the two worlds with the test suites, defining which tests should be executed in which context of application and environments.

Your continuous testing loops continue when your users start interacting with the application.

Get Continuous User Feedbacks with Continuous Test Operations

Continuous Testing in production environments can meet various objectives and consequently relies on different implementations. The most obvious test is the one of user experience followed by the practices used to improve reliability and performance.

The monitoring of the user experience is commonly referred to as Customer Journey Monitoring. They enable the team to keep the user perspective in their various measurements efforts. Google has widely spread the concept of SRE, aiming at measuring service level indicators based on key user services. You can natively use Cerberus Testing to schedule monitoring campaigns reusing your non-regression tests and specific tags.

Figure 5: Cerberus Testing supports the Customer Journey Monitoring use-case.

The second objective of improving reliability and performance is about specific practices usually referred to as “Test in production”. We find here Chaos engineering, progressive deployments and web performance. They are more technically oriented practices even if at the end of the day, they improve the user experience. Cerberus Testing allows you to implement various non-functional tests you can integrate with the various tools available. For example, you can have a specific post-deployment campaign to run just after 1% of the infrastructure has the new version available.

While we close the continuous testing loop from design to production, we still need to improve the big picture. This is where we can use reporting.

Continuous improvement with Continuous Test Measurement

All our Continuous Testing effort is about creating value for the actors and the users. We have to make an impact at the end of the day, else we are just busy, not focusing on the right priorities. This is where a global measurement across the chain avoids local optimizations and pitfalls.

Our first question in Continuous Testing is to know the “as is”. This is where reporting functionalities are useful. We need to know how many applications we have, how many tests and their associated states, the results of our campaigns including their execution time and stability ratio. Cerberus Testing provides you the native reporting capabilities of the test repository and test execution. All the meta-data available in the tests are then very helpful to drill-down and improve the visibility.

Figure 6: The native Test Analytics of Cerberus Testing to enable actionable insights

Our second type of questions will focus on improving what we are doing and its impacts. In the data domain, we differentiate reporting with analytics; the latest being more focused on insights and making decisions. In the world of test automation, the analytics must provide a view over-time of your various test assets. In Cerberus Testing, you can leverage native dashboards to answer questions like “Is the experience stable on all devices for all countries?”, “What is the flakiness ratio of our campaigns over-time?”, “Which features and use-cases are more unstable at each release?”.

Reporting and analytics support this need for continuous measurement to drive improvements.

Continuous Testing is leveraging Continuous Feedback Loops

This is what Continuous Testing is about, using fast and continuous feedback loops to improve our user experience as soon as possible. This focus on the users is key to align the most valuable tests, else we risk losing effort for non-necessary activities.

Continuous Testing has to be available anywhere and anytime in the software delivery value-chain. A test automation framework with native integration, composition and scalability capabilities is a must have.

Cerberus Testing provides these integration capabilities while supporting a variety of test techniques and test typologies. You can implement mobile tests in CI/CD, while having a monitoring campaign in production for your web and app experiences.

There is no time to code a framework, our users have no time to wait for improvements.

Stop coding and start testing with Cerberus Testing for free.