From ebee6330e86a58515fddfb36511dc1b4107f8620 Mon Sep 17 00:00:00 2001 From: andyblundell Date: Mon, 10 Mar 2025 11:44:34 +0000 Subject: [PATCH 01/26] Update testing.md --- practices/testing.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/practices/testing.md b/practices/testing.md index 2a085966..e7610307 100644 --- a/practices/testing.md +++ b/practices/testing.md @@ -26,6 +26,7 @@ - **Quality is the whole team's responsibility** - Education on testing and testing principles should be important to the whole team. - Quality approaches should be driven as a team and implemented by everyone. + - Teams should consider running regular coaching / mentoring sessions to support colleagues who are less experienced in testing to grow their skills, for example by no-blame group discussions to identify edge-case tests which have so far been missed, tests positioned incorrectly in the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html), etc. - **Combining business knowledge with testing knowledge yields better quality outcomes** - Include business knowledge and critical thinking as part of assurance @@ -49,6 +50,7 @@ - Eg. using API testing tools to aid exploratory API testing - **Testing should be continually improved** + - [Peer reviews](../patterns/everything-as-code.md#code-review) must consider tests as a first-class concern - this includes tests that are present / have been added (e.g. whether they are positioned appropriately in the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html), whether they are triggered appropriately in CI builds, etc) and any tests that are missing, e.g. edge-cases not yet considered - **Testing is continuous** - Testing is a continuous activity, not a phase of delivery From ba552294ae628b8436a3af118c9ffc0b8eb163de Mon Sep 17 00:00:00 2001 From: andyblundell Date: Mon, 10 Mar 2025 13:08:06 +0000 Subject: [PATCH 02/26] Update structured-code.md --- practices/structured-code.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/practices/structured-code.md b/practices/structured-code.md index 639efd32..52fe6b24 100644 --- a/practices/structured-code.md +++ b/practices/structured-code.md @@ -9,6 +9,15 @@ - These notes are part of a broader set of [principles](../principles.md) - These practices should be read in conjunction with [architect for flow](../patterns/architect-for-flow.md) +## Benefits + +The benefits of well-structured & clean code are profound & widespread, some highlights are: + +- Promoting *maintainability* by generally making the code easier and safer to work on +- Supporting *building for testability*, which hugely reduces the risk and effort of practicing good testing + +The above are fundamental to supporting the (little and often)[../patterns/little-and-often.md] delivery approach, which itself has many benefits and is at the heart of this framework + ## Details - Good code structure is essential for maintainability. From 2cef90f256356de9f974f612d8e3b6e79273c6dc Mon Sep 17 00:00:00 2001 From: andyblundell Date: Mon, 10 Mar 2025 13:08:32 +0000 Subject: [PATCH 03/26] Update structured-code.md --- practices/structured-code.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/practices/structured-code.md b/practices/structured-code.md index 52fe6b24..68c35c6e 100644 --- a/practices/structured-code.md +++ b/practices/structured-code.md @@ -16,7 +16,7 @@ The benefits of well-structured & clean code are profound & widespread, some hig - Promoting *maintainability* by generally making the code easier and safer to work on - Supporting *building for testability*, which hugely reduces the risk and effort of practicing good testing -The above are fundamental to supporting the (little and often)[../patterns/little-and-often.md] delivery approach, which itself has many benefits and is at the heart of this framework +The above are fundamental to supporting the [little and often](../patterns/little-and-often.md) delivery approach, which itself has many benefits and is at the heart of this framework ## Details From 30212ef67b0ce34876d605853e9d674001919dd4 Mon Sep 17 00:00:00 2001 From: andyblundell Date: Mon, 10 Mar 2025 13:12:47 +0000 Subject: [PATCH 04/26] Update everything-as-code.md --- patterns/everything-as-code.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/patterns/everything-as-code.md b/patterns/everything-as-code.md index e99b950f..1ba96bd8 100644 --- a/patterns/everything-as-code.md +++ b/patterns/everything-as-code.md @@ -70,6 +70,9 @@ While effective testing is the best way to detect bugs or non-functional problem - Is the code clear and simple? - Is the code layout and structure consistent with agreed style and other code? (please see [enforce code formatting](enforce-code-formatting.md)) - Would it easily allow future modification to meet slightly different needs, e.g. ten times the required data size or throughput? +- Is it [built for testability](#)? +- Are the automated tests positioned appropriately in the [test pyramid](#), triggered appropriately in CI builds, etc? +- Are there any missing [automated tests](#), e.g. edge-cases that have not yet been considered? - Have the non-functional requirements been considered (performance, scalability, robustness, etc)? - Are common security issues guarded against (e.g. [OWASP Top 10](https://owasp.org/www-project-top-ten/))? Including: - Is any new input data being treated as potentially hostile? From a8af75f3c9ab6dc64cdd20788e11408108b70086 Mon Sep 17 00:00:00 2001 From: andyblundell Date: Mon, 10 Mar 2025 13:14:07 +0000 Subject: [PATCH 05/26] Update everything-as-code.md --- patterns/everything-as-code.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/patterns/everything-as-code.md b/patterns/everything-as-code.md index 1ba96bd8..f9a7fb09 100644 --- a/patterns/everything-as-code.md +++ b/patterns/everything-as-code.md @@ -70,7 +70,7 @@ While effective testing is the best way to detect bugs or non-functional problem - Is the code clear and simple? - Is the code layout and structure consistent with agreed style and other code? (please see [enforce code formatting](enforce-code-formatting.md)) - Would it easily allow future modification to meet slightly different needs, e.g. ten times the required data size or throughput? -- Is it [built for testability](#)? +- Is it [built for testability](../practices/structured-code.md)? - Are the automated tests positioned appropriately in the [test pyramid](#), triggered appropriately in CI builds, etc? - Are there any missing [automated tests](#), e.g. edge-cases that have not yet been considered? - Have the non-functional requirements been considered (performance, scalability, robustness, etc)? From d70377ccfb8a4299c68fa84bbc7a844ad5b6282e Mon Sep 17 00:00:00 2001 From: andyblundell Date: Mon, 10 Mar 2025 13:15:39 +0000 Subject: [PATCH 06/26] Update everything-as-code.md --- patterns/everything-as-code.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/patterns/everything-as-code.md b/patterns/everything-as-code.md index f9a7fb09..aac42c1b 100644 --- a/patterns/everything-as-code.md +++ b/patterns/everything-as-code.md @@ -71,8 +71,8 @@ While effective testing is the best way to detect bugs or non-functional problem - Is the code layout and structure consistent with agreed style and other code? (please see [enforce code formatting](enforce-code-formatting.md)) - Would it easily allow future modification to meet slightly different needs, e.g. ten times the required data size or throughput? - Is it [built for testability](../practices/structured-code.md)? -- Are the automated tests positioned appropriately in the [test pyramid](#), triggered appropriately in CI builds, etc? -- Are there any missing [automated tests](#), e.g. edge-cases that have not yet been considered? +- Are the automated tests positioned appropriately in the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html), triggered appropriately in CI builds, etc? +- Are there any missing [automated tests](../practices/testing.md), e.g. edge-cases that have not yet been considered? - Have the non-functional requirements been considered (performance, scalability, robustness, etc)? - Are common security issues guarded against (e.g. [OWASP Top 10](https://owasp.org/www-project-top-ten/))? Including: - Is any new input data being treated as potentially hostile? From 16043456696b2e714ee0c4dedf49927826d9d233 Mon Sep 17 00:00:00 2001 From: andyblundell Date: Mon, 10 Mar 2025 13:18:13 +0000 Subject: [PATCH 07/26] Update principles.md --- principles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/principles.md b/principles.md index 361142a0..0f45643c 100644 --- a/principles.md +++ b/principles.md @@ -50,7 +50,7 @@ The following practices support the principle of building quality in. **[Test automation.](practices/testing.md)** Use test-driven development: Write the tests hand in hand with the code it is testing to ensure code is easily testable and does just enough to meet the requirements. -**[Protect code quality](patterns/everything-as-code.md)** to keep code easy to maintain. +**[Protect code quality](patterns/everything-as-code.md)** to keep code easy to maintain and to [build for testability](patterns/everything-as-code.md#benefits). **Write less code.** Treat code as a liability rather than an asset: the more code, the more there is to go wrong. Incremental delivery and test driven development both help keep the codebase small and simple. From 1f1e6c319bbe43adc36414fed841c0b4ffc3a205 Mon Sep 17 00:00:00 2001 From: andyblundell Date: Mon, 10 Mar 2025 13:21:33 +0000 Subject: [PATCH 08/26] Update principles.md --- principles.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/principles.md b/principles.md index 0f45643c..d9140f35 100644 --- a/principles.md +++ b/principles.md @@ -48,9 +48,9 @@ The following practices support the principle of building quality in. **Pair programming**. Avoid quality issues by combining the skills and experience of two developers instead of one. Take advantage of navigator and driver roles. Also consider cross-discipline (e.g. dev-test) pairing. -**[Test automation.](practices/testing.md)** Use test-driven development: Write the tests hand in hand with the code it is testing to ensure code is easily testable and does just enough to meet the requirements. +**[Test automation.](practices/testing.md)** and **[build for testability](practices/structured-code.md#benefits)** Use test-driven development: Write the tests hand in hand with the code it is testing to ensure code is easily testable and does just enough to meet the requirements. -**[Protect code quality](patterns/everything-as-code.md)** to keep code easy to maintain and to [build for testability](patterns/everything-as-code.md#benefits). +**[Protect code quality](patterns/everything-as-code.md)** to keep code easy to maintain. **Write less code.** Treat code as a liability rather than an asset: the more code, the more there is to go wrong. Incremental delivery and test driven development both help keep the codebase small and simple. From fc9a2e581101e86c12315d09a88cdf9fc0b6d88c Mon Sep 17 00:00:00 2001 From: andyblundell Date: Mon, 10 Mar 2025 13:22:08 +0000 Subject: [PATCH 09/26] Update principles.md --- principles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/principles.md b/principles.md index d9140f35..debb6540 100644 --- a/principles.md +++ b/principles.md @@ -48,7 +48,7 @@ The following practices support the principle of building quality in. **Pair programming**. Avoid quality issues by combining the skills and experience of two developers instead of one. Take advantage of navigator and driver roles. Also consider cross-discipline (e.g. dev-test) pairing. -**[Test automation.](practices/testing.md)** and **[build for testability](practices/structured-code.md#benefits)** Use test-driven development: Write the tests hand in hand with the code it is testing to ensure code is easily testable and does just enough to meet the requirements. +**[Test automation.](practices/testing.md)** and **[build for testability](practices/structured-code.md)** Use test-driven development: Write the tests hand in hand with the code it is testing to ensure code is easily testable and does just enough to meet the requirements. **[Protect code quality](patterns/everything-as-code.md)** to keep code easy to maintain. From f62c2463b45a1966f7c9b164f47209a580787f7d Mon Sep 17 00:00:00 2001 From: andyblundell Date: Mon, 10 Mar 2025 13:28:59 +0000 Subject: [PATCH 10/26] Update review.md --- insights/review.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/insights/review.md b/insights/review.md index 8afda90d..1e3d913f 100644 --- a/insights/review.md +++ b/insights/review.md @@ -167,7 +167,10 @@ You may wish to score each individual component or system separately for these a #### 9. Testing - We have great test coverage. -- Testing is everyone's responsibility. +- Testing is everyone's responsibility and test is a first-class concern. +- We support all team members to practice good testing, including by holding no-blame sessions to discuss any automation tests we should have added, and what we can learn from having missed them initially. +- We build code for testability. +- Tests are part of our standard peer-review process. - Repetitive tests are automated. - Testing is considered before each work item is started and throughout its delivery. - We use the right mix of testing techniques including automated checks and exploratory testing. From 41a7db03758617de4601ab70fe013bd15f54aa22 Mon Sep 17 00:00:00 2001 From: andyblundell Date: Mon, 10 Mar 2025 13:30:13 +0000 Subject: [PATCH 11/26] Update review.md --- insights/review.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/insights/review.md b/insights/review.md index 1e3d913f..0a60eb0c 100644 --- a/insights/review.md +++ b/insights/review.md @@ -170,7 +170,7 @@ You may wish to score each individual component or system separately for these a - Testing is everyone's responsibility and test is a first-class concern. - We support all team members to practice good testing, including by holding no-blame sessions to discuss any automation tests we should have added, and what we can learn from having missed them initially. - We build code for testability. -- Tests are part of our standard peer-review process. +- Tests (including both test code and test coverage & whether there are gaps) are part of our standard peer-review process. - Repetitive tests are automated. - Testing is considered before each work item is started and throughout its delivery. - We use the right mix of testing techniques including automated checks and exploratory testing. From 50687f638910c0da8372aa53adabfdcc04aa45d9 Mon Sep 17 00:00:00 2001 From: andyblundell Date: Mon, 10 Mar 2025 17:36:54 +0000 Subject: [PATCH 12/26] Update patterns/everything-as-code.md Co-authored-by: Alex Young --- patterns/everything-as-code.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/patterns/everything-as-code.md b/patterns/everything-as-code.md index aac42c1b..6f83f274 100644 --- a/patterns/everything-as-code.md +++ b/patterns/everything-as-code.md @@ -71,7 +71,7 @@ While effective testing is the best way to detect bugs or non-functional problem - Is the code layout and structure consistent with agreed style and other code? (please see [enforce code formatting](enforce-code-formatting.md)) - Would it easily allow future modification to meet slightly different needs, e.g. ten times the required data size or throughput? - Is it [built for testability](../practices/structured-code.md)? -- Are the automated tests positioned appropriately in the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html), triggered appropriately in CI builds, etc? +- Are the automated tests positioned appropriately in the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html), triggered appropriately in CI builds, and do they block the build when they fail? - Are there any missing [automated tests](../practices/testing.md), e.g. edge-cases that have not yet been considered? - Have the non-functional requirements been considered (performance, scalability, robustness, etc)? - Are common security issues guarded against (e.g. [OWASP Top 10](https://owasp.org/www-project-top-ten/))? Including: From c5f213042010af960bb8b37b9554e3507f36c1c4 Mon Sep 17 00:00:00 2001 From: andyblundell Date: Mon, 10 Mar 2025 17:37:25 +0000 Subject: [PATCH 13/26] Update practices/testing.md Co-authored-by: Alex Young --- practices/testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/practices/testing.md b/practices/testing.md index e7610307..5c5f4b12 100644 --- a/practices/testing.md +++ b/practices/testing.md @@ -26,7 +26,7 @@ - **Quality is the whole team's responsibility** - Education on testing and testing principles should be important to the whole team. - Quality approaches should be driven as a team and implemented by everyone. - - Teams should consider running regular coaching / mentoring sessions to support colleagues who are less experienced in testing to grow their skills, for example by no-blame group discussions to identify edge-case tests which have so far been missed, tests positioned incorrectly in the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html), etc. + - Teams should consider running regular coaching / mentoring sessions to support colleagues who are less experienced in testing to grow their skills, for example by no-blame group discussions to identify edge-case tests which have so far been missed, tests positioned incorrectly in the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html), and pairing with a tester navigating so that the driver learns the necessary skills. - **Combining business knowledge with testing knowledge yields better quality outcomes** - Include business knowledge and critical thinking as part of assurance From 36a29b0671dc881479c2d7196b03e3beb99300d3 Mon Sep 17 00:00:00 2001 From: andyblundell Date: Mon, 10 Mar 2025 17:37:46 +0000 Subject: [PATCH 14/26] Update insights/review.md Co-authored-by: Alex Young --- insights/review.md | 1 + 1 file changed, 1 insertion(+) diff --git a/insights/review.md b/insights/review.md index 0a60eb0c..9ac3f938 100644 --- a/insights/review.md +++ b/insights/review.md @@ -168,6 +168,7 @@ You may wish to score each individual component or system separately for these a - We have great test coverage. - Testing is everyone's responsibility and test is a first-class concern. +- A failing test suite in CI gets immediate attention. - We support all team members to practice good testing, including by holding no-blame sessions to discuss any automation tests we should have added, and what we can learn from having missed them initially. - We build code for testability. - Tests (including both test code and test coverage & whether there are gaps) are part of our standard peer-review process. From 07db966e51ba9529c59cb8e9ff9dbf291f936e89 Mon Sep 17 00:00:00 2001 From: Dan Stefaniuk Date: Tue, 11 Mar 2025 09:03:49 +0000 Subject: [PATCH 15/26] Add and improve the 'General testing principles' section --- practices/testing.md | 111 ++++++++++++++++++++++++++++++++++++++----- 1 file changed, 100 insertions(+), 11 deletions(-) diff --git a/practices/testing.md b/practices/testing.md index 5c5f4b12..d850a44b 100644 --- a/practices/testing.md +++ b/practices/testing.md @@ -23,31 +23,120 @@ ## General testing principles +- **Design for testability**, and [shift testing left and right](https://www.redhat.com/en/topics/devops/shift-left-vs-shift-right) + + Testing is most effective when it is baked into the system design and runs across the entire lifecycle-from development to production. Teams should build systems that are inherently testable and support both early validation ("shift left") and ongoing validation in live environments ("shift right"). Key Practices: + + - Shift left, aka test early + - Testing starts at the design and coding phase, not after. + - Pre-commit hooks, linting, static code analysis, and unit tests run locally before code even hits a branch. + - [Test-Driven Development (TDD)](https://www.thoughtworks.com/en-gb/insights/blog/test-driven-development-best-thing-has-happened-software-design) and [Behavior-Driven Development (BDD)](https://www.thoughtworks.com/en-gb/insights/blog/applying-bdd-acceptance-criteria-user-stories) encourage writing tests before or alongside code, ensuring clarity of requirements and better design. + - Test planning is informed by risk analysis and [architectural decisions](../any-decision-record-template.md) made early on. + - Design for testability + - Systems are designed with observability, modularity, and controllability in mind. + - Expose clear APIs, provide injection points for test doubles (mocks/stubs), and avoid tight coupling. + - Feature toggles and dependency injection help test components in isolation without complex setups. + - Make non-functional testing (performance, security, resilience) a first-class concern, with hooks and controls to simulate adverse conditions. + - Design for reproducibility + - Tests should be idempotent and easily repeatable in any environment (local, test, staging, production). + - Shift right, aka test in production + - Testing does not stop at deployment-continuous validation in production is essential. + - Implement real-time monitoring, synthetic checks, health probes, and user behavior tracking. + - Use canary deployments, blue-green releases, and feature flags to roll out changes gradually, monitoring for issues as they surface. + - Employ chaos engineering to test system resilience under real-world failure conditions. + - Instrument systems to detect anomalies, performance degradation, or unexpected behaviors automatically. + + In a high-throughput environment-where deploying at least once a day is the norm, adhering to the design for testability principle is paramount. The benefits include: 1) *faster feedback loops* – early testing catches issues when they are cheapest to fix, while testing later in the cycle ensures real-world readiness; 2) *increased confidence* – testing at all stages validates assumptions, improves system reliability, and supports safe, frequent releases; and 3) *higher quality by design* – systems built for testability are easier to maintain, scale, and evolve. + - **Quality is the whole team's responsibility** - Education on testing and testing principles should be important to the whole team. - Quality approaches should be driven as a team and implemented by everyone. - Teams should consider running regular coaching / mentoring sessions to support colleagues who are less experienced in testing to grow their skills, for example by no-blame group discussions to identify edge-case tests which have so far been missed, tests positioned incorrectly in the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html), and pairing with a tester navigating so that the driver learns the necessary skills. + - Testing is a shared team concern, not a tester’s job alone. Developers own testing for their code, including pipeline tests, monitoring their deploys, infrastructure and on-call responsibilities, as deploying at least once a day requires autonomy and accountability. - **Combining business knowledge with testing knowledge yields better quality outcomes** - - Include business knowledge and critical thinking as part of assurance - - Intrinsic knowledge and mindset of the team is key to driving quality outcomes + - Include business knowledge and critical thinking as part of technical assurance. + - Intrinsic knowledge and mindset of the team is key to driving quality outcomes. - **Testing is prioritised based on risk** - - A testing risk profile is defined and understood by the whole team, including the customer - - Risk appetite should be worked across the whole team, including customers and/or users - - Solution Assurance risks and Clinical Safety hazards must also be considered when prioritising risks + - A testing risk profile is defined and understood by the whole team, including the customer. + - Risk appetite should be worked across the whole team, including customers and/or users. + - Assurance risks and Clinical Safety hazards must also be considered when prioritising risks. - **Testing is context driven** - Context should be considered when deciding on test techniques and tools to use. +- **Test data management is a first-class citizen** + + Frequent deployments require reliable and consistent test environments. Data drift or stale data can undermine test confidence. + + - Test data should be easy to generate, isolate and reset. + - Use factories, fixtures or synthetic data generation. + - Anonymised production data can improve test relevance, especially in performance or exploratory testing. + +- **Consistent, CLI-driven test execution across all environments** + + Tests and test processes should execute consistently in every environment, ranging from local developer workstations to cloud-based CI/CD pipelines. Using a CLI-driven approach ensures standardisation, portability and reliability. + + - Command-Line interface (CLI) as the default test runner + - All tests (unit, integration, functional, performance) must be executable through straightforward, repeatable CLI commands. + - Ensure a single, consistent command can run the complete test suite, facilitating rapid local and remote execution , e.g. `make test` + - Consistent environment configuration + - Clearly defined and documented dependencies (IaC) ensure that test environments are reproducible, reducing "it works on my machine" scenarios. + - Use Infrastructure as Code (IaC) or containerised test environments (e.g. Docker) to guarantee identical configurations between local machines and cloud pipelines. + - Reproducibility and portability + - Tests must behave identically when run locally and remotely. No tests should rely on hidden state, manual configuration, or proprietary local tooling. + - Standardise environment configuration through version-controlled configuration files or scripts, enabling teams to replicate exact test runs on any workstation or CI/CD environment effortlessly. + - Dependency isolation and management + - Dependencies should be explicitly declared and managed using tools appropriate to your technology stack (e.g. Python’s requirements.txt, Node’s package.json, etc.). + - Employ dependency management tools (e.g. virtual environments, containers, package managers) to enforce consistency. + - Environment parity between development and production + - Aim to eliminate differences between local, staging and production environments. Running tests consistently across environments ensures that deployment to production is predictable and low-risk. + - Teams regularly validate environment parity through automated checks or smoke tests. + - Clear and consistent documentation + - Standardised CLI test commands and environment setups must be clearly documented (e.g. README.md) and version-controlled. + - Onboarding documentation should guide new developers to execute the same tests consistently across their local and cloud environments. + +- **Validate continuously through observability** + + Effective testing does not stop once software reaches production. By integrating observability into testing, teams gain real-time insights and continuously validate system behavior under real-world conditions. Observability-driven testing means using telemetry data, such as metrics, logs, tracing and user analytics, to shape test approach, validate assumptions, detect regressions early and drive continuous improvement. + + - Instrument systems for visibility + - Implement consistent instrumentation (metrics, logs, tracing) across services to capture detailed runtime behavior. + - Ensure telemetry data clearly maps back to business functionality, enabling both technical and business stakeholders to interpret the data. + - Continuous monitoring in production + - Use dashboards and automated alerts to monitor system health continuously, proactively detecting anomalies, performance degradation or unexpected behaviors. + - Regularly verify production health checks and synthetic monitoring results as part of your ongoing testing activities. + - Real-user monitoring (RUM) + - Observe and analyze how real users interact with the system, capturing actual usage patterns, performance characteristics and edge-case scenarios. + - Leverage this data to refine existing automated tests or identify new scenarios worth automating. + - Distributed tracing to inform testing + - Use distributed tracing data (such as OpenTelemetry, AWS X-Ray or Azure Monitor Application Insights) to understand how requests flow through services, identify latency hotspots and pinpoint complex dependency issues. + - Translate tracing insights into targeted integration tests and service-level tests, improving test precision. + - Alerting and proactive issue detection + - Set clear, actionable alerts based on predefined thresholds that matter to users and the business. + - Tie production alerts back into automated test scenarios, ensuring tests reflect actual production conditions and preventing similar issues from recurring. + - Feedback loops into test planning + - Regularly analyze observability data (logs, metrics, user sessions) during sprint planning or retrospectives to identify gaps in testing coverage. + - Treat production incidents as opportunities for testing improvements, each incident should trigger analysis of whether similar risks are sufficiently covered by automated or exploratory tests. + - Testing resilience and failure modes + - Observability supports chaos engineering practices by providing detailed visibility into system behavior under fault conditions. + - Proactively test recovery procedures, failovers and resilience strategies based on observed patterns of failure or degradation from production data. + + Applying this principle reduces mean-time-to-detection and recovery, improving reliability, enables teams to validate assumptions using real data rather than guesswork, enhances the quality of future releases by continuously learning from real-world usage patterns. Increases confidence when releasing frequently, knowing production issues can be quickly identified, understood and addressed. + - **Testing is assisted by automation** - - Appreciate that not everything can be automated - - Identify good candidates for automation - particular focus on high risk and repeatable areas - - Automated tests should be used to provide confidence to all stakeholders. This includes test analysts themselves who should be familiar with what the tests are doing to allow them to make decisions on what they want to test. + + Test automation is critical for maintaining rapid, frequent deployments while consistently ensuring quality. It provides scalable confidence in software changes, reduces repetitive manual efforts, and frees up human activities for high-value exploratory testing. Automated testing should be seen as a core enabler of development workflow, particularly when combined with a robust approach to design for testability. + + - Appreciate that not everything can be automated, however automated testing, supported by intentional design for testability, increases delivery speed, confidence and adaptability. + - Identify good candidates for automation - particular focus on high risk and repeatable areas. + - Automated tests should be used to provide confidence to all stakeholders. This includes test analysts themselves who should be familiar with what the tests are doing to allow them to make decisions on what they want to test. + - It defines clear, technology-neutral contracts and behaviors. This provides stable reference points when migrating or re-implementing systems in new languages or platforms. Automated contract tests (e.g. consumer-driven contract tests) enable safe technology swaps, helping confirm system compatibility across evolving stacks. - Automated test packs should be maintained regularly to ensure they have suitable coverage, are efficient and providing correct results. - - Consider using testing tools to enhance other test techniques. - - Eg. using record and play tools to aid exploratory UI testing - - Eg. using API testing tools to aid exploratory API testing + - Consider using testing tools to enhance other test techniques, e.g. + - using record and play tools to aid exploratory UI testing, + - using API testing tools to aid exploratory API testing. - **Testing should be continually improved** - [Peer reviews](../patterns/everything-as-code.md#code-review) must consider tests as a first-class concern - this includes tests that are present / have been added (e.g. whether they are positioned appropriately in the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html), whether they are triggered appropriately in CI builds, etc) and any tests that are missing, e.g. edge-cases not yet considered From bbf08022141c55349d74e284ac9d08dc1d5beb17 Mon Sep 17 00:00:00 2001 From: andyblundell Date: Tue, 11 Mar 2025 18:04:35 +0000 Subject: [PATCH 16/26] Remove reference to support --- practices/testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/practices/testing.md b/practices/testing.md index d850a44b..157871f6 100644 --- a/practices/testing.md +++ b/practices/testing.md @@ -52,7 +52,7 @@ - Education on testing and testing principles should be important to the whole team. - Quality approaches should be driven as a team and implemented by everyone. - Teams should consider running regular coaching / mentoring sessions to support colleagues who are less experienced in testing to grow their skills, for example by no-blame group discussions to identify edge-case tests which have so far been missed, tests positioned incorrectly in the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html), and pairing with a tester navigating so that the driver learns the necessary skills. - - Testing is a shared team concern, not a tester’s job alone. Developers own testing for their code, including pipeline tests, monitoring their deploys, infrastructure and on-call responsibilities, as deploying at least once a day requires autonomy and accountability. + - Testing is a shared team concern, not a tester’s job alone. Developers own testing for their code. - **Combining business knowledge with testing knowledge yields better quality outcomes** - Include business knowledge and critical thinking as part of technical assurance. From 3fce283ca315ef6654b7f45bc7b5d4d926ebde99 Mon Sep 17 00:00:00 2001 From: andyblundell Date: Wed, 12 Mar 2025 14:23:35 +0000 Subject: [PATCH 17/26] Update design for testability section --- practices/testing.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/practices/testing.md b/practices/testing.md index 157871f6..5061f9e2 100644 --- a/practices/testing.md +++ b/practices/testing.md @@ -33,7 +33,7 @@ - [Test-Driven Development (TDD)](https://www.thoughtworks.com/en-gb/insights/blog/test-driven-development-best-thing-has-happened-software-design) and [Behavior-Driven Development (BDD)](https://www.thoughtworks.com/en-gb/insights/blog/applying-bdd-acceptance-criteria-user-stories) encourage writing tests before or alongside code, ensuring clarity of requirements and better design. - Test planning is informed by risk analysis and [architectural decisions](../any-decision-record-template.md) made early on. - Design for testability - - Systems are designed with observability, modularity, and controllability in mind. + - Build systems as small units, each of which can be tested in isolation. - Expose clear APIs, provide injection points for test doubles (mocks/stubs), and avoid tight coupling. - Feature toggles and dependency injection help test components in isolation without complex setups. - Make non-functional testing (performance, security, resilience) a first-class concern, with hooks and controls to simulate adverse conditions. @@ -42,9 +42,9 @@ - Shift right, aka test in production - Testing does not stop at deployment-continuous validation in production is essential. - Implement real-time monitoring, synthetic checks, health probes, and user behavior tracking. - - Use canary deployments, blue-green releases, and feature flags to roll out changes gradually, monitoring for issues as they surface. - - Employ chaos engineering to test system resilience under real-world failure conditions. - - Instrument systems to detect anomalies, performance degradation, or unexpected behaviors automatically. + - Use canary deployments and feature flags to support testing changes as they are deployed. + - When safe to do so, employ chaos engineering to test system resilience under real-world failure conditions. + - Instrument systems to detect anomalies, performance degradation, or unexpected behaviors automatically - to support good quality canary deployments. In a high-throughput environment-where deploying at least once a day is the norm, adhering to the design for testability principle is paramount. The benefits include: 1) *faster feedback loops* – early testing catches issues when they are cheapest to fix, while testing later in the cycle ensures real-world readiness; 2) *increased confidence* – testing at all stages validates assumptions, improves system reliability, and supports safe, frequent releases; and 3) *higher quality by design* – systems built for testability are easier to maintain, scale, and evolve. From 223282f6a703318d059719b3a594d4026cab7545 Mon Sep 17 00:00:00 2001 From: andyblundell Date: Wed, 12 Mar 2025 14:25:27 +0000 Subject: [PATCH 18/26] Typos --- practices/testing.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/practices/testing.md b/practices/testing.md index 5061f9e2..91839384 100644 --- a/practices/testing.md +++ b/practices/testing.md @@ -40,13 +40,13 @@ - Design for reproducibility - Tests should be idempotent and easily repeatable in any environment (local, test, staging, production). - Shift right, aka test in production - - Testing does not stop at deployment-continuous validation in production is essential. + - Testing does not stop at deployment: continuous validation in production is essential. - Implement real-time monitoring, synthetic checks, health probes, and user behavior tracking. - Use canary deployments and feature flags to support testing changes as they are deployed. - When safe to do so, employ chaos engineering to test system resilience under real-world failure conditions. - Instrument systems to detect anomalies, performance degradation, or unexpected behaviors automatically - to support good quality canary deployments. - In a high-throughput environment-where deploying at least once a day is the norm, adhering to the design for testability principle is paramount. The benefits include: 1) *faster feedback loops* – early testing catches issues when they are cheapest to fix, while testing later in the cycle ensures real-world readiness; 2) *increased confidence* – testing at all stages validates assumptions, improves system reliability, and supports safe, frequent releases; and 3) *higher quality by design* – systems built for testability are easier to maintain, scale, and evolve. + In a high-throughput environment where deploying at least once a day is the norm, adhering to the design for testability principle is paramount. The benefits include: 1) *faster feedback loops* – early testing catches issues when they are cheapest to fix, while testing later in the cycle ensures real-world readiness; 2) *increased confidence* – testing at all stages validates assumptions, improves system reliability, and supports safe, frequent releases; and 3) *higher quality by design* – systems built for testability are easier to maintain, scale, and evolve. - **Quality is the whole team's responsibility** - Education on testing and testing principles should be important to the whole team. From e725a910d2ec141c134be98c64a877192be86f66 Mon Sep 17 00:00:00 2001 From: andyblundell Date: Wed, 12 Mar 2025 14:26:38 +0000 Subject: [PATCH 19/26] Update testing.md --- practices/testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/practices/testing.md b/practices/testing.md index 91839384..d343b6c2 100644 --- a/practices/testing.md +++ b/practices/testing.md @@ -46,7 +46,7 @@ - When safe to do so, employ chaos engineering to test system resilience under real-world failure conditions. - Instrument systems to detect anomalies, performance degradation, or unexpected behaviors automatically - to support good quality canary deployments. - In a high-throughput environment where deploying at least once a day is the norm, adhering to the design for testability principle is paramount. The benefits include: 1) *faster feedback loops* – early testing catches issues when they are cheapest to fix, while testing later in the cycle ensures real-world readiness; 2) *increased confidence* – testing at all stages validates assumptions, improves system reliability, and supports safe, frequent releases; and 3) *higher quality by design* – systems built for testability are easier to maintain, scale, and evolve. + In a high-throughput environment where deploying at least once a day is the norm, adhering to the design-for-testability principle is paramount. The benefits include: 1) *faster feedback loops* – early testing catches issues when they are cheapest to fix, while testing later in the cycle ensures real-world readiness; 2) *increased confidence* – testing at all stages validates assumptions, improves system reliability, and supports safe, frequent releases; and 3) *higher quality by design* – systems built for testability are easier to maintain, scale, and evolve. - **Quality is the whole team's responsibility** - Education on testing and testing principles should be important to the whole team. From 1f7867511bef59dcee535750ca1b4ea417975e27 Mon Sep 17 00:00:00 2001 From: andyblundell Date: Wed, 12 Mar 2025 14:29:07 +0000 Subject: [PATCH 20/26] Update testing.md --- practices/testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/practices/testing.md b/practices/testing.md index d343b6c2..679d86e8 100644 --- a/practices/testing.md +++ b/practices/testing.md @@ -51,7 +51,7 @@ - **Quality is the whole team's responsibility** - Education on testing and testing principles should be important to the whole team. - Quality approaches should be driven as a team and implemented by everyone. - - Teams should consider running regular coaching / mentoring sessions to support colleagues who are less experienced in testing to grow their skills, for example by no-blame group discussions to identify edge-case tests which have so far been missed, tests positioned incorrectly in the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html), and pairing with a tester navigating so that the driver learns the necessary skills. + - Teams should consider running regular coaching/mentoring sessions to support colleagues who are less experienced in testing to grow their skills, for example by no-blame group discussions to identify edge-case tests which have so far been missed, tests positioned incorrectly in the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html), and pairing with a tester navigating so that the driver learns the necessary skills. - Testing is a shared team concern, not a tester’s job alone. Developers own testing for their code. - **Combining business knowledge with testing knowledge yields better quality outcomes** From 74df23b52b86aef2da48774324f184cbee125975 Mon Sep 17 00:00:00 2001 From: andyblundell Date: Wed, 12 Mar 2025 14:32:35 +0000 Subject: [PATCH 21/26] Update testing.md --- practices/testing.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/practices/testing.md b/practices/testing.md index 679d86e8..f150655a 100644 --- a/practices/testing.md +++ b/practices/testing.md @@ -51,7 +51,9 @@ - **Quality is the whole team's responsibility** - Education on testing and testing principles should be important to the whole team. - Quality approaches should be driven as a team and implemented by everyone. - - Teams should consider running regular coaching/mentoring sessions to support colleagues who are less experienced in testing to grow their skills, for example by no-blame group discussions to identify edge-case tests which have so far been missed, tests positioned incorrectly in the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html), and pairing with a tester navigating so that the driver learns the necessary skills. + - Teams should consider running regular coaching/mentoring sessions to support colleagues who are less experienced in testing to grow their skills, for example by: + - No-blame group discussions to identify edge-case tests which have so far been missed, tests positioned incorrectly in the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html). + - Pairing developers with a tester navigating so that the driver learns the necessary testing skills. - Testing is a shared team concern, not a tester’s job alone. Developers own testing for their code. - **Combining business knowledge with testing knowledge yields better quality outcomes** From b4821e1148fe3cfe7b9b20ad7f62b95bc2eec684 Mon Sep 17 00:00:00 2001 From: andyblundell Date: Wed, 12 Mar 2025 14:33:29 +0000 Subject: [PATCH 22/26] Update testing.md --- practices/testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/practices/testing.md b/practices/testing.md index f150655a..9eb97097 100644 --- a/practices/testing.md +++ b/practices/testing.md @@ -52,7 +52,7 @@ - Education on testing and testing principles should be important to the whole team. - Quality approaches should be driven as a team and implemented by everyone. - Teams should consider running regular coaching/mentoring sessions to support colleagues who are less experienced in testing to grow their skills, for example by: - - No-blame group discussions to identify edge-case tests which have so far been missed, tests positioned incorrectly in the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html). + - Holding no-blame group discussions to identify edge-case tests which have so far been missed and tests positioned incorrectly in the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html). - Pairing developers with a tester navigating so that the driver learns the necessary testing skills. - Testing is a shared team concern, not a tester’s job alone. Developers own testing for their code. From 829269fe6c87c18609fe0e7bd36206066283ee0d Mon Sep 17 00:00:00 2001 From: andyblundell Date: Wed, 12 Mar 2025 14:36:52 +0000 Subject: [PATCH 23/26] Update testing.md --- practices/testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/practices/testing.md b/practices/testing.md index 9eb97097..e99eba51 100644 --- a/practices/testing.md +++ b/practices/testing.md @@ -74,7 +74,7 @@ - Test data should be easy to generate, isolate and reset. - Use factories, fixtures or synthetic data generation. - - Anonymised production data can improve test relevance, especially in performance or exploratory testing. + - Make sure that you can generate test data of a scale and complexity representative of the production system, to ensure that performance and exploratory testing is realistic. - **Consistent, CLI-driven test execution across all environments** From e77e5a1bd861b6968f74a22a1aa9bf7237edea81 Mon Sep 17 00:00:00 2001 From: andyblundell Date: Wed, 12 Mar 2025 14:44:02 +0000 Subject: [PATCH 24/26] Update testing.md --- practices/testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/practices/testing.md b/practices/testing.md index e99eba51..1c7c135d 100644 --- a/practices/testing.md +++ b/practices/testing.md @@ -90,7 +90,7 @@ - Tests must behave identically when run locally and remotely. No tests should rely on hidden state, manual configuration, or proprietary local tooling. - Standardise environment configuration through version-controlled configuration files or scripts, enabling teams to replicate exact test runs on any workstation or CI/CD environment effortlessly. - Dependency isolation and management - - Dependencies should be explicitly declared and managed using tools appropriate to your technology stack (e.g. Python’s requirements.txt, Node’s package.json, etc.). + - Dependencies should be explicitly declared and managed using tools appropriate to your technology stack (e.g. Python’s requirements.txt, Node’s package.json, etc.). Use these tools to ensure that specific versions are locked. - Employ dependency management tools (e.g. virtual environments, containers, package managers) to enforce consistency. - Environment parity between development and production - Aim to eliminate differences between local, staging and production environments. Running tests consistently across environments ensures that deployment to production is predictable and low-risk. From a50ea9c9db9fa9827e624718dc2c77b01677645c Mon Sep 17 00:00:00 2001 From: andyblundell Date: Wed, 12 Mar 2025 14:46:58 +0000 Subject: [PATCH 25/26] Update testing.md --- practices/testing.md | 24 +----------------------- 1 file changed, 1 insertion(+), 23 deletions(-) diff --git a/practices/testing.md b/practices/testing.md index 1c7c135d..db5804c5 100644 --- a/practices/testing.md +++ b/practices/testing.md @@ -101,29 +101,7 @@ - **Validate continuously through observability** - Effective testing does not stop once software reaches production. By integrating observability into testing, teams gain real-time insights and continuously validate system behavior under real-world conditions. Observability-driven testing means using telemetry data, such as metrics, logs, tracing and user analytics, to shape test approach, validate assumptions, detect regressions early and drive continuous improvement. - - - Instrument systems for visibility - - Implement consistent instrumentation (metrics, logs, tracing) across services to capture detailed runtime behavior. - - Ensure telemetry data clearly maps back to business functionality, enabling both technical and business stakeholders to interpret the data. - - Continuous monitoring in production - - Use dashboards and automated alerts to monitor system health continuously, proactively detecting anomalies, performance degradation or unexpected behaviors. - - Regularly verify production health checks and synthetic monitoring results as part of your ongoing testing activities. - - Real-user monitoring (RUM) - - Observe and analyze how real users interact with the system, capturing actual usage patterns, performance characteristics and edge-case scenarios. - - Leverage this data to refine existing automated tests or identify new scenarios worth automating. - - Distributed tracing to inform testing - - Use distributed tracing data (such as OpenTelemetry, AWS X-Ray or Azure Monitor Application Insights) to understand how requests flow through services, identify latency hotspots and pinpoint complex dependency issues. - - Translate tracing insights into targeted integration tests and service-level tests, improving test precision. - - Alerting and proactive issue detection - - Set clear, actionable alerts based on predefined thresholds that matter to users and the business. - - Tie production alerts back into automated test scenarios, ensuring tests reflect actual production conditions and preventing similar issues from recurring. - - Feedback loops into test planning - - Regularly analyze observability data (logs, metrics, user sessions) during sprint planning or retrospectives to identify gaps in testing coverage. - - Treat production incidents as opportunities for testing improvements, each incident should trigger analysis of whether similar risks are sufficiently covered by automated or exploratory tests. - - Testing resilience and failure modes - - Observability supports chaos engineering practices by providing detailed visibility into system behavior under fault conditions. - - Proactively test recovery procedures, failovers and resilience strategies based on observed patterns of failure or degradation from production data. + Effective testing does not stop once software reaches production. By integrating [observability](observability.md) into testing, teams gain real-time insights and continuously validate system behavior under real-world conditions. Observability-driven testing means using telemetry data, such as metrics, logs, tracing and user analytics, to shape test approach, validate assumptions, detect regressions early and drive continuous improvement. Applying this principle reduces mean-time-to-detection and recovery, improving reliability, enables teams to validate assumptions using real data rather than guesswork, enhances the quality of future releases by continuously learning from real-world usage patterns. Increases confidence when releasing frequently, knowing production issues can be quickly identified, understood and addressed. From 0f7239c2a0b3ce54c582c85e9e2556fbb9891507 Mon Sep 17 00:00:00 2001 From: andyblundell Date: Wed, 12 Mar 2025 14:48:06 +0000 Subject: [PATCH 26/26] Update testing.md --- practices/testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/practices/testing.md b/practices/testing.md index db5804c5..d6acad3b 100644 --- a/practices/testing.md +++ b/practices/testing.md @@ -103,7 +103,7 @@ Effective testing does not stop once software reaches production. By integrating [observability](observability.md) into testing, teams gain real-time insights and continuously validate system behavior under real-world conditions. Observability-driven testing means using telemetry data, such as metrics, logs, tracing and user analytics, to shape test approach, validate assumptions, detect regressions early and drive continuous improvement. - Applying this principle reduces mean-time-to-detection and recovery, improving reliability, enables teams to validate assumptions using real data rather than guesswork, enhances the quality of future releases by continuously learning from real-world usage patterns. Increases confidence when releasing frequently, knowing production issues can be quickly identified, understood and addressed. + Applying this principle reduces mean-time-to-detection and recovery, improving reliability, enables teams to validate assumptions using real data rather than guesswork, and enhances the quality of future releases by continuously learning from real-world usage patterns. It increases confidence when releasing frequently, knowing production issues can be quickly identified, understood and addressed. - **Testing is assisted by automation**