Recoding Engineering Culture

Making quality part of every product

Agnes Pasztor and Franziska Ströbel, Senior Software Engineers on the AMBOSS Quality team explain how they shifted the mindset of the AMBOSS product and engineering teams regarding quality from one that’s passive and outside of any individual’s control, to one that’s part of every product built.

Written by Agnes Pasztor and Franziska Ströbel

Traditional software development separates development from testing

Developers develop, then testers test. This involves creating and using test cases and plans, applying heuristics to strategically approach a problem, follow up on their discoveries by creating issue tickets, and finally escalating and triaging defects according to severity and priority.

Putting the testing process late in the software development lifecycle costs both time and money — bugs found by the testers get handed back to developers who then need to make changes to things that were already considered ‘done’. Rinse and repeat, and you’ll find yourself dealing with bottlenecks and delays.

Eventually, this forces stakeholders to make a choice — release bugs, or postpone and address technical debt.

When the development cycle continues without addressing technical debt, a product eventually becomes unattractive or simply unusable. In an attempt to overcome this problem, teams often consider hiring more testers. This however doesn’t negate the systemic nature of this issue.

To combat the ideological and professional separation of these two deeply intertwined activities, we’re bringing around a better system at AMBOSS. Quality Engineering.

What is Quality Engineering?

In a nutshell, it’s a culture of positive accountability and responsibility. One that lets us align globally and across teams in order to build quality products. By having common goals, unified processes, and sharing our learnings, we’re able to better use our resources.

“Quality Engineering is a culture of positive accountability and responsibility.”

We can establish standards and create documentation that’s easily accessible at all times and insightful for everyone. From a more technical, and practical, point of view the Quality Engineering (QE) mindset aspires to make testing possible along every step of the software development lifecycle.

Continuous and early testing combined with easily accessible and transparent test reporting and data repositories is by far the clearest, most structured, cost effective and swift way to keep a product functioning well while retaining paying customers.

This is why we’re shifting left. Scalable solutions are rarely based on lots of manual overhead. Having a solid foundation of unit tests as well as the continued automation of business critical, regression, or other high priority test suites, ensures future scalability.

Monitoring our QE metrics, such as the number of critical bugs, code coverage, amount of regressions and so on, uncovers our blind spots and guides us when planning our testing and development efforts.

Switching from a traditional Quality Assurance to a modern Quality Assistance way of working is similarly scalable to automated testing — we can ensure that quality related skill sets are made available to all members of our engineering teams. When everyone operates in a tester role, more defects, design flaws, and vanity implementations are found and pass through the crowd-sourced filter made up of diverse points of view.

“When everyone operates in a tester role, more defects, design flaws, and vanity implementations are found and pass through the crowd-sourced filter made up of diverse points of view.”

Introducing testing early on in the development lifecycle is the only way to make sure testing can be efficiently scaled. Considering testability when designing a feature, ensuring that specifications are documented, and that automated tests are updated to reflect the requirements, seldom works well when left until after features have been implemented.

However, it’s important to note that this approach does not attempt to dictate how to do things on an individual or team level. We want every team to be able to choose how to adopt and implement it in a way that’s best for their way of working. Team level quality rituals, processes and metrics should be carefully and empathetically tailored around each teams’ needs.

What is Quality Assistance?

Quality Assistance decentralises all quality considerations. It is the methodology to the QE framework. It enables every team member in a cross functional product team to contribute to quality by distributing the necessary skills. It’s called assistance because of how information is channeled back to engineers, just like a private tutor in school.

“Delivering less broken software is easier as a supportive team that has been given the right set of tools to operate autonomously and confidently.”

Using QE, teams are surrounded by quality-minded individuals all contributing to product and development. Our approach aims to empower and positively impact everyone involved. We’re doing away with the false assumptions that developers don’t have what it takes to “break” a product — especially their own. In truth, a product isn’t broken by a tester. It’s already broken when the tester gets to work.

Delivering less broken software is easier as a properly supportive team that has been given the right set of tools to operate autonomously and confidently.

We can do this by adjusting our existing processes to modern requirements and ways of working:

• Pair or team testing sessions: using our Test Management System’s guided test runs via a Zoom session with the whole team or a select audience
• Adjusting the Definition of Ready and Definition of Done to specifically require testing implications: steps to test, edge and error case considerations, and automation
• Issue and bug type tickets are estimated and retain story points
• Having clear team-tailored defect reporting and triage processes
• Making team-wide use of test case repositories: own your test cases
• Team-established quality metrics dashboards enabling everyone to investigate and assess data
• Making heavy use of test automation: unit, API, integration, e2e tests
• Establishing shared best practices and automate deployments accordingly: code style, design patterns, architecture, linting, git hygiene, code coverage, CI configurations

Changing mindsets and habits is hard. It requires patience and a whole lot of positivity and perseverance to take effect. People may not react positively to what you’re trying to do, or they may be suspicious of the new way of working.

Adjustment periods should be expected and are a personal thing to go through. We never force people to do things they don’t agree with, and always try to show that we’re here to help them and work with them.

“Much like regular mentoring, quality specialists assist other team members who haven’t spent as much time working with quality.”

Much like regular mentoring, quality specialists assist other team members who haven’t spent as much time working with quality. Since mentoring and changing processes are not one-way-streets, patience and empathetically driving this mindset forward are our utmost priorities!

What’s in it for everyone?

Product owners neither want to see their products decline in quality, nor compromise on introducing new features.

Continuous testing creates a technical landscape allowing for the development of high-quality products, enabling faster feature development and A/B testing, while significantly adding to a better user experience, retention, and customer lifetime value.

Educated test planning and a distributed, concerted execution of testing combined with an empowered, team-centric release management process are surefire ways to build up release reliability and product confidence.

This way, firefighting, ad-hoc fixes, missed deadlines, spontaneous releases and anxiety are past areas of stress.

As an engineer, designer, product owner or engineering manager you’re likely already testing to make sure your implementation works as expected and doesn’t break anything.

As an engineer, sharing the workload of manual testing will provide you with a unique and valuable insight into areas of the product that might otherwise be hidden.

You’ll gain a more holistic understanding of the product landscape at your company and access to a wealth of data, aiding release-related considerations. It will also keep you more engaged with your own work. Usually engineers are more invested if they contribute to testing and creating a quality mindset for themselves and their peers.

Your professional toolbox gets expanded, making you more self-sufficient, technically versed and consequently much more employable. Oftentimes being able to see a product through the entire software development lifecycle is a sign of seniority, enabling you to reach higher professional spheres!

On top of that, decentralising and crowdsourcing manual testing overhead comes with the expressed benefit of QA tasks taking much less time, leaving story points for nice-to-haves and can’t-do-nows.

Hosting a team-owned testing session for regression testing of release candidates, which includes coordinating who tests what on which devices and in what environments while using proper test suites as guides, has taken time for pre-release testing down from 3 days with two people, to approximately 30 to 45 minutes with four to eight people.

This way, you’ll enable automation to progress faster. By chipping in on maintaining said automation you can hope to never be blocked again by flaky tests. By realistically planning both implementation and testing, you can be sure that your feature will make it into the release and won’t be taken out at the last minute due to uncertainty.

As a designer, you know best where a planned flow may go haywire. Your team will benefit from your insight and you’ll get the opportunity to directly contribute to making sure that everything works as expected.

Product and Engineering Managers will also gain more in-depth and unique understandings of their work and people. Observing testing sessions, listening to team members consider quality and anticipate issues (such as checking regression and old crashes) will help you make decisions that aren’t always supported by large, peer-reviewed data sets.

Since responsibilities have shifted, pressure eases and job satisfaction may grow. Taking risks can be done in a calculated way, allowing more freedom within product development.

Last but not least, a good user experience is heavily dependent on both design and technical stability. Product significance measured against metrics, such as customer lifetime value, is only positively influenced by a well designed and implemented product.

Recoding a culture is a monumental and collective undertaking

It’s vital that old experiences and assumptions are challenged in a positive and empathetic way, and are slowly replaced by new and better impressions of how your world could look.

Taking tiny steps one day at a time, we have the power and ability to undo an ancient, unjust ideological and practical separation holding us back from what could be.

With the evolution of Agile to Continuous, it’s worth considering how product and marketing methodologies should adjust to fit our shift in a more appropriate way.

Principles of Continuous could also be applied to these disciplines, making for a better-fitting global strategy.

In our next quality-related article, we’ll focus on test automation in a QE context. Stay tuned to nerd out with us!

Frequently asked questions & answers

Who’s going to be the QA engineer on my team?

The role of holding each other accountable and taking responsibility for the quality of the product doesn’t vanish when a team no longer hires a dedicated QA engineer to do all that.

We look at it as a role that can ideally be embodied by anyone on the team.

What kind of team metrics are useful when using the Quality Engineering approach?

There is no canned response for that. It depends on the team and the product they’re creating. A product manager may find certain crash events important to monitor and prevent in the future.

The team may decide that they want to start tracking how many bugs pour in after a release, or track the reduction of their tech debt by keeping an eye on the number/age of bugs that are already open.

Ultimately, you’ll need the metrics which make you confident about sending something out the door — it’s your choice.

How does rolling out the Quality Engineering approach actually work?

We start by talking to representatives of the team to determine their current quality situation.

We ask questions such as:

• Who usually does the testing?
• How are they doing it?
• What bad habits should be broken?

If for example, the team lets the Product Owner get away with not writing user stories, that’s a problem. We look into the issues they know they have and try to explore the team’s blind spots together.

Once this is established, we come up with a roadmap completely tailored to the team and that is regularly adjusted.

We may think doing one workshop about the basics of manual testing is enough to kick things off and everyone will be able to start coming up with ideas for what to test.

Then, we find out that they have no idea how to solve issues in their automated tests and deploys are blocked. They would rather just comment out all the tests. We aim to address all those challenges as part of the process.

Other resources

The Culture Code: Secrets of Highly Successful Groups by Daniel Coyle

Lessons Learned in Software Testing: A Context-Driven Approach by Bret Pettichord, Cem Kaner, and James Marcus Bach

Continuous resources by Martin Fowler; software developer, author and international speaker