Part 1 of 4: Rethinking the quality operating model for AI-native engineering

Here’s a scenario I’d suspect most engineering leaders reading this will recognise:

An incident slips into production. Not a sev-1 or 2, more like a sev-3 that the team commits to resolving in their backlog rather than through on-call, the kind that generates a Slack thread and a slightly awkward retro instead of a proper postmortem. And when you go looking for where it could have been discovered earlier in the process, you find it went through five pairs of hands, that all gave a thumbs up, before it landed in prod and became an incident…

There was the initial “power of three” conversation – the PO, a dev, and a tester, sitting together early in the process, talking through the work before a line of code existed. Nobody raised the risk that eventually caused the incident. It came up again, and the risk got missed again, during grooming and sizing conversations, where the user story could/should have been tested for accuracy, ambiguity, assumptions, and risks, and somehow wasn’t. Then the PR review missed it, because the reviewer was focused on the diff, not the behaviour. The automated checks missed it, because nobody had written a test for that specific scenario, because test automation checks the explicitly documented expectations within the acceptance criteria, and can’t cover un-expectations. And it wasn’t discovered during exploratory testing, because the tester had to batch a team’s worth of feature changes into a handful of test charters to keep up with the throughput demands, and not be seen as a bottleneck…

Five different points where testing is being conducted throughout the SDLC. But is this really a testing failure? I don’t think so. I think it’s an operating model failure.

Testing isn’t dying. It’s evolving. And “testing” was never the same thing as “quality”.

I want to be precise about what I’m meaning here, because this is the bit that tends to get misconstrued.

I am not saying testers are becoming less important. I think the opposite is true (although I do think testers should utilise AI tools more to enhance and evolve their testing), and I’ll spend a good chunk of this series making that case. I’m also clearly not saying testing and quality are the same thing, because they’re not, and treating them as interchangeable is a big part of how we ended up here. Testing is a discipline, a set of practices and judgement calls. Quality is the outcome, the thing that emerges when testing, development, AI agents, and leadership decisions all pull in the same direction.

What I am saying is that testing as a narrowly-viewed, bolted-on activity – the thing that happens after code has been written or a feature has been deployed on a test environment – owned by one or two people with a tester title within each of the dev teams, often gated behind a big single set of automated checks, is evolving into something else entirely. Even the not-so-new evolved view of testing – the view where testing is all about discovery, investigation, exploration, unknowns, variables, and providing information… This is evolving into something else too. It has to! The volume and velocity of product changes have outgrown the idea that quality is something someone does to software after it’s been built.

What doesn’t change, and what becomes more valuable as this evolves, is the discipline underneath it. The skill of thinking in “risk”. The instinct for what’s likely to break and why. The judgement to know which 20% of a system carries 80% of the danger. That expertise doesn’t disappear as testing stops being a bolted-on activity. It gets distributed across the system instead, and the system needs more of it, not less.

So when I say quality has outgrown the old operating model, I mean the organisational-system around it, the mindset people have about it, and the old org-wide testing strategies that just don’t fit anymore. Not the people in quality-based roles, and not the discipline of testing/assessing quality itself.

Why quality is now a network, not a workflow

Think about how quality information actually moves through a modern engineering org right now. Requirements live in Jira, or YouTrack, or three different docs depending on who wrote them last. Code gets written by a developer, reviewed by another developer, occasionally touched by an AI agent, sometimes all three on the same PR. Testing happens continuously, in PRs, in feature branches, in staging, in production monitoring – not in one succinct, tidy phase. And the actual signals about quality, the failed test, the weird log entry, the flaky test nobody’s investigated yet, the risk someone flagged in a comment three weeks ago, are scattered across GitHub, CI, your test runner, and someone’s memory.

Does anyone genuinely believe they still hold that whole picture in their heads, or even in a document? I doubt it, even if the org chart suggests otherwise.

It helps me to think about this the way you’d think about a distributed system, because that’s honestly what it is! You’ve got nodes, your developers, testers, and increasingly your AI agents. You’ve got messages passing between them, PR comments, test failures, risk signals, logs and monitoring. You’ve got protocols, or at least you should, the workflows and quality gates meant to keep everyone roughly in sync. And you’ve got observability, or the lack of it, which is all too often whatever dashboard someone built eighteen months ago that half the team has quietly stopped trusting.

Quality, in this view, isn’t a deliverable a team hands you. It’s an emergent property of how well that system is coordinated. And here’s the uncomfortable bit: most engineering orgs are running a distributed system with no coordination layer at all.

You’ll recognise the symptoms if you’ve got this problem. Duplicate or contradictory tests nobody’s reconciled. PRs that merge with no quality signal attached, because the signal existed somewhere, just not anywhere the merger could see it. AI-generated tests piling up that nobody’s actually reviewed, because generating them was the easy part. Bugs that surface embarrassingly late because the person who spotted the risk early flagged it in a tool three other people never check. Test debt quietly outpacing test coverage, which is a strange thing to say out loud, but I’d bet it’s true for more of you than would like to admit.

That’s not a tooling gap. That’s entropy. And it’s what distributed quality looks like without anything coordinating it.

AI can be a connective tissue

This is where AI earns its place in this whole conversation, and I want to keep this part tool-agnostic, because the principle matters more than any specific product at the moment.

AI is getting good, with the right context setting, at the connective work that used to require a person to manually relay context between tools that were never designed to talk to each other. It can read a PR comment describing a behaviour and decipher what’s actually being asked. It can read a log file and a test failure and tell you whether they’re describing the same problem. It can sit between two tools and surface something neither one could tell you on its own.

That’s the role I think AI is actually playing in this shift. Not replacing the discipline of quality. Becoming the connective tissue the distributed system was always missing, the thing that lets the nodes actually talk to each other instead of each working in its own silo.

What this actually looks like

As you know, I’ve been testing SmartBear’s BearQ agentic testing system against my own apps over the last few months, and a handful of scenarios kept coming up that illustrate this shift well. It’s definitely worth walking through it as an example, because this stuff stops being theoretical fast once you see it in practice, but it’s also worth me pointing out that BearQ is unlike other AI testing products that have been popping up because it’s a system of agents that work as “teammates” with a focus on assessing quality to build a holistic picture of application integrity.

PR-driven test generation. A developer leaves a comment on a PR describing a behaviour, nothing formal, just plain English. A BearQ AI agent reads it, understands the intent, and generates tests that attach back to that same PR. No handoff. No separate ticket. The test exists because the behaviour was described, not because someone remembered to write it later.

Story-driven test creation. A story gets created or updated in your backlog tool. A BearQ agent  reads it and generates acceptance tests or exploratory charters directly from it… There’s no person sitting in the middle translating “what the story says” into “what we should test”. PS – that translation step is exactly the kind of thing that quietly gets skipped when a team’s under pressure.

Behaviour-driven bug reporting. A test fails, or a screen does something it shouldn’t. Instead of a screenshot and a hopeful Slack message, the system captures what actually happened and turns it into a structured, reproducible bug report. (Anyone who’s tried to action a bug report that just says “this looks wrong” just knows why that matters).

Cross-tool quality intelligence. BearQ pulls signals together across GitHub, your backlog, your test runs, your logs, into a single risk picture a leader can actually read in five minutes instead of five tabs. This is the one that matters most at the leadership level, because it’s the difference between guessing where your risk sits and actually knowing.

None of this is evidence that “the philosophy is right because BearQ exists”. It’s actually fully the other way round: BearQ is a useful, concrete example of what the philosophy looks like when someone builds for it properly!

What this means for engineering leaders

If quality is genuinely a distributed system now, the job changes. You’re not just managing a testing team anymore. You’re designing a system, and that part requires a different skill set entirely.

Practically, I think that means a few things become non-negotiable:

Elevating testers into quality system designers, the people who direct AI agents, judge risk, and interpret signals, rather than treating the role as something AI quietly absorbs. For me, this is the single most important shift in this whole article, and it’s really unfortunate that it’s the one I think gets missed most often.

Designing the operating model itself, not just staffing a team and hoping coordination happens naturally. It won’t.

Defining the protocols between your people and your AI agents: Who reviews what. Who has the final say. Where human judgement actually sits in the loop.

Building real observability across the whole quality system, and not just a dashboard for one team’s slice of it.

Treating quality debt as a first-class risk on the same footing as technical debt, not as background noise addressed “when there’s time.”

Enabling your teams to actually work inside AI-augmented workflows, which is a different ask than rolling out a new tool and hoping adoption happens on its own.

And making sure compliance and traceability come out of the system by design, rather than as a separate audit exercise someone scrambles to assemble every quarter.

A simple model for the new quality operating system

I find it useful to think about this as five layers, mostly because it gives you and your team a shared language for where a given problem actually sits.

What comes in. PRs, stories, behaviours, production signals, AI-generated insights. The raw material.

What connects it. The layer, increasingly AI-driven, that links your tools, agents, and workflows together and turns scattered noise into something that looks like a signal.

What runs it. Your actual test execution, environments, CI/CD, agentic QA workflows doing the work.

What it tells you. Risk clustering, structured testing notes, flaky test detection, coverage intelligence, the compliance evidence you’ll eventually need to produce anyway.

What comes back. Feedback into your PRs, your stories, your dashboards, and ultimately the decisions you’re making as a leader.

A tool like BearQ can be extra effective in layers two, and four – the connective and intelligence layers. And the other layers are still there for you to design too.

Quality systems are the next leadership frontier

The old model of testing being bolted on as one team’s activity at the end of the pipeline – it cannot scale against AI-accelerated engineering. I don’t think that’s a controversial claim anymore, even if not everyone’s operating model has caught up with it yet.

But the discipline at the core of it, the judgement, the risk instinct, the eye for what breaks and why, has never mattered more. If anything, the leaders who win, over the next few years, will be the ones who figure out how to spread that discipline through the whole system instead of leaving it stranded in one team’s queue.

That’s the system half of the equation. The next piece is harder, and it’s the one I think most leaders are underestimating… you can design the perfect operating model and still watch it fail, because the real bottleneck isn’t architecture. It’s whether your team actually trusts it enough to let go of the old ways of working.

Thanks to SmartBear for giving me early access to BearQ, and the opportunity to write about my thoughts about AI in Quality.