AI testing and debugging

AI testing and debugging is the model pointed at one question: does the code work? It writes tests, reads a stack trace or a log and says where to look, turns a bug report into a reproduction, narrows a failure down to the change that caused it, and proposes a fix. Increasingly it also probes running software from the outside, the way an attacker would.

It is the most undersold category in the taxonomy, and the one that makes every rung safer. A builder on the base rung, an agent in the editor and a delegated agent all produce code whose correctness someone has to establish. This is how establishing it gets cheaper. It is also how it gets faked.

What it is for

• Writing the tests nobody wrote, especially around code you are about to change.
• Turning a bug report into a failing test before anyone touches the fix.
• Explaining an unfamiliar stack trace, error or log, and suggesting where the cause might be.
• Generating the inputs a person would not think of: the empty list, the negative amount, the name with an apostrophe, the request sent twice.
• Narrowing a failure down: which commit, which input, which line.

Who it suits

Everyone, on every rung, and most of all anyone reviewing code they did not write. It suits engineers who already know what the code is supposed to do, because a test is only as good as the requirement it checks. It suits teams whose suite is thin, provided someone reads what gets added to it.

How to tell this is your category

• You are asking whether something works, or why it does not, rather than asking for something new.
• The output is a test, a reproduction, a diagnosis or a fix for a known failure.
• If you are judging a change someone else made, whether it is right and whether it belongs, that is AI code review, refactoring and maintenance. The two work together: review asks the questions, and tests make the answers repeatable.

What it costs you later

• Tests that describe the code instead of checking it. A test generated from the implementation records what the code does and calls that the expectation. Its usual shapes are easy to miss in review: the expected value copied from the current output, a mock that returns whatever the code needs so the test exercises the mock, an assertion so loose it cannot fail. One quick check catches most of them. Break the code on purpose and see whether a test goes red. If nothing fails, the suite was not checking that line.
• The symptom fixed, the cause kept. Asked to make an error go away, a model can do exactly that: catch the exception, add a null check, retry the call, widen the type. The failing test passes and the bug moves somewhere quieter. Ask for the cause in one sentence before any fix, and do not accept a fix without a test that failed before it and passes after.
• Edited tests. When the instruction is to make the suite green, changing the test is the shortest path there. Read changes to test files before changes to code, and treat any change to an assertion as a change to the requirement, which needs a reason.
• A suite nobody reads. Generated tests are cheap, so suites grow faster than anyone understands them. Slow, duplicated and flaky tests follow, and a test nobody understands is the first to be skipped when it fails. Keep the tests you can explain, and delete the ones that only repeat others.
• Finding outpaces fixing. Agents that probe software for faults can surface them faster than a team can fix them. A finding without a reproduction is noise, and a queue of real findings nobody has time for is a list of known exposures. AI is finding bugs faster than teams can fix them makes the case on the security side.
• What you paste travels. Stack traces, logs and failing payloads carry more than the error: customer data, tokens, internal addresses. Know where they go and how long they are kept before you paste them into a tool, and strip what the diagnosis does not need.

Tests written from the requirements

The defence against a suite that confirms whatever the code does is order. Write the acceptance tests from the requirements before the code exists, or have them written that way and read them first. The agent can add tests of its own, but it does not change those without a reason you accept. Writing requirements an AI coding agent can build from covers making requirements testable in the first place.

Example tests check the cases someone thought of. Property-based tests check a rule across many generated inputs, and they find the edge cases that pass the example tests and break in production. A property is a requirement restated as something that must hold for every input:

- For any order and any sequence of refunds, the total refunded never
  exceeds the amount paid.
- For any refund request sent twice with the same idempotency key,
  exactly one refund is recorded.
- For any amount held in minor units, formatting it for display and
  parsing it back returns the same whole number.

Each line becomes one test, run against many generated orders, keys and amounts. When one fails, the test reports the smallest failing input it can find, which is usually the edge case nobody wrote down.