Taking over a generated prototype

This is for the moment a prototype from an AI app builder has real users and someone has to make it a codebase a team can change. Maybe you built it, maybe you were handed it. The Prompt to app layer page gives the route in four steps: characterise what exists, write the requirements it never had, put tests around the paths that would hurt most, then change it one reviewed step at a time. This guide is the working method for each step, in the order that keeps the data safe.

One rule holds until the last step: nothing changes. Mapping and pinning come first, because a fix made while you are still finding out what the application does changes the behaviour you are trying to record. The one exception is a live exposure, covered in the first two steps.

The example is a small shop with sign-in, a cart, discount codes and orders, the kind of application these builders produce in an afternoon.

Before you start

• Get the code into a repository you control. Export it, commit it exactly as received, and tag that commit so there is always a way back to the starting point.
• Stop changing it through the builder. From here the repository is the source of truth. Asking the builder for another change regenerates code you are in the middle of pinning, and leaves two versions of the application that disagree.
• Work against a copy of the data. Restore a backup somewhere other than production, with personal data removed or replaced. If restoring it fails, you have found the first thing to fix, and found it on a quiet day. Never characterise against live data.
• Find every key now. A key in code the browser downloads is readable by any visitor, and it is exposed today rather than in some later release. Search for them first, and rotate any you find in client code before doing anything else.

grep -rnIiE "api[_-]?key|secret|token|password" src public

The search over-matches on purpose. Read every hit, and move each real secret to the server and the platform's secret store.

Step 1: Map what exists

Write the document the application never had. It records what is there, not what should be, and it is where every question goes that you cannot answer yet.

## Data
 
| Table   | One row is   | Written by                 | Notes                     |
| ------- | ------------ | -------------------------- | ------------------------- |
| orders  | one checkout | `createOrder`, orders API  | `total` is a float        |
| refunds | (no table)   |                            | `orders.refunded_at` only |
 
## Access, as the rules are written today
 
- orders: a customer can read rows where `user_id` is their own.
  Any signed-in customer can update any row (rule `orders_update`).
 
## Outside services
 
- Payments: secret key read on the server. Webhook has no signature check.
- Email: key was in `src/lib/email.js`, shipped to the browser. Rotated.
 
## Unknowns
 
- Is the discount meant to come off shipping? Today it does.
- Why does `status` hold both `cancelled` and `canceled`?

Four sections do most of the work.

• Data. Every table, what one row means, and what writes to it. For each entity, ask whether it can happen more than once. A refund held as a timestamp on the order cannot record a partial refund or a failed attempt, and Designing a feature an AI coding agent cannot improvise explains why that matters before more orders arrive.
• Access. Who can read and change each table, in one sentence each, from the rules as they are written. Write what the rule says, not what it was meant to say. The gap between the two is the most important thing this document can find.
• Outside services. Every provider the application calls, where its key lives, and what happens when the call fails.
• Unknowns. Everything you could not explain. Each one becomes a question for whoever owns the product, not a change.

Step 2: Pin today's behaviour

A characterisation test records what the code does now, quirks included, so that a later change which alters it fails a test instead of passing unnoticed. Start with the paths that would hurt most if they changed by accident: money, identity, and data you cannot recreate.

// Characterisation: what orderTotal returns today, not what it should.
// The expected values were captured by running the current code, then
// checked by hand. A failure here means behaviour changed, on purpose or not.
import { orderTotal } from "../../src/orders/total.js";
 
test("a 10% code on one item at 19.99 with 4.50 shipping", () => {
  // Pinned as found: the discount also comes off shipping, and the total
  // keeps a fraction of a cent. Both are listed in docs/inventory.md.
  const total = orderTotal({
    lines: [{ price: 19.99, quantity: 1 }],
    shipping: 4.5,
    code: "TEN",
  });
  expect(total).toBe(22.041);
});
 
test("an expired code is ignored without an error", () => {
  const total = orderTotal({
    lines: [{ price: 19.99, quantity: 1 }],
    shipping: 4.5,
    code: "SUMMER",
  });
  expect(total).toBe(24.49);
});

Three habits make these tests useful.

• Pin the wrong behaviour too. Where today's result looks wrong, as 22.041 does, pin it anyway and add it to Unknowns. The test changes later, on purpose, in the commit that changes the behaviour.
• Capture by running, then check by hand. Here, copying the current output into the expected value is the point. AI testing and debugging warns against exactly that in an ordinary test, because there the expectation should come from a requirement. This application has no requirements yet, and the job of these tests is to notice change.
• Test through the edges. Call the function a screen calls, or the API, rather than the internals. The refactor that comes later will move the internals, and tests tied to them break for the wrong reason.

Access is tested as a requirement

One kind of test is not characterisation, because the answer is not in doubt: one customer must not be able to read or change another customer's records. Write it as a requirement, sign in as two real users through the real sign-in path, and try to cross the line by changing an identifier.

// Requirement, not characterisation. If either test fails on the
// prototype, fixing it is the first change, before anything else.
// signInAs goes through the real sign-in path. A helper that skips
// sign-in would be testing itself, not the access rules.
import { request, signInAs } from "../helpers/session.js";
 
test("a customer cannot read another customer's order", async () => {
  const ada = await signInAs("ada@example.test");
  const ben = await signInAs("ben@example.test");
  const order = await request(ben, "POST", "/api/orders", { lines: [{ sku: "A1", quantity: 1 }] });
 
  const res = await request(ada, "GET", `/api/orders/${order.body.id}`);
  expect([403, 404]).toContain(res.status);
});
 
test("a customer cannot change another customer's order", async () => {
  const ada = await signInAs("ada@example.test");
  const ben = await signInAs("ben@example.test");
  const order = await request(ben, "POST", "/api/orders", { lines: [{ sku: "A1", quantity: 1 }] });
 
  const res = await request(ada, "PATCH", `/api/orders/${order.body.id}`, { status: "cancelled" });
  expect([403, 404]).toContain(res.status);
});

A flaw of this kind is in the logic rather than the configuration, so a scanner looking for known signatures passes it, the point AI is finding bugs faster than teams can fix them makes. In the inventory above, the second test fails. That is the exception to nothing changing yet, because it is an exposure on live data: fix the rule, keep both tests in the suite, and carry on mapping.

Step 3: Write the requirements it never had

Go through the inventory and turn each decision the application embodies into a requirement, in the form Writing requirements an AI coding agent can build from sets out. Then sort every requirement into one of three bins.

• Keep. Today's behaviour is right. The requirement records it, and the characterisation test that pins it becomes that requirement's test.
• Change. Today's behaviour is wrong. The requirement states the new behaviour, and the characterisation test is rewritten in the commit that makes the change.
• Decide. Nobody knows which is right. These go to whoever owns the product, because they are product decisions, not engineering ones.

1. THE SYSTEM SHALL store and calculate money in integer minor units.
   Today: floats. Change. Needs a migration of `orders.total`.
 
2. WHEN a discount code is applied, THE SYSTEM SHALL discount the
   subtotal only, excluding shipping and tax.
   Today: shipping is discounted too. Decide: the owner confirms.
 
3. IF a submitted discount code has expired, THEN THE SYSTEM SHALL
   reject it and show the date it expired.
   Today: ignored without a message. Change.
 
4. THE SYSTEM SHALL let a customer read and change only their own
   orders.
   Today: fixed in step 2. Keep. Tested in tests/access/orders.test.js.

The Today line under each requirement is what turns this from a wish list into a plan: it says what the code does now, and whether closing the gap is a change, a decision or nothing.

Where a requirement changes what exists, such as a refund that should be a row of its own, the design comes next, and the design post covers the data model, the states and the order of writes.

Step 4: Put the risky paths under test

The characterisation tests pin what the application does. The tests in this step check what it must do, written from the Keep and Change requirements, for money, identity and data you cannot recreate. For money, a property-based test states a rule across many generated inputs, such as the total refunded never exceeding the amount paid, and finds the edge cases nobody wrote down.

Then make the suite run on every change. One command, a clean checkout, the restored copy of the data, and a check that has to pass before anything deploys. From here, a change that breaks a pinned behaviour is a red build rather than a surprise.

Step 5: Make one change

Pick one change from the Change bin, small enough to review in one sitting, and make only that.

• One behaviour per change. The characterisation tests that pinned the old behaviour are updated in the same change, visibly, with the requirement's number in the message.
• Everything else still passes. Every other characterisation test is the evidence that nothing else moved. A test that changes without a requirement behind it is a question to answer before merging.
• Review it against the requirement. Reviewing a change an agent made sets out the order. If an agent makes the change, give it the inventory, the requirements and a brief that names the tests it must not edit.

The change that touches stored data needs one more step. Moving orders.total from a float to integer minor units is a migration: add the new column, fill it from the old one, check the two agree, and drop the old column only once they do.

-- Orders whose integer total disagrees with the old float total by more
-- than half a minor unit. Expect no rows before dropping orders.total.
SELECT id, total, total_minor
FROM orders
WHERE ABS(total * 100 - total_minor) > 0.5;

Run it against the restored copy first, then against production after the migration, and keep the result with the change.

When the map says rebuild

Sometimes the inventory shows the code costs more to keep than to replace: the schema is wrong at its root, nothing in it can be explained, and the data is small. Mapping was not wasted work in that case. The requirements, the characterisation tests and the data are what a rebuild needs, because together they describe what the users already rely on. What you throw away is only the code, and Vibe coding builds fast, but can you explain it later? is about why the code was never the expensive part. The understanding was.