Why Test Data That Doesn't Join Is Just Noise

The checkout test is green. It has been green for three weeks.

Then the release goes out, and within an hour support is looking at orders that exist in the orders table and nowhere else. No customer attached. No payment. The report that joins the three tables together returns a number that is obviously wrong, and nobody can tell whether it is a code bug or a data bug, because in the test environment it never had to be right.

I keep coming back to the same cause in setups like this. The test data was three separate piles. Customers in one file, orders in another, payments in a third — each one plausible on its own, none of them actually connected to each other.

A row can look perfect and still be useless

Open a generated customer row and check it by eye. The name reads like a name. The email has an @ in it. The city matches the postcode, more or less. It passes every check you would do with your eyes, and it still fails the only check that matters: does anything else in the database point at this customer?

Most test data gets made table by table. Someone writes a script for customers, a different script for orders, maybe on a different day, and the IDs are invented independently in both. So order 10482 belongs to customer 9917, and there is no customer 9917. There never was.

What happens next is quiet, which is why it survives. An inner join drops the orphan rows without complaining, so your totals come out low and your tests, which were written against those same totals, pass. A left join keeps the rows and hands you nulls, and now a screen shows blank names and somebody files it under "front-end issue". Either way you are debugging your data instead of your code, and you usually find out late.

Fixtures written by hand are worse, not better. Ten customers, ten orders, everything matched up by a person who was careful that afternoon. Real data is not careful. Real data has a customer with 400 orders and a customer with none, a refund three weeks after the payment, an order cancelled before it was paid for. A test set where everything lines up neatly will pass tests that production data fails before breakfast.

The difference is not the rows themselves. It is whether the rows know about each other.

What "good" actually looks like here

Good test data is boring to describe. Every foreign key points at a row that exists. Every order belongs to a customer, every payment belongs to an order, every ticket belongs to somebody. That is the whole standard, and a surprising amount of test data does not meet it.

The rest is shape. If your customer IDs look like CUST-000042, the test ones should look like that too — CUST-000001, CUST-000002, in sequence — because plenty of bugs live in formats: a column that is too short, a parser that expects the prefix, a sort that goes wrong on the ninth row. Dates should fall in the ranges your real dates fall in. Amounts should be skewed the way real amounts are skewed, which is to say a lot of small ones and a few embarrassing large ones, not a tidy spread between ten and a hundred.

And the volumes should be uneven. Give one customer a long history and most customers almost none, because that is what your database actually holds, and queries behave differently on uneven data. A page that loads instantly against 50 balanced rows can crawl against a realistic spread.

Check the joins before you run a single test

Here is the habit I would push on any team, whatever tool they use. Before the test suite runs, ask three dull questions of the data itself.

How many orders have no customer? How many payments have no order? How many rows in each table are pointed at by nothing at all?

If the first two numbers are not zero, stop. There is no point reading test results built on data that could not exist in production. The third number is allowed to be non-zero — customers who never ordered are real, and you want some — but you should know the number because you chose it, not because the generator left gaps by accident.

This is the check we built into DataGenStudio, because people kept skipping it by hand. You define the tables and how they link, generate, and the app runs a link check over the result: every reference is followed to see whether it lands on a real parent row, and you get a report instead of a shrug. If a link is broken, you find out before the data leaves the tool, not three weeks later in a release.

A small way to start

You do not need to rebuild your whole test setup to fix this. Pick the three tables your team argues about most — for most product teams that is customers, orders and payments — and generate those as one linked set, with your own ID formats and your own ranges. Load that into staging, run the orphan checks above, and then run the test suite you already have.

Two things tend to happen. A few tests fail that used to pass, and when you look, the test was relying on data that could never happen. And a bug or two shows up that the old fixtures had been politely hiding. Both are cheaper to meet on a Tuesday in staging than on release day.

That is really the argument for synthetic data, stripped of the pitch. Not more data. Data that hangs together, describes nobody, and can be regenerated whenever a test needs a fresh copy.

If you want to try it with your own schema, DataGenStudio is free to start: bring a SQL file, an ER diagram, or just describe your tables in plain English, and generate a linked set you can check for yourself.

Try it free at data.teragenai.com — and if you are testing AI systems as well as ordinary software, read Your AI Agents Need Data That Doesn't Exist Yet next.