AI-Ready Data Is Decided at Ingestion, Before the Context Layer
In September 2026, the context layer became a product category. At the dbt Summit, Fivetran, which merged with dbt Labs in June, announced a Context Layer built from semantic layers, internal documents and application metadata. It also announced general availability of the dbt MCP server, which hands any agent the models, metrics, lineage and test results of a dbt project. Other vendors have been building towards the same thing all year, and context is becoming something data teams ship.
The reasoning behind them holds. An agent has no colleague to ask. Everything a colleague would have supplied, which revenue definition applies or who owns a table, has to be written down where a machine can read it.
Every one of these layers starts from the warehouse as it is. Whatever happened to the records on their way in has already happened, and nobody wrote it down.
What a context layer can see
A context layer describes tables. It knows their columns, their definitions, their lineage back to a source, the tests that ran on them and the people who own them.
It cannot see a value as it was before a connector converted it. It cannot see a record that never arrived. Every row it describes reached the warehouse in good shape, because putting rows in good shape is what ingestion does.
That is fine when the cleaning was harmless. Often it was a decision, made by a connector at load time, with no trace kept.
5 things ingestion loses before data is AI-ready
Each of these is ordinary. In every case the run finishes green, and every check downstream passes the output.
- A rejected row
The target requires a primary key, some source rows have none, and the connector drops them. The policy is defensible. The missing rows share a trait, though: they are the records without a natural identifier. In most business systems, that means the oldest ones, or the ones a particular integration created. A trend computed over that table runs over a population with a hole in the shape of a category. - A coerced type
A field arrives as03/12/2026. A parser picked 3 December or 12 March. The result is a valid date. Unless the original string was kept beside it, nobody can tell later which reading won, and in any dataset that mixes locales it is wrong for some rows. - An invented key
Rather than drop rows without a key, a pipeline can generate surrogate keys. It is often the better choice. It is also a claim that these records are distinct entities, made by nobody, and the first downstream join makes it permanent. - A flattened structure
A nested array carried a relationship. Stored as a JSON string in a single column, the relationship survives in theory and disappears in practice, because nobody queries it. - A merged null
The most common case, and the most damaging. 3 different facts arrive as the same empty value: the source does not know, the source did not send the field, the pipeline could not process what it received. The schema accepts all 3. An agent averaging that column averages 3 facts and returns one number.
In all 5 cases the loss happens before any check runs, and the output looks correct. A context layer can document the policy that caused it. It cannot give the value back.
Why AI agents make these losses matter
These losses are as old as ETL. What changed this year is who reads the result.
An analyst who sees a March total lower than expected asks why. Someone spends a day on it and finds a few hundred rows that never loaded. That question was the safety net. Nobody designed it. It existed because a person who has once presented a wrong number to their management distrusts figures that look too clean.
An agent has no such history. Asking why a total looks low costs it time and earns it nothing, so it does not ask, and every choice the pipeline made becomes an assumption it works from without knowing it. Its tone is the same whether the column underneath is sound or full of holes, so the person reading the answer cannot tell the difference either.
The doubt an analyst used to supply has to become a value in a field. That is what AI-ready data means in practice: data that carries the record of what happened to it on the way in.
What AI-ready ingestion has to publish
Ingestion is the only step that knows what it did. So it has to write it down, as data, next to the data.
- Counts per run: Read, written, rejected, and the reason for each rejection class, computed by the pipeline itself rather than reconstructed from a row count afterwards.
- A success flag per field, per record: If a date failed to parse on row 412, row 412 says so in a column.
- The original value beside the converted one: The most valuable item on this list, because it turns a coercion into something you can audit.
- Nulls that say why: Not known, not sent, rejected. A small enumeration or a companion flag is enough.
- A declared schema: Schema inference reads the rows it sees first, which are usually the cleanest in the file.
The flags and the original values cost almost nothing to produce. The pipeline copies each source record before it touches it, and writes every derived field in a new column beside the original. Each derived field gets a companion flag that says whether the conversion worked, so a reader can keep only the rows where it did.
When a conversion fails, the record keeps what arrived. A date field that received "not_a_date_at_all" ends up with an empty date and a failure flag, and the original string is still there beside them.
That original string is the difference between a null you can investigate and a null you can only guess about. It belongs in the bar a pipeline has to clear before anyone calls it finished.
Where the context layer fits
Business meaning does not come out of a pipeline. Which revenue definition finance uses, or why one subsidiary is left out of a metric: only people know that. A context layer is the right place to write it down.
Provenance is different. Gartner lists it among the 3 core components of a context layer for AI agents, alongside semantics and operational state, and it can only come from one place: the step that handled the data. A context layer built on records that kept their original values, with a flag and a reason wherever something failed, can describe them accurately. A context layer built on tables that were quietly cleaned will present the cleaning as if it were the source.
Before funding the layer on top, open the warehouse and check 2 things. Can you find the original value of a field that failed to convert? Can you count the rows a connector rejected last night, and say why? If both answers are no, the context you are about to write down will be precise about data that is wrong.
PrettyWhale.ai is an Engineering AI for ingestion and integration code: a specialised harness around a small model trained on that code alone. It works from a real sample of the source and hands back a deliverable that has already run on that sample and passed its tests.
You have any questions on PrettyWhale.ai ?