Estimating Data Ingestion Work: A Unit-of-Work Method

Software estimation is hard everywhere. Data estimation is hard in a specific way that generic estimation advice does not address, and that specificity is the whole problem.

On application work, the unknown lives in the scope. You reduce it by asking questions: what should this screen do, what happens on error, who can see it. Discovery is a conversation, and a good analyst can close most of the gap before anyone commits to a number.

On ingestion work, the unknown lives in the data. No conversation closes it. The client genuinely believes the export is clean, because the person you are talking to has never opened it. You find out in week three, and by then the number is already in a signed document.

This article is a method for producing ingestion estimates that survive contact with the data. It is aimed at whoever actually writes the estimate.

1. Stop estimating tasks, start estimating units of work

The common approach is a task breakdown. Read the source, two days. Write the transformations, three days. Tests, one day. Documentation, half a day. Total, six and a half days, add a buffer.

This fails for a reason worth understanding: the breakdown is a description of the work, not a prediction of it. Every line is an estimate of something nobody has looked at yet, and the errors compound in the same direction, because every unexamined line looks simpler than it is.

The alternative is to estimate at the level of a unit that recurs. In ingestion, that unit is the source.

One source, ingested to production standard, is a repeatable deliverable. It has variants, and the variants are classifiable. Once you know what a standard-tier source costs you, an eleven-source project is arithmetic rather than a guess.

This only works if two things are true. The definition of "to production standard" has to be written down and identical across sources, otherwise you are averaging different deliverables. And your reference figures have to come from your own history, because nobody else's productivity applies to your team.

2. Building your tier grid from your own history

3 tiers because the grid has to be usable in a conversation with a client who is not going to sit through a taxonomy.

• Step one: take your last ten to fifteen ingestion deliveries. Across clients, same category of work.
• Step two: reconstruct actual effort. Not the estimate, the actual. Timesheets if they are trustworthy, otherwise commit history plus a five-minute conversation with whoever did it. Elapsed engineering effort from source access to production, excluding client waiting time.
• Step three: sort them by effort, then look for the natural breaks. There usually are some, and they usually correspond to the characteristics below.

• Step four: write down the classification criteria, not just the numbers. In six months someone else will need to classify a source, and if the criteria only exist in your head the grid decays into a lookup table nobody trusts.

The output of this exercise is two hours of work and it is the only credible estimation basis that exists for your team. Published industry benchmarks are worse than useless here, because they average across practices with different definitions of done.

3. The qualification phase, in detail

This is the part that converts a gamble into an estimate, and it is the part most proposals skip.

A qualification phase is short and produces a deliverable. Two to five days depending on the number of sources.

What you do during this phase

• Get real access to every source. Not a specification, not a screenshot. Access.
• Pull a genuine sample from each, large enough to contain the ugly rows. The first hundred records of a file are usually the cleanest hundred records in it.
• Profile each sample: null rates, distinct counts, format variants per field, encoding, volume, and whether the claimed unique identifier is unique.
• Identify the change indicator. Is there a reliable updated-at column, a change log, a sequence. If the answer is no, that source is complex regardless of how simple its format looks.
• Classify each source against your criteria, and record why.

What you deliver
A short document: the inventory of sources with tier and rationale, the profile of each sample, the list of remaining unknowns, and the estimate that follows.

Why the client could pay for it
Because it has standalone value. It documents their data estate, and it almost always surfaces quality problems they did not know about. More than once the qualification report is the most useful artefact the client receives all quarter, and it is produced before any commitment on the main work.

The clause that follows from it.
The estimate covers the qualified source, meaning the source as sampled. If what arrives in delivery differs materially from the sample, it gets requalified. Stated up front this reassures rather than alarms, because it demonstrates that you know where the risk lives.

4. What the estimate should actually contain

An estimate that is a single number invites haggling over that number. An estimate that shows its structure moves the conversation to scope.

The exclusions section is what makes the rest credible. A proposal that only lists what is included reads like optimism.

One more line worth including on any estimate over a certain size: what the client has to provide, and by when. Source access, credentials, a named technical contact, and a decision on ambiguous business rules. Half of all data project overruns are waiting time, and waiting time that was never named in the estimate becomes your problem.

5. The recalibration loop

An estimation grid built once and never revisited drifts within a year, usually toward optimism.

After each delivery, record three numbers: the tier you assigned, the effort you estimated, the effort it actually took. It takes 10 minutes per project.

Every quarter, look at the deltas. You are looking for three patterns.

• Systematic underestimation in one tier: Adjust the reference figure. This is the normal outcome and it is why the loop exists.
• A tier that no longer separates cleanly: If your standard and complex tiers now overlap, your criteria need work, or the shape of your incoming sources has changed.
• A misclassification pattern: If the same characteristic keeps pushing sources over their tier, that characteristic belongs in your criteria for the next tier up. The absence of a reliable change indicator is the most common one.

The reward for maintaining this loop is that your contingency shrinks. A practice with three years of calibrated history can quote with a five percent buffer where it once needed thirty, and that difference is either margin or competitiveness, depending on how you choose to spend it.

6. Estimating when you have no history

The method above assumes around 10 past deliveries. Plenty of practices do not have them, either because the data practice is new or because nobody recorded the effort. Three substitutes, in descending order of reliability.

• Borrow from an adjacent capability: If your team has built integrations, API work, or file processing, that effort is closer to your ingestion effort than any published benchmark. Adjust for the parts that differ and use it as a starting point.
• Run one deliberately instrumented delivery: Take a single source, deliver it to your written standard, and measure carefully. One real data point from your own team beats twenty from someone else's. Sell that first one on time and materials precisely so that the measurement is not distorted by pricing pressure.
• Estimate the qualification only, and stage the commitment: Quote the qualification phase firmly, and quote the production work as a range with the commitment deferred until qualification completes. Some clients will refuse. Many will accept, because a firm that says "here is what I can commit to now and here is what I will commit to in five days" reads as competent rather than evasive.

What not to do: take a figure from a vendor page or a conference talk and treat it as a reference. Those numbers come from a different team, with a different definition of done, on sources you have not seen. Using them produces an estimate with the appearance of rigour and none of the substance, which is worse than an honest range.

7. Three estimation traps specific to ingestion

• The clean sample: Whatever the client sends you as an example is the cleanest data they have. Always ask for a full period rather than a sample, and if you get a sample, ask specifically for the rows that caused problems downstream.
• The invisible change indicator: No reliable way of knowing which records changed since the last run. No updated-at column, no change log, no sequence. That leaves full refresh, or comparison logic. Both are significantly more expensive to build and to run than reading one column. This characteristic alone moves more sources up a tier than any other, and it is the one least likely to come up in a scoping conversation.
• Volume that is not volume yet: A source that is small today and growing changes tier at a threshold. Ask about growth, and about any planned acquisition or migration that would multiply it. The cost of finding out later is a rebuild, not an adjustment.

What to take away

To accurately estimate intake, it's not so much a matter of technique as it is of having your own data.

Reconstruct the actual effort on your last ten to fifteen deliveries and find the natural tier breaks. Write down the classification criteria and not just the figures. Bill a qualification phase if possible and commit against the qualified source rather than the described one. Record estimate versus actual after every delivery, and revise quarterly.

None of that is difficult. It is just unglamorous, and it is the difference between an estimate you defend and an estimate you trust.

What you then do with those numbers commercially, including whether to commit them as a fixed price, is a separate decision.

PrettyWhale.ai produces the complete ingestion deliverable from a source sample: code, tests, quality checks, schemas, configuration and documentation. Two effects on estimation: the definition of done stops varying by engineer, and effort shifts toward qualification and review, which are the parts you can actually scope in advance. If you are working on how your data practice estimates, we can talk.