Data quality is where AI projects quietly fail, and the usual response is disproportionate: a general data programme that takes two years and blocks everything behind it. The useful question is narrower. Is this data good enough for this specific question?
Answering that lets most businesses ship something useful while the wider cleanup happens in the background, if it happens at all.
The problems that genuinely break systems
- Duplicates, because a system cannot tell which record is authoritative and will average across both.
- Inconsistent values for the same concept, where three spellings of a status become three categories.
- Missing fields the logic depends on, particularly where blank means something different from zero.
- Stale records nobody has retired, which are indistinguishable from current ones.
These break things because they change what the system concludes rather than merely making it untidy.
The problems you can usually live with
Inconsistent formatting, minor typos, free-text fields with varied phrasing and gaps in fields the task does not use are all tolerable. Language models handle surface variation well, which is one of the genuine advantages over rule-based systems, and cleaning it first is frequently wasted effort.
Nobody needs a perfect data set. They need to know which fields the answer depends on, and whether those specific fields can be trusted.
Lena Fischer, Solutions Architect, Engineered With AI
Test against the actual question
Take twenty real cases, work out the correct answer by hand, then run the system and compare. Where it is wrong, look at whether the cause was the model or the record it was given. That distinction is the entire point of the exercise, and it usually shows the problem is concentrated in one or two fields rather than spread across everything.
Fix the field, not the database
If two fields carry the failures, fixing those two is a week of work. Fixing everything is a programme. Scope the cleanup to what the system depends on and revisit if you extend its remit later.
Stop the source of new bad data
Cleaning historic records while the intake keeps producing new ones is a losing effort. Where a free-text field should be a defined list, change the form. Where duplicates enter through two systems, fix the matching rule. This is usually less work than the cleanup and it is what prevents doing it again next year.
Design the system to fail visibly
Where data may be missing or contradictory, the system should say so rather than infer. A response that states it cannot determine the answer from the records available is far more useful than a plausible guess, because it points at a specific record somebody can fix.
That behaviour also converts data quality from an invisible drag into a queue of concrete corrections, which is the point at which it starts improving. Our automation index covers how this is handled per sector.
Who owns a field being correct
Data quality degrades when no one owns a field. Someone owns the customer record in principle, and in practice three teams write to it with different conventions and nobody reconciles them. Naming an owner per field that matters, with the authority to define what a valid value is, does more for quality over a year than any cleanup exercise.
Ownership also settles the arguments that stall these projects. When two systems disagree about a customer’s status, someone has to decide which one is authoritative, and that decision is organisational rather than technical. Making it explicitly, once, removes a class of ambiguity that would otherwise surface in every report built afterwards.
Measure quality where it matters
A general completeness score across a database is close to meaningless, because most fields are irrelevant to any given question. Measuring the fields the system depends on, as a simple weekly count of blanks, duplicates and out-of-range values, gives you something actionable and small enough that somebody will actually look at it.
That number also tells you when the system can be extended. A process whose input fields are consistently clean is a candidate for more autonomy; one whose inputs are still noisy needs to keep the human review step, and the measurement makes that a decision rather than a debate.
Not sure if your data is good enough?
We will test it against the specific question you want answered, which is a far cheaper check than a general data audit.
