Why Front-Line Data Is Bad, And Voice Changes Everything
Julien Vantyghem CTO of Quintess AI, automating industrial maintenance workflows.
gettyAsk a maintenance leader what they want from AI and the answer is unanimous: better data quality.
Their system was supposed to hold a structured record of every intervention. What comes back is vague or incorrect failure codes chosen because they close the order fastest, and zero diagnostic reasoning. Researchers at NIST and the University of Western Australia describe technician-entered work orders (PDF download) as “often inconsistent, error-filled, and replete with domain specific jargon.”
Managers reach for a people explanation: insufficient incentives, perceived value or training. But the cause is elsewhere.
Working alongside technicians on-site for the past two years, I learned that the cause is not the human organization, it is the interface to the system of record.
Maintenance is where I see this most closely, but it generalizes. A survey of 9,600 front-line workers across eight industries found 55% were learning digital tools on the fly and one-third lacked the right technology for the job. Field service, nursing, inspection and logistics each have their version of this problem.
And the interface breaks in several ways.
A desktop terminal can’t be where work is happening because of physical and sometimes safety constraints. Mobile phones and tablets closed part of that distance, but their greater availability came at the cost of usability.
Both form factors also demand channels the technician has already committed to. Hands first: gloved, holding a multimeter, inside a panel. Eyes second: on the component, on the reading. A screen asks for both, so either the work stops or the record waits.
As is logical, the record waits, and evidence perishes. A symptom that clears after a power cycle takes the diagnosis with it. Retrospective capture is also worse than people believe: When researchers gave chronic pain patients paper diaries and instrumented the binders with photosensors logging every opening, reported compliance was 90% and measured compliance was 11%.
A form is a projection of a database record, and behind it sits a schema whose entities and relationships are fixed for every customer who bought the same license.
This rigidity is why it takes five clicks to reach the screen you actually need, and why it fails to capture the complex nature of front-line work.
Then the mandatory selection arrives: a failure category from a tree of several hundred nodes (none of which describes what happened) that exists because it feeds an accounting classification three departments away. NIST and UWA researchers call this a mismatch between designated schemas and the technician’s needed semantic flexibility (download required). What he knows goes nowhere, because there was no field for it.
Structured records are not going away, and someone has to reconcile physical work against a schema. The mistake is which party we gave the job to. Constraints, enumerations and referential integrity are miserable work for a person in front of an open gearbox. But they are ideal material for a language model. We are finally able to invert the dependency: The worker describes what happened, the model produces the row.
Voice is the channel that survives the moment of work, because it leaves the hands where they are and the eyes on the component. At CUI 2022, researchers had 24 participants document a repair either through a conversational interface while working or in a report afterward. The conversational group saved time and produced higher-quality reports, with no added cognitive load.
Most teams are not replacing their system of record and should not have to. Build a knowledge overlay instead: Capture what the form has no field for, attached to the intervention it came from. The corroded terminal, the location, 30 seconds of video of the noise, the hypothesis that was ruled out. Those accumulate into run books built from real work.
Voice products differ in one respect: what the system can do after someone speaks.
Speech becomes text. Every other piece of software work remains.
Prompted, fine-tuned or given retrieval over static domain material, it holds a competent conversation about the work. It cannot read or write your systems, so it cannot answer an operational question. Its value runs the other way: it asks the follow-up the form never asked and elicits what the worker knows.
It can query history, search approved documentation, draft a record, change state. Now it can answer because it is connected to the systems the work depends on.
It also sees the current screen and drives it: opens the record, highlights a component, attaches a reading to the correct step, shows what changed. Pipecat’s screen-aware architecture splits this into a voice agent and a UI worker that reads the page and acts on it.
I’d argue the gap between three and four is the one that matters, and the one most road maps skip because it is where the screen earns its place. Voice is strong at intent, questions, evidence and correction. Screens are strong at schematics, comparisons and confirmation. The display is shared ground between worker and agent, and where trust is built: Measurements stay exact, drafts stay editable, and consequential changes wait for human approval.
The front-line data problem has been read as a workforce problem for 20 years. It is an interface problem, and the interface finally has somewhere else to go.
After someone speaks, what can your product actually do? The worker should stay with the work. The agent should handle the software around it. The screen should show what was found, recorded and decided.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
