Models That Read Images, Audio and Video: Where the Errors Hide

Shown a chart, a model can describe its trend with confidence and still report a value the chart does not contain. The description reads well, the number looks plausible, and nothing in the output flags the mistake. Images, audio and video reach a model through encodings that discard detail, and the model fills what it cannot resolve with its expectations. Knowing where the detail goes is the first step toward catching the errors it causes.

What the model actually receives

An image is converted into visual tokens or embeddings before the model reads it. The conversion depends on resolution, scaling and compression, and many systems resize or divide the image before processing. A label a few pixels tall may not survive that step. Audio is typically transcribed or encoded into features, and timing and speaker information can be simplified along the way. Video is usually handled as a sequence of sampled frames plus an audio track, so anything that happens between samples may never reach the model.

These details vary between systems. The consequence is shared by all of them: the model works from a representation, and the representation has already lost part of the source.

Charts and numbers

Charts produce a particular kind of error. A model may read the wrong axis, miss a logarithmic scale, confuse two series whose colors are close, or estimate a value when the chart carries labels that state it exactly. It may describe a real trend and misstate its size. Stacked and overlapping elements are hard, and so are charts with dense annotations. The narrative around the numbers is often the most fluent part of the answer, and it is also the part least tied to the data.

A useful habit is to ask for each value together with where it appears in the image. The request does not make the reading more accurate, but it makes the reading checkable in seconds. Where the underlying data is available, compare the model's values with it, since the image is only a rendering of that data and the model sees nothing else.

Documents, tables and handwriting

Scanned documents combine several difficulties. Merged cells and irregular layouts can misalign rows and columns, so a value ends up beside the wrong label. Low-contrast scans, stamps and marginal notes can hide digits, and digits do the most damage when wrong. A one and a seven, or a zero and an eight, can be hard to tell apart at low resolution. Handwriting varies in ways no training set covers completely. None of these failures announces itself. The output looks like a clean table, and a few cells in it are wrong.

Audio and video

Audio errors cluster around overlapping speech, where a transcript may merge two speakers or attribute a sentence to the wrong person. Accents, background noise, crosstalk and clipped recordings all raise the error rate, and numbers and proper names are the usual casualties. Heavily compressed audio, such as a phone call recorded at low quality, may lose the detail that separates similar-sounding names, so transcripts of such calls deserve closer checking than studio recordings. Timestamps can drift, so a claim attributed to minute twelve may have come from minute eleven.

Video adds the sampling problem. A quick gesture, a brief warning on a screen, or a single frame showing a reading may fall between sampled frames and never reach the model. The model may then infer what happened from the frames it did receive, producing a plausible account of something that was not in the sample.

Testing with real media, and keeping a person in the loop

Build the test set from the same sources the system will use. Captures from the actual cameras, scanners and recording setups are more informative than clean examples gathered elsewhere, because the failures depend on capture conditions. Include poor-quality inputs deliberately, such as blurred images, quiet audio and low-resolution scans. For each input, record a human-verified answer. Then tally the errors by type: misread digits, wrong axis, merged speakers, missed events and so on. The distribution is more informative than overall accuracy, because it shows which failures to design around. The test set should also include inputs the system ought to refuse or flag, such as a scan too blurred to read or audio with long stretches of silence. A model that answers anyway is failing even when its answer looks reasonable, so measure how often it flags such inputs as well as how accurate its answers are.

For high-stakes output, the model should prepare the reading and a person should verify it. Make verification fast by showing the source beside the extracted values, highlighting the regions the model cited and making corrections easy. Set thresholds in advance. A value outside the expected range, or one that disagrees with a second check, should go to review automatically. Track how often reviewers change the model's output. A high change rate means the feature is not yet saving time, and a near-zero rate can mean reviewers have stopped checking.

Practice: ten items with known answers

Choose ten images or recordings from ordinary work. Write the correct answer for each, including the values, speakers or events it contains, and note the capture quality. Run them through the model and grade each answer against the key. Sort the errors into the categories above. Then take the category with the most errors and ask what preprocessing or review step would have caught it. Answering that question for a real case is more useful than any general warning about multimodal models.