Databricks Uses Structured Extraction to Improve Agent Chart Understanding

Enterprises increasingly want agents to answer questions using proprietary documents, but crucial information frequently resides in figures and charts. [1] Customers have discovered that agents have difficulty answering questions that involve reading and counting chart values. [1] A text-based retrieval system only searches text, and generating a caption often misses the data required for detailed questions about the numbers in a chart. [1] Consequently, the system might retrieve an incorrect page or lack sufficient information to answer the query. [1]

In a test asking how many local maxima were on a chart, a frontier agent passed only the image reasoned for 50 seconds and gave an incorrect answer of 17. [1] Databricks Genie, using structured extraction via ai_parse_document, arrived at the correct answer of 18. [1] Databricks found that structured extraction from charts enhances both retrieval and answer quality on chart-based questions. [1] This approach was evaluated on a subset of ViDoRe V3 and a synthetic dataset called Chart-RAG, performing competitively with large single-vector and multi-vector multimodal embedding models. [1]

As enterprise AI moves beyond text to complex, multimodal documents, extracting precise data from charts is becoming a critical bottleneck. Databricks' approach highlights the limitations of relying purely on large multimodal models (LMMs) for zero-shot reasoning on visual data, as seen with the unnamed "frontier agent" that failed the local maxima test. By utilizing a structured extraction method (ai_parse_document), Databricks positions its Genie product as a more reliable tool for rigorous data analysis, especially against baselines in benchmarks like ViDoRe V3. This suggests hybrid approaches combining specialized parsing with LLM reasoning are necessary for enterprise-grade accuracy. What the sources don't address: How the latency and computational cost of structured extraction compare to pure multimodal embedding models in large-scale deployment.