Turning image-heavy technical documents into agent-ready Markdown
Learn how image taxonomy and parallel document processing convert image-heavy PDF, PPTX, and DOCX files into structured, agent-ready Markdown.
7-minute read time
Standard document parsers lose the meaning imparted by images
Ask an agent a question about a 300-page technical document, and more often than not the right answer lives in the images. It may be in a datasheet's timing diagram, a test report's waveform plot, or in a slide deck's block diagram. In technical documents, images often contain information that the surrounding text leaves out.
A standard document parser can drop these images or replace them with generic one-liners such as a line chart showing data. By the time your agent starts reasoning, the required data it needs may already be gone. Large documents create another problem. An agent that tries to process hundreds of pages in one pass also runs into timeouts and context limits before it reaches the end. This is a common failure mode in technical document processing.
We built two agents on Vectara's agent platform to address both problems. They convert large, image-heavy PDF, PPTX, and DOCX files into Markdown, with each image classified and captioned by type. The resulting Markdown becomes a structured artifact that downstream agents can use in larger workflows.
A two-agent pipeline
The pipeline separates two tasks. It understands the visual vocabulary of the document(s) using the first agent, Image Taxonomy Discovery. The agent identifies the kinds of images in the source material. Then, the second agent, File2Markdown, converts the document(s) in parallel, using that vocabulary at scale, into a single Markdown artifact.
The taxonomy is produced once and reused throughout the conversation. Every image that File2Markdown captions follows the instructions for its assigned category.
Agent 1: Image Taxonomy Discovery
Why start with a taxonomy?
Different image types require different descriptions. For a waveform plot, useful details include axes, units, and signal behavior. For a block diagram, they include components and connections. A photograph of a device requires a different description altogether. So a generic image-captioning prompt does not know which information matters for each visual.
The Image Taxonomy Discovery agent addresses that by reading the images in a document set and proposing up to 10 categories that fit the material. Each category includes its own descriptions and captioning instructions. Downstream agents use those instructions rather than applying a single prompt to every image.
How it works
Input: One or more large PDF, PPTX, or DOCX files.
- Extract in batches: The agent extracts 8 images at a time. This balances visual description quality with model context limits.
- Propose: From the first batch, it drafts a base taxonomy of at most 10 categories.
- Refine: As it processes subsequent batches, it can add, merge, or refine categories while maintaining consistency across the document set.
- Finalize: After the final batch, the resulting taxonomy becomes the output.
This agent runs sequentially rather than in parallel because each refinement step depends on the taxonomy produced by the step before it.
Output: A JSON object whose categories field holds the taxonomy. Each category has a name, a description, and a description_prompt (the captioning instructions). Other fields record document metadata, such as the number of images processed and the artifact IDs associated with each category. The result is a set of instructions informing the next agent what to look for and how to describe it.
Agent 2: File2Markdown
Why split the work?
A downstream agent that processes a very large file in a single pass can hit timeouts and context limits. Its image captions also remain generic because the agent has no image-specific guidance. File2Markdown divides the work into manageable pieces, processes those pieces in parallel, and captions each image according to its taxonomy category.
How it works
Input: One or more large PDF, PPTX, or DOCX files, plus an optional JSON taxonomy, usually the output of Agent 1.
- Split: A document-conversion tool divides a file into two-page chunks, small enough to process reliably. For our current infrastructure, two pages gives us enough local context around figures while keeping each chunk well within sub-agent context limits and reducing the risk of timeouts.
- Fan out: Multiple sub-agents run in parallel, each owning one chunk from start to finish. The number of sub-agents depends on available compute. We started with 7 in our deployment. This number can scale with available resources.
- Classify, then caption: Within each chunk, the agent assigns each image to a taxonomy category and captions it using that category's description_prompt. Images within a chunk are processed in parallel. Their captions are inserted into the text where the images appeared.
- Recombine: Deterministic lambda tools merge the chunks, in order, into one Markdown document. They also write per-category image manifests: JSON lists used to audit classification accuracy and filter extracted images by type.
How images are embedded in the Markdown: The converted document refers to each image only through an artifact reference, which resolves only inside the agent’s session. To fix this, a deterministic lambda tool reads each captioned image’s bytes through Vectara’s read_artifact tool, which returns them as base64-encoded text. It then writes that text into the Markdown as a data URI in place of the artifact reference.
Output: A JSON result with fields including success, document_id, taxonomy_artifact_id, and final_markdown_artifact_id. The finished Markdown is stored as a session artifact. A parent agent can use its artifact ID directly or fetch the file through the session artifact API. That artifact handoff matters because File2Markdown is designed as part of a larger agent workflow instead of a standalone document converter.
Example: Converting a sample technical paper
We evaluated the multi-agent pipeline on a sample technical paper to demonstrate the quality and detail of the generated image descriptions.
The clearest way to show the difference is to caption the same image both ways:
The generic caption identifies the subject. The taxonomy-guided version captures the structure, labels, relationships, and orientation that a downstream task may need. The pipeline also preserves the source text alongside these enriched image descriptions. We verified text preservation by running character-level diffs and comparing the total character counts between the extracted source PDF text and the generated Markdown artifacts across the document suite. We also validated heading hierarchies, paragraph boundaries, and list ordering programmatically. The result is a lossless representation of the source text, with richer descriptions added for the document’s images.
Limitations
Although the pipeline improves how image-heavy documents are processed but does not eliminate all sources of ambiguity.
- Taxonomy ceiling: In our initial findings, 8 to 10 categories worked best, though this threshold can be refined with additional empirical data. Ten categories may be sufficient for a specific document family, whereas mixed corpora may require additional categories or a distinct taxonomy per document type.
- Chunk boundaries: A strict two-page split can separate a figure from its surrounding explanatory text on adjacent pages. To reduce this context loss, sub-agents carry forward sliding text windows across boundaries to maintain local cross-page references.
- Classification errors: If an image is assigned to the wrong category, it receives incorrect captioning instructions. The system addresses this risk with a confidence score threshold. When classification confidence is low, it falls back to a general technical-image prompt.
What comes next
Parsers that treat images as structured document content play a critical role in providing agents with the correct context when answering users’ questions. By combining image-specific taxonomies, parallel document processing, and artifact-based handoff, ingestion pipelines can preserve the information from complex technical documents that produces this context. Future work focuses on evaluating downstream retrieval performance, testing taxonomy reuse across broader document corpora, and enhancing structured table extraction.
Working with image-heavy technical documents? Explore how Vectara's agents can help you process complex files, preserve more of their useful context, and prepare the results for downstream agent workflows.
