Can our AI system find the relevant information in our semiconductor data? Any semiconductor team that depends on AI must answer this question with hard evidence. This evidence may be a waveform, a diagram, or a passage of text that an agent needs to retrieve before it can use it to answer a question.

We built Boomerang V2, our multimodal embedding model, with semiconductor data as a priority. It maps text and images into one shared vector space so they can be searched together. We tested it against Boomerang V1 and external embedders from OpenAI and Google to see how that approach performs.

Several experiments cover general-purpose image data, semiconductor image data, and mixed semiconductor image and text data, with gemini-embedding-2 and openai-text-embedding-3-large as the external references. For the text-only models, images are represented by generated descriptions from our production description pipeline.

Finding the right semiconductor data

Our semiconductor retrieval benchmark was built using an internally generated dataset comprising 2,059 document images. We searched them with three kinds of queries:

• Basic queries: Short search queries generated by an LLM after viewing each image. We then test whether the generated query retrieves the source image.
• Semiconductor questions: Real questions from the dataset, each asking about a specific document, and feed the question verbatim as the query. This tests whether the model can associate a natural-language question with the image of the document containing the relevant information.
• Ground-truth answers: We use the ground-truth answer to each question as the query. This tests whether the model can retrieve the source image based on the information contained in the answer itself.

We measured hit@5, the share of queries that retrieved their target among the first five results. Figure 1 shows that V2 had the highest hit@5 score across all three query types.

Mixed semiconductor data: Searching text and images together

Text documents and images often coexist in semiconductor data. So we tested questions against three versions of the internal semiconductor index: 2,059 document images, 6,200 text chunks, and a mixed index containing both. Each question in our semiconductor dataset was generated from a specific document that served as the retrieval target.

V2 led on the image-only and mixed indexes, as shown in Figure 2; Gemini led on text-only. The mixed index result is especially relevant when an application needs to search for evidence across both text and images. All four models scored lower in that setting than in either single-modality index, but V2 retained the highest score.

Module and part identifiers help retrieval

We searched with the same semiconductor questions before and after removing their module and part identifiers. The point is to measure how strongly each model picks up on semiconductor-specific terminology. In the example below, the module name pipelined_alu is removed from a question about a waveform.

Figure 3 shows that every model scored lower without the identifiers. V2 showed the largest absolute drop, but still had the highest score after removal.

Testing on public semiconductor documents

To test whether the gains generalize to documents closer to what our customers actually ingest, we conducted an additional experiment focused on retrieving individual image parts. This is particularly important in semiconductor applications, where answering questions often requires retrieving specific visual evidence, such as waveforms, micrographs, or distribution plots.

We ingested the same 20 public semiconductor documents into two corpora, one embedded with V1 and the other with V2. These documents include textbooks, research papers, and failure-analysis reports. V1 represented each image with a generated description, whereas V2 embedded the image with a short caption. Using an LLM, we generated 100 image-related questions from these documents and retrieved up to 10 parts per question from each corpus.

Because a question can be answered by several figures and these public documents lack relevance labels, we evaluated retrieval quality using graded relevance rather than a binary hit rate. We used a Retrieval Relevancy Evaluation Agent built on the UMBRELA prompt, the same LLM-as-a-judge method behind the retrieval metric in our open-source Open-RAG-Eval framework (see the UMBRELA metric in our earlier post on RAG evaluation for details). The judge assigned a score from 0 to 3: 0 when the part has nothing to do with the question, 1 when it is related but does not answer it, 2 when it partly answers it or the answer is buried in other content, and 3 when it directly contains the answer. UMBRELA’s automatic judgments have been shown to correlate well with human judgments, allowing us to evaluate retrieval on document sets without a golden answer set. For image parts, our agent first fetches and views the image, then scores it based on what it sees. From these scores, we computed, for each of the 100 questions and each corpus, DCG@10 over the retrieved image parts, so that systems that rank more relevant parts higher are rewarded.

As the table below shows, V2’s average DCG@10 over relevant image parts is 3.85, compared with 2.95 for V1, a gain of 0.90 that is highly significant (p < 0.0001). In plain terms, for the same question, V2 retrieves more relevant images and ranks them higher among the retrieved results, making them more likely to be read and used by the downstream agent that answers the question. One practical note: this gain comes not only from V2’s stronger image embedding but also from its ability to embed longer text chunks, which leaves more room for image parts within the 10-part limit per question.

General-purpose images tell a different story

For a broader comparison, we evaluated 1,000 images from DOCCI, a dataset with detailed human-written descriptions. We searched with short LLM-generated queries, full descriptions from the dataset, and queries targeting specific visual details such as objects, attributes, and spatial relationships.

For V1 and OpenAI’s text-only model, the indexed image descriptions were generated with gemma-4-31B-it. The supplementary section below explains the query types and shows the detailed results.

V2 improved on V1 across all three query types. On the more difficult dimension-aware queries, hit@5 rose from 0.490 to 0.592, an increase of 10.2 percentage points. V2 was competitive with OpenAI’s text-only setup, while Gemini remained ahead.

Choose the model for the documents your team uses

V2’s strongest comparative results came from the semiconductor material we built it to support. It led the four-model comparison on semiconductor images and on text and images searched together. It also improved on V1 in the separate public-document evaluation.

The general-image results help explain why testing on your own material matters. A model that leads one benchmark may not lead on the documents your engineers need to search. For teams working with semiconductor text and images, V2’s results provide a concrete reason to test it on their own documents and questions, then assess the answers generated from the retrieved evidence.

A closer look at the general image benchmark

The following breakdowns provide additional detail on DOCCI. Its human-written descriptions capture objects, attributes, spatial relationships, text in images, world knowledge, and scene information.

The three query types test different ways of finding the same target image:

• Basic queries: An LLM is given the image and generates a short, natural-language query that a person might use to search for it. We then evaluate whether the query retrieves the target image.
• Full-description queries: The image’s ground-truth description from the dataset is used directly as the query.
• Dimension-aware queries: An LLM generates queries that explicitly target the dataset’s dimensions, augmented with an object and relation taxonomy. This allows us to evaluate model performance across different object types, attributes, and relations.

Retrieval by visual detail

Figure 5 splits the dimension-aware queries by the DOCCI dimension they target. V2 improves on V1 in every dimension, and the ordering of the four models is stable across dimensions. Objects and attributes, along with spatial relationships, are among the harder categories for all four models, which we examine in more depth below.

Figure 6 goes one level deeper, splitting by the kind of attribute a query references. V2 clearly beats V1 on color, material, size, and shape. Interestingly, lighting appears to be a difficult attribute for all of them.

Retrieval by spatial relationship

Figure 7 examines spatial relations, which matter to us because they correlate with understanding of diagrams and graphs. Generally, there is considerable variability in performance here.

Overall, V2 is a clear improvement over V1 on general images and performs on par with OpenAI's text-only embedder, but it does not fully close the gap to Gemini. That said, we built V2 for semiconductor data, and the next section analyzes its performance on that data.

The takeaway

Boomerang V2 is a clear upgrade over V1 across every benchmark we ran: general images, semiconductor image data, and mixed data. Against the external embedders, V2 delivers solid general performance and is worth considering for semiconductor data.

If you’re evaluating retrieval for semiconductor documents, start with the Boomerang V2 introduction, then test it with the documents and questions your team works with. Our guide to RAG evaluation and the UMBRELA metric explains how to assess retrieval relevance. For a closer look at the general-image benchmark used here, explore the DOCCI dataset and sample illustration from Onoe et al., Google Research, ECCV 2024.