From semiconductor training data to Mockingbird V3 MM

In the first post in this series, Building Verified Multimodal Training Data for Semiconductor AI Agents, we described how we built semiconductor-specific multimodal training data with answers derived and verified using authoritative engineering artifacts such as RTL simulations, manufacturer datasheets, pin mappings, timing traces, waveforms, diagrams, and specifications.

This training data enabled us to build Mockingbird V3 MM, a model for semiconductor AI agents that can produce grounded answers and work through multi-step engineering tasks.

Mockingbird V3 MM adapts an open-weight multimodal foundation model for semiconductor answer generation and agent workflows. We used two distinct post-training phases. The first phase applies LoRA-based supervised fine-tuning to semiconductor multimodal data. The second phase uses GRPO reinforcement learning to target planning and tool calling. For semiconductor questions that included images, the factual consistency score increased from 0.2919 to 0.4539 after fine-tuning.

Phase 1: Semiconductor fine-tuning with LoRA

We used LoRA-based supervised fine-tuning (SFT) to train the model on examples from our in-house semiconductor dataset. The training material includes technical documentation, specifications, diagrams, pin maps, waveforms and timing traces, RTL code and simulation outputs, tables, charts, and other engineering artifacts. We used Low-Rank Adaptation (LoRA), a parameter-efficient method that trains a small set of added parameters while keeping the original weights frozen. This phase focuses on using semiconductor material to produce answers grounded in the supplied evidence.

Higher factual consistency on questions with images

On semiconductor questions that included images, Mockingbird V3 MM scored 0.4539 on HHEM, compared with 0.2919 for the base multimodal model before we applied post-training. HHEM, the Hughes Hallucination Evaluation Model, evaluates whether the generated text is consistent with the reference material. Higher scores indicate stronger factual consistency. The score is not the percentage of questions answered correctly.

For a semiconductor engineer evaluating an agent, this result is relevant when the supporting evidence is visual: a signal trace or pin diagram may contain information the model needs to answer the question. The higher score indicates greater consistency with the reference material for the image-bearing questions tested; it does not establish correctness for all types of engineering images.

Phase 2: Training for planning and tool use

The second phase targets how a model plans a task and uses tools. We used Group Relative Policy Optimization (GRPO), a reinforcement-learning method, with task-specific reward functions to guide training toward the intended behavior. The training data combines open-source agentic data, including Toolathlon-style runnable tool-use tasks, with Vectara’s in-house examples. The in-house material covers tool selection and multi-step engineering workflows, including trajectories that record a sequence of planning and tool-calling steps.

How Mockingbird compares

Apart from comparing Mockingbird V3 MM with the base multimodal model, we also compared it with three other models: Gemma4 31B Base, our production small language model (SLM) baseline; GPT-6 Astra, a frontier API model; and Inkling-Small, an open-source model served through the Hugging Face router. This comparison provides insight into how the specialized model performs relative to the production baseline and broader models.

Benchmark snapshot

The following table reports results on our internal Semiconductor Test; MMCircuitEval, a public multimodal circuit benchmark; and MMLU-Pro, a broader knowledge and reasoning benchmark. Higher reported scores are better within each column.

On MMCircuitEval, Mockingbird scored 0.4763, which was 0.0033 below GPT-6 Astra and above the Gemma4 production baseline's score. On our internal semiconductor benchmark, Mockingbird scored 0.6573, slightly above GPT-6 Astra at 0.6545. On MMLU-Pro, GPT-6 Astra led at 0.9017, while Mockingbird’s 0.7315 was the lowest score among these four models. GPT-6 Astra optimizes for broad frontier capability. Inkling-Small provides a strong open-source generalist reference point. Gemma4 represents the smaller production baseline we wanted to improve upon. Mockingbird, by contrast, is deliberately specialized for multimodal semiconductor agent workloads.

What the results mean for semiconductor agents

For teams evaluating an agent to answer questions from semiconductor documents and visual artifacts, Mockingbird’s domain results make it worth testing on their own material. The smaller model also offers deployment control and specialization advantages. These evaluations cover model responses in benchmark settings and do not measure a complete production agent’s reliability, engineering time saved, or deployment cost.

What we learned from training Mockingbird V3 MM

• Evaluating performance outside our internal training pipeline is essential. Since MMCircuitEval was excluded from training, it serves as a valuable external check for circuit-specific tasks. However, using a public benchmark cannot replace performing an independent evaluation of Mockingbird.
• Visual questions also need explicit attention. The higher HHEM score on semiconductor questions with images shows why these questions should be a part of the evaluation. Visual data may carry the evidence needed for an answer. So a text-only evaluation would not assess how well the model uses evidence present in waveforms and diagrams to answer questions.
• Domain specialization and agent behavior need separate objectives. Our first phase teaches the model the semiconductor domain and its artifacts. The second phase targets planning and tool-use behaviors through reward signals. Each objective needs its own evaluation.
• Reinforcement learning requires validation during training. Adding GRPO does not make a checkpoint better simply because the optimization method is more sophisticated. The reward functions, trajectories, and evaluation suite must demonstrate that the target behavior actually improved.

Model selection should be workload-driven. For this workload, the strongest finding is that fine-tuning the base multimodal model improved factual consistency on the tested semiconductor questions with images. Mockingbird V3 MM is designed to help semiconductor agents use the evidence engineers trust, from specifications to waveforms, to deliver grounded answers and guide tool use.

To learn more about how Vectara supports secure AI agents for specialized enterprise and engineering workloads, contact us for a demo.