# A scientific agent that chooses the right tools

> Our scientific agent can use a wide range of tools, pipelines, databases, analysis operations and method guides, but in production it often misses the right one. We built the first benchmark measuring ability to discover and select the right tool for scientific research tasks. Later on we trained a Jev-like selector that outperforms Claude Code, Codex and Biomni on the same set of resources.

<HeroFigure id="coverAnim" viewBox="0 0 1600 800" label="Animation. A request asks how a small molecule binds a protein. Without the selector, word matching reaches six tools, the agent picks a wrong one, and a card lists why: the best tool was not on the shortlist, nine structure tools look alike, each had only a name and one line, and strengths, limits and GPU needs were unknown. With the selector, the whole catalog is in view, a line connects the request to boltz2, and its tool card opens: best for protein plus ligand with affinity, not for a plain structure, alternative chai1, runs on a T4 GPU." caption="Without the selector, the agent chose from a keyword shortlist with one line per tool. With the selector, it sees the right tool and reads its card." />

Our scientific agent has access to an incredible catalog of 720 specialized tools, pipelines, and databases but it often misses the right one. In practice, the agent only searches the catalog on 15% of turns. Worse, 35% of the resources it needs are never even considered. And when it does search, the best tool shows up in the top five less than half the time. Even worse, when running pipelines, the agent defaults to heavyweight processes, like launching full nf-core workflows for small jobs that simpler, faster solutions could handle with ease. All this power, yet so much of it goes wasted

To fix this, we can simply load all 720 resource descriptions into the agent’s context like some common agent harnesses. This would bloat the agent context, causing unnecessary confusion to the agent and increased latency overall. So we built a small selection model instead: one fast model call per user turn that reads the request against the catalog and select up to four resources  for the agent at the beginning of the task. 

To rigorously evaluate this routing upgrade, we built a 120-case benchmark, designed around the failure patterns observed in production. This benchmark provided robust test scenarios to assess multiple routing strategies effectively

Here, we first look at how the agent misses resources in production, then describe the benchmark and two selectors built on off-the-shelf models, Gemini 3.8 Flash and TypeSafe's Jev decision model; we also compare against Claude Code, Codex and Biomni given the same resources catalog, and with examples of what the selector changes in practice.

## The agent often misses the right resource
We took 7 days of production traffic and kept about 400 sessions that could plausibly have used a specialized resource. An LLM labelled the resources each turn needed, and we traced what the agent did with each of them.

<Figure id="outcomefig" title="Figure 1 · Outcome of each needed resource in production" caption="Most misses happen before retrieval, because the agent never searches. Share of needed resources by outcome, about 400 sessions from 7 days of relevant production traffic. Shares are rounded, so they sum to 101%.">
<FigureCanvas id="outcomeViz" viewBox="0 0 720 150" label="Stacked bar: 35% of needed resources were never searched for, 11% searched but not surfaced, 3% surfaced and ignored, 16% used this turn, 36% used earlier in the session" />
</Figure>

The first problem is discovery. The agent searched the catalog on 15% of turns and never searched for 35% of the resources that requests needed, and when it did search, the needed resource reached the top five 41% of the time. This inherently due to the model sometimes just focuses on a well-defined task using standard tools and its memorized practices instead of leverage existing platform tools/helpers or available skills.

The second problem is selection. Once the resources are available in the model context, it still has three decisions to make. First is which resource to use. Structure prediction alone has nine endpoints, the antibody numbering, structure, developability and design tools look alike without being interchangeable, and a request may need a database or a method guide alongside the tool. The second is where to run it. For a few samples, stage workers that already hold the reference index start at once; for many samples, a Nextflow pipeline is slower to launch but scales and records QC and provenance. The third is whether to use anything at all, since a scientist who asks how ESMFold differs from AlphaFold wants a straight explanation, not launching a whole expensive GPU job.

Getting these decisions wrong costs hours. For example, the agent can launch a six-sample mouse pipeline that failed on an unsupported genome key, and the relaunch ran for a few hours while it can simply use accelerated tool with already built-in index to make it faster.

## Tool selection benchmark
We created a benchmark designed to mirror the inputs our agent sees in production. Each case includes the user request sometimes paired with earlier turns and an answer key listing the resources the agent should explore or use before tackling the task, the answer key was labeled by multiple LLMs and finally reviewed by our team of 3 domain experts. Below are some examples in the benchmark: 

### What the cases look like
| Kind | Example request | Right resources | The trap |
| --- | --- | --- | --- |
| Look-alike tools | "How does this glucose analogue sit in my 280-residue sugar transporter? I only have the protein sequence and the ligand's SMILES." | a co-folding model: `boltz2`, `chai1`, `protenix` or `openfold3` | `esmfold` predicts the protein alone, without the ligand |
| Protocol before scale | "We sequenced one 10x V(D)J library from tumour-infiltrating lymphocytes. Please call clonotypes and pair the receptor chains." | `nf-core-airrflow` | a single library, but raw 10x V(D)J reads still need this pipeline |
| Scale | "Pilot run, three yeast RNA-seq libraries. I only want read counts per gene to see whether the deletion strain looks different." | stage tools: `star`, `salmon` or `kallisto` | the full RNA-seq pipeline is right for 60 samples, wasteful for 3 |
| Database | "Which proteins are predicted to interact with the human kinase PLK4? A network figure is enough." | `query_stringdb` | KEGG also has pathways, but the question is about an interaction network |
| Several types at once | "Pull the public human liver atlas from GEO, cluster the hepatocytes, and cite the paper that defined the zonation markers." | the GEO lookup, clustering ops, a single-cell skill, a literature lookup | a selector that stops after one skill misses the data and the ops |
| Needs nothing | "Can you shorten that summary to three bullet points?" (follow-up) | none | pinning a report-writing skill here only adds noise |

### How the benchmark was built
The mix of cases follows the failures we saw in production:

- 49 tool cases, many built around look-alikes: co-folding versus docking, fast versus accurate structure prediction, and antibody tools that sound alike.
- 20 cases where the right choice depends on the number of samples: the same analysis asked with a few samples and with many.
- 20 database cases: 16 for biological resources (datasets and atlases, genes and proteins, variants, pathways, chemical and immune data) and four for literature.
- 20 omics-operation cases, from Seurat QC to deconvolution and flow cytometry, plus one planning case led by a skill.
- 10 controls that need nothing, such as conceptual questions.


Scoring works per resource type. For each type a case needs, a best pick scores 1, an acceptable pick 0.5 and a miss 0, and the combined score averages the types within a case and then the cases. Controls score 1 only if every pick is allowed. 



## Resources selection framework
### The framework
Our framework makes one fast model call per user turn, in parallel with work the agent already does, reads the request and earlier user turns against compact catalog cards, which say what each resource is and is not best for, and pins between zero and four resources in a short note on the current message. 

<Figure id="frameworkfig" wide title="Figure 2 · One request through the selector" caption="The selector runs once per user turn. Click the options to follow one request through each design compared in this post. The example request, prompt excerpt, question and agent note format are real; the scores and reasons are illustrative.">
<FigureCanvas id="frameworkViz" viewBox="0 0 720 560" label="Interactive diagram. The 720 resources (tools, pipelines, databases, omics ops and guidelines) are retrieved as the whole catalog or a keyword shortlist; together with the request, three yeast RNA-seq libraries for quick gene counts, they go to the selector, which is either an LLM (for example Gemini 3.8 Flash) or a decision model (Jev, or our fine-tuned Drylab selector). The LLM writes a list of ids with reasons; the decision model answers one yes/no question per resource and code selects the top four. Pinned: kallisto and star. What the agent receives: a relevant_assets block appended to the user message, listing each pinned resource with its type, a reason and a usage line such as client.accelerated_tools.run of kallisto." />
</Figure>

### A language model as selector
For this we use Gemini 3.8 Flash at low reasoning effort to output a single structured call. The prompt holds the routing rules and the compact cards, grouped by type, followed by the request and earlier turns. The model returns up to four resource ids, most important first, each with a short reason, as a typed object. We found no difference in performance between reasoning efforts and Gemini 3.8 flash model is also the best one among all the lightweight frontier LLMs we tried.

<Figure id="llmfig" wide title="Figure 3 · How a language model selects" caption="The language model reads one prompt with the rules, every resource card and the request, and writes its own list of ids with reasons; plain code then cleans it up. Card text and output are shortened from a real call.">
<FigureCanvas id="llmViz" viewBox="0 0 720 300" label="Diagram. One prompt is assembled for every user turn: routing rules (choose 0 to 4 ids the agent should use for this request), 720 compact resource cards, grouped by type, such as kallisto for transcript counts, star as a splice-aware aligner, salmon for transcript counts and nf-core-rnaseq as the full pipeline, and the request with earlier turns, here three yeast libraries for read counts only. A language model reads it in one call of about 3.3 seconds and writes a typed object: ids kallisto and star, with the reasons quick counts for 3 samples and splice-aware counts. Code checks drop unknown ids, keep one per group of equivalents and keep the first four. Pinned: kallisto and star." />
</Figure>

### A decision model as selector
A decision model like TypeSafe's Jev does not act like a traditional LLM by spitting out the list of decisions but instead passes the question once and answers many typed questions about it in parallel, each with a probability. Our selector asks one yes/no question per resource ("should the agent use this resource for this request?", with the resource's id, kind and card), plus three gate questions that check whether the request needs any resource at all: does it ask to analyse data (run something on the user's files), to look something up (in a database or the literature), or does it name a specific tool or pipeline? 

<Figure id="decisionfig" wide title="Figure 4 · How a decision model selects" caption="The model reads the request once and scores every question in parallel; plain code then turns the probabilities into at most four pins. Probabilities and the T and G lines are illustrative.">
<FigureCanvas id="decisionViz" viewBox="0 0 720 330" label="Diagram. A request, three yeast RNA-seq libraries for quick read counts, goes to a decision model that answers all questions in one pass in 0.35 seconds. Gate questions: analyse data? 0.97, look something up? 0.08, name a tool? 0.03; the gate is open because the request asks for an analysis. Resource questions: kallisto 0.94, star 0.88, salmon 0.81, nf-core-rnaseq 0.12. Then plain code selects in four steps: 1, the gate is open because the request asks to analyse data; 2, named tools are kept first, none here; 3, one per group of equivalents, so salmon is dropped as equivalent to kallisto; 4, keep resources with probability at least T, at most four, so nf-core-rnaseq is dropped. Pinned: kallisto and star." />
</Figure>

Selection then follows a straightforward process. If all three gate answers are no, the request needs no resource and nothing is selected. The selector starts by keeping any tools or pipelines explicitly named in the request. Next, it ranks the remaining resources by probability, picks one from each group of equivalent options, and applies a threshold to filter out the rest. In the end, up to four resources are chosen.

 

The thresholds T, G (gate) and N (named tool) are fixed in advance. Because every question is independent, all 720 answers come from a single pass over the request at very fast speed, at a median of 0.35 seconds.

### The Drylab selector: fine-tuning our own decision model
Gemini and Jev fail in opposite places. Gemini covers databases, operations and skills but is slow, while Jev is fast and strong on tools but tends to stop after the first resource (see [Section 4](#results) for detailed results)

Therefore, we also decided that we're gonna train our own decision model to get Gemini's coverage while being close to Jev's speed. 

We started from [Kev](https://github.com/jaredpalmer/kev), an open decision model built on Qwen3.5 that serves TypeSafe's System One API. Kev adds a low-rank adapter (LoRA) and a small pointer head to a frozen backbone, and the head scores each answer option against its question.

The training data starts from production turns and is then topped up with targeted synthetic requests and contrastive pairs, as Figure 5 shows.

<Figure id="ftpipelinefig" wide title="Figure 5 · From production turns to training records" caption="Real traffic is filtered for isolation and labelled, then topped up with targeted synthetic requests and contrastive pairs; 517 records that failed label checks are removed before the split. The bar is log scale">
<FigureCanvas id="ftPipelineViz" viewBox="0 0 720 360" label="Pipeline: 17,974 production turns; 4,744 eligible after isolation; 1,007 real turns labelled; plus 2,012 synthetic requests and 688 contrastive-pair requests; 2,875 training and 315 development records" />
</Figure>

Real traffic contains few of the hard cases. We took every eligible real turn in which production used a tool, operation, database or skill, and capped plain turns at a quarter of the set. That gave 1,007 real turns, but still far from sufficient training data for a decision model. We filled the gap with synthetic requests: 1,418 written for specific resources that had fewer than three positive examples, and 594 that need several resource types at once. Each one was labelled by the same independent pipeline, so a request counts for a resource only if the labeller agrees.

Following the Nimble recipe, we also wrote 344 pairs of near-identical requests that differ in one fact that changes the right resource: 3 versus 200 samples, mouse versus human, a sequence versus a structure file, "explain" versus "run", or a GEO accession versus an existing data file. We filtered out pairs that leaked a resource name or stated the rule in the text, and kept a pair only if its two labels actually differ.

| Stage | Count |
| --- | --: |
| Real production turns labelled | 1,007 |
| Synthetic requests (uncovered resources, multi-resource needs) | 2,012 |
| Contrastive pair requests (344 pairs) | 688 |
| Removed by label checks | 517 |
| Training / development records, split by session | 2,875 / 315 |
| Positive / negative questions in training | 10,285 / 119,099 |

We trained with the following settings:

| Training setting | Value |
| --- | --- |
| Adapter | LoRA rank 16 on attention, MLP and linear-attention projections |
| Learning rate, epochs | 2e-5, one epoch |
| Replay | 2,000 records from Kev's original training set |
| Context | 6,144 tokens per record |
| Calibration | one temperature, fitted on development data |
| Hardware | one H100 |

Generating all 720 questions for every training turn would make each record about 62,000 tokens—15 times larger than the 4,000 tokens in a sampled record. To keep training efficient, we use sampled records. At evaluation, however, we still use the full 720-question request as deployed in production. Checkpoint selection, threshold tuning, and all reported scores are based on these full requests. Thresholds are tuned using development data and locked before the benchmark is scored.

## Results
### Evaluation procedure
Gemini and Jev are each run three times per case, because their answers vary from call to call, which gives 2,520 graded calls across seven configurations (five Gemini and two Jev variants, including ablations not shown here); a case's score is the mean of its three runs. The Drylab selector is deterministic, so we only ran it once. Its thresholds are fitted on its own development set, never on these benchmark cases, and frozen before scoring.

Each run returns a set of at most four resource ids. The scorer maps equivalent ids to one class, such as a database playbook and its query function, or an omics operation and the tool that runs the same model, and then grades the set.

### Metrics
| Metric | What it measures | How it is computed |
| --- | --- | --- |
| Per-type hit | whether the selector found a right resource for each need | for each resource type a case needs: 1 if a best resource was picked, 0.5 if only an acceptable one, 0 otherwise |
| Combined score | overall routing quality | the mean of the per-type hits within a case, then the mean over cases; the headline number |
| Tools, databases, omics ops, skills | quality per resource type | the per-type hit, averaged over every case that needs that type |
| False pick rate | restraint | on the 10 no-resource controls, the share of runs that picked anything not allowed |
| Latency | speed | median wall time of one selection call |

### Overall
Every result below compares the same three selectors: Gemini, Jev, and the Drylab selector.

| Metric | Gemini | Jev | Drylab |
| --- | --: | --: | --: |
| Combined | 0.85 | 0.83 | **0.87** |
| Tools | 0.89 | 0.96 | 0.94 |
| Databases | 0.75 | 0.63 | 0.75 |
| Omics ops | 0.60 | 0.45 | **0.78** |
| Skills | 0.83 | 0.64 | 0.80 |
| Follow-up cases | 0.84 | 0.80 | **0.86** |
| Multi-type cases | 0.78 | 0.64 | **0.79** |
| Look-alike cases | 0.88 | 0.84 | **0.90** |
| False picks on no-resource controls | 0.00 | 0.07 | 0.00 |
| Median latency | 3.28 s | 0.35 s | 0.80 s |

The Drylab selector matches or beats Gemini on the combined score and on omics operations, keeps Jev's lead on tools, and makes no false picks on the controls. It also meets the targets we set before training: within 0.03 of Gemini on the combined score, within 0.05 on databases, operations and skills, and no more false picks.  And of course, much faster than a Gemini call.

<Figure id="v14overviewfig" title="Figure 6 · Selector scores on the benchmark" caption="Seeing every resource lifts Gemini on tools and omics ops. Jev is strongest on tools but trails on databases, ops and skills. The Drylab selector keeps Jev's tool strength and leads on omics ops.">
<FigureCanvas id="v14OverviewViz" viewBox="0 0 720 300" label="Dot plot with the Drylab selector. Combined: shortlist 0.79, all resources 0.85, Jev 0.83, Drylab selector 0.87. Tools: 0.78, 0.89, 0.96, 0.94. Databases: 0.80, 0.75, 0.63, 0.75. Omics ops: 0.41, 0.60, 0.45, 0.78. Skills: 0.85, 0.83, 0.64, 0.80." />
</Figure>

### Against whole agent harnesses
A selector is only one part of an agent. To test the approach most harnesses take, we installed the same 720 Drylab resources into three agent systems through each one's native discovery mechanism, and ran the same benchmark through the whole harness, once each:

- Claude Code (Claude Opus 5): every resource becomes a skill folder in `.claude/skills/`, with the card as its `SKILL.md`. Claude Code keeps each skill's name and description in context and opens the full file when it decides to use it.
- Codex (GPT-6.1 terra): the same skill folders in `.agents/skills/`, discovered the same way.
- Biomni: tools, pipelines and omics operations registered as tools, databases as data-lake items and method skills as know-how documents, then selected by Biomni's own resource retriever.

Each harness could read and open skills but not run anything. It ended its turn by naming at most four resources it would use, which we scored with the same scorer as the selectors.

| Metric | Claude Code | Codex | Biomni | Drylab |
| --- | --: | --: | --: | --: |
| Combined | 0.86 | 0.70 | 0.71 | **0.87** |
| Tools | 0.92 | 0.88 | 0.87 | 0.94 |
| Databases | **0.93** | 0.28 | 0.71 | 0.75 |
| Omics ops | 0.58 | **0.90** | 0.38 | 0.78 |
| Skills | 0.80 | 0.38 | 0.65 | 0.80 |
| Multi-type cases | **0.81** | 0.54 | 0.68 | 0.79 |
| False picks on no-resource controls | 0.40 | 0.10 | 0.90 | 0.00 |
| Median time per request | 40 s | 30 s | 3.8 s | 0.80 s |

Claude Code comes closest to the selectors on quality and leads on databases and multi-type requests, because it reads the skill files before deciding. It takes about 40 seconds and a median of five model turns per request to get there, and it pins something on four of the ten requests that need nothing. Codex hit its skill-listing budget: with 720 skills installed, it dropped every description and left 333 skills out of the list the model sees, so it routed mostly by name, which shows in its database and skill scores. Biomni's retriever is built to over-select. It returned 45 resources per request on average; we scored its first four, which rarely included the right database or operation.

This comparison grades how well each harness finds our resources, not its overall ability. Each was run once with its default skill and tool discovery, and its picks come from its own final answer.

## What the selector changes in practice
Below are cases that both Gemini and Jev get wrong but the Drylab selector recovers.

| Request | Before: production selector | After: Drylab |
| --- | --- | --- |
| Follow-up after a table of approved bispecific antibodies: "Now summarise which cell-based potency assays were used for each of them." | nothing pinned | a literature search (`query_pubmed`) and the citation guide |
| "Apply the single-cell QC, clustering and annotation workflow to GSE150728, and the spatial transcriptomics workflow to GSE211895." | four method guides, no data access | the GEO lookup, the QC and clustering operations, and the annotation and spatial guides |
| "Annotate the cell types in this human tonsil single-cell dataset, with special attention to the follicular dendritic cells." | two annotation guides | a cell-marker database and an expression-summary operation beside the annotation guide |
| "I have sequences for 6 bacterial enzymes and SMILES for 40 fragment hits. Which enzyme-fragment pairs are most likely to bind?" | an affinity predictor and a general tools guide | a co-folding model that starts from sequence: `boltz2` or `chai1` |
| "Humanize this rabbit antibody's variable domains, keep the CDRs, and score how human each variant looks against natural human repertoires." | humanization tools, a numbering tool and a general tools guide | humanization tools and the antibody-repertoire database (`query_oas`) |
| "This Arabidopsis gene has no known function. Predict what it does; it is induced by drought and localises to the nucleus." (gene ID withheld) | four databases, no homology search | protein and domain databases plus `diamond` for homology |

The previous system's picks follow a common failure pattern. The selector understands the request, but it spends its four slots on general guides and stops before the database, operation or sequence-based tool that does the actual work. The Drylab selector was trained on requests of exactly this shape: requests that need several resource types, look-alikes labelled as wrong answers, and pairs in which one fact changes the right tool.

## Conclusion
On a well-defined task, a small decision model can match a much larger language model. Jev matched Gemini overall at a tenth of the latency, and after fine-tuning on targeted data, the Drylab selector reaches Gemini's coverage of databases and skills, clearly beats both models on omics operations, and keeps Jev's strength on tools selection. We expect more cases like this as agent systems grow: small, specialised components that are not only cheaper and faster than a general-purpose LLM, but also better at their one job.
