Is Agentic Search Actually Better for E-Commerce Search?

Say, a shopper asks for a bicycle without pedals. The right answer is a balance bike, but a literal search can also surface child seats, training wheels, and pedal accessories. I gave my agent three tools to search and let it decide how to separate the product from the near-matches.

The agent used GLM-5.2, hosted on Together AI, and could call five tools:

• grep_products: literal search over product text
• semantic_search_products: ColGREP/ColBERT retrieval
• visual_search_products: text-to-image retrieval with SigLIP
• inspect_products: read the full fields of retrieved products
• finish_search: return a grounded ranking or abstain

This is the agentic search setup I wanted to test that allows an agent to choose retrievers, observe results, refine its strategy, inspect candidates, and decide when to stop.

The question was simple: does giving an agent access to several retrieval tools produce better search than running those retrievers directly?

TL;DR: In our experiment, the agent was competitive on the core commerce queries, but standalone SigLIP still led the full judged benchmark, as shared in the above benchmark image. The reason is more interesting than a simple model loss: retrieval usually surfaced strong candidates, while final selection remained difficult.

The catalog came from a frozen subset of the Amazon ESCI shopping-query dataset. Each product could include listing text, structured metadata such as brand and color, and a product image. This was not a database of clean filter fields alone.

• 100 queries
• 2,231 unique products
• 2,377 query-product judgments
• five heuristic intent families: exact, visual, composed, semantic, and managed/awkward queries

Each system ran on every query in two settings:

1. Judged candidates: rank only the products with labels for that query. This isolates ranking quality.
2. Full catalog: search all 2,231 products. These relevance scores are lower bounds because products outside the labeled pool are unjudged, not necessarily irrelevant.

The five evaluated systems were grep, ColGREP, SigLIP, reciprocal-rank fusion, and the adaptive GLM-5.2 agent. That produced 1,000 system-query results. The agent saw product data and tool outputs, but never the relevance labels.

I used NDCG@5 as the main ranking metric, with ESCI labels weighted Exact=3, Substitute=2, Complement=1, and Irrelevant=0. Latencies are local measurements, and model cost uses the token pricing recorded at run time.

These were not three implementations of the same ranking function. Each exposed a different representation of the catalog.

grep_products: literal evidence

This was a small field scanner, not BM25 and not the Unix grep command. It lowercased the query, removed a short stop-word list, and looked for each remaining term as a substring in the title, brand, color, description, and bullet fields. Products were ranked by how many field-term matches they accumulated, normalized by the number of query terms.

That made it useful for model numbers, brands, colors, and phrases already present in the listing. It was also permissive: one matching term was enough to enter the result set, repeated matches across fields increased the score, and it had no representation of synonyms, intent, or negation. Its judged-track median was 0.8 ms.

semantic_search_products: semantic-only ColGREP

ColGREP can support hybrid patterns, but this experiment invoked it with --semantic-only. It used the AnswerAI ColBERT-small ONNX model over text files containing the product fields. ColBERT keeps token-level representations and uses late interaction, so a query can align with several parts of a product description without requiring the same surface words.

This was the tool for vocabulary mismatch, such as asking for a product by purpose rather than by its catalog phrasing. The tradeoff in this local setup was latency: its judged-track median was 1,428 ms. That number describes this implementation and hardware, including the ColGREP process and index path, not an inherent latency for every ColBERT deployment.

visual_search_products: text against product images

SigLIP searched a different source of evidence. I embedded 2,192 product images in advance as 768-dimensional vectors. At query time, the SigLIP text encoder produced one normalized vector, then the retriever ranked images by cosine similarity.

It could recover color, pattern, shape, and style when listing text was weak. It could also be fooled by the image itself. A hero image may contain a shoe even when the product being sold is a Bluetooth pedal. With the image vectors already computed, its judged-track median was 32 ms.

The fixed comparison view made these differences concrete. For the deliberately vague query right tick shoes where I wanted to see if any results give Nike shoes, you can see what are the different results that I got.

• Grep followed surface words and returned candy sticks and unrelated black products.
• ColGREP understood that the request was about footwear.
• Interestingly, SigLIP returned one result with Nike logo that means visually it was able to capture that right tick logo. Now if you see closely, for the other retrieved images the actual product there belonged to Bluetooth pedal products whose listing image happened to include a sneaker.

No single column was universally correct. This is what made the agent setup interesting. Instead of deciding beforehand that every query should use lexical, semantic, or visual retrieval, I could give the agent access to all three and let it search differently depending on the query.

The question was whether that flexibility actually helped.

So I compared the agent against each retriever on its own, along with a fixed reciprocal-rank-fusion baseline that combined their rankings without an LLM making the routing decisions.

On the judged-candidate track, standalone SigLIP performed best.

The n/a cells do not mean the local retrievers are free in production; this experiment did not price their infrastructure since all the models are open-source and are running in local. The table shows the marginal LLM charge and, more importantly here, the latency added by orchestration.

The agent beat SigLIP on 26 queries, tied on 18, and lost on 56. Its median latency was roughly 186 times SigLIP's measured median. Across both tracks, 85% of agent latency came from model calls rather than retrieval tools.

The gap was much smaller on the predefined 60-query exact + visual + composed slice. SigLIP scored 0.796 NDCG@5 and the agent scored 0.762. On that slice, the agent matched ColGREP and came within 0.034 of SigLIP.

I replayed the judged-track traces against the relevance labels and separated the products the agent observed from the products it finally returned.

• An Exact product appeared in the agent's retrieved candidate pool for 95 of 100 queries.
• An Exact product survived into the final ranking for 68 of 100.
• Of the 32 final rankings with no Exact product, 27 had already observed one.

An oracle reranker, using the hidden labels to sort only the candidates surfaced by the agent's actual tool calls, would score 0.931 NDCG@5, compared with the agent's 0.562. Choosing the best standalone retriever separately for each query would score 0.771.

These are diagnostic ceilings, not deployable systems: a production selector does not have relevance labels. But they locate the lost potential. The candidate pool was strong, yet selection remained the larger bottleneck. The system often did not need another search. It needed to use the evidence it already had.

This was not a comparison between an LLM and SigLIP. SigLIP was one of the LLM's tools.

Across 200 runs, the agent called grep 190 times, semantic search 181 times, visual search 55 times, and product inspection 35 times. It used more than one retriever on 90 of the 100 judged queries. The most common complete trajectories were:

grep → semantic → finish             87 runs
grep → semantic → visual → finish    22 runs
semantic → grep → finish             15 runs
grep → semantic → inspect → finish   12 runs
grep → finish                         11 runs

This is genuinely iterative retrieval, not a one-shot query rewrite. The agent often used the first result set to choose a second search surface. Visual search remained selective: it appeared in 13 of the 20 judged visual-family queries rather than being applied to every request.

For black and white striped shirt for a girl, the agent combined literal matches with SigLIP results and returned products whose images actually showed the requested pattern.

For bicycle without pedals, the agent used literal and semantic evidence to identify balance bikes, while explicitly excluding child seats and training wheels that shared many query terms.

It also matters for composed requests. The long-parka query requires color, length, insulation, hood, pockets, and weather resistance. The agent used all three retrievers, returned ten candidates, and preserved uncertainty around artificial filling and the difference between waterproof and water-resistant.

These examples show where an agent can add something beyond a single ranking: exclude accessory matches, reconcile several constraints, and preserve uncertainty when catalog fields do not fully support the request. The cost is additional retrieval, tool-output context, and model time.

On the judged track, the agent used multiple retrievers on 90 queries. In 39 of those runs, a later retriever contributed a final selected product that the first retriever had not found.

That 43% figure is a narrow selection-yield metric: an extra retrieval could still have helped the model reject a bad candidate or increase confidence. But it reveals a useful operational question:

What did the extra tool call change?

Counting calls is not enough. An agent evaluator should record whether an escalation adds relevant candidates, changes the final ranking, verifies a constraint, or merely adds latency and tokens.

The agent did have genuine wins. For black and gold warm ups, it combined grep and semantic search to score 1.0, versus SigLIP's 0.213. It recognized that the catalog contained partial matches rather than a complete black-and-gold warm-up suit. This is the kind of constraint-aware synthesis that a single embedding ranking does not perform by itself.

But occasional synthesis wins did not offset the remaining selection failures and orchestration cost across the full query set.

The result does not argue for removing agents. It argues against placing an unconstrained agent in front of every search.

The direct retrievers already define a useful first split:

Only 6 judged queries used grep as their sole retriever. Those runs had a 4,328 ms median, compared with 0.8 ms for grep alone. The agent may add value by filtering or explaining results, but if the task is only to return the literal ranking, the model call is pure overhead.

That last row needs a high bar. On the 91 judged runs where the agent called semantic search, median end-to-end latency was 6,097 ms and average model cost was $0.00933. Those queries were not randomly assigned, so their relevance score cannot be compared causally with the other routes. The operational point is simpler: if one semantic query is enough, call ColGREP directly. Put an agent around it only when the system must do something after retrieval, such as reconcile a hard constraint, inspect details, combine modalities, or recover from weak evidence.

A better first production design is probe, then plan:

query → cheap retrieval probe → inspect the result set
                                ├─ sufficient → rank
                                ├─ vocabulary mismatch → semantic retrieval
                                ├─ visual mismatch → SigLIP
                                └─ conflicting constraints → agent + inspect

This differs from asking an agent to predict the best retriever from the query alone. The planner first sees what the catalog actually returned, then intervenes for a specific failure. JD.com's recent Probe-then-Plan work develops this idea for industrial ecommerce search; my experiment did not test that architecture, but its motivation matches the failure here unusually well.

The fast retriever should be the default. The agent should be invoked when it has a specific job: reconcile constraints, inspect details, reformulate after weak evidence, or explain why no credible match exists. Each escalation should earn its execution cost and the context it adds.

There is also a scale boundary. Scanning all 2,231 products is reasonable in this experiment; letting an agent roam millions of listings is not. At larger scale, I would use first-stage retrieval to build a bounded candidate workspace, then allow grep, inspection, and visual comparison inside it. Recent work on retrieving interaction spaces makes the same architectural move for much larger text corpora: retrieval creates the space in which the agent can investigate, rather than trying to produce the final answer by itself.

The next evaluation should compare the current free-form agent with:

• an explicit prompt rule that requires SigLIP for appearance-heavy queries
• a deterministic intent router
• a constrained agent that receives retrieval-quality and latency priors
• another tool-calling model using the same tools and evaluation setup

The metric should not only be relevance. It should be incremental relevance per additional second, dollar, and token of tool output, with routing, retrieval, and selection measured separately.

Providing tools makes a workflow agentic. It does not make the resulting search policy good. That policy includes which retriever to call, what to ask it, what evidence to inspect, which candidates to keep, when to search again, and when to stop.

The agent used its retrieval tools actively and produced strong candidate pools. It still trailed the best standalone retriever overall because choosing among those candidates is its own ranking problem, and the orchestration added substantial model latency.

That is not a disappointing result. It is a concrete engineering result: before adding more tools or a larger model, make the full search policy observable, evaluate every stage, and require each escalation to justify its cost.

This conclusion is deliberately narrow. I tested single-turn retrieval against a frozen catalog, not personalization, clarification across turns, live inventory, review synthesis, or cross-store comparison. Agents may be easier to justify when shopping becomes an investigation. For ordinary catalog lookup, the burden of proof should remain with the agent.

The experiment used one agent sample per query with temperature 0. The capability-family labels were heuristically selected rather than human-certified. The semantic slice deliberately includes malformed and customer-service-like inputs. Full-catalog relevance is a conservative lower bound due to incomplete judgments. Local latency depends on hardware, cache state, and precomputed embeddings.