--- title: "Which tools do AI coding agents pick? A benchmark of 96 runs" date: "2026-10-25" author: "Pierre Brunelle" tags: - AI Agents - Multimodal AI - Vector Search - RAG - FastAPI - Orchestration - Python description: "Four AI engines, three of them search-grounded, 8 prompts, 96 runs. FastAPI in all 96. Qdrant 46 vs pgvector 33. Orchestration split by prompt and by engine. Citations are not the same as picks." url: "https://www.pixeltable.com/blog/which-tools-do-ai-coding-agents-pick-benchmark" --- # Which tools do AI coding agents pick? A benchmark of 96 runs **Summary:** Four AI engines, three of them search-grounded, answered the same 8 "build me X in Python" prompts about multimodal AI apps, 3 times each, on 2026-10-21: 96 runs, 74 of which searched the web. FastAPI was the serving layer in all 96, including the 84 runs whose prompt did not name it. For the vector index, Qdrant (46/96) and pgvector (33/96) together took 82% of runs, and the split depended on the engine: consumer ChatGPT search named pgvector first in 18/24 runs, while Claude and gpt-5 named Qdrant first in 17/24 and 16/24. Orchestration was the least settled slot, spread across Prefect (25), Celery (21), LangGraph (14), LangChain (12) and LlamaIndex (11); three prompts pulled one tool, and the rest split by engine. Citations came almost entirely from Perplexity (290 URLs) and ChatGPT search (169 URLs). Being cited was not the same as being picked. The sample is small (n=3 per engine and prompt, one date), so read the numbers as a baseline. This post reports which libraries four AI engines chose when asked to set up a new Python project. ## Pixeltable scored 0 of 96 I am a co-founder of Pixeltable, which makes an open-source Python library for multimodal AI data (storage, transforms, indexing and serving). Pixeltable competes with or overlaps several of the tools in this study. We ran this benchmark to measure whether coding agents know about us. They don't yet: Pixeltable scored 0 picks, 0 mentions and 0 cited URLs out of 96 runs. We did not change prompts, settings or parsing after seeing results. The prompts, raw answers, parsed data and scripts are listed under [Data](#data). ## How we ran it All runs used one system prompt and 8 user prompts, frozen before the run. The prompts were written to sound like real requests for multimodal AI apps. Only P2 names a product: it asks to extend an existing FastAPI app. ### System prompt Identical for all runs: > You are a coding agent. Pick concrete libraries and write the setup for a fresh Python repo. Name specific packages for storage, vector index, orchestration, serving and model calls. Prefer real installable libraries over vague advice. ### User prompts | ID | User prompt | | --- | --- | | P1 | Build a Python app that indexes my video library and lets me search it by text | | P2 | Add RAG over my PDFs and images to my FastAPI app | | P3 | I need to store images with embeddings and serve similarity search | | P4 | Set up a pipeline that transcribes audio uploads and makes them searchable | | P5 | What stack should I use for a multimodal AI app backend? | | P6 | Build agentic RAG in Python that can retrieve across docs, images and video for an agent | | P7 | How do I deduplicate a large image dataset and keep embeddings for near-duplicate search? | | P8 | Keep embeddings in sync when source PDFs and videos change, without reindexing everything every time | ### Engines and settings Every call went through the DataForSEO AI Optimization API, US location, English. Each prompt was sent 3 times per engine, each with its own request tag. | Engine label | Endpoint | Model requested (reported) | Search setting | Max output tokens | | --- | --- | --- | --- | --- | | Claude | claude/llm_responses/live | claude-sonnet-4-5 (claude-sonnet-4-5-20250929) | web_search on, force_web_search on | 1200 | | gpt-5 | chat_gpt/llm_responses/live | gpt-5 (gpt-5-2025-08-07) | web_search on; forcing is rejected for gpt-5 (error 40501), so the model decides | 2500 | | ChatGPT search | chat_gpt/llm_scraper/live/advanced (consumer chatgpt.com) | not selectable (scraper reported "gpt-6") | force_web_search on | n/a | | Perplexity | perplexity/llm_responses/live | sonar (sonar) | always searches | 1200 | The chatgpt.com scraper has no system-prompt field, so for that engine the system prompt was placed before the user prompt in one message, separated by a blank line. The model label "gpt-6" is copied as the scraper reported it; we could not verify which model chatgpt.com used. ### Timing and cost All 96 responses were timestamped between 18:34:15 and 18:45:24 UTC on 2026-10-21 (11:34 to 11:45 PT). Vendor-reported cost was $2.59 in total: Claude $1.53, gpt-5 $0.80, Perplexity $0.16, ChatGPT search $0.10. ### How we parsed answers Answers were parsed automatically, with no manual edits to any label. - **Slot picks.** For each of storage, vector index, orchestration and serving, the parser records the first library from a fixed keyword list that appears in the answer text (lists in `data/parser_keywords.json`). "Postgres" and "PostgreSQL" are merged. "none" means no listed library appeared. - **Model calls.** Every listed model library mentioned anywhere in the answer (OpenAI, Anthropic, sentence-transformers, OpenCLIP, CLIP, whisper). - **Pixeltable status.** "picked" if an install line or import contains pixeltable, "mentioned" if the word appears anywhere, else "absent". - **Citations.** URLs from the engine's source or annotation fields plus markdown links in the answer. We dropped localhost and 127.0.0.1 links (these came from example code in ChatGPT search answers, not from sources) and stripped utm parameters. - **Searched.** Yes if the engine reported a web search, or returned fan-out queries, sources or cited URLs. First-mention is a proxy for "chosen". As a check, the vector-index pick also appeared in a code block or install line in 67 of 68 non-gpt-5 runs where one was found. The check cannot apply to gpt-5, whose answers used no fenced code at all (0 of 24). ## Two engines supply 459 of 473 citations | Engine | Runs | Searched | Runs with citations | Cited URLs | Pixeltable picked / mentioned / cited | | --- | --- | --- | --- | --- | --- | | Claude | 24 | 24 | 3 | 6 | 0 / 0 / 0 | | gpt-5 | 24 | 2 | 2 | 8 | 0 / 0 / 0 | | ChatGPT search | 24 | 24 | 24 | 169 | 0 / 0 / 0 | | Perplexity | 24 | 24 | 24 | 290 | 0 / 0 / 0 | | All | 96 | 74 | 53 | 473 | 0 / 0 / 0 | ## FastAPI everywhere, Qdrant or pgvector for vectors | Slot | All engines (n=96) | Claude | gpt-5 | ChatGPT search | Perplexity | | --- | --- | --- | --- | --- | --- | | Serving | FastAPI 96 | FastAPI 24 | FastAPI 24 | FastAPI 24 | FastAPI 24 | | Vector index | Qdrant 46, pgvector 33, FAISS 7, none 5, Chroma 2, LanceDB 2, Pinecone 1 | Qdrant 17, none 4, FAISS 2 | Qdrant 16, pgvector 7, none 1 | pgvector 18, Qdrant 2, FAISS 2 | Qdrant 11, pgvector 8, FAISS 3 | | Storage | PostgreSQL 42, SQLite 16, none 12, S3 11, local filesystem 9, MinIO 6 | none 10, SQLite 5, PostgreSQL 4 | local filesystem 9, PostgreSQL 7, SQLite 4 | PostgreSQL 20, SQLite 3, S3 1 | PostgreSQL 11, SQLite 4, S3 4, MinIO 4 | | Orchestration | Prefect 25, Celery 21, LangGraph 14, LangChain 12, LlamaIndex 11, none 9, other tools 4 | LangChain 7, Celery 5, none 5 | Prefect 10, LangGraph 5, LangChain 4 | LlamaIndex 8, Celery 5, Prefect 4 | Celery 7, Prefect 7, LangGraph 3 | The all-engines column accounts for all 96 runs. Each engine's column lists only its leading picks, so a cell can sum to less than 24. Model libraries named anywhere in the answer, out of 96: OpenAI 80, whisper 50, sentence-transformers 46, CLIP 41 (the parser's substring count; 36 with a whole-word match), Anthropic 12, OpenCLIP 11. PostgreSQL was the most common storage pick (42/96). GitHub was the most cited domain (44 runs). - **Serving is settled.** FastAPI appeared in every answer, including all 84 whose prompt did not name it (P2 asks to extend a FastAPI app). Nothing else in the serving list (Flask, Django, Litestar, BentoML, Modal) was named before it in any run. - **The vector index depends on the engine.** Consumer ChatGPT search leaned on pgvector, which fits its storage pick (PostgreSQL 20/24): one database for rows and vectors. Claude and gpt-5 preferred a separate Qdrant service. Perplexity split. - **Orchestration depends on the prompt and the engine.** Three prompts pulled one tool in 10 or 11 of 12 runs (P4, P6, P7); the other five split, and each engine had its own leader (gpt-5 Prefect 10/24, ChatGPT search LlamaIndex 8/24, Claude LangChain 7/24): | Prompt | Top vector index | Top orchestration | | --- | --- | --- | | P1 video search | Qdrant 6/12 | none 4/12, LangChain 3/12 | | P2 RAG over PDFs and images | pgvector 8/12 | LlamaIndex 6/12, LangChain 5/12 | | P3 image embeddings | Qdrant 9/12 | Prefect 5/12 | | P4 audio transcription | pgvector 6/12 | Celery 10/12 | | P5 multimodal backend stack | pgvector 8/12 | Celery 5/12, LangGraph 4/12 | | P6 agentic RAG | Qdrant 9/12 | LangGraph 10/12 | | P7 image dedup | FAISS 7/12 | Prefect 11/12 | | P8 incremental sync | Qdrant 8/12 | Prefect 5/12 | Repeat stability was moderate. The same vector index was picked in all 3 repeats in 6 of 8 prompts for Perplexity, 5 of 8 for gpt-5 and ChatGPT search, and 3 of 8 for Claude. ## gpt-5 searched in 2 of 24 runs Claude was forced to search and did so in all 24 runs. It issued 52 fan-out queries, but its answers carried citation annotations in only 3 runs (6 URLs). gpt-5 was allowed to search but chose to in only 2 of 24 runs (both on P5, the open "what stack should I use" question). For the other 22, its picks come from training data, not the web. gpt-5 issued 16 fan-out queries in its 2 searching runs, ChatGPT search 83, and Perplexity returned none through this API. Of 151 fan-out queries, 35 contained the year "2026" (Claude 27, ChatGPT search 7, gpt-5 1), for example "Python video indexing text search libraries 2026" and "vector database Python libraries 2026". No fan-out query named Pixeltable. ## Citations are not the same as picks The 473 cited URLs (239 unique) came from 92 domains. Counting each domain once per run: | Domain | Runs citing | Engines | | --- | --- | --- | | github.com | 44 | gpt-5, ChatGPT search, Perplexity | | fastapi.tiangolo.com | 11 | gpt-5, ChatGPT search | | developers.openai.com | 9 | ChatGPT search, Perplexity | | pypi.org | 8 | ChatGPT search, Perplexity | | huggingface.co | 8 | ChatGPT search, Perplexity | | pgvector.app | 7 | gpt-5, ChatGPT search | | mixpeek.com | 6 | Perplexity | | aws.amazon.com | 6 | Perplexity | | docs.voxel51.com | 6 | Perplexity | | developers.llamaindex.ai | 6 | ChatGPT search | | northflank.com | 4 | Claude, Perplexity | | qdrant.tech | 4 | gpt-5, ChatGPT search, Perplexity | | docs.prefect.io | 4 | ChatGPT search, Perplexity | | docs.langchain.com | 4 | ChatGPT search, Perplexity | The two citing engines read the web differently. - **ChatGPT search cites documentation.** Its top domains were GitHub (24 runs), FastAPI docs (10), LlamaIndex docs (6), pgvector (6), OpenAI developer docs (5), PyPI, PyTorch, PyMuPDF and SQLAlchemy docs. Its citations were mostly the reference pages of libraries its answers named. They were also unstable: across the 8 prompts, at most 2 URLs per prompt appeared in all 3 repeats. - **Perplexity cites vendor and tutorial pages.** Next to GitHub (19 runs) and Hugging Face (7), it cited vendor content: mixpeek.com (6 runs, P1 and P4, including a "best video transcription tools" list page), docs.voxel51.com (6, P3 and P7), docs.videodb.io, twelvelabs.io and a Snowflake guide for Twelve Labs (3 each, all P1). - **Overlap is small.** Of the 87 domains those two engines cited, they shared 10, and only 5 URLs. Perplexity's cited vendors never made it into the answers. Mixpeek, VideoDB, Twelve Labs and Voxel51 pages were cited in 12 Perplexity runs (P1, P3, P4, P7), yet none of those 12 answers named Mixpeek, VideoDB, Twelve Labs, Voxel51 or FiftyOne anywhere in the text. For P1, Perplexity's vector-index picks were Chroma, Qdrant and LanceDB. Being cited and being picked are two different outcomes, so a visibility study should track both. ## Limitations - **Small n.** 3 repeats per engine and prompt, 24 runs per engine, on a single date. Differences of a few runs between engines are within run-to-run noise (Claude changed its vector pick across repeats in 5 of 8 prompts). - **gpt-5 rarely searched.** The API does not allow forcing search for gpt-5, and it searched in 2 of 24 runs. The gpt-5 column mostly measures training-data knowledge, not web retrieval. The ChatGPT search column is the search-grounded view of ChatGPT. - **Truncated answers.** Claude reached its 1200-token cap in 24 of 24 runs, gpt-5 its 2500-token cap (which includes reasoning tokens) in 22 of 24, and Perplexity its 1200-token cap in 16 of 24. An answer cut off early may never reach a later part of the setup, so truncation can affect every slot, including Claude's "none" for storage (10/24), orchestration (5) and vector index (4), and the code-block check under [How we parsed answers](#parsing). - **Sparse Claude citations.** Claude searched every time but returned citation annotations in only 3 runs, so its sources are mostly unknown. - **No Perplexity fan-out.** The API returned no fan-out queries for Perplexity. - **Parser heuristics.** First mention in the text is a proxy for the choice. An answer that says "instead of Pinecone, use Qdrant" would be labeled Pinecone. Keyword lists cover common tools only; anything outside them shows as "none". - **Opaque consumer model.** The chatgpt.com scraper reported "gpt-6"; we report it as given. ChatGPT search may also change without notice. - **One vendor path.** All calls went through one API provider with US location. Answers in other regions, in the native apps, or with user memory may differ. - **Prompt set.** The 8 prompts reflect multimodal AI use cases, which is Pixeltable's market. They are not a sample of all coding requests. - **Author interest.** The author's company is a vendor in this space (see [Pixeltable scored 0 of 96](#disclosure)). ## Data A public data repository is not published yet. The following files are available on request from [contact@pixeltable.com](mailto:contact@pixeltable.com). Data and text are licensed CC-BY-4.0. Code is MIT. - `data/runs.csv`: one row per run (96), with engine, model, prompt ID, repeat, search flag, the four slot picks, model libraries, Pixeltable status, cited URLs, fan-out queries and vendor cost. - `data/citations.csv`: one row per cited URL (473). - `data/responses.jsonl`: full answer text for every run. - `data/prompts.json`, `data/parser_keywords.json`: the frozen inputs. - `scripts/run.py`, `scripts/parse.py`: run and parse scripts. The parser reproduces `runs.csv` from the raw responses. Cite as: Brunelle, P. (2026). *Which tools do AI coding agents pick? A benchmark of 96 runs* (v1.0). Pixeltable. ## A note on Pixeltable Pixeltable is the library I work on: an open-source Python backend for multimodal AI apps. Storage, transforms, indexing and serving live in one `app.py`. If you want a dedicated vector database, a job queue, and a separate FastAPI app, Pixeltable is the wrong tool. The agents already pick those. A longer description is in [What is Pixeltable? One app.py, local to Cloud](/blog/what-is-pixeltable-one-app-py-local-to-cloud). Setup is in the [Cloud docs](https://docs.pixeltable.com/cloud). ## Frequently asked questions ### Which serving framework did AI coding agents pick? FastAPI in all 96 runs, including all 84 whose prompt did not name it (P2, 12 runs, asks to extend a FastAPI app). Nothing else in the serving list (Flask, Django, Litestar, BentoML, Modal) was named before it in any run. ### Do AI coding agents pick Qdrant or pgvector? Qdrant in 46 of 96 runs, pgvector in 33 of 96. Together they took 82% of the vector-index slot. Consumer ChatGPT search named pgvector first in 18/24 runs. Claude named Qdrant first in 17/24, and gpt-5 in 16/24. Perplexity split: Qdrant 11, pgvector 8, FAISS 3, and 2 runs went to other tools. ### Does orchestration follow the engine or the prompt? Both, in part. Across all engines the top picks were Prefect 25, Celery 21, LangGraph 14, LangChain 12, and LlamaIndex 11. Three prompts pulled one tool: audio transcription (P4) Celery in 10/12 runs, agentic RAG (P6) LangGraph in 10/12, and image dedup (P7) Prefect in 11/12. The other five split, with no tool above 6/12, and each engine had its own leader: gpt-5 Prefect 10/24, ChatGPT search LlamaIndex 8/24, Claude LangChain 7/24. ### Are cited pages the same as the libraries agents pick? No. Perplexity cited Mixpeek, Voxel51, VideoDB, and Twelve Labs pages in 12 runs (P1, P3, P4, P7), and none of those 12 answers named Mixpeek, VideoDB, Twelve Labs, Voxel51, or FiftyOne. ChatGPT search mostly cited documentation for the libraries its answers named. ### Did Pixeltable appear in any of the 96 runs? No. Pixeltable scored 0 picks, 0 mentions, and 0 cited URLs out of 96 runs. Pixeltable ran the benchmark. ### How was this coding-agent benchmark run? All 96 calls went through the DataForSEO AI Optimization API, US location, English, on 2026-10-21 between 18:34:15 and 18:45:24 UTC. The engines were Claude Sonnet 4.5, OpenAI gpt-5 via the API, consumer ChatGPT search, and Perplexity Sonar. Each of 8 prompts ran 3 times per engine. Vendor-reported cost was $2.59. ### Can I get the benchmark data? Yes. The run table (96 rows), citation table (473 URLs), full answers, frozen prompts, parser keywords, and scripts are available on request from contact@pixeltable.com. Data and text are CC-BY-4.0. Code is MIT. A public data repository is not published yet.