--- kind: ideas granularity: day period_start: '2026-06-03T00:00:00' period_end: '2026-06-04T00:00:00' run_id: 89bc13dd-4469-48a8-bcb4-fb31cdf03275 status: succeeded topics: - LLM agents - coding benchmarks - software engineering - MCP security - LLM serving - observability - agent tooling tags: - recoleta/ideas - topic/llm-agents - topic/coding-benchmarks - topic/software-engineering - topic/mcp-security - topic/llm-serving - topic/observability - topic/agent-tooling language_code: en pass_output_id: 227 pass_kind: trend_ideas upstream_pass_output_id: 226 upstream_pass_kind: trend_synthesis --- # AI toolchain verification ## Summary MCP server teams can add description-code consistency checks before tools reach agents. SDK documentation teams can give review agents a retrieval layer that finds cross-file evidence for claims. LLM serving teams can add differential diagnosis runs that compare intermediate model states when output quality drops without a crash. ## CI checks for MCP tool descriptions against implementation behavior Teams shipping MCP servers should treat tool descriptions as executable contracts and check them in CI when a tool changes. The check can extract the tool name, schema, natural-language description, entry function, local helper calls, and sensitive API calls, then flag omitted behavior, overclaimed capability, state mutation, resource use, or data leakage for human review. The case is concrete because the measured failure rate is no longer hypothetical. DCIChecker found description-code inconsistency in 9.93% of 19,200 tool pairs across 2,214 real MCP servers. These inconsistencies can mislead tool selection because the LLM plans from exposed descriptions and usually cannot inspect implementation code at call time. A practical first deployment is a CI job on changed MCP tools, with stricter review for filesystem, network, auth, analytics, and state-changing tools. The output should include the descriptor text, the code path that caused the flag, and the inconsistency subtype so maintainers can fix the description or the implementation before an agent can call it. ### Sources - [Description-Code Inconsistency in Real-world MCP Servers: Measurement, Detection, and Security Implications](../Inbox/2026-06-03--description-code-inconsistency-in-real-world-mcp-servers-measurement-detection-and-security-implications.md): Defines description-code inconsistency for MCP tools, describes DCIChecker, and reports 9.93% inconsistent pairs across 19,200 tool pairs from 2,214 servers. - [Description-Code Inconsistency in Real-world MCP Servers: Measurement, Detection, and Security Implications](../Inbox/2026-06-03--description-code-inconsistency-in-real-world-mcp-servers-measurement-detection-and-security-implications.md): Explains that LLMs usually plan from tool descriptions without inspecting code, and gives an example of undisclosed state and analytics side effects. ## Repository evidence retrieval for SDK documentation review agents SDK documentation teams can add a tool-callable retrieval step to every AI-assisted API comment review and tutorial validation pass. The retrieval layer should index source files, API references, tests, examples, and upstream documentation, then return ranked snippets with file metadata for each claim the agent wants to verify. Context-as-a-Service gives a workable pattern. In two production-SDK case studies, a CaaS-augmented agent found the same five missing public-member docs as a baseline agent with normal repository tools, plus eight more issues: cross-file factual errors, underspecified API comments, an executable URI bug, an API-usage improvement, and missing prerequisites. It also cut wall-clock time and input tokens in the reported runs, although it used more LLM calls. The first useful adoption point is high-risk generated docs: API references for public methods, tutorials with executable snippets, and docs that depend on lifecycle behavior or objects created in other files. Review should keep only findings tied to concrete evidence snippets and source paths, so the doc owner can accept or reject a fix without rerunning the agent’s search. ### Sources - [Context-as-a-Service: Surfacing Cross-File Dependency Chains for LLM-Generated Developer Documentation](../Inbox/2026-06-03--context-as-a-service-surfacing-cross-file-dependency-chains-for-llm-generated-developer-documentation.md): Describes CaaS as a retrieval layer for documentation agents and reports retained findings rising from 5 to 13 across two production-SDK workflows. - [Context-as-a-Service: Surfacing Cross-File Dependency Chains for LLM-Generated Developer Documentation](../Inbox/2026-06-03--context-as-a-service-surfacing-cross-file-dependency-chains-for-llm-generated-developer-documentation.md): Lists the extra findings surfaced by CaaS and reports 22% to 34% wall-clock reduction across the two tasks. - [Context-as-a-Service: Surfacing Cross-File Dependency Chains for LLM-Generated Developer Documentation](../Inbox/2026-06-03--context-as-a-service-surfacing-cross-file-dependency-chains-for-llm-generated-developer-documentation.md): Explains why ordinary repository tools miss non-obvious dependency chains behind documentation claims. ## Differential diagnosis runs for silent LLM serving errors Teams running vLLM, SGLang, or similar serving engines should add a reproducible differential diagnosis path for quality regressions that do not crash. When a known prompt, model, and serving configuration starts producing wrong or lower-accuracy output, the system should run the target engine beside a reference implementation such as HuggingFace Transformers, collect intermediate activations and call sequences, align matching components, and rank the first component where outputs diverge sharply. Ekka shows this workflow can move silent-error triage beyond output comparison. Its study of real vLLM and SGLang issues reports 80% pass@1 and 88% pass@5 diagnosis accuracy. The issue set also shows why component-level evidence is needed: root causes were spread across serving logic, model implementation, kernel backends, and numerical precision. A small operational version can start with nightly canary prompts for models that have custom kernels, quantization, paged attention, or sliding-window attention enabled. When an accuracy metric drops, the diagnosis artifact should include the prompt, model revision, engine revision, aligned component path, activation-difference summary, and candidate faulty layer for the serving engineer. ### Sources - [Ekka: Automated Diagnosis of Silent Errors in LLM Inference](../Inbox/2026-06-03--ekka-automated-diagnosis-of-silent-errors-in-llm-inference.md): Summarizes Ekka’s differential debugging method and reports 80% pass@1 and 88% pass@5 diagnosis accuracy on real silent serving errors. - [Ekka: Automated Diagnosis of Silent Errors in LLM Inference](../Inbox/2026-06-03--ekka-automated-diagnosis-of-silent-errors-in-llm-inference.md): Explains why optimized LLM serving engines can return responses without explicit errors while output quality degrades.