Why your agent needs access to your documentation

What 1,192 agent conversations taught us about knowledge base search A few months ago we shipped an agent inside our own product. It lives in our web app and lets users ask questions about their deployment,... What 1,192 agent conversations taught us about knowledge base search A few months ago we shipped an agent inside our own product. It lives in our web app and lets users ask questions about their deployment, things like “how many Slack bot questions have users asked in the last month?” We built it because the analytics tooling we had shipped (clustering, tagging, filters) never quite covered every use case, and we wanted to see whether a chat interface could. The agent has around 30 native tools to interact with our platform, such as search_conversations and display_chart. It also has a single search_knowledge_base tool that can read documentation, code examples, support FAQs, and API reference. We expected the native tools to do most of the work. But when we looked at the last 1,192 conversations, search_knowledge_base was the most used tool, almost as much as all the native tools combined. This pattern is worth flagging for the cloud-native community in particular. Across the CNCF landscape, from Kubernetes and Prometheus to Envoy, Argo, and OpenTelemetry, documentation is one of the primary interfaces between a project and its users, and it tends to be large, fast-moving, and spread across many surfaces. As more teams add agents alongside that documentation, the question of how much the agent should read the docs versus call purpose-built tools becomes a practical design decision. Here is what we found when we looked at why documentation search mattered so much. Finding #1: Documentation became the fallback The main use case for knowledge base search was acting as a failover when users asked questions that no native tool could answer. That accounted for 32.1% of conversations . These were reasonable product questions like: “How do I set up the Slack integrat

What 1,192 agent conversations taught us about knowledge base search A few months ago we shipped an agent inside our own product. It lives in our web app and lets users ask questions about their deployment,... What 1,192 agent conversations taught us about knowledge base search A few months ago we shipped an agent inside our own product. It lives in our web app and lets users ask questions about their deployment, things like “how many Slack bot questions have users asked in the last month?” We built it because the analytics tooling we had shipped (clustering, tagging, filters) never quite covered every use case, and we wanted to see whether a chat interface could. The agent has around 30 native tools to interact with our platform, such as search_conversations and display_chart. It also has a single search_knowledge_base tool that can read documentation, code examples, support FAQs, and API reference. We expected the native tools to do most of the work. But when we looked at the last 1,192 conversations, search_knowledge_base was the most used tool, almost as much as all the native tools combined. This pattern is worth flagging for the cloud-native community in particular. Across the CNCF landscape, from Kubernetes and Prometheus to Envoy, Argo, and OpenTelemetry, documentation is one of the primary interfaces between a project and its users, and it tends to be large, fast-moving, and spread across many surfaces. As more teams add agents alongside that documentation, the question of how much the agent should read the docs versus call purpose-built tools becomes a practical design decision. Here is what we found when we looked at why documentation search mattered so much. Finding #1: Documentation became the fallback The main use case for knowledge base search was acting as a failover when users asked questions that no native tool could answer. That accounted for 32.1% of conversations . These were reasonable product questions like: “How do I set up the Slack integration?” “Why am I getting CORS errors on the widget?” “How often do website crawls ingest?” Without search_knowledge_base, the agent would have had to refuse or guess. We had built an analytics agent, but users treated it as a catch-all product agent, and the documentation tool was what let it meet that expectation. Finding #2: Documentation adds context to the native tools The next pattern was conversations where the agent used both kinds of tools, around 7% of the total . These were smaller in volume but they made the division of labor clear. The native tools answer what is true about a user’s account right now. search_knowledge_base answers what it means. In one example, a user asked what type of MCP integration they had set up and how it differed from the other options. list_integrations answered the first half by reporting the current configuration. To answer the second half, the agent had to read the docs, because the comparison between integration types lives there. Neither tool could have answered on its own. The native tool knew the state; the knowledge-base tool knew the product. This is the role most people expect a docs-reading tool to play, and it was the smallest of the three patterns we found. Finding #3: Documentation helps the agent use its own tools This was the most interesting pattern, and the one we did not expect. In some conversations, search_knowledge_base was not called to answer the user. It was called so the agent could figure out what to do. The clearest example: a user asked the agent to find conversations with negative sentiment. There is no sentiment filter and no sentiment field anywhere in the product. An agent with only native tools would either refuse or guess at a tool that does not exist. Instead, the agent searched the documentation, found that negative signals are captured as downvotes and feedback comments, and then called the right native tool with the right filters. The docs were not the answer; they were the briefing the agent needed before it could act. The knowledge-base tool was doing planning work. We have started thinking of this as a third job the knowledge-base tool does, alongside answering and contextualizing: it teaches the agent what the product can actually do, so the agent can pick the right native tool. The takeaway If you are building a product agent, a knowledge-base tool is likely more important than it first appears. It catches the questions your native tools were never designed for, which tend to be more numerous than expected. It explains what your native tools return when the raw output is not enough. And it tells the agent which native tool to call when the user’s language does not match your schema. We shipped 30 native tools and one tool that reads documentation. The tool that reads documentation is what makes the other 30 useful. For projects with deep, evolving documentation, which describes much of the cloud-native ecosystem, that is a useful thing to keep in mind when designing an agent: the documentation is not just a fallback, it is part of how the agent reasons.

What 1,192 agent conversations taught us about knowledge base search A few months ago we shipped an agent inside our own product. It lives in our web app and lets users ask questions about their deployment,... What 1,192 agent conversations taught us about knowledge base search A few months ago we shipped an agent inside our own product. It lives in our web app and lets users ask questions about their deployment, things like “how many Slack bot questions have users asked in the last month?” We built it because the analytics tooling we had shipped (clustering, tagging, filters) never quite covered every use case, and we wanted to see whether a chat interface could. The agent has around 30 native tools to interact with our platform, such as search_conversations and display_chart. It also has a single search_knowledge_base tool that can read documentation, code examples, support FAQs, and API reference. We expected the native tools to do most of the work. But when we looked at the last 1,192 conversations, search_knowledge_base was the most used tool, almost as much as all the native tools combined. This pattern is worth flagging for the cloud-native community in particular. Across the CNCF landscape, from Kubernetes and Prometheus to Envoy, Argo, and OpenTelemetry, documentation is one of the primary interfaces between a project and its users, and it tends to be large, fast-moving, and spread across many surfaces. As more teams add agents alongside that documentation, the question of how much the agent should read the docs versus call purpose-built tools becomes a practical design decision. Here is what we found when we looked at why documentation search mattered so much. Finding #1: Documentation became the fallback The main use case for knowledge base search was acting as a failover when users asked questions that no native tool could answer. That accounted for 32.1% of conversations . These were reasonable product questions like: “How do I set up the Slack integration?” “Why am I getting CORS errors on the widget?” “How often do website crawls ingest?” Without search_knowledge_base, the agent would have had to refuse or guess. We had built an analytics agent, but users treated it as a catch-all product agent, and the documentation tool was what let it meet that expectation. Finding #2: Documentation adds context to the native tools The next pattern was conversations where the agent used both kinds of tools, around 7% of the total . These were smaller in volume but they made the division of labor clear. The native tools answer what is true about a user’s account right now. search_knowledge_base answers what it means. In one example, a user asked what type of MCP integration they had set up and how it differed from the other options. list_integrations answered the first half by reporting the current configuration. To answer the second half, the agent had to read the docs, because the comparison between integration types lives there. Neither tool could have answered on its own. The native tool knew the state; the knowledge-base tool knew the product. This is the role most people expect a docs-reading tool to play, and it was the smallest of the three patterns we found. Finding #3: Documentation helps the agent use its own tools This was the most interesting pattern, and the one we did not expect. In some conversations, search_knowledge_base was not called to answer the user. It was called so the agent could figure out what to do. The clearest example: a user asked the agent to find conversations with negative sentiment. There is no sentiment filter and no sentiment field anywhere in the product. An agent with only native tools would either refuse or guess at a tool that does not exist. Instead, the agent searched the documentation, found that negative signals are captured as downvotes and feedback comments, and then called the right native tool with the right filters. The docs were not the answer; they were the briefing the agent needed before it could act. The knowledge-base tool was doing planning work. We have started thinking of this as a third job the knowledge-base tool does, alongside answering and contextualizing: it teaches the agent what the product can actually do, so the agent can pick the right native tool. The takeaway If you are building a product agent, a knowledge-base tool is likely more important than it first appears. It catches the questions your native tools were never designed for, which tend to be more numerous than expected. It explains what your native tools return when the raw output is not enough. And it tells the agent which native tool to call when the user’s language does not match your schema. We shipped 30 native tools and one tool that reads documentation. The tool that reads documentation is what makes the other 30 useful. For projects with deep, evolving documentation, which describes much of the cloud-native ecosystem, that is a useful thing to keep in mind when designing an agent: the documentation is not just a fallback, it is part of how the agent reasons.