Building an AI agent that searches your knowledge base before answering

An LLM answering from its training data is a confident guesser. An LLM that searches your knowledge base first, then answers from what it found, is a research assistant. The gap between those two behaviors is almost entirely about giving the model a retrieval tool and instructing it to reach for that tool before it speaks.

This walkthrough shows the shape of an agent that does exactly that. The specifics use Kognita's search endpoint and the Claude API, but the pattern is portable to any retrieval backend and any tool-capable model.

The core loop

An agent that grounds its answers runs a simple loop:

1. Receive the user's question.
2. Decide whether it needs to look something up (for a real knowledge task, it should).
3. Call a search_knowledge_base tool with a query.
4. Read the retrieved passages.
5. Answer from those passages — and cite them.

The model drives steps 2–4 itself. Your job is to define the tool and set the instructions so "look it up" is the default instinct, not an afterthought.

Step 1: Define the search tool

Give the model a tool whose description makes its purpose unambiguous. The description is not documentation for you — it's the model's only guide to when to call it, so write it for the model.

tools = [
    {
        "name": "search_knowledge_base",
        "description": (
            "Search the company knowledge base for relevant information. "
            "Use this for ANY question about company products, policies, "
            "documentation, or internal processes. Always search before "
            "answering such questions rather than relying on prior knowledge."
        ),
        "input_schema": {
            "type": "object",
            "properties": {
                "query": {
                    "type": "string",
                    "description": "A focused search query capturing the user's intent.",
                }
            },
            "required": ["query"],
        },
    }
]

Step 2: Back the tool with real retrieval

When the model calls the tool, you run the actual search. Here that's Kognita's hybrid search endpoint — vector plus keyword, with reranking — which returns the passages most relevant to the query along with their sources.

import requests

def search_knowledge_base(query: str) -> list[dict]:
    resp = requests.post(
        f"https://api.kognita.dev/v1/knowledge-bases/{KB_ID}/search/hybrid",
        headers={"X-API-Key": API_KEY, "X-Organization-Id": ORG_ID},
        json={"query": query, "limit": 5},
    )
    resp.raise_for_status()
    return resp.json()["results"]   # each: {content, source, score}

Hybrid search matters here: an agent phrases queries in natural language, but users often ask about exact names, error codes, or product terms that pure vector search can miss. Fusing keyword and semantic retrieval — then reranking — gives the agent the best shot at surfacing the right passage on the first call.

Step 3: Run the agent loop

Now wire it together. Send the question with the tool available; if the model calls the tool, execute the search, feed results back, and let it produce the final grounded answer.

import anthropic

client = anthropic.Anthropic()

def ask(question: str) -> str:
    messages = [{"role": "user", "content": question}]

    while True:
        response = client.messages.create(
            model="claude-fable-5",
            max_tokens=1024,
            system=(
                "You are a support assistant. For any question about the "
                "company, ALWAYS call search_knowledge_base first and answer "
                "only from the results. Cite the source of each claim. If the "
                "search returns nothing relevant, say you don't have that "
                "information — never guess."
            ),
            tools=tools,
            messages=messages,
        )

        if response.stop_reason == "tool_use":
            messages.append({"role": "assistant", "content": response.content})
            tool_results = []
            for block in response.content:
                if block.type == "tool_use" and block.name == "search_knowledge_base":
                    results = search_knowledge_base(block.input["query"])
                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": block.id,
                        "content": format_passages(results),
                    })
            messages.append({"role": "user", "content": tool_results})
            continue   # let the model read results and answer (or search again)

        return response.content[0].text   # final grounded answer

The while loop is what makes this an agent rather than a single retrieval call: the model can search, look at what came back, decide it's insufficient, and search again with a refined query before committing to an answer.

Step 4: The system prompt does the heavy lifting

Notice how much work the system prompt does. Three instructions turn a guesser into a researcher:

• "ALWAYS call search_knowledge_base first" — makes retrieval the default, not a maybe.
• "answer only from the results" — forecloses blending real retrieval with hallucinated training-data recall.
• "if the search returns nothing, say you don't have that information — never guess" — this is the one that builds trust. An agent that admits ignorance is worth far more than one that confabulates. Grounding without this instruction still leaves the model free to fill gaps with invention.

The MCP shortcut

Everything above is the manual version, and it's worth understanding because it shows what's actually happening. But if your knowledge base already speaks MCP — as every Kognita knowledge base does — you can skip the tool-definition and glue code entirely. Point an MCP-compatible client at the server and the search_knowledge_base tool appears automatically, wired and ready. The loop above is exactly what the client runs for you.

Either way, the principle is the same: give the model a way to look things up, and make looking up the first thing it does. That single instinct — retrieve before you answer — is most of the distance between an AI that embarrasses you and one you'd put in front of a customer.