Modified
An agent is only as good as its memory
A large model with poor retrieval performs worse than a small model with accurate context. The critical link is not which LLM you choose — it is how your agent finds the right information before it starts answering.
RAG: Retrieval-Augmented Generation
is a four-step pipeline. First the documents are split into pieces () which are then converted into vectors by an and stored in a such as or . This is the preparation step — it happens once.
When the agent then receives a query the following happens: the query is converted into a vector → the database returns the nearest vectors ( chunks) → those chunks are passed into the LLM context → the LLM answers based on actual information, not just training data.
Chroma is the vector database in this system. is the R in . The quality of that step determines everything that comes after.
Chunking: how you split your data
The embedding model quality only shows up if the chunk contains the right thing. Chunks that are too large dilute the signal — you pull in three pages to get one paragraph. Chunks that are too small lose context and coherence.
A rule of thumb: 400–800 tokens per chunk with 10–20% overlap. The overlap ensures sentences at chunk boundaries do not disappear. Add metadata (filepath, heading, timestamp) on every chunk — it makes filtering and re-ranking after retrieval possible.
Recursive text splitting
Splits at paragraphs → sentences → words until the chunk size is reached. Preserves semantic coherence. The default choice in LangChain and LlamaIndex.
Semantic chunking
Splits at embedding distance, not character count. New chunk when the sentence "changes topic". Better precision but requires an embedding pass already at indexing time.
Code-specific chunking
Split at function boundaries, not line counts. One function = one chunk. Otherwise you risk retrieving half functions without signature or return type.
Three retrieval levels
Recall measures what share of the relevant documents your retrieval actually returns. An agent with 50% recall misses every other relevant chunk — and hallucinates the rest.
No retrieval
The agent answers purely from training data. Works well for general knowledge — fails consistently for internal, domain-specific or fresh information. Hallucinations are not a model problem, they are a memory problem.
Keyword search
BM25 or full-text search. Finds exact word matches but misses paraphrases. "Agentminne" does not find "agent memory". Works like without synonyms. Fast and simple but precision drops quickly on varied phrasing.
Semantic search
An embedding model converts query and chunks into vectors in the same space. Similar meaning = nearby coordinates. "Agent optimization" finds "improve AI agents" and vice versa — in Swedish, English or German. Multilingual-e5-large gives a 1024-dim representation with built-in support for 100+ languages.
MCP as the optimization layer
An agent talking to a raw ChromaDB without built-in embedding does keyword search. The agent sends a text string, Chroma matches against stored strings. Recall ~50–60%.
An agent talking to an that internally runs and (the screen-watch pattern) gets: the agent sends a natural-language query in any language → the embeds the query server-side → returns semantically relevant chunks → recall ~85–90%.
The agent has no idea about embeddings. It just calls search("agent memory") and gets the right answer. The complexity is encapsulated in the layer.
Without embedding in the MCP
Chroma + raw text query. Quick to set up. Works well if you search on exact terms — API method names, file names, specific IDs. Falls apart when phrasing varies.
With embedding in the MCP
fastembed and multilingual-e5-large built into the MCP process. The agent is only exposed to a natural search service. All embedding and vector handling happens server-side. Screen-watch follows this pattern.
Hybrid retrieval
BM25 (keyword) + semantic search in parallel, results merged with RRF (Reciprocal Rank Fusion). Captures both exact terms and semantic paraphrases. Justified for production systems with high precision requirements.
Context window and top-k
Top-k controls how many chunks are passed into the LLM context. k=3 gives precision but may miss relevant information. k=20 gives coverage but dilutes the context — the LLM starts ignoring information in the middle of a long prompt window (the lost in the middle problem).
A practical strategy: retrieve k=15–20 with semantic search, then run a re-ranker that reorders and keeps the 4–6 most relevant. Re-ranking is cheap (a small classifier model) and gives a noticeable precision boost without increasing LLM token usage.
- Chunk size: 400–800 tokens, 15% overlap
- Top-k retrieval: 15–20 candidates
- Re-rank down to 4–6 chunks
- Metadata filter before retrieval where possible (shrinks the search space)
- Embedding model: if you have Swedish in the stack
- Check recall regularly — the index degrades as data changes
When you pick a local LLM via Ollama you implicitly pick whether you have semantic search or not. Comparison of embedding models and LLM alternatives. Go to Language models →
RAG is a memory tool. How the agent decides when to use it — and how you build the rest of the agent toolchain. Go to Agents →