#Chunking

The retrieval unit is a chunk, not a whole file. Chunks that are too large dump unrelated sentences into the prompt. Chunks that are too small split a fact across two vectors, so neither looks like the answer.

Starting points (then tune on your corpus):

Character counts are enough for Chinese. If you count tokens, stay well below the embedding model’s limit (often 8k); retrieval chunks should be far smaller.

#Overlap

Neighbors share a tail so an answer that sits on the cut is not lost. Use about 10%–20% of chunk length. Too much overlap duplicates storage; too little drops cross-boundary facts.

def window_split(text: str, size: int = 500, overlap: int = 80) -> list[str]:
    if size <= overlap:
        raise ValueError("size must be greater than overlap")
    chunks: list[str] = []
    i = 0
    n = len(text)
    while i < n:
        chunks.append(text[i : i + size].strip())
        if i + size >= n:
            break
        i += size - overlap
    return [c for c in chunks if c]

This is a sliding window with no notion of meaning. Separators help.

#Split on structure, then on length

For Markdown and policy docs, split on headings first, then window-split long sections:

import re

def split_markdown(md: str, size: int = 500, overlap: int = 80) -> list[dict]:
    parts = re.split(r"(?m)^(#{1,3} .+)$", md)
    sections: list[tuple[str, str]] = []
    title = "(root)"
    buf = []
    for part in parts:
        if re.match(r"^#{1,3} ", part):
            if buf:
                sections.append((title, "".join(buf)))
            title, buf = part.strip(), []
        else:
            buf.append(part)
    if buf:
        sections.append((title, "".join(buf)))

    out: list[dict] = []
    for heading, body in sections:
        text = f"{heading}\n{body}".strip()
        for i, chunk in enumerate(window_split(text, size, overlap)):
            out.append({"heading": heading, "i": i, "text": chunk})
    return out

Keep heading and filename in metadata for filters and citations.

Do not put “clause 3” in one chunk and its sub-items in another non-adjacent chunk. Keep lists, steps, and Q+A pairs together.

#Other strategies (when to use them)

LangChain’s RecursiveCharacterTextSplitter, LlamaIndex node parsers, and Dify General / Parent-child / Q&A are product names for the same ideas. See LangChain RAG, LlamaIndex, Dify Knowledge Base.

#Chinese notes

• Do not split Chinese on spaces (often there are none). Prefer periods, newlines, headings.
• “300 characters” is a short policy paragraph; 300 English characters is often too small.
• Do not bisect proper nouns (product codes, clause numbers).

After splitting, spot-check 20 chunks: half tables, half functions, question in one chunk and answer in another. That review beats tuning ANN parameters.

#Sanity check

chunks = window_split(open("policy.md", encoding="utf-8").read())
print(len(chunks), min(map(len, chunks)), max(map(len, chunks)))
print(chunks[0][:200])

Tens of thousands of chunks: you window-split a novel with huge overlap. Exactly one chunk: you forgot to split (only OK for a tiny FAQ).

#Next steps

• Embeddings — turn chunks into vectors
• Chroma — persist them
• Pipeline