Kenhuang Academy
  • English

      • Loaders and Nodes

      An LLM never sees “files on disk”—only the text you place in context. LlamaIndex splits ingestion into Load (Document) → Transform (chunk to Node, metadata, embeddings) → Index / Store. Same idea as this site’s RAG tutorial, different class names.

      Document vs Node

      ObjectMeaningTypical source
      DocumentA full source: body + metadata (path, page, origin)Readers / Document(text=...)
      NodeA retrievable chunk; Document subclasses NodeSentenceSplitter, internals of from_documents

      Chunks too small: fragmented hits, missing context. Too large: noise in the prompt, window overflow. Start around chunk_size=512 with modest overlap, then inspect source_nodes on real questions.

      Hand-written document:

      from llama_index.core import Document

doc = Document(
    text="Kenhuang Academy covers LlamaIndex RAG and FunctionAgent.",
    metadata={"filename": "intro.md", "lang": "en"},
)

      SimpleDirectoryReader

      The starter reader. It understands common Markdown, PDF, Word, PPT, and some media. Top-level only unless recursive=True.

      from llama_index.core import SimpleDirectoryReader

documents = SimpleDirectoryReader("data").load_data()

reader = SimpleDirectoryReader(
    input_dir="data",
    recursive=True,
    required_exts=[".md", ".txt", ".pdf"],
)
documents = reader.load_data()

      Or pass input_files=["./data/a.pdf"]. Production often uses a LlamaHub reader (database, Notion, S3) instead of one recursive folder scan.

      from llama_index.readers.database import DatabaseReader

reader = DatabaseReader(
    scheme="postgresql",
    host="localhost",
    port=5432,
    user="postgres",
    password="...",
    dbname="app",
)
documents = reader.load_data(query="SELECT id, body FROM articles")

      Confirm the package name on LlamaHub (e.g. llama-index-readers-database).

      Splitting into Nodes

      VectorStoreIndex.from_documents(documents) chunks using global Settings (or a transformations list). To control size:

      from llama_index.core import Settings, VectorStoreIndex
from llama_index.core.node_parser import SentenceSplitter

splitter = SentenceSplitter(chunk_size=512, chunk_overlap=50)
Settings.text_splitter = splitter

index = VectorStoreIndex.from_documents(
    documents,
    transformations=[splitter],
)

      Or a declarative IngestionPipeline (split, extract titles, embed):

      from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.openai import OpenAIEmbedding

pipeline = IngestionPipeline(
    transformations=[
        SentenceSplitter(chunk_size=512, chunk_overlap=50),
        OpenAIEmbedding(model="text-embedding-3-small"),
    ]
)
nodes = pipeline.run(documents=documents)

      Build Nodes yourself and skip Document splitting:

      from llama_index.core.schema import TextNode
from llama_index.core import VectorStoreIndex

nodes = [
    TextNode(text="chunk one", id_="n1", metadata={"source": "manual"}),
    TextNode(text="chunk two", id_="n2"),
]
index = VectorStoreIndex(nodes)

      Why metadata matters

      After retrieval you often filter by team, date, or file type. Write those fields at ingest time; query-time metadata filters are covered in Retrieval. Automatic extractors such as TitleExtractor cost extra LLM calls—fine for offline batches, not for every user question.

      Practice:

      • Prefer sentence/paragraph splits; do not cut inside words
      • Scanned PDFs need OCR or LlamaParse; otherwise the reader returns garbage
      • The same embedding model must be used at index time and query time

      Next steps

      评论