Packaging & Context Handling

Packaging & Context Handling — a set of techniques for selecting, compressing, arranging, and delivering extracted knowledge fragments into the context of LLMs within Retrieval-Augmented Generation (RAG). The goal is to maximize the utility of a limited token budget, improve accuracy and robustness of responses, and ensure traceable source citation. "Packaging" refers not only to forming a list of fragments but also to their compression, ordering, grouping, and model instructions, including stuff, map-reduce, refine, and tree-of-chunks strategies.

Definition and Motivation

In RAG systems, the quality of the final response is determined not only by retrieval but also by how the selected fragments are incorporated into the prompt. Context limitations and token costs require balancing between completeness and precision: excessive fragments increase the risk of lost-in-the-middle and extend latency, while aggressive filtering/compression may remove key evidence. In classic RAG, sources serve as external "non-parametric memory," providing relevance and citability when packaging allows the LLM to reliably operate with facts and references.

Chunking and Content Extraction

The segmentation policy (chunking) defines chunk size/overlap, normalization, and granularity (document→passage→sentence). Typical approaches:

• Fixed-size rule (by characters/tokens) with overlap to preserve coherence between chunks;
• Semantic chunking (boundaries based on embedding similarity), reducing "meaning breaks."
• Sentence-window retrieval — sentences are initially indexed; during retrieval, relevant sentences are extracted with a window of neighboring sentences before/after to restore local context.
• Passage-level indexing — splitting Wikipedia into ~100-word passages became standard in open QA (DPR), reflecting the benefit of fine granularity for initial stages.
• Normalization and cleaning (removing noise, headers/footers, whitespace unification), tracking sources/pages/offsets at the metadata level for traceability.
• Deduplication of candidate chunks (exact and near-duplicate): shingles + MinHash/LSH to reduce repetition.

Diversification and Selection (MMR and Others)

When forming the context set, high relevance and low redundancy are required. The classic Maximal Marginal Relevance function selects the next fragment considering proximity to the query and maximum similarity with already selected items (penalty for duplicates):

$\mathrm{M}\mathrm{M}\mathrm{R}(d_i)=\arg \underset{d_i\in D\setminus S}{\max }[\lambda \cdot \mathrm{s}\mathrm{i}\mathrm{m}(q,d_i)-(1-\lambda )\cdot \underset{d_j\in S}{\max }\mathrm{s}\mathrm{i}\mathrm{m}(d_i,d_j)],\lambda \in [0,1]$.

Combining signals: hybrid retrieval (BM25 + dense) → fusion (e.g., Reciprocal Rank Fusion, RRF) → reranking with cross-encoder/ColBERT:

• RRF: a simple and effective unsupervised scheme for merging rankings from heterogeneous retrievers.
• Cross-encoder rerankers (BERT/MonoT5/modern commercial APIs) significantly improve top-k precision but add latency.
• Multi-vector retriever ColBERT (late interaction) often serves as an effective reranker/first-level retriever on large corpora.
• Hybrid search (BM25F+vector) is implemented in industrial engines and libraries with configurable weight/fusion (alpha, RRF, etc.).

Context Compression

Reducing context volume without losing facts is critical for cost and latency:

• Extractive compression (extracting key sentences/phrases); abstractive summarization (paraphrasing/compression). The classic perspective — Nenkova & McKeown.
• Query-guided / instruction-guided compression: summarization tailored to query/task (highlighting evidence and removing irrelevant content).
• Prompt/context compression using LLM filtering/token pruning (e.g., LLMLingua/LLMLingua-2) reduces token budget with minimal quality loss but requires careful faithfulness validation.
• Compression is a quality↔cost↔latency tradeoff: aggressive compression increases the risk of missing nuances/premises and degrades fact attribution.

Packaging Strategies (stuff/map-reduce/refine/tree)

Below are four basic schemes for arranging sources in the prompt and typical scenarios for their application (see also the comparison table).

Stuff (direct input)

Concatenate selected fragments (after possible compression) and feed them entirely. Simple and fast, but limited by volume and susceptible to lost-in-the-middle on long inputs.

Map-Reduce

In the map stage, locally answer/summarize each fragment/document, then reduce aggregates (comparison, voting, merging). Scales well with the number of sources, reducing load on a single prompt; risk of losing cross-source connections with naive aggregation.

Refine

Sequential improvement: initial answer from the first fragment, then iterative refine considering the next fragment (adding/correcting). Convenient when source order matters; risk of "sticking" to early errors and accumulating distortions.

Tree-of-chunks

Hierarchical compression/summarization: local summaries by chunks → section-level rollups → final summary. Useful for long documents; requires careful passing of source identifiers between levels for correct attribution.

Comparison of Packaging Strategies

Strategy	Idea	Cost/Latency	Context Loss Risk	When to Apply	Sources
Stuff	All fragments at once in one prompt	Low (up to context limit)	High on long inputs (lost-in-the-middle)	Small volume, simple questions
Map-Reduce	Local answers → aggregation	Medium/high (many calls)	Medium (depends on reduce quality)	Many sources, scalability needed
Refine	Sequential answer improvement	Medium	Order-dependent, error reinforcement risk	When order/answer evolution matters
Tree-of-chunks	Hierarchical summaries	Medium/high	Detail loss at upper levels	Long documents/collections

Source Ordering and Positioning

LLMs use information from the middle of long contexts less effectively; useful facts are better placed at the beginning/end, grouped by topics/sources, and marked with headers and IDs. Reranking with query-aware importance and diversification helps bring key fragments closer to the beginning.

Integration into RAG Pipeline (fusion → rerank → packaging)

A typical multi-stage pipeline: hybrid retrieval (BM25 + dense) → fusion (RRF/weighted mix) → rerank (Cross-Encoder/ColBERT) → packaging (one of the strategies) → generation + citation. Hybrid search and RRF are robust to scoring incompatibility of different retrievers; cross-encoder improves LLM input precision, saving tokens.

Quality Evaluation and Ablations

Evaluation is conducted at retrieval, packaging, and generation levels:

• Retrieval: Recall@k, nDCG@k, MRR — standard IR metrics.
• Faithfulness/groundedness: proportion of statements supported by citations; automated frameworks (RAGAS, TruLens) + manual attribution validation.
• End-to-end QA: EM/F1/ROUGE depending on task/dataset (NQ/HotpotQA, etc.).
• Efficiency: latency p50/p95, token count, $-cost; comparison of packaging strategies and compression levels by quality↔cost.
• Ablations: disabling MMR/deduplication/compression/changing order to measure each component's contribution (as of 2025-09-10, RAG research practice recommends clearly documenting k, λ, chunk sizes, and token limits).

Practical Recommendations and Checklist

• k and diversification: start with k=20–40 candidates from hybrid retrieval; apply MMR with λ≈0.5–0.8; strictly penalize duplicates by URL/ID/text hash.
• Chunking: 200–400 tokens with 10–20% overlap for fixed chunking; for legal-technical documents, sentence/window scheme often helps.
• Compression: use extractive filtering by query and careful abstraction; decrease/increase LLMLingua-like methods depending on faithfulness on your data (A/B validation required).
• Order: important/high-confidence fragments — at the beginning of the prompt; group by sources/topics, explicitly mark IDs and headers; consider the lost-in-the-middle effect (duplicating key facts at the beginning and end may help).
• Rerank: if budget allows, add Cross-Encoder/ColBERT on top-k (k≈50–200) before packaging — this saves generation tokens and improves precision.
• Fallback strategies: (1) insufficient facts → request additional sources; (2) token limit exceeded → switch stuff→refine or enable compression; (3) low confidence/contradictions → refusal response with explicit list of missing IDs (see template below).

Packaging Pipeline Pseudocode

# Input: query q
cands = retrieve(q, K_sparse, K_dense)          # search BM25, DPR etc.
cands = diversify_MMR(cands, lambda=0.7)        # diversification (MMR)
snips = compress(query=q, items=cands, mode="extractive|abstractive", budget=tokens)
pkg   = package(snips, strategy="stuff|map_reduce|refine|tree")
resp  = generate(prompt=build_prompt(q, pkg), citations=True)  # LLM with citations

Prompt Template Skeleton (Fragment)

[USER QUERY]
{q}

[SOURCES]
{# Each fragment with ID, title, and link #}
- [{id}] {title} — {url}
{content_snippet}

[REQUIREMENTS]
1) Use only facts from sources, reference [ID].
2) If data is insufficient, say so and request clarification/additional sources.
3) Maintain response structure and list used [IDs].

Limitations and Open Questions

• Hallucinations and aggregation in map-reduce/refine: abstractive summaries may introduce new facts; clear attribution instructions and citation verification mechanisms are critical.
• Detail loss with aggressive compression/hierarchical summarization; storing back-references to original source/page/offset is important.
• Domain transferability of retrievers/rerankers and compressors; adaptation/fine-tuning on domain corpora is required.
• Privacy/PII and LLM memorization: generation without strict grounding may leak private strings; apply filters, private storage, and refusal policies.
• Trainable "packagers", adaptive ordering/arrangement, RLHF/feedback loops for improving faithfulness, multilingual and ultra-long context — active research directions.

External links

• LangChain: Summarization (stuff/map_reduce/refine). [29]
• LangChain: Text splitters. [30]
• LlamaIndex: Response Synthesizers (refine/tree). [31]
• LlamaIndex: Node Parsers / SentenceSplitter / SemanticSplitter. [32]
• Haystack: SentenceWindowRetriever. [33]
• Haystack: PreProcessors / DocumentSplitter. [34]
• Weaviate: Hybrid search. [35]
• Pinecone: Hybrid search. [36]
• Cohere: Rerank API. [37]
• RAGAS (repo/docs). [38] [39]
• TruLens (docs). [40]
• Context engineering — Wikipedia

Bibliography

• Manning, C. D., Raghavan, P., Schütze, H. (2008). Introduction to Information Retrieval. Cambridge University Press. ISBN 978-0521865715.
• Nenkova, A., McKeown, K. (2011). Automatic Summarization. FnT IR, 5(2–3), 103–233. DOI:10.1561/1500000015.
• Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS. arXiv:2005.11401.
• Khattab, O., Zaharia, M. (2020). ColBERT. SIGIR'20. DOI:10.1145/3397271.3401075.
• Izacard, G., Grave, E. (2021). Fusion-in-Decoder. EACL. arXiv:2007.01282.
• Ji, Z., et al. (2023). Survey of Hallucination in NLG. ACM CS. DOI:10.1145/3571730.
• Gao, S., et al. (2024). RAG for LLM: A Survey. arXiv:2312.10997.