Notes on Context: 2M Tokens and What to Do With Them

Context windows grew like disk drives: every year bigger, every year more useless unless you build the machinery to actually use them. A 2M-token context is not a bigger prompt. It is a library, and libraries need indexes.

The room, not the flood

We keep two numbers: the context window, and the attended context. Most systems advertise the first and silently constrain the second. The engine attends to the full window — but attention is a budget, so we made the budget spendable.

const ctx = bb.context({
  window: 2_000_000,        // the room
  index: true,              // build the card catalog
  focus: ["ledger", "proof"],  // spend attention here
});
 
// retrieval is a query, not a scroll
const hits = ctx.query("QA-0417 load test results");

Practical rules we learned

• Index at ingest, not at query — retrieval latency stays flat at any window size.
• Compress losslessly. A 2M window that requires the model to re-read everything is a 2M-token tax.
• Surface provenance per token. Every fact in a long context should know which document it came from.

A context window is not memory. Memory is what you can return to without re-reading.

The full engineering notes run to 40 pages. The short version: big contexts are only useful when they behave like well-run libraries — indexed, quiet, and instantly navigable.