Document Ingestion in Next.js 15, Chunking, Tokens, and Async Embeddings

Uploading a PDF is one click. Turning it into chunks the chat can retrieve is a pipeline: parse, split, embed, insert, then mark the document ready. If you do all of that inside the request that accepted the file, you will hit body size limits, OpenAI rate limits, and a proxy timeout.

AgenticPacks OSS has chat and Postgres. It has no ingest path: Agentic-Packs/agenticpacks-oss. Pro ships rag/chunker.ts, embeddings, cosine search, and /api/rag/ingest. How the column is stored is in Prisma with pgvector. Why that column lives in Postgres instead of Pinecone is in pgvector vs Pinecone.

The problem

A typical first version looks like this:

1. Server Action receives the file.
2. Parse the PDF in the same action.
3. Call the embeddings API for every chunk.
4. Insert rows.
5. Return 200 when it is all done.

A 40 page PDF is thousands of tokens. Embedding it takes longer than Cloudflare, Caddy, or a browser will wait. If the request dies at chunk 80, you have no resume point and you may have already spent the embedding budget.

The upload request should only accept the file and enqueue work.

What the upload action should do

1. Check that the user belongs to the workspace. See Server Actions authorization.
2. Debit an estimate from the team credit pool so a 200 page dump cannot run for free.
3. Store the file. Disk, object storage, or Postgres bytea for small docs.
4. Write a Document row with status = uploaded and return.

A worker then parses and embeds. In Pro that worker is the same Node app plus a Redis queue. AgenticPacks OSS has no worker. If you only have AgenticPacks OSS, keep ingest off the MIT repo and add a job later, or cap files to a few megabytes and accept that a proxy may still cut you off.

Chunk by tokens, not by characters

Embedding models have a token window. Character slicing splits words and citations in the middle. Pro uses about 500 tokens per chunk with a small overlap so a sentence that sits on a boundary still appears in retrieval.

const SIZE = 500;
const OVERLAP = 50;

export function chunkByTokens(pieces: string[]): string[] {
  const chunks: string[] = [];
  for (let i = 0; i < pieces.length; i += SIZE - OVERLAP) {
    chunks.push(pieces.slice(i, i + SIZE).join(""));
  }
  return chunks;
}

Use a tokenizer that matches the embedding model. For OpenAI text-embedding-3-small that means a cl100k-compatible tokenizer, not string.length.

PDFs need a text extract step first. Scanned pages need OCR. If OCR is not the product, reject those files in v1 and say so in the UI.

Batch embed, then resume

const BATCH = 64;

for (let i = startOffset; i < chunks.length; i += BATCH) {
  const slice = chunks.slice(i, i + BATCH);
  const vectors = await embedMany(slice);
  await insertChunks(documentId, slice, vectors);
  await prisma.document.update({
    where: { id: documentId },
    data: { offset: i + slice.length },
  });
}

On HTTP 429, wait and retry that batch. Do not restart the PDF from page one. The offset column is how a crash resumes.

Insert with a ::vector cast. Build an IVFFlat index after the first real load, not on an empty table. Those SQL details are in the pgvector migration post.

Tenant scope and UI status

Every chunk row needs organizationId. Retrieval must include that column in WHERE. A vector index without a tenant filter is a data leak.

Statuses the UI can poll:

Chat RAG only searches ready.

Ingest is not a LangGraph. Retrieval is a tool that chat or a researcher node calls later. See AI SDK tool calling.

Tired of a weekend ingest script that is not in the repo?

AgenticPacks OSS is chat plus Postgres: Agentic-Packs/agenticpacks-oss.

The chunker, embeddings, and ingest route are in the document ingest pipeline in Pro.

---

Where to Go From Here

You do not have to hand-roll a PDF pipeline to ship contextual RAG.

Option A: Free open-source baseline (MIT)

• GitHub: github.com/Agentic-Packs/agenticpacks-oss
• What is included: Next.js 15, Postgres, Prisma, Auth.js, streaming chat, single-plan Stripe. No ingest, no pgvector.
• Cost: free, MIT.

---

Option B: Ship with AgenticPacks Pro ($149 one-time)

If documents should land in the same database as users:

• Automated Hetzner provisioning: Terraform, Caddy, Compose, Redis queue.
• Production multi-agent core: Chunker, pgvector, LangGraph, search tool.
• Usage & Unit Economics: Stripe, team pools, token ledger on embed plus chat.