AI Book Chat

01 / problem

Can you trust an answer about a long book?

A plain chat model answers from general knowledge and cannot show where a claim comes from. For a long PDF you want answers grounded in the text, pages to check them against, and an honest signal when the book does not cover the question.

02 / solution

Retrieval with re-ranking and citations

A worker parses the PDF, detects sections (outline first, then heading heuristics, then flat pages) and cuts chunks of about 600 tokens with 15% overlap that never cross a section boundary. Each question is rewritten into a standalone query from the recent turns, matched by cosine search in pgvector and re-ranked by an LLM before generation. Sources are streamed first, so the UI shows citations while the answer is still being written; when nothing relevant is found, the answer says so up front.

• [screenshot 1]

  [caption]

• [screenshot 2]

  [caption]

03 / architecture

How it works

A React frontend talks HTTP and SSE to a FastAPI API backed by PostgreSQL with pgvector (HNSW, cosine). Uploads are queued in Redis and processed by a Celery worker: PyMuPDF parsing, chunking and batch embedding with local Ollama (nomic-embed-text). Chat, query rewriting and re-ranking use any OpenAI-compatible endpoint, Groq by default. pnpm workspaces and Turborepo drive both apps.

04 / results

Outcome

05 / lessons

What I learned

• Embeddings are deliberately not switchable: the vector column has a fixed width, so another model means a migration and a re-ingest. The app refuses to start when the configured dimensions and the column disagree, instead of returning plausible but meaningless results.
• Follow-up questions like "and what about his brother?" only work because of the rewrite step; retrieval alone sees no context. Still out of scope: cross-document queries, OCR for scanned PDFs and accounts.