Prism/plan.md at main · quickSilverShanks/Prism

Plan

This file has a scratchpad meant to contain the working plans and thoughts, milestones to track the development of this project and a section on developer notes that mentions frequently used code snippets, design choices and their rationale for future reference.

The original work is in AWS Bedrock with milvus db, s3 bucket, lambda functions being used. This repo aims to present an open-source alternative of the commercial product with local llm models, postgresql, milvus and a forward looking architecture as compared the original project it takes inspiration from.

Scratchpad

Input pdf file should be ingested through a script and saved as a structured table with chunked data in postgres and as a vectordb in milvus. This will use docling and other libraries installed, milvus db with local server running, postgres schema created.

Docker compose has below codes that need prepared minimally to begin with:

backend.main:app
frontend/app.py

run required models to begin with as below

docker exec -it prism-ollama ollama pull qwen3:8b

Updates to be done later:

At initial stage, create json outputs instead of directly writing to postgresdb. Once chunking and vectorizing works fine, start adding postgres elements for every relevant output. Until then, schema and db initialization codes reamin unused.
For now, use smaller pdf files and insert entire chunked data into milvus collection at once. Add batch mode later.
Also, all the classes/methods need refinement as in the parameters that should be there for init and the ones that should be moved to methods.
The initialization parameter self.auto_id will be removed from vectorize_milvusdb.py later when chunk ids will be obtained from postgres db directly.
Add dozzle in the container to view logs for now, later replace it with logstash + something appropriate to capture agentic logs separately.
The vectorize router of fastapi does chunking before indexing in milvus. After chunks are available in postgres, this won't be required.
If there are more than 5 levels of headings, all headings post heading 4 should be appended separated with '|' to create the new heading_5.
Add a script to factory reset the entire knowledge base and aplication data.

Streamlit UI Updates planned for later:

Improve the visual appearance a bit by introducing a reusable sidebar component. It would include:

Prism logo
Application version
Backend health indicator (🟢/🔴)
Connected services status
GitHub repository link
Documentation link

Add stopword removal in stats_utils.py and word cloud plots later. For stopwords, below can be used.

from wordcloud import STOPWORDS, WordCloud
PRISM_STOPWORDS = STOPWORDS.union(
  {"figure", "table", "tables", "figures", "section", "sections", "page", "pages", "chapter", "chapters", "appendix", "document", "paper", "file", "files"}
)
wordcloud = WordCloud(stopwords=PRISM_STOPWORDS).generate(text)

Its also possible to vastly optimize code stats_utils.py and get most of the headings level stats by just importing and summarizing the chunks json.

The 2nd tab, data exploration, in document intelligence page has constants to declare how much of ingested project will be shown as a preview/sample. Later, this should be moved to projects_utils so that entire project data is never read to begin with.

The 3rd tab, llm chat, appends the entire chat and does not try to save chat memory anywhere. The memory element must be added later. Proper memory management and logging/observability is required here.

The 4th tab would have option to choose retrieval strategy initially. This is just for experimentation and would be removed later. It would be fixed in env file based on evaluation results and user should not be able to modify it mid conversation.

The prompt file will later have all prompt structures stored like: Query Rewriting, HyDE, Summarization, Context Compression, Router, Agent Prompts etc.

For now, semantic_retriever.py leverages the search method available in vectorizer class. Later its own search function needs to be implemented since this current search method is suppsoed only to serve as a validation for vectorization success.

Modify RAGChatRequest later to ensure frontend can send minimal required parameters to api endpoint and still get the response rather than redeclaring and passing all parameters. Modified model should be somehting like below:

class RAGChatRequest(BaseModel):
    project_name: str
    messages: list[ChatMessage]
    retrieval: RetrievalSettings = Field(default_factory=RetrievalSettings)
    generation: GenerationSettings = Field(default_factory=GenerationSettings)

Milestones

Milestones (Future Updates)

Developer Notes

When user uploads the document:

load_document() → chunk_document() → save_chunks_to_postgres() → generate_embeddings() → store_in_milvus() → save_metadata_to_postgres()

uploaded document: data/uploads/project.pdf
ingestion/document_loader/pdf2md.py (output stored in data/processed)
[optional, planned for later use] ingestion/document_loader/pdf2md_multimodal.py (output stored in data/processed)
chunking/document_chunker.py
embeddings/generate_embeddings.py
vectorstore/milvus_client.py (will have the milvus specific helper codes)

Sample usage for later

# ingestion/document_loader/pdf2md.py
from backend.ingestion.document_loader.pdf_parser import PDFParser

parser = PDFParser(
    filename="model_validation",
    input_path="data/uploads/",
    output_root="data/processed"
)

docs = parser.parse()


# chunking/document_chunker.py
from backend.chunking.document_chunker import DocumentChunker

chunker = DocumentChunker(
    project_name="model_validation",
    data_dir="data/processed"    
)

final_split_docs = chunker.chunk()


# vectorstore/vectorize_milvusdb.py
from backend.vectorstore.vectorize_milvusdb import Vectorizer

vectorizer = Vectorizer(
    data_dir="data/processed",
    project_name="model_validation",
    db_name="prism",
    replace_collection=False,
    repalce_db=False
)

vectorizer.vectorize(split_docs=final_split_docs)

-- since metadata is JSONB, below would be possible
SELECT *
FROM document_chunks
WHERE metadata ->> 'page' = '15';

# executing db initialization from within docker
docker exec -it prism-postgres \
psql -U prism -d postgres -f /path/to/init.sql

How to use FastAPI docs @`http://localhost:8000/docs`

Open the page and it will show available APIs that can be used in below order after either uploading a pdf through streamlit ui or manually putting the pdf in folder data/uploads. For below example, lets say the file we upload is Arxiv_Attention_is_all_you_need.pdf

01> Ingestion (POST ingestion/pdf2md): Set filename to Arxiv_Attention_is_all_you_need and leave default values for others. Execute. If its working fine, it will have output similar to below and the ingested figures/amrkdown/tables/metadata can be found in data/processed sub-directories.

{
  "status": "success",
  "message": "PDF parsed successfully.",
  "paragraphs_imported ": 36471
}

02> Chunking (POST chunking/document): Set filename to Arxiv_Attention_is_all_you_need and leave default values for others. Execute. If its working fine, it will have output similar to below and the chunked data can be found in data/processed/chunks.

{
  "status": "success",
  "chunks_created": 112
}

03> Vector Store (POST vectorstore/vectorize): Set project_name to Arxiv_Attention_is_all_you_need, set export to false(if you have already run chunking api with export set as true) and leave default values for others. Execute. If its working fine, it will have output similar to below. The Attu UI @ localhost:3000 can be accessed to view the new indexed collection created with the name Arxiv_Attention_is_all_you_need.

{
  "status": "success",
  "message": "Documents indexed in Milvus."
}

04> Retrieval (GET retrieval/search): Set project_name to Arxiv_Attention_is_all_you_need, set query to Do we use softmax in attention architecture?, set top_k to 2 and leave default values for others. Execute. If its working fine, it will have output similar to below.

{
  "status": "success",
  "retrieved_docs": [
    [
      {
        "chunk_id": 467421221080008200,
        "distance": 0.5743760466575623,
        "entity": {
          "header_2": "3.2 Attention",
          "header_3": "3.2.1 Scaled Dot-Product Attention",
          "header_4": "",
          "header_5": "",
          "content": "We call our particular attention \"Scaled Dot-Product Attention\" (Figure 2). The input consists of queries and keys of dimension d k , and values of dimension d v . We compute the dot products of the query with all keys, divide each by √ d k , and apply a softmax function to obtain the weights on the values.",
          "source": "Arxiv_Attention_is_all_you_need",
          "header_1": "3 Model Architecture",
          "chunk_id": 467421221080008200
        }
      },
      {
        "chunk_id": 467421221080008260,
        "distance": 0.6365917921066284,
        "entity": {
          "header_2": "",
          "header_3": "",
          "header_4": "",
          "header_5": "",
          "content": "In this work, we presented the Transformer, the first sequence transduction model based entirely on attention, replacing the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention.",
          "source": "Arxiv_Attention_is_all_you_need",
          "header_1": "7 Conclusion",
          "chunk_id": 467421221080008260
        }
      }
    ]
  ]
}

05> Chat (POST /chat): Set request Body to below json and hit execute. The output response response should be available to see in a while.

{
  "model": "qwen3:1.7b",
  "system_prompt": "You are a helpful assistant.",
  "messages": [
    {
      "role": "user",
      "content": "Describe Agentic RAG in less than three sentences."
    }
  ],
  "temperature": 0.2,
  "top_p": 0.95,
  "max_tokens": 1000
}

Generated sample response:

{
  "response": "Agentic RAG integrates Retrieval-Augmented Generation with agent-based systems to enhance generative models by dynamically retrieving and integrating knowledge from diverse sources. It enables agents to adaptively retrieve and synthesize information, making it suitable for complex, evolving tasks requiring real-time knowledge updates."
}

06> Chat (POST /chat/rag): Set request Body to below json and hit execute. The output response response should be available to see in a while.

{
  "project_name": "Arxiv_Attention_is_all_you_need",
  "messages": [
    {
      "role": "user",
      "content": "What is attention architecture in transformers?"
    }
  ],
  "retrieval": {
    "strategy": "semantic",
    "top_k": 5,
    "query_preprocessing": {
      "multi_query": false,
      "self_query": false,
      "hyde": false
    },
    "post_processing": {
      "rerank": false,
      "compression": false
    }
  },
  "generation": {
    "model": "qwen3:1.7b",
    "temperature": 0.2,
    "top_p": 0.95,
    "max_tokens": 2048
  }
}

Generated sample response:

{
  "response": "Attention architecture in transformers refers to the use of **multi-headed self-attention mechanisms** to enable the model to focus on relevant parts of the input sequence. This architecture replaces recurrent layers (common in traditional encoder-decoder models) with attention mechanisms, allowing the model to dynamically attend to different positions in the input sequence. Key features include:  \n1. **Multi-headed self-attention**: Multiple attention heads process different parts of the input, enabling parallelism and flexibility.  \n2. **Encoder-decoder attention**: Queries from the decoder attend to the encoder's hidden states, mimicking sequential modeling in sequence-to-sequence tasks.  \n3. **Parallelization**: Attention mechanisms allow simultaneous processing of the entire input sequence, enabling efficient training and inference.  \n\nThis approach enables the Transformer to capture global dependencies between input and output while maintaining parallelism, leading to state-of-the-art performance in tasks like translation.",
  "sources": [
    {
      "source": "Arxiv_Attention_is_all_you_need",
      "chunk_id": 467669583792215940,
      "score": 0.43600785732269287,
      "content": "In this work, we presented the Transformer, the first sequence transduction model based entirely on attention, replacing the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention."
    },
    {
      "source": "Arxiv_Attention_is_all_you_need",
      "chunk_id": 467669583792215940,
      "score": 0.47291189432144165,
      "content": "We are excited about the future of attention-based models and plan to apply them to other tasks. We plan to extend the Transformer to problems involving input and output modalities other than text and to investigate local, restricted attention mechanisms to efficiently handle large inputs and outputs such as images, audio and video. Making generation less sequential is another research goals of ours."
    },
    {
      "source": "Arxiv_Attention_is_all_you_need",
      "chunk_id": 467669583792215900,
      "score": 0.5032509565353394,
      "content": "The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely"
    },
    {
      "source": "Arxiv_Attention_is_all_you_need",
      "chunk_id": 467669583792215940,
      "score": 0.5177544355392456,
      "content": "The Transformer uses multi-head attention in three different ways:  \n[FOOTNOTE]: - In \"encoder-decoder attention\" layers, the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder. This allows every position in the decoder to attend over all positions in the input sequence. This mimics the typical encoder-decoder attention mechanisms in sequence-to-sequence models such as [38, 2, 9]."
    },
    {
      "source": "Arxiv_Attention_is_all_you_need",
      "chunk_id": 467669583792215900,
      "score": 0.5438987612724304,
      "content": "In this work we propose the Transformer, a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output. The Transformer allows for significantly more parallelization and can reach a new state of the art in translation quality after being trained for as little as twelve hours on eight P100 GPUs."
    }
  ]
}

Live development environment

The Dockerfile has a line COPY . . which copies local project codes to docker image and this alone would not enable us to do live modifications without rebuilding the images/containers. But due to docker compose later mounting the local folder we can actually do this live. But we can't remove copy command from dockerfile because its required for application to work later when we don't have our own system to mount the project folder(maybe git pull cna be coupled in the final release to make this simpler). Just a sidenote for future considerations.

Streamlit, too, will be able to automatically detect code changes. But the FastAPI server won't be able to reload code changes unless we enable it. Adding reload in docker compose allows us to do this.

command: >
  uvicorn backend.main:app
  --host 0.0.0.0
  --port 8000
  --reload

Docker Compose GPU Issue

`docker compose build` command returns error: `Error response from daemon: could not select device driver "nvidia" with capabilities: [[gpu]]`

Solution: First check if wsl has access to GPU and nvidia driver is installed:

nvidia-smi

If nvidia driver is installed within wsl, check for runtime visibility

docker info | grep -i runtime

Its expected to see Runtimes: nvidia runc. If only Runtimes: runc is visible, toolkit is not configured/installed. Use below command to see if toolkit is installed. If it gives no output then toolkit is not installed.

dpkg -l | grep -i nvidia-container

If it gives these in the output then toolkit is installed: nvidia-container-toolkit, libnvidia-container1, libnvidia-container-tools

Below command can also be used. If it gives a version number then toolkit is installed. Otherwise, needs to be installed:

nvidia-ctk --version

If toolkit is not installed, follow below steps to get it installed and configured to be used in docker:

# STEP 1 — Remove broken NVIDIA repo file (if it was created wrong, like in my case it was created as an html file)
sudo rm /etc/apt/sources.list.d/nvidia-container-toolkit.list

# STEP 2 — Verify your Linux distribution string (used for repo compatibility checks)
. /etc/os-release
echo $ID$VERSION_ID
# Expected output example: ubuntu22.04

# STEP 3 — Add NVIDIA GPG key (modern keyring method, replaces deprecated apt-key)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
| sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
# This stores the trusted key used to verify NVIDIA packages

# STEP 4 — Add NVIDIA container toolkit repository (stable universal repo)
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
| sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
# This registers the package source with signed-by keyring validation

# STEP 5 — Update apt package index (should complete without <!doctype> errors)
sudo apt update
# Expected: NVIDIA repo fetched successfully, no source list errors

# STEP 6 — Install NVIDIA Container Toolkit (enables Docker GPU runtime)
sudo apt install -y nvidia-container-toolkit
# Installs nvidia-container-runtime and supporting libraries

# STEP 7 — Configure Docker to use NVIDIA runtime + restart service
sudo nvidia-ctk runtime configure --runtime=docker
sudo service docker restart
# Registers NVIDIA runtime inside Docker daemon

# HOW TO CONFIRM FIX — Verify repo file contains valid deb entries (not HTML)
cat /etc/apt/sources.list.d/nvidia-container-toolkit.list
# Expected: Lines starting with "deb [signed-by=...] https://nvidia.github.io/..."

# HOW TO CONFIRM FIX — Verify Docker detects NVIDIA runtime
docker info | grep -i runtime
# Expected output should include: "nvidia" alongside "runc"

# HOW TO CONFIRM FIX — Test GPU access inside a container (did not work for me as i did not have cuda toolkit installed)
docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi
# Expected: GPU table output showing driver, CUDA, and device utilization

Postgres volume mount issue

error: chmod: changing permissions of '/var/lib/postgresql/data': Operation not permitted

Above(or similar) error happens if I mount local folder from windows ntfs filesystem while using wsl2 to build container and run postgres service. Solution is to do below replacement, and this won't be required if someone uses unix native filesystem, either host or wsl itself:

# currently it is as below
postgres:
  ...
  volumes:
    - ./data/postgres:/var/lib/postgresql/data

# replace with below
postgres:
  ...
  volumes:
    - postgres_data:/var/lib/postgresql/data
# and add in the bottom
volumes:
  postgres_data:

FilesExpand file tree

plan.md

Latest commit

History