Add Document Ingestion module for curated docs and web search

.gitignore

@@ -7,7 +7,8 @@
 .env
 .env.example
 JuliaHealthLLM_exp_raw_combined.txt
+cacert.pem
 FunSQLQueries/
 benchmark_result_*
 nul
-synthea_1M_3YR.duckdb
+synthea_1M_3YR.duckdb

Project.toml

@@ -4,6 +4,7 @@ version = "0.1.0"
 authors = ["ParamThakkar123 <paramthakkar864@gmail.com> and TheCedarPrince <jacobszelko@gmail.com>"]
 
 [deps]
+HTTP = "cd3eb016-35fb-5094-929b-558a96fad6f3"
 HuggingFaceHub = "d0076355-e2c0-48e6-a044-05906e51b7fc"
 JSON3 = "0f8b85d8-7281-11e9-16c2-39a750bddbf1"
 LibPQ = "194296ae-ab2e-5f79-8cd4-7183a0a5a0d1"
@@ -13,10 +14,10 @@ RAGTools = "16ddad29-bbe8-45a7-857d-3d9514eb0023"
 Serialization = "9e88b42a-f829-5b0c-bbe9-9e923198166b"
 SparseArrays = "2f01184e-e22b-5df5-ae63-d93ebab69eaf"
 Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
+URIs = "5c2747f8-b7ea-4ff2-ba2e-563bfd36b1d4"
 
 [compat]
-Aqua = "0.4"
-HTTP = "1.11.0"
+HTTP = "1.11"
 HuggingFaceHub = "0.1.2"
 JSON3 = "1.14.3"
 LibPQ = "1.18.0"
@@ -26,17 +27,5 @@ RAGTools = "0.7.0"
 Serialization = "1.10"
 SparseArrays = "1.10"
 Statistics = "1.10"
-Test = "1.10"
+URIs = "1.6"
 julia = "1.10"
-
-[extras]
-Aqua = "4c88cf16-eb10-579e-8560-4a9242c79595"
-DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
-DrWatson = "634d3b9d-ee7a-5ddf-bec9-22491ea816e1"
-DuckDB = "d2f5444f-75bc-4fdf-ac35-56f514c445e1"
-FunSQL = "cf6cc811-59f4-4a10-b258-a8547a8f6407"
-HTTP = "cd3eb016-35fb-5094-929b-558a96fad6f3"
-Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
-
-[targets]
-test = ["Aqua", "DataFrames", "DrWatson", "DuckDB", "FunSQL", "HTTP", "Test"]

README.md

@@ -1,11 +1,37 @@
-# HealthLLM
+# HealthLLM.jl
 
 [![Stable](https://img.shields.io/badge/docs-stable-blue.svg)](https://ParamThakkar123.github.io/HealthLLM.jl/stable/)
 [![Dev](https://img.shields.io/badge/docs-dev-blue.svg)](https://ParamThakkar123.github.io/HealthLLM.jl/dev/)
 [![Build Status](https://github.com/ParamThakkar123/HealthLLM.jl/actions/workflows/CI.yml/badge.svg?branch=master)](https://github.com/ParamThakkar123/HealthLLM.jl/actions/workflows/CI.yml?query=branch%3Amaster)
 [![Coverage](https://codecov.io/gh/ParamThakkar123/HealthLLM.jl/branch/master/graph/badge.svg)](https://codecov.io/gh/ParamThakkar123/HealthLLM.jl)
 [![Code Style: Blue](https://img.shields.io/badge/code%20style-blue-4495d1.svg)](https://github.com/invenia/BlueStyle)
 
-## Citing
+`HealthLLM.jl` is a Julia package for retrieval-augmented workflows over health-oriented corpora and structured clinical data. It focuses on a small set of building blocks for preparing corpora, building RAG indexes, generating query-oriented answers, and storing embeddings in PostgreSQL with `pgvector`.
 
-See [`CITATION.bib`](CITATION.bib) for the relevant reference(s).
+## What the project contains
+
+- Corpus preparation utilities for collecting files and writing combined sources.
+- RAG pipeline helpers built around `RAGTools` and `PromptingTools`.
+- Query-generation helpers for retrieval-backed, FunSQL-oriented workflows.
+- PostgreSQL embedding storage utilities targeting `pgvector`.
+
+Detailed setup, usage, and testing instructions live in the hosted docs:
+https://paramthakkar123.github.io/HealthLLM.jl/dev/
+
+## Repository layout
+
+- `src/`: package source code.
+- `docs/`: Documenter site and docs-specific environment.
+- `test/`: test suite and test-specific environment.
+- `FunSQLQueries/`: query-related assets used by the project.
+- `.env.example`: example environment variable template.
+- `CITATION.bib`: citation metadata.
+
+## Documentation
+
+- API and usage guide: `docs/`
+- Hosted docs: https://paramthakkar123.github.io/HealthLLM.jl/dev/
+
+## Citation
+
+See [`CITATION.bib`](CITATION.bib) for citation details.

docs/make.jl

@@ -14,6 +14,8 @@ makedocs(;
     ),
     pages=[
         "Home" => "index.md",
+        "Getting Started" => "getting-started.md",
+        "Document Ingestion" => "ingestion.md",
     ],
 )
 

docs/src/getting-started.md

@@ -0,0 +1,108 @@
+# Getting Started
+
+## Installation
+
+```julia
+using Pkg
+Pkg.add(url="https://github.com/ParamThakkar123/HealthLLM.jl")
+```
+
+## Quick start
+
+```julia
+using HealthLLM
+
+files = collect_files_with_extensions("data", [".md", ".jl"])
+corpus = write_combined_file(files, "combined.txt")
+
+register_models("llama3.2", "nomic-embed-text")
+index = build_index_rag(RAGTools.SimpleIndexer(), files)
+
+answer = generate_funsql_query(
+    index,
+    "nomic-embed-text",
+    "llama3.2",
+    "Context: {input_query}. Answer concisely.",
+    "What is the FunSQL query to query for patients with Hypertension?"
+)
+```
+
+## Full workflow
+
+### 1. Collect source files and build a corpus
+
+Use `collect_files_with_extensions` to recursively gather files by extension, then concatenate them with `write_combined_file`:
+
+```julia
+using HealthLLM
+
+files = collect_files_with_extensions("data", [".jl", ".md"])
+corpus = write_combined_file(files, "corpus.txt")
+```
+
+### 2. Register models
+
+Register chat and embedding models for use with `PromptingTools`. Supports Ollama, HuggingFace, and other backends:
+
+```julia
+register_models("llama3.2", "nomic-embed-text")
+register_models("hf:facebook/opt-350m", "hf:sentence-transformers/all-mpnet-base-v2")
+```
+
+### 3. Build a RAG index
+
+```julia
+index = build_index_rag(RAGTools.SimpleIndexer(), files)
+```
+
+### 4. Validate embeddings
+
+```julia
+embedding_dim = 384
+num_chunks = length(files)
+embeddings = rand(embedding_dim, num_chunks)
+
+validate_embeddings_inputs(embeddings, files, embedding_dim)
+```
+
+### 5. Store embeddings in PostgreSQL
+
+```julia
+using LibPQ
+
+conn = LibPQ.Connection("postgresql://user:pass@localhost:5432/healthdb")
+store_embeddings_pgvector(conn, embeddings, files, embedding_dim)
+```
+
+### 6. Query the RAG system
+
+```julia
+answer = generate_funsql_query(
+    index,
+    "nomic-embed-text",
+    "llama3.2",
+    "Context: {input_query}. Answer concisely.",
+    "What is the recommended treatment for hypertension?"
+)
+```
+
+## Testing
+
+Default test runs execute the offline unit tests:
+
+```julia
+using Pkg
+Pkg.test()
+```
+
+Integration-style tests that depend on network or large datasets are opt-in:
+
+```bash
+HEALTHLLM_RUN_INTEGRATION_TESTS=true julia --project=. -e 'using Pkg; Pkg.test()'
+```
+
+## Environment notes
+
+- Keep secrets out of git-tracked files.
+- Use `.env.example` as a local template.
+- Configure provider keys in your environment before running model-backed workflows.

docs/src/index.md

@@ -4,11 +4,32 @@ CurrentModule = HealthLLM
 
 # HealthLLM
 
-Documentation for [HealthLLM](https://github.com/ParamThakkar123/HealthLLM.jl).
+HealthLLM provides a compact Julia interface for retrieval-augmented workflows over health-related text and structured datasets.
 
-```@index
+## Package scope
+
+The package centers on five areas:
+
+- collecting source files and writing combined corpora
+- ingesting curated docs and web-search results into an index (see [Document Ingestion](ingestion.md))
+- building retrieval indexes through `RAGTools`
+- generating retrieval-backed answers for query construction
+- storing embeddings in PostgreSQL with `pgvector`
+
+## Package surface
+
+After `using HealthLLM`, the package API and its RAG dependencies are available from one entrypoint:
+
+```julia
+using HealthLLM
+
+register_models("llama3.2", "nomic-embed-text")
+files = collect_files_with_extensions("data", [".md", ".jl"])
+index = build_index_rag(RAGTools.SimpleIndexer(), files)
 ```
 
+More detailed setup, testing commands, and the end-to-end walkthrough are in [Getting Started](getting-started.md).
+
 ```@autodocs
-Modules = [HealthLLM, HealthLLM.Utils]
+Modules = [HealthLLM, HealthLLM.Utils, HealthLLM.Database, HealthLLM.Query, HealthLLM.Ingestion]
 ```