BBC Ingester

End-to-end pipeline to ingest BBC-style JSON articles, deduplicate and chunk text, generate embeddings, persist to Parquet and LanceDB (local or S3-backed), and query via a small FastAPI service.

Key Components

• combine_s3_json.py: Lists and combines multiple JSON files from an S3 prefix into one JSON (writes locally and can upload back to S3).
• embedder.py: Loads/combines JSON, normalizes/deduplicates, chunks text, computes embeddings, writes Parquet, uploads to S3, and writes to LanceDB.
• upload_parquet_to_lancedb.py: Reads a Parquet file from S3 and writes/appends it to a LanceDB table (local path or S3-backed DB). Falls back to local ./lancedb if no cloud URI is set.
• lance_query.py: Minimal FastAPI app to query the LanceDB table with vector-only or hybrid search.
• crawler.py, helpers: Project-specific ingestion utilities.

Prerequisites

• Python 3.9+ in a virtual environment.
• AWS credentials with s3:GetObject, s3:ListBucket, and (for uploads) s3:PutObject on your bucket.
• LanceDB storage:
  • Local filesystem directory (no creds), or
  • LanceDB Cloud (db:// URI + API key), or
  • S3-backed LanceDB (uses your AWS creds).
• Hugging Face (only if you use gated models). Open models are recommended.

Environment (.env) Create .env in the repo root (dotenv format: KEY=VALUE, no export):

• AWS

  • AWS_REGION=eu-west-2
  • S3_BUCKET=ansleysawsbucket
  • S3_PREFIX=bbc/news/
  • S3_OUTPUT_KEY=bbc/news/combined.json (optional; auto-derived if omitted)

• LanceDB (pick one)

  • Cloud: LANCEDB_URI=db://your-db-id, LANCEDB_API_KEY=sk_..., LANCEDB_REGION=us-east-1
  • Local: no env needed; scripts fall back to ./lancedb
  • S3-backed: pass --lancedb-uri s3://your-bucket/lancedb/mydb to writer script

• Embedding/runtime (optional)

  • EMBED_MODEL=sentence-transformers/all-MiniLM-L6-v2
  • EMBED_DEVICE=cpu (or mps on Apple once tested)
  • EMBED_BATCH=64
  • MAX_CHUNKS=0 (0 = all)
  • MIN_CHARS=20

Install

• macOS-friendly base (avoids xformers/CUDA wheels):
  • pip install --no-cache-dir "torch==2.3.1" "transformers<4.46" "sentence-transformers==2.7.0"
  • pip install --no-cache-dir boto3 python-dotenv lancedb pyarrow pandas tqdm ftfy simhash tiktoken fastapi uvicorn
• Or use the provided requirements.txt if you’ve tailored it for your environment.

Typical Workflow

1. Combine JSON files from S3

   • python combine_s3_json.py --bucket ansleysawsbucket --prefix bbc/news/ --output combined.json --output-s3-key bbc/news/combined.json

2. Generate embeddings and write outputs

   • python embedder.py
   • Outputs:
     • Local Parquet: embeddings/embeddings-<timestamp>.parquet
     • S3 (if S3_BUCKET set): s3://<bucket>/<prefix>/embeddings/run=<timestamp>/embeddings.parquet
     • LanceDB table: creates/overwrites embeddings_table at configured DB/URI

3. Import Parquet from S3 into LanceDB (optional)

   • Local LanceDB:
     • python upload_parquet_to_lancedb.py --bucket ansleysawsbucket --key bbc/news/embeddings/run=<ts>/embeddings.parquet --table embeddings_table --lancedb-uri ./lancedb
   • S3-backed LanceDB:
     • python upload_parquet_to_lancedb.py --bucket ansleysawsbucket --key bbc/news/embeddings/run=<ts>/embeddings.parquet --table embeddings_table --lancedb-uri s3://ansleysawsbucket/lancedb/mydb

4. Start the query API (FastAPI)

   • uvicorn lance_query:app --reload
   • Open docs: http://127.0.0.1:8000/docs
   • Example body for POST /query: {"query":"News about Anglian Water", "limit":5, "hybrid": false}

Notes on Embeddings

• Use open, CPU/MPS-friendly models to avoid xformers requirements:
  • Recommended default: sentence-transformers/all-MiniLM-L6-v2 (384-d)
• embedder.py:
  • Deduplicates (SimHash), chunks text, encodes with encode(..., convert_to_numpy=True, normalize_embeddings=True)
  • Stores vectors as float32 lists; guards against NaNs/zeros and drops zero vectors
  • Writes LanceDB and attempts index creation; gracefully skips if API signature differs

FastAPI Query Service

• lance_query.py provides:
  • GET / health check
  • POST /query with JSON body {query, limit, hybrid}
  • GET /table-info to inspect columns and a small sample
• Always use the same embedding model as in embedder.py and specify vector_column="embedding" when searching.

S3 + LanceDB Paths

• S3 examples:
  • Combined: s3://<bucket>/<prefix>/combined.json
  • Embeddings: s3://<bucket>/<prefix>/embeddings/run=<timestamp>/embeddings.parquet
• LanceDB (S3-backed):
  • DB URI: s3://<bucket>/lancedb/mydb
  • Table directory typically ends with .lance, e.g., .../embeddings_table.lance

Troubleshooting

• xformers build error on macOS:
  • Cause: Some models require xformers; Apple clang lacks OpenMP by default
  • Fix: Use all-MiniLM-L6-v2 (no xformers), or install LLVM+OpenMP and build (not recommended)
• urllib3 NotOpenSSLWarning:
  • Harmless warning about LibreSSL on macOS; for a proper fix, use a Python built against OpenSSL (Homebrew/pyenv)
• Gated HF models:
  • huggingface-cli login and request access, or switch to open models
• Dataset/path not found:
  • Point to the table directory (e.g., .../embeddings_table.lance), not a single internal data file
• Vector column/index issues:
  • Search with vector_column="embedding"; re-create index with the correct metric if needed