Object Search Engine

A production-grade, hybrid (BM25 + kNN) enterprise search pipeline built on OpenSearch, Redis, Kafka, and MinIO — deployable via Docker Compose (local dev) or Minikube / Kubernetes (server).

Pipeline	Directory	What it does
Ingestion	`workers/`, `infrastructure/`, `docker-compose.yml`	File upload → Kafka → Tika extract → chunk → embed → OpenSearch
Search	`Search-Engine/`	REST query → Go Gateway → gRPC PyWorker → OpenSearch hybrid query → Redis cache

Architecture Overview

┌─────────────── INGESTION PIPELINE ───────────────────────────────┐
│                                                                    │
│  MinIO (S3)  ──PUT event──▶  Kafka (3-node)  ──▶  Ingestion      │
│  :9000/:9001                 :29092               Worker          │
│                                                    │              │
│                                       (Jina CLIP v2 local embed)  │
│                                                    │              │
│                              Tika  ◀──────── OpenSearch :9200    │
│                              :9998                  │              │
└──────────────────────────────────────────── Redis :6379 ─────────┘
                                                      │
┌─────────────── SEARCH PIPELINE ──────────────────▼──────────────┐
│                                                                    │
│  Frontend (Next.js)  ──▶  Go Gateway  ──▶  PyWorker-2 (gRPC)    │
│  :3000                     :8080            :50052                │
│                              │                │                    │
│                           Redis            OpenSearch             │
│                         (cache HIT)      (kNN + BM25)            │
│                           :6379             :9200                 │
└───────────────────────────────────────────────────────────────────┘

Prerequisites

Target platforms: WSL 2 (Ubuntu 22.04 on Windows) · RHEL 10 x86_64 · Fedora 43 x86_64

Hardware Requirements

Resource	Docker Compose (min)	Minikube (min)	Recommended
CPU	4 cores (x86_64)	4 cores (x86_64)	6+ cores
RAM	8 GB	12 GB	16 GB
Disk	25 GB free	45 GB free	60 GB free

WSL 2 (Windows 11 / Windows 10 22H2+)

1. Enable WSL 2 + Ubuntu

# Run in PowerShell (Administrator)
wsl --install -d Ubuntu-22.04
wsl --set-default-version 2

Restart Windows, then open the Ubuntu 22.04 app and create your UNIX user.

2. Docker Desktop

Install Docker Desktop for Windows from docs.docker.com/desktop/windows.

In Docker Desktop → Settings → Resources → WSL Integration, enable your Ubuntu distro.

# Verify inside WSL terminal
docker --version          # Docker version 24.x or higher
docker compose version    # Docker Compose version v2.20 or higher

3. OpenSearch kernel setting

WSL does not persist sysctl between reboots — add to ~/.bashrc so it re-applies each session:

sudo sysctl -w vm.max_map_count=262144
echo 'sudo sysctl -w vm.max_map_count=262144 > /dev/null 2>&1' >> ~/.bashrc

4. Minikube + kubectl

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube && rm minikube-linux-amd64

curl -LO "https://dl.k8s.io/release/$(curl -sL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install kubectl /usr/local/bin/kubectl && rm kubectl

minikube config set driver docker

5. Git & AWS CLI

sudo apt-get install -y git awscli

RHEL 10 — x86_64

RHEL 10 ships with DNF5; commands below use dnf5 config-manager syntax.

1. Docker CE

sudo dnf5 config-manager addrepo \
  --from-repofile=https://download.docker.com/linux/rhel/docker-ce.repo

sudo dnf install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo systemctl enable --now docker
sudo usermod -aG docker $USER && newgrp docker

docker --version          # Docker version 26.x or higher
docker compose version    # Docker Compose version v2.27 or higher

2. OpenSearch kernel setting

sudo sysctl -w vm.max_map_count=262144
echo "vm.max_map_count=262144" | sudo tee /etc/sysctl.d/99-opensearch.conf
sudo sysctl --system

3. Minikube + kubectl

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube && rm minikube-linux-amd64

curl -LO "https://dl.k8s.io/release/$(curl -sL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install kubectl /usr/local/bin/kubectl && rm kubectl

minikube config set driver docker

4. Git & AWS CLI

sudo dnf install -y git unzip
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o awscliv2.zip
unzip awscliv2.zip && sudo ./aws/install
rm -rf awscliv2.zip aws/

Fedora 43 — x86_64

Fedora 43 ships with DNF5 by default.

1. Docker CE

sudo dnf5 config-manager addrepo \
  --from-repofile=https://download.docker.com/linux/fedora/docker-ce.repo

sudo dnf install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo systemctl enable --now docker
sudo usermod -aG docker $USER && newgrp docker

docker --version
docker compose version

2. OpenSearch kernel setting

sudo sysctl -w vm.max_map_count=262144
echo "vm.max_map_count=262144" | sudo tee /etc/sysctl.d/99-opensearch.conf
sudo sysctl --system

3. Minikube + kubectl

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube && rm minikube-linux-amd64

curl -LO "https://dl.k8s.io/release/$(curl -sL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install kubectl /usr/local/bin/kubectl && rm kubectl

minikube config set driver docker

4. Git & AWS CLI

sudo dnf install -y git unzip
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o awscliv2.zip
unzip awscliv2.zip && sudo ./aws/install
rm -rf awscliv2.zip aws/

Quick prerequisite check (all platforms)

echo "=== Prerequisite Check ===" && \
  docker --version && \
  docker compose version && \
  (command -v minikube && minikube version || echo "minikube: not installed") && \
  (command -v kubectl  && kubectl version --client 2>/dev/null || echo "kubectl: not installed") && \
  echo "vm.max_map_count=$(sysctl -n vm.max_map_count) (need >= 262144)" && \
  echo "=== All checks done ==="

Repository Structure

HPE/
├── Search-Engine/                  # Search pipeline (Roles 4 & 5)
│   ├── gateway/                    # Go API Gateway
│   │   ├── cache/redis.go          # Redis cache — Steps 1 & 4 of search flow
│   │   ├── handlers/search.go      # REST handler (HIT/MISS logic)
│   │   ├── grpcclient/client.go    # gRPC client → PyWorker-2
│   │   ├── merger/merger.go        # Top-K dedup + score sort
│   │   ├── proto/                  # Generated gRPC stubs (Go)
│   │   ├── main.go
│   │   ├── Dockerfile
│   │   └── go.mod
│   ├── pyworker/                   # PyWorker-2 gRPC server (NLP + embed + search)
│   │   ├── search_worker.py        # gRPC servicer
│   │   ├── nlp_parser.py           # spaCy NLP parser
│   │   ├── embedding_service.py    # SentenceTransformer (all-MiniLM-L6-v2)
│   │   ├── config.py               # All env-var config
│   │   ├── proto/                  # Generated gRPC stubs (Python)
│   │   └── Dockerfile
│   ├── frontend/                   # Next.js HPE-themed search UI
│   ├── docker-compose.yml          # Search-side stack (standalone)
│   └── .env.example
│
├── workers/
│   ├── ingestion/                  # Ingestion worker (Kafka consumer)
│   │   ├── main.py                 # Entry point — Kafka → process → index
│   │   ├── tika_extractor.py       # Apache Tika text & metadata extraction
│   │   ├── chunker.py              # Sliding-window text chunker
│   │   ├── image_handler.py        # Image pipeline (resize → embed)
│   │   ├── model_client.py         # Local Jina CLIP v2 embedding inference
│   │   └── opensearch_client.py    # Bulk upsert into OpenSearch
│
├── backend/
│   ├── cache/redis_cache.py        # Python Redis cache layer (search results)
│   └── search/opensearch_query_builder.py  # Hybrid BM25+kNN query reference
│
├── infrastructure/
│   ├── opensearch/index-mapping.json  # kNN-enabled index schema (hpe-search-docs)
│   ├── kafka/                         # Kafka topic scripts
│   ├── minio/                         # MinIO bucket + event-notification config
│   └── startup.sh                     # One-shot bootstrap script
│
├── k8s/                            # Kubernetes / Minikube manifests
│   ├── 00-namespace.yaml
│   ├── 01-configmap.yaml
│   ├── 02-pvcs.yaml
│   ├── infrastructure/             # Kafka, MinIO, Tika, OpenSearch, Redis
│   ├── ingestion/                  # ingestion-worker
│   ├── search/                     # pyworker, go-gateway, frontend
│   └── deploy.sh                   # One-shot deploy script for Minikube
│
├── tests/
│   └── test_opensearch.py          # OpenSearch + Redis integration tests
│
├── docker-compose.yml              # Unified full-stack compose (14 services)
├── requirements.txt                # Python dependencies (ingestion + backend)
├── pytest.ini
└── .env.example                    # All environment variables documented

Quick Start — Docker Compose

# 1. Clone the repo
git clone <repo-url> && cd HPE

# 2. Apply kernel setting for OpenSearch (Linux only)
sudo sysctl -w vm.max_map_count=262144

# 3. Configure environment
cp .env.example .env
# Edit .env only if you need non-default values (host, credentials, etc.)

# 4. Start all 14 services
docker compose up --build -d

# 5. Follow startup logs (wait ~2 min for all services to be healthy)
docker compose logs -f opensearch ingestion-worker model-server

Open http://localhost:3000 once the stack is healthy.

Quick Start — Minikube

The k8s/deploy.sh script handles everything — Minikube startup, image builds, manifest apply, and init job sequencing.

# 1. Clone the repo
git clone <repo-url> && cd HPE

# 2. Run the deploy script (takes ~5–10 min on first run — downloads models)
chmod +x k8s/deploy.sh
./k8s/deploy.sh

# Full reset (wipes namespace and redeploys from scratch):
./k8s/deploy.sh --reset

After deployment, the script prints access URLs:

Frontend:              http://<minikube-ip>:30300
Go Gateway (API):      http://<minikube-ip>:30080
MinIO Console:         http://<minikube-ip>:30901
OpenSearch Dashboards: http://<minikube-ip>:30601

Get your Minikube IP with:

minikube ip

Common Minikube commands

# Watch all pods
kubectl get pods -n hpe-search -w

# Tail a service's logs
kubectl logs -n hpe-search deploy/model-server -f
kubectl logs -n hpe-search deploy/ingestion-worker -f

# Open a service in the browser
minikube service frontend -n hpe-search
minikube service go-gateway -n hpe-search

# Stop Minikube (preserves data)
minikube stop

# Delete entire cluster
minikube delete

Service Ports

Docker Compose

Service	Port(s)	Description
Frontend	`3000`	Next.js search UI
Go Gateway	`8080`	REST API (`GET /search`, `GET /health`)
PyWorker-2	`50052`	gRPC — NLP parse → embed → OpenSearch
OpenSearch	`9200`	Hybrid BM25 + kNN search database
OpenSearch Dashboards	`5601`	Index inspection UI
Redis	`6379`	Search result cache
Apache Tika	`9998`	Text & metadata extraction
MinIO S3 API	`9000`	Object upload endpoint
MinIO Console	`9001`	MinIO web UI
Kafka broker 1	`29092`	External Kafka listener
Kafka broker 2	`29093`	External Kafka listener
Kafka broker 3	`29094`	External Kafka listener

Minikube (NodePort)

All services are reachable at http://$(minikube ip):<NodePort> — no port-forwarding needed.

Service	NodePort	Description
Frontend	`30300`	Next.js search UI
Go Gateway	`30080`	REST API (`GET /search`, `GET /health`)
MinIO S3 API	`30900`	S3-compatible upload endpoint
MinIO Console	`30901`	MinIO web UI
OpenSearch Dashboards	`30601`	Index inspection UI

Environment Variables

Copy .env.example → .env and adjust as needed.

# OpenSearch — must match the index created by the ingestion pipeline
OPENSEARCH_HOST=localhost
OPENSEARCH_PORT=9200
OPENSEARCH_INDEX=hpe-search-docs

# Search tuning
SEARCH_KNN_K=50
SEARCH_BM25_BOOST=0.4
SEARCH_KNN_BOOST=0.6

# Redis cache (Go Gateway Steps 1 & 4)
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_TTL_DEFAULT=300         # seconds — default query TTL
REDIS_TTL_POPULAR=1800        # seconds — TTL for frequently hit queries
REDIS_POPULAR_THRESHOLD=10    # hit count to qualify as "popular"

# Ingestion tuning
OPENSEARCH_BULK_CHUNK_SIZE=200
OPENSEARCH_MAX_RETRIES=5

# Kafka
KAFKA_BOOTSTRAP_SERVERS=localhost:29092
KAFKA_TOPIC=file-upload-events

# MinIO
MINIO_ENDPOINT=localhost:9000
MINIO_ACCESS_KEY=minioadmin
MINIO_SECRET_KEY=minioadmin123

Note: When running on Minikube, most of these values are set in k8s/01-configmap.yaml and injected into pods as environment variables. The .env file is only needed for Docker Compose deployments.

OpenSearch Index

The pipeline uses an index named hpe-search-docs with kNN + BM25 hybrid mapping.

The index is created automatically on startup by the opensearch-init service. The mapping lives in infrastructure/opensearch/index-mapping.json.

To create it manually:

curl -X PUT http://localhost:9200/hpe-search-docs \
  -H 'Content-Type: application/json' \
  -d @infrastructure/opensearch/index-mapping.json

How the Search Pipeline Works

1. Browser → Frontend (port 3000)
2. Frontend → Go Gateway GET /search?q=<query>   (port 8080)
3. Go Gateway → Redis: cache lookup by SHA-256(query)
   └─ HIT  → return cached JSON immediately  [X-Cache: HIT]
   └─ MISS → continue ↓
4. Go Gateway → PyWorker-2 gRPC ProcessQuery
5. PyWorker-2:
   a. spaCy NLP parse → intent text + keywords + filters
      - Exact calendar dates (e.g. "May 15th") → 24-hour range filter
      - Relative dates (last week / month / year / yesterday / today)
      - File type, extension, and size filters
   b. Jina CLIP v2 local embed query → 512-dim vector
   c. OpenSearch query:
      - kNN semantic search using the embedded vector (k=50, HNSW cosine)
      - BM25 keyword search with fuzziness:AUTO (handles typos like "informaton" → "information")
      - Filter-only query (no keywords) → match_all + filter clauses
   d. Merge & Rank:
      - Blend kNN and BM25 scores (configurable boosts: 0.6 / 0.4)
      - Deduplicate chunks (keep highest scoring chunk per file)
      - Drop irrelevant matches (combined_score < 0.45 threshold)
   e. Abstractive Summarization:
      - google/flan-t5-small generates a human-readable 1–2 sentence summary
      - Loaded once at startup; runs on CPU within the pyworker container
6. PyWorker-2 returns ranked proto results to Go Gateway
7. Go Gateway caches and returns the final results to the UI
8. Go Gateway → Redis: store result with TTL    [X-Cache: MISS]
9. Go Gateway → Frontend → render results

How the Ingestion Pipeline Works

1. Client uploads file to MinIO bucket "uploads"
2. MinIO publishes s3:ObjectCreated event to Kafka topic "file-upload-events"
3. Ingestion Worker (Python) consumes Kafka message:
   a. Download file bytes from MinIO
   b. Apache Tika: extract text + metadata
   c. Route by content type:
      - Image  → Local Jina CLIP v2 inference → 512-dim vector → single chunk doc
      - Text   → TextChunker (sliding window) → Local Jina CLIP v2 inference → N chunk docs
   d. OpenSearch bulk upsert (object_key/chunk_index = document ID, idempotent)

Natural Language Query Examples

Query	What PyWorker extracts
`quarterly report pdf`	keywords: `quarterly report` + `type:pdf` filter
`images bigger than 10MB`	`type:image` + `size_gt:10MB` filter
`contracts from last week`	keywords: `contracts` + `date:last_week` filter
`invoices from May`	keywords: `invoices` + `month:may` filter
`marketing deck .pptx`	keywords: `marketing deck` + `extension:pptx` filter
`files uploaded on May 15th`	`exact_date:may_15_<year>` filter → 24-hour range query
`pdfs`	`type:pdf` filter → `match_all` (no keywords needed)

Uploading Files

Docker Compose

MinIO is exposed on localhost:9000. Configure AWS CLI once:

aws configure set aws_access_key_id     minioadmin
aws configure set aws_secret_access_key minioadmin123
aws configure set default.region        us-east-1

# Upload a file
aws s3 cp ./file.pdf s3://uploads/file.pdf --endpoint-url http://localhost:9000

# Upload a folder
aws s3 cp ./folder/ s3://uploads/ --recursive --endpoint-url http://localhost:9000

# List bucket contents
aws s3 ls s3://uploads/ --endpoint-url http://localhost:9000 --recursive

Minikube

MinIO's S3 API is exposed on NodePort 30900 — no port-forwarding needed. Use the provided script:

# Upload a single file (bucket defaults to "uploads")
./infrastructure/upload.sh ./yourfile.pdf

# Upload an entire folder
./infrastructure/upload.sh ./your-folder/

# Upload to a specific bucket
./infrastructure/upload.sh ./yourfile.pdf my-bucket

Or call AWS CLI directly:

MINIO_URL="http://$(minikube ip):30900"

# Upload a file
aws s3 cp ./file.pdf s3://uploads/file.pdf --endpoint-url $MINIO_URL

# Upload a folder
aws s3 cp ./folder/ s3://uploads/ --recursive --endpoint-url $MINIO_URL

# List bucket contents
aws s3 ls s3://uploads/ --endpoint-url $MINIO_URL --recursive

How it triggers ingestion: Every upload PUT fires a MinIO event → Kafka file-upload-events → Ingestion Worker → Tika extract → chunk → embed → OpenSearch. Files become searchable within seconds.

Debugging

Docker Compose

# OpenSearch health
curl -s http://localhost:9200/_cluster/health | python3 -m json.tool
curl -s http://localhost:9200/hpe-search-docs/_count | python3 -m json.tool

# Redis cache
docker exec hpe-search-redis redis-cli FLUSHDB   # flush stale cache
docker exec hpe-search-redis redis-cli DBSIZE

# Service logs
docker logs hpe-search-ingestion-worker -f
docker logs hpe-search-go-gateway -f
docker logs hpe-search-pyworker-2 -f

# Kafka topics
docker exec hpe-search-kafka-1 /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka1:9092 --list

# MinIO — list files
aws s3 ls s3://uploads/ --endpoint-url http://localhost:9000 --recursive
# Web console: http://localhost:9001  (minioadmin / minioadmin123)

Minikube

MINIO_URL="http://$(minikube ip):30900"

# Pod status
kubectl get pods -n hpe-search -o wide
kubectl get pods -n hpe-search -w          # watch live, catches CrashLoopBackOff

# Why is a pod restarting?
kubectl describe pod <pod-name> -n hpe-search   # events section at bottom is key
kubectl logs <pod-name> -n hpe-search --previous  # logs from BEFORE last crash

# OpenSearch health (via exec, no port-forward needed)
kubectl exec -n hpe-search deploy/opensearch -- curl -s localhost:9200/_cluster/health?pretty
kubectl exec -n hpe-search deploy/opensearch -- curl -s localhost:9200/_cat/shards/hpe-search-docs?v

# Or via NodePort dashboard
curl -s "http://$(minikube ip):30601"

# Redis cache — flush stale results after a deploy
kubectl exec deploy/redis -n hpe-search -- redis-cli FLUSHALL

# Service logs
kubectl logs -n hpe-search deploy/ingestion-worker -f
kubectl logs -n hpe-search deploy/go-gateway -f
kubectl logs -n hpe-search deploy/pyworker -f
kubectl logs -n hpe-search deploy/model-server -f

# Kafka — consumer group lag (diagnoses ingestion stalls/restarts)
kubectl exec kafka-0 -n hpe-search -- /opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 --describe --group <ingestion-consumer-group-id>

# Kafka topics
kubectl exec kafka-0 -n hpe-search -- /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 --list

# Watch Kafka upload events in real-time
kubectl exec kafka-0 -n hpe-search -- /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 --topic file-upload-events --from-beginning

# Resource pressure (common cause of silent kills)
kubectl top pods -n hpe-search
kubectl describe node

# MinIO — list files (direct NodePort, no port-forward)
aws s3 ls s3://uploads/ --endpoint-url $MINIO_URL --recursive
# Web console: http://$(minikube ip):30901  (minioadmin / minioadmin123)

Stopping

Docker Compose

# Stop containers (preserves volumes)
docker compose down

# Stop and remove all data volumes
docker compose down -v

Minikube

# Stop cluster (preserves data)
minikube stop

# Delete cluster and all data
minikube delete

Running Tests

# Install Python dependencies
pip install -r requirements.txt

# Unit tests only (no running services needed)
pytest tests/test_opensearch.py -v -m unit

# Integration tests (requires OpenSearch + Redis running on localhost)
pytest tests/test_opensearch.py -v -m integration

# Full E2E pipeline test
pytest workers/ingestion/test_e2e_full_pipeline.py -v

Known Issues & Fixes

kNN search fails with `Field 'embedding' is not knn_vector type`

OpenSearch auto-creates fields when data is indexed before the init job runs, resulting in embedding being typed as float instead of knn_vector. Fix:

# Drop the incorrectly mapped index
kubectl exec -n hpe-search deploy/opensearch -- curl -s -X DELETE localhost:9200/hpe-search-docs

# Recreate with the correct knn_vector mapping
kubectl exec -n hpe-search deploy/opensearch -- curl -s -X PUT localhost:9200/hpe-search-docs \
  -H "Content-Type: application/json" \
  -d "$(cat infrastructure/opensearch/index-mapping.json)"

# Re-upload files to trigger re-ingestion
MINIO_URL="http://$(minikube ip):30900"
aws s3 cp ./yourfile.pdf s3://uploads/ --endpoint-url $MINIO_URL

Ingestion worker crashes with `UNKNOWN_TOPIC_OR_PART` on fresh deploy

The file-upload-events Kafka topic must exist before the ingestion worker starts. This is handled by:

The kafka-init Kubernetes Job (k8s/infrastructure/kafka-init-job.yaml) — creates topic with 3 partitions, replication-factor 3
An explicit kubectl exec step in k8s/deploy.sh that creates the topic directly after Kafka is ready

Both use --if-not-exists so they are safe to run on every deploy.

Zero results despite documents being indexed

If kNN fails, the combined score will be at most 0.4 (BM25-only). A threshold above 0.4 will return zero results. The current threshold is 0.45 in pyworker/search_worker.py. Raise it for precision or lower it for recall.

Recent Enhancements

Timezone-Aware Date Queries: The NLP parsing engine now seamlessly supports timezone-relative date range queries (e.g., "get files uploaded on july 7th" or "today"). It intelligently injects local offsets into OpenSearch queries, reconciling the user's local timezone with UTC ingestion timestamps to guarantee precise file retrieval bounds without missing boundary-case uploads.

Name		Name	Last commit message	Last commit date
Latest commit History 51 Commits
Search-Engine		Search-Engine
backend		backend
docs		docs
infrastructure		infrastructure
k8s		k8s
shared		shared
tests		tests
workers		workers
.env.example		.env.example
.gitignore		.gitignore
CLAUDE.md		CLAUDE.md
Claude_resume.txt		Claude_resume.txt
Project-dessc.txt		Project-dessc.txt
README.md		README.md
TECHNICAL_DOCUMENTATION.md		TECHNICAL_DOCUMENTATION.md
docker-compose.yml		docker-compose.yml
pytest.ini		pytest.ini
requirements.txt		requirements.txt
role-assignments.md		role-assignments.md

Folders and files

Latest commit

History

Repository files navigation

Object Search Engine

Table of Contents

Architecture Overview

Prerequisites

Hardware Requirements

WSL 2 (Windows 11 / Windows 10 22H2+)

1. Enable WSL 2 + Ubuntu

2. Docker Desktop

3. OpenSearch kernel setting

4. Minikube + kubectl

5. Git & AWS CLI

RHEL 10 — x86_64

1. Docker CE

2. OpenSearch kernel setting

3. Minikube + kubectl

4. Git & AWS CLI

Fedora 43 — x86_64

1. Docker CE

2. OpenSearch kernel setting

3. Minikube + kubectl

4. Git & AWS CLI

Quick prerequisite check (all platforms)

Repository Structure

Quick Start — Docker Compose

Quick Start — Minikube

Common Minikube commands

Service Ports

Docker Compose

Minikube (NodePort)

Environment Variables

OpenSearch Index

How the Search Pipeline Works

How the Ingestion Pipeline Works

Natural Language Query Examples

Uploading Files

Docker Compose

Minikube

Debugging

Docker Compose

Minikube

Stopping

Docker Compose

Minikube

Running Tests

Known Issues & Fixes

kNN search fails with Field 'embedding' is not knn_vector type

Ingestion worker crashes with UNKNOWN_TOPIC_OR_PART on fresh deploy

Zero results despite documents being indexed

Recent Enhancements

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

kNN search fails with `Field 'embedding' is not knn_vector type`

Ingestion worker crashes with `UNKNOWN_TOPIC_OR_PART` on fresh deploy