CoRe Retrieval API

FastAPI service implementing a multimodal retrieval pipeline for skin lesion analysis. The system combines image segmentation, multimodal embeddings, vector search, and reranking to perform retrieval-based prediction.

Processing Pipeline

Ingestion

1. Receive image, text, and label.
2. Segment the lesion using UNet.
3. Generate a multimodal embedding with Visualized-BGE-m3.
4. Store the embedding and payload in Qdrant.

Inference

1. Receive image and text.
2. Segment the lesion.
3. Generate the multimodal embedding.
4. Retrieve the Top-K nearest neighbours from Qdrant.
5. Rerank the retrieved candidates using the cross-encoder.
6. Return the highest-ranked prediction together with the retrieved candidates.

Project Structure

core-api/
├── app/
│   ├── main.py                     # FastAPI entrypoint and application lifespan
│   ├── core/
│   │   ├── config.py               # Application configuration
│   │   ├── logging_config.py
│   │   └── image_utils.py          # Image decoding utilities
│   │
│   ├── models/
│   │   ├── unet.py                 # Lesion segmentation
│   │   ├── embedder.py             # Visualized-BGE-m3 wrapper
│   │   ├── reranker.py             # Cross-encoder reranker
│   │   └── visual_bge/
│   │       ├── modeling.py
│   │       └── eva_clip/
│   │
│   ├── services/
│   │   ├── pipeline.py             # Retrieval pipeline orchestration
│   │   └── vector_store.py         # Qdrant wrapper
│   │
│   ├── routers/
│   │   ├── health.py
│   │   ├── retrieval.py            # Retrieval endpoints
│   │   └── documents.py            # Qdrant document management API
│   │
│   └── schemas/
│       ├── schemas.py
│       └── vector_store.py
│
├── models/
│   ├── unet/
│   │   └── lesion_segmentation.pth
│   └── bge/
│       └── Visualized_m3.pth
|
├── tools/
│   └── injest.py                   # Injest pipeline for the Qdrant VectorStore DB
|
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
└── .env.example

---

Model Checkpoints

The following model checkpoints must be available before starting the application.

models/
├── unet/
│   └── lesion_segmentation.pth
└── bge/
    └── Visualized_m3.pth

---

Running with Docker

Create the model directories and place the required checkpoints before starting the services.

mkdir -p models/unet models/bge

cp .env.example .env

docker compose up --build

After startup:

Service	URL
API	http://localhost:8000
Swagger UI	http://localhost:8000/docs
Qdrant	http://localhost:6333
Qdrant Dashboard	http://localhost:6333/dashboard

---

Retrieval API

POST /ingest

Adds a new document to the vector database.

Request

multipart/form-data

Field	Type	Description
image	File	Lesion image
text	String	Associated clinical description
label	String	Ground-truth label

Example

curl -X POST http://localhost:8000/ingest \
    -H "Authorization: Bearer $API_AUTH_TOKEN" \
    -F "image=@lesion.jpg" \
    -F "text=Irregular skin lesion before surgery" \
    -F "label=True"

---

POST /predict

Performs retrieval and reranking to predict the most likely label.

Request

multipart/form-data

Field	Type	Description
image	File	Query image
text	String	Clinical description
pression_category	Integer	Pressure category
k	Integer (optional, default: 15)	Number of retrieved candidates before reranking

Example

curl -X POST http://localhost:8000/predict \
    -H "Authorization: Bearer $API_AUTH_TOKEN" \
    -F "image=@query.jpg" \
    -F "text=Irregular skin lesion before surgery" \
    -F "k=15"

Response

{
  "predicted_label": "True",
  "best_match": {
    "id": "...",
    "score": 0.87,
    "text": "...",
    "label": "True"
  },
  "candidates": [
    ...
  ]
}

---

Document Management API

The /documents endpoints expose a wrapper around the underlying Qdrant collection, allowing document inspection and payload management without interacting directly with Qdrant.

These operations affect only the stored metadata unless explicitly stated. Updating a document does not recompute its embedding.

---

GET /documents

Returns a paginated list of stored documents.

Query Parameters

Parameter	Type	Description
limit	Integer	Number of documents to return (1–500)
offset	String	Pagination token returned by the previous request

Response

{
  "documents": [...],
  "next_offset": "..."
}

---

GET /documents/{id}

Retrieves a single document by its identifier.

Response

{
  "id": "...",
  "text": "...",
  "label": "...",
  "image_path": "..."
}

---

PATCH /documents/{id}

Updates the payload associated with an existing document.

Only the fields included in the request are modified.

The embedding vector remains unchanged.

Request

{
  "text": "Updated description",
  "label": "True",
  "image_path": "/path/image.jpg"
}

---

DELETE /documents/{id}

Deletes a single document from the collection.

Response

{
  "id": "..."
}

---

DELETE /documents?confirm=true

Deletes all documents from the collection.

This operation requires explicit confirmation through the confirm=true query parameter.

Response

{
  "deleted_count": 1250
}

---