Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 33 additions & 5 deletions docs/docs/advanced/intrinsics.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,11 @@ description: "Adapter-accelerated RAG quality checks using LoRA/aLoRA adapters w
# diataxis: how-to
---

**Prerequisites:** `pip install "mellea[hf]"` for LocalHFBackend (GPU or Apple
Silicon Mac recommended), or `pip install mellea` for OpenAIBackend with a
[Granite Switch](/reference/glossary#granite-switch) model served via vLLM.
**Prerequisites:** use `uv sync --extra hf` for runtime LoRA/aLoRA adapter
functions and local [Granite Switch](/reference/glossary#granite-switch)
checkpoints. Both local paths require a GPU or Apple Silicon Mac. An
OpenAIBackend using a Granite Switch model served via vLLM uses
`uv sync --extra switch` when it downloads embedded adapter metadata.

Adapter functions are adapter-accelerated operations for RAG quality checks. They use
LoRA/aLoRA adapters loaded directly into the Hugging Face backend — faster and more
Expand All @@ -15,7 +17,10 @@ reliable than prompting a general-purpose model for these specialized micro-task
> **Backend note:** Adapter functions work with two backends:
>
> - **LocalHFBackend** — loads LoRA/aLoRA adapters from the catalog at runtime.
> All adapter functions are available. Requires a GPU or Apple Silicon Mac.
> A local Granite Switch checkpoint can instead use
> `load_embedded_adapters=True`; install `mellea[hf]` first. Only
> adapter functions embedded in the checkpoint are then available. Requires a
> GPU or Apple Silicon Mac.
> - **OpenAIBackend** — uses a Granite Switch model served via vLLM with
> `load_embedded_adapters=True`. Only adapter functions embedded in the model are
> available — check the model's `adapter_index.json` for the list.
Expand All @@ -33,6 +38,29 @@ from mellea.backends.huggingface import LocalHFBackend
backend = LocalHFBackend(model_id="ibm-granite/granite-4.1-3b")
```

## Use a local Granite Switch checkpoint

Granite Switch checkpoints contain their adapter functions already. Pass the
checkpoint to `LocalHFBackend` with `load_embedded_adapters=True`; existing
helper functions such as `rag.check_answerability()` work unchanged.

```python
# Requires: mellea[hf]
# Returns: LocalHFBackend
from mellea.backends.huggingface import LocalHFBackend
from mellea.backends.model_ids import IBM_GRANITE_SWITCH_4_1_3B_PREVIEW

backend = LocalHFBackend(
model_id=IBM_GRANITE_SWITCH_4_1_3B_PREVIEW,
load_embedded_adapters=True,
)
```

Only adapter functions listed in the checkpoint's `adapter_index.json` are
available. Mellea warns once if Granite Switch's installed package metadata
does not yet include Mellea's resolved Transformers version; the warning
disappears after Granite Switch publishes compatible metadata.

Or, with a Granite Switch model via the OpenAI backend:

```python
Expand Down Expand Up @@ -297,7 +325,7 @@ implementations, not whether a composed `Adapter` can be registered directly:

| Backend | `LocalFileBinding` (LocalFile/PEFT) | `EmbeddedBinding` (Embedded/Granite Switch) | `ServerMediatedBinding` |
| --- | --- | --- | --- |
| `LocalHFBackend` | ✅ shipping — `add_adapter` accepts a `LocalFileBinding` directly | 🔜 planned (#1018) | — |
| `LocalHFBackend` | ✅ shipping — `add_adapter` accepts a `LocalFileBinding` directly | ✅ shipping — `load_embedded_adapters=True`, via the deprecated `EmbeddedIntrinsicAdapter` shim | — |
| `OpenAIBackend` | — | ✅ shipping, via the deprecated `EmbeddedIntrinsicAdapter` shim above, which builds an `EmbeddedBinding` internally | — |

`ServerMediatedBinding` has no backend implementation yet — see discussion #1486.
Expand Down
3 changes: 2 additions & 1 deletion docs/docs/advanced/lora-and-alora-adapters.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ Hugging Face account.
> They do not work with Ollama, OpenAI, or other remote backends.
>
> Granite Switch models ship with pre-trained adapter functions embedded in the
> model weights, which can be used via `OpenAIBackend` with
> model weights. Use them through `OpenAIBackend` with a served checkpoint, or
> through `LocalHFBackend` with a local checkpoint and
> `load_embedded_adapters=True`. See [Adapter functions](./intrinsics.md) for details.

## LoRA vs aLoRA
Expand Down
16 changes: 9 additions & 7 deletions docs/docs/reference/glossary.md
Original file line number Diff line number Diff line change
Expand Up @@ -273,7 +273,7 @@ Granite Libraries is the collective name for the three curated collections of
[adapter functions](#adapter-function) published by IBM Granite:

| Collection | Purpose |
|-----------|---------|
| ---------- | --------- |
| **Granite Libraries Core** | General-purpose capabilities: certainty checking, requirement verification, context attribution |
| **Granite Libraries RAG** | Retrieval-Augmented Generation pipeline: answerability, citations, hallucination detection, query rewriting |
| **Granite Libraries Guardian** | Safety and compliance: guardian checks, policy guardrails, factuality detection and correction |
Expand Down Expand Up @@ -472,10 +472,11 @@ See: [Safety Guardrails](../how-to/safety-guardrails#policy-compliance)
Granite Switch is the architecture and toolchain for composing adapter functions into a
single deployable Granite model. The composer embeds LoRA/aLoRA adapter function weights
directly into a base Granite model checkpoint, producing a self-contained model file that
carries its adapter functions with it. When that checkpoint is served via vLLM and
accessed through `OpenAIBackend` with `load_embedded_adapters=True`, adapter functions
are available without runtime adapter loading. Only adapter functions embedded in the
checkpoint are available — check the model's `adapter_index.json`.
carries its adapter functions with it. It can be used through `OpenAIBackend`
when served via vLLM, or directly through `LocalHFBackend`; both use
`load_embedded_adapters=True` to register the checkpoint's adapter functions
without runtime adapter loading. Only adapter functions embedded in the checkpoint
are available — check the model's `adapter_index.json`.

The resulting model file is sometimes called a **Granite Switch checkpoint** or simply a
**checkpoint**. The toolchain that produces it is the **Granite Switch composer**.
Expand Down Expand Up @@ -567,8 +568,9 @@ See: [Use Images and Vision Models](../how-to/use-images-and-vision)
in Mellea's implementation. It is a backend-level primitive — a structured generation
operation backed by a LoRA/aLoRA adapter with special input/output handling (e.g.,
constrained decoding, RAG retrieval). `LocalHFBackend` loads `Intrinsic` adapters at
runtime; `OpenAIBackend` uses them when backed by a [Granite Switch](#granite-switch)
checkpoint with `load_embedded_adapters=True`.
runtime or uses adapter functions embedded in a local [Granite Switch](#granite-switch)
checkpoint with `load_embedded_adapters=True`. `OpenAIBackend` also supports
embedded adapter functions when backed by a Granite Switch deployment.

> **Note:** The Python symbol `Intrinsic` (and related classes such as `IntrinsicAdapter`)
> will be renamed to `AdapterFunction` / `Adapter` in a future phase of Epic #929
Expand Down
45 changes: 30 additions & 15 deletions docs/examples/granite-switch/README.md
Original file line number Diff line number Diff line change
@@ -1,35 +1,49 @@
# Granite Switch Examples

This directory contains examples for running Mellea adapter functions through an
OpenAI-compatible backend using Granite Switch models.
This directory contains examples for running Mellea adapter functions with
Granite Switch models, either locally through `LocalHFBackend` or through an
OpenAI-compatible vLLM deployment.

## What is Granite Switch?

Granite Switch models ship with LoRA and aLoRA adapters pre-baked into the model
weights. Instead of loading adapters at runtime (as `LocalHFBackend` does), these
embedded adapters are activated via control tokens injected by the model's chat
template. Only the I/O transformation configs are downloaded — no adapter weights
are transferred.
weights. Unlike runtime LoRA/aLoRA adapters on a standard `LocalHFBackend`,
these embedded adapters are activated via control tokens injected by the model's
chat template. Only the I/O transformation configs are downloaded — no adapter
weights are transferred.

## Prerequisites

1. A Granite Switch model hosted via [vLLM](https://docs.vllm.ai/):
### Local Hugging Face inference

Run the local example on a GPU or Apple Silicon Mac:

```bash
python -m vllm.entrypoints.openai.api_server \
--model <granite-switch-model-id> \
--dtype bfloat16 \
--enable-prefix-caching
uv sync --extra hf
```

2. `pip install mellea`
### OpenAI-compatible inference

1. Host a Granite Switch model with [vLLM](https://docs.vllm.ai/).
2. Install the `switch` extra to download embedded adapter metadata:

```bash
uv sync --extra switch
```

## Available adapters

Not all adapter functions are embedded in every Granite Switch model. You should check the model's `adapter_index.json` file for a definitive list. For granite switch models pre-built by IBM, we include a list of models in the Mellea `model_id`.
Not all adapter functions are embedded in every Granite Switch model. Check the
model's `adapter_index.json` for a definitive list. For Granite Switch models
pre-built by IBM, Mellea includes a list of models in `model_id`.

## Files

### answerability_local_hf.py

Demonstrates `rag.check_answerability()` against a local Granite Switch
checkpoint using `LocalHFBackend(load_embedded_adapters=True)`.

### answerability_openai.py

Demonstrates `rag.check_answerability()` using `OpenAIBackend` with
Expand All @@ -49,10 +63,11 @@ you only need a subset of adapters or want more control over adapter
registration.

## Architecture

![Granite Libraries Software Stack Architecture in Mellea](../../docs/images/granite-libraries-mellea-architecture.png)

## Related

- [`../intrinsics/`](../intrinsics/) — the same adapter functions using `LocalHFBackend`
- [`../intrinsics/`](../intrinsics/) — runtime LoRA/aLoRA adapter functions
- [Adapter Functions Documentation](../../docs/docs/advanced/intrinsics.md)
- [Official Granite Switch Documentation](https://github.com/generative-computing/granite-switch)
- [Official Granite Switch Documentation](https://github.com/generative-computing/granite-switch)
29 changes: 29 additions & 0 deletions docs/examples/granite-switch/answerability_local_hf.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# pytest: e2e, huggingface, skip

"""Run the answerability adapter function through a local Granite Switch checkpoint.

Requires a GPU or Apple Silicon Mac and:

uv sync --extra hf

To run from the Mellea source tree:

uv run python docs/examples/granite-switch/answerability_local_hf.py
"""

from mellea.backends.huggingface import LocalHFBackend
from mellea.backends.model_ids import IBM_GRANITE_SWITCH_4_1_3B_PREVIEW
from mellea.stdlib.components import Document, Message
from mellea.stdlib.components.intrinsic import rag
from mellea.stdlib.context import ChatContext

backend = LocalHFBackend(
model_id=IBM_GRANITE_SWITCH_4_1_3B_PREVIEW, load_embedded_adapters=True
)

context = ChatContext().add(Message("assistant", "Hello! How can I help you?"))
question = "What is the square root of 4?"
documents = [Document("The square root of 4 is 2.")]

result = rag.check_answerability(question, documents, context, backend)
print(f"Answerability: {result}")
24 changes: 16 additions & 8 deletions docs/examples/intrinsics/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,9 @@ This directory contains examples for using Mellea's adapter functions - speciali
- **Adapter System**: Using LoRA/aLoRA adapters for specific tasks
- **RAG Evaluation**: Assessing retrieval-augmented generation quality
- **Quality Metrics**: Measuring relevance, groundedness, and accuracy
- **Backend Integration**: Adding adapters to different backend types (LocalHFBackend with runtime adapters, OpenAIBackend with Granite Switch embedded adapters)
- **Backend Integration**: Running adapter functions through LocalHFBackend
(runtime adapters or local Granite Switch checkpoints) and OpenAIBackend
(Granite Switch deployments)

## Basic Usage

Expand All @@ -29,16 +31,22 @@ result = core.check_certainty(ctx, backend)
print(f"Certainty score: {result}")
```

OpenAIBackends also support a type of embedded adapter for Granite Switch models:
Granite Switch models can run locally with `LocalHFBackend`:

```python
backend = OpenAIBackend(
model_id=IBM_GRANITE_SWITCH_4_1_3B_PREVIEW.hf_model_name,
load_embedded_adapters=True, # Auto-loads adapters from Hugging Face repo.
...
from mellea.backends.huggingface import LocalHFBackend
from mellea.backends.model_ids import IBM_GRANITE_SWITCH_4_1_3B_PREVIEW

backend = LocalHFBackend(
model_id=IBM_GRANITE_SWITCH_4_1_3B_PREVIEW,
load_embedded_adapters=True,
)
```

Install this path with `uv sync --extra hf`.

The underlying adapter functions can also be utilized directly when generating:

```python
from mellea.stdlib.components import Intrinsic
import mellea.stdlib.functional as mfuncs
Expand All @@ -54,8 +62,8 @@ out, new_ctx = mfuncs.act(
)
```

For complete runnable examples using the OpenAI backend with Granite Switch,
see [`../granite-switch/`](../granite-switch/).
For complete runnable Granite Switch examples, see
[`../granite-switch/`](../granite-switch/).

> **Note:** Not all adapter functions are embedded in every Granite Switch model. You should check
> the model's `adapter_index.json` file for a definitive list. For granite switch models
Expand Down
16 changes: 6 additions & 10 deletions docs/examples/intrinsics/intrinsics.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,18 +11,14 @@
# for helper functions.

backend = LocalHFBackend(model_id=model_ids.IBM_GRANITE_4_1_3B)
# --- Alternative: OpenAI backend with Granite Switch (requires vLLM server) ---
# Requires the adapter for this intrinsic to be embedded in the Granite Switch
# model. See docs/examples/granite-switch/ for a full runnable example.
# from mellea.backends.openai import OpenAIBackend
# --- Alternative: local Granite Switch checkpoint ---
# Requires: uv sync --extra hf
# See docs/examples/granite-switch/answerability_local_hf.py for a runnable example.
# from mellea.backends.huggingface import LocalHFBackend
# from mellea.backends.model_ids import IBM_GRANITE_SWITCH_4_1_3B_PREVIEW
# from mellea.formatters import TemplateFormatter
#
# backend = OpenAIBackend(
# model_id=IBM_GRANITE_SWITCH_4_1_3B_PREVIEW.hf_model_name,
# formatter=TemplateFormatter(model_id=IBM_GRANITE_SWITCH_4_1_3B_PREVIEW.hf_model_name),
# base_url="http://localhost:8000/v1", # vLLM server URL
# api_key="EMPTY",
# backend = LocalHFBackend(
# model_id=IBM_GRANITE_SWITCH_4_1_3B_PREVIEW,
# load_embedded_adapters=True,
# )
# --- End alternative ---
Expand Down
12 changes: 6 additions & 6 deletions mellea/backends/adapters/_core.py
Original file line number Diff line number Diff line change
Expand Up @@ -725,7 +725,8 @@ class EmbeddedBinding:
source (str): Base model identifier this binding activates adapters
against — the backend's `base_model_name` (e.g. `granite-4.1-3b`
for a backend built against `ibm-granite/granite-4.1-3b`).
Stamped by `OpenAIBackend.add_adapter` at registration; not
Stamped by `OpenAIBackend.add_adapter` or
`LocalHFBackend.add_adapter` at registration; not
otherwise used by `apply_activation`.
"""

Expand Down Expand Up @@ -776,14 +777,13 @@ async def apply_activation(
invocation-complete event here would have to guess an `outcome` that
this method cannot know, which is worse than not firing it: it would
report `outcome="success"` for calls that go on to fail. Wiring a
real invocation-complete signal in requires the caller (currently
`OpenAIBackend._generate_from_intrinsic`) to fire it once generation
and parsing resolve — tracked as a follow-up, not part of this method.
real invocation-complete signal requires the caller to fire it once
generation and parsing resolve — tracked as a follow-up, not part of
this method.

This method is `async` (unlike the rest of `EmbeddedBinding`'s
surface) purely because hook dispatch (`invoke_hook`) is async; its
own work is synchronous. Its one caller,
`OpenAIBackend._generate_from_intrinsic`, is already a coroutine, so
own work is synchronous. Its callers already run in coroutines, so
`await`ing here — rather than bridging through
`_run_async_in_thread`, which is for calling async code from sync
code — avoids spawning a throwaway event loop and thread per call.
Expand Down
8 changes: 4 additions & 4 deletions mellea/backends/adapters/adapter.py
Original file line number Diff line number Diff line change
Expand Up @@ -644,10 +644,10 @@ def resolve_adapter(self, name: str) -> _AdapterCore:
for a in EmbeddedIntrinsicAdapter.from_source(
repo_id, intrinsic_name=name
):
# EmbeddedIntrinsicAdapter is only valid for backends whose
# add_adapter accepts the full Adapter type (e.g. OpenAIBackend).
# LocalHFBackend.add_adapter expects LocalHFAdapter; HF backends
# never set _uses_embedded_adapters=True.
# EmbeddedIntrinsicAdapter is valid only for backends whose
# add_adapter supports the Embedded/Granite Switch reality
# (currently OpenAIBackend and LocalHFBackend when configured
# with load_embedded_adapters=True).
self.add_adapter(a)
else:
# AdapterType.LORA is the pre-Phase-1 default (mirrors old _util.py).
Expand Down
Loading
Loading