feat: add SIMBAUQSamplingStrategy

.github/CODEOWNERS

@@ -1,2 +1,8 @@
 # Default: request review from maintainers
 * @generative-computing/mellea-maintainers
+
+# SIMBA-UQ sampling strategy: pulls in scikit-learn / sentence-transformers and
+# has confidence-estimation footguns worth a specialist's eye on changes.
+/agent-utilities/mellea_contribs/agent_utilities/core/simbauq.py @radum2275
+/agent-utilities/tests/test_simbauq.py @radum2275
+/agent-utilities/examples/simbauq/ @radum2275

agent-utilities/README.md

@@ -12,6 +12,7 @@ generation, evaluation, or testing of m-programs.
 | `top_k` | Generic Top-K LLM-as-judge selector. Pick the best K of N candidate items using a comparison prompt. |
 | `double_round_robin` | Pairwise tournament selector. Runs A-vs-B and B-vs-A across all pairs and ranks by accumulated wins. |
 | `benchdrift_runner` | BenchDrift integration for robustness testing of Mellea m-programs against semantic problem variations. |
+| `simbauq` | SIMBA-UQ confidence-aware sampling strategy. Generates samples across temperatures and selects the most confident one via similarity-based uncertainty quantification. |
 
 ## Install
 
@@ -20,6 +21,9 @@ pip install mellea-contribs-agent-utilities
 
 # With BenchDrift robustness extras
 pip install "mellea-contribs-agent-utilities[robustness]"
+
+# With SIMBA-UQ sampling extras (scikit-learn, sentence-transformers, tqdm, datasets)
+pip install "mellea-contribs-agent-utilities[simbauq]"
 ```
 
 ## Usage
@@ -60,6 +64,34 @@ for item, score in ranked:
     print(item["name"], score)
 ```
 
+### SIMBA-UQ sampling
+
+Confidence-aware sample selection. Requires the `simbauq` extra for the
+`sbert` metric and the `classifier` confidence method:
+
+```python
+from mellea import start_session
+from mellea_contribs.agent_utilities.core.simbauq import SIMBAUQSamplingStrategy
+
+m = start_session()
+result = m.instruct(
+    "What is the capital of France?",
+    strategy=SIMBAUQSamplingStrategy(
+        temperatures=[0.3, 0.5, 0.7, 1.0],
+        n_per_temp=3,
+        similarity_metric="rouge",
+        confidence_method="aggregation",
+        aggregation="mean",
+    ),
+    return_sampling_results=True,
+)
+
+best = result.result
+print(best._meta["simba_uq"]["confidence"], str(best))
+```
+
+See `docs/simbauq.mdx` and `examples/simbauq/` for the full guide.
+
 ### BenchDrift robustness
 
 Install with the `robustness` extra and have an Ollama server running:

agent-utilities/docs/simbauq.mdx

@@ -0,0 +1,100 @@
+---
+title: SIMBA-UQ Sampling Strategy
+description: Confidence-aware sample selection for Mellea via similarity-based uncertainty quantification.
+---
+
+# SIMBA-UQ Sampling
+
+`SIMBAUQSamplingStrategy` is a confidence-aware sample selector. It generates
+multiple samples across a range of temperatures, computes a pairwise similarity
+matrix between them, and selects the sample with the highest estimated
+confidence.
+
+Based on the SIMBA-UQ framework (Bhattacharjya et al., 2025),
+[SIMBA UQ: Similarity-Based Aggregation for Uncertainty Quantification in Large Language Models](https://arxiv.org/abs/2510.13836).
+
+---
+
+## What SIMBA-UQ Does
+
+- Generates `len(temperatures) * n_per_temp` samples for a single instruction.
+- Builds an `N x N` pairwise similarity matrix (rouge, jaccard, sbert, difflib, or levenshtein).
+- Estimates per-sample confidence with one of two methods:
+  - **aggregation** (data-free): aggregates each sample's similarity to the others.
+  - **classifier**: a trained probabilistic classifier predicts `P(correct)` from similarity features.
+- Returns the most confident sample, with metadata stored on the selected
+  `ModelOutputThunk` under `mot._meta["simba_uq"]`.
+
+<Callout>
+Use SIMBA-UQ when you can afford multiple generations and want the answer the
+model is most self-consistent about, rather than a single greedy sample.
+</Callout>
+
+---
+
+## Install
+
+The strategy's `sbert` and `classifier` paths need extra dependencies:
+
+```bash
+pip install "mellea-contribs-agent-utilities[simbauq]"
+```
+
+`rouge`, `jaccard`, `difflib`, and `levenshtein` metrics with the
+`aggregation` method work without the extra.
+
+---
+
+## When to Use SIMBA-UQ
+
+### Recommended for:
+- Factual / short-answer questions where self-consistency signals correctness.
+- Selecting among several candidate generations without a reference answer.
+- Confidence-gated pipelines (abstain when confidence is low).
+
+### Avoid for:
+- Single-sample deterministic generation (`temperatures` of length 1, `n_per_temp=1`).
+- Long, open-ended outputs where surface similarity is a poor proxy for agreement.
+
+---
+
+## Core API
+
+### Aggregation (data-free)
+
+```python
+from mellea import start_session
+from mellea_contribs.agent_utilities.core.simbauq import SIMBAUQSamplingStrategy
+
+m = start_session()
+result = m.instruct(
+    "What is the capital of France?",
+    strategy=SIMBAUQSamplingStrategy(
+        temperatures=[0.3, 0.5, 0.7, 1.0],
+        n_per_temp=3,
+        similarity_metric="rouge",
+        confidence_method="aggregation",
+        aggregation="mean",
+    ),
+    return_sampling_results=True,
+)
+
+best = result.result
+meta = best._meta["simba_uq"]
+print(meta["confidence"], str(best))
+```
+
+### Classifier (trained)
+
+Provide either `training_samples` + `training_labels` (each group sized
+`len(temperatures) * n_per_temp`) or a pre-fitted `classifier` with
+`predict_proba`. See `examples/simbauq/` for the full four-variant walkthrough,
+including Hugging Face training-data generation.
+
+---
+
+## Attribution
+
+Original author: Radu Marinescu ([@radum2275](https://github.com/radum2275)),
+IBM Research. Ported into mellea-contribs from upstream mellea
+[PR #785](https://github.com/generative-computing/mellea/pull/785).

agent-utilities/examples/README.md

@@ -1,5 +1,9 @@
 # Examples
 
-Runnable examples for `mellea-contribs-agent-utilities` will live here. For now,
-see the README at the package root and the `tests/` directory for working usage
-of `top_k`, `double_round_robin`, and `benchdrift_runner`.
+Runnable examples for `mellea-contribs-agent-utilities`.
+
+- `simbauq/` — SIMBA-UQ confidence-aware sampling strategy, demonstrating all
+  four confidence-estimation variants against Ollama. See `simbauq/README.md`.
+
+For `top_k`, `double_round_robin`, and `benchdrift_runner`, see the README at
+the package root and the `tests/` directory for working usage.