Workstream F: CI hardening, packaging, exceptions, loader dedup (C3/P1/T1/B3/B4)

.github/workflows/ci.yml

@@ -35,7 +35,6 @@ jobs:
         run: python -m black --check src/prkit tests/prkit
 
       - name: Type check
-        continue-on-error: true
         run: python -m mypy src/prkit
 
       - name: Test package

CHANGELOG.md

@@ -9,6 +9,10 @@ Production releases follow semantic versioning. TestPyPI validation builds use P
 
 ### Added
 
+- **`OpenAIModel` custom endpoint support** — new keyword-only constructor params `base_url`, `api_key`, and `api_key_env` allow routing to any proxy or gateway that implements the OpenAI Responses API (`POST /v1/responses`) with an explicit key or key from a named environment variable. Backward-compatible: omitting all three preserves existing `OPENAI_API_KEY` + default endpoint behaviour.
+- **`OllamaModel` explicit auth params** — new keyword-only constructor params `api_key` and `api_key_env` forward a `Bearer` token as the `Authorization` header to `ollama.Client`, providing API-key parity with other providers. Works for cloud endpoints (e.g. `base_url="https://ollama.com"`).
+- **Remote-safe Ollama preflight** — when `base_url` or `OLLAMA_HOST` points to a non-local host, a failed startup connectivity check now emits a warning instead of raising `ConnectionError`; precise errors surface at `chat()` call time.
+- **"Extending prkit" contract documented** — `DATASETS.md` and `CORE.md` now document the stable external extension points: registering a custom `DatasetHub` loader/downloader from outside the package, local-directory loading without a downloader, `OpenAIModel` / `OllamaModel` custom-endpoint construction, and `register_model_client` for additional providers.
 - **`prkit` command-line interface** (`prkit list`, `prkit info <dataset>`, `prkit download <dataset>`, `prkit --version`) for dataset workflows, installed via the `prkit` console script.
 - PEP 561 typing support: ships a `py.typed` marker and the `Typing :: Typed` classifier.
 - `ruff` linting + import sorting, a `.pre-commit-config.yaml`, a `Makefile`, and GitHub Actions CI (lint, format check, type check, tests on Python 3.10–3.12) plus a release workflow.
@@ -30,8 +34,13 @@ Production releases follow semantic versioning. TestPyPI validation builds use P
 - Coverage enforcement for `prkit` now uses a 60% minimum and keeps unit tests in pytest format.
 - Provider-model test targets were updated for OpenAI, Gemini, Anthropic, Ollama, DeepSeek, xAI, and DashScope clients.
 
+### Changed
+
+- **(internal)** Model-output JSON extraction consolidated: the duplicate `extract_json_object` in `prkit.evaluation.llm_judge.parse` and the unreachable helpers `_iter_braced_json_candidates`, `_try_parse_json_object`, `_JSON_FENCE_RE`, and the thin `_extract_json_object` wrapper in `prkit.semantics.inference.calls` are removed. All call sites now delegate to the single canonical `extract_json_object` / `extract_json_payload` in `prkit.core.model_clients.structured_output`. Public API and parsing semantics are unchanged.
+
 ### Fixed
 
+- **`DatasetHub` registration-ordering bug** — calling `DatasetHub.register(name, Loader)` before any built-in dataset was touched caused all built-in loaders and downloaders to be permanently suppressed. Built-ins are now seeded idempotently (via `setdefault`) at the start of every public mutating method, so external registrations can safely happen in any order.
 - JEEBench loader handling for numeric answer categories and retained metadata.
 - Workflow module behavior in domain assessment, theorem review, and workflow composition paths.
 

CORE.md

@@ -195,6 +195,81 @@ text = client.chat(
 print(text)
 ```
 
+#### Custom OpenAI Responses-API endpoints
+
+`OpenAIModel` accepts `base_url` and `api_key` / `api_key_env` keyword arguments for
+routing to a proxy or gateway that implements the OpenAI **Responses API**
+(`POST /v1/responses`). These are **not** available through `create_model_client` (which
+is routing-only); construct `OpenAIModel` directly:
+
+```python
+from prkit.core.model_clients.openai import OpenAIModel
+
+# Explicit key + custom endpoint
+client = OpenAIModel("gpt-4.1-mini", base_url="https://gw.example/v1", api_key="sk-…")
+
+# Key from a named env var
+client = OpenAIModel("gpt-4.1-mini", base_url="https://gw.example/v1", api_key_env="GW_KEY")
+
+# No args → uses OPENAI_API_KEY and the default OpenAI endpoint (backward-compatible)
+client = OpenAIModel("gpt-4.1-mini")
+```
+
+Key-resolution precedence: explicit `api_key` → `api_key_env` env lookup → `OPENAI_API_KEY`.
+Omitting `base_url` lets the OpenAI SDK default apply (honouring `OPENAI_BASE_URL` if set).
+
+> **Note:** `OpenAIModel` only calls `client.responses.create` (the Responses API). It is not
+> suitable for Chat-Completions-only gateways.
+
+#### Ollama local and cloud usage
+
+`OllamaModel` supports both local Ollama runtimes and cloud endpoints. The `base_url` and
+`api_key` / `api_key_env` keyword arguments give explicit control over the connection:
+
+```python
+from prkit.core.model_clients.ollama import OllamaModel
+
+# Local (default: http://localhost:11434 or OLLAMA_HOST env)
+client = OllamaModel("qwen3-vl:8b")
+
+# Local with explicit host
+client = OllamaModel("qwen3-vl:8b", base_url="http://192.168.1.10:11434")
+
+# Cloud endpoint with explicit key
+client = OllamaModel("llama3:70b-cloud", base_url="https://ollama.com", api_key="ol-…")
+
+# Cloud endpoint with key from env var
+client = OllamaModel("llama3:70b-cloud", base_url="https://ollama.com", api_key_env="OLLAMA_CLOUD_KEY")
+
+# Env-var auth only (lib auto-reads OLLAMA_API_KEY when api_key/api_key_env not supplied)
+client = OllamaModel("llama3:70b-cloud", base_url="https://ollama.com")
+```
+
+Key-resolution precedence: explicit `api_key` → `api_key_env` env lookup → library
+auto-reads `OLLAMA_API_KEY`. For remote hosts (`base_url` pointing to a non-localhost
+address) a failed startup preflight emits a warning instead of raising `ConnectionError`;
+precise errors surface at `chat()` call time.
+
+#### Registering additional providers
+
+Use `register_model_client` to add new providers or override routing without modifying
+built-in code:
+
+```python
+from prkit.core.model_clients import register_model_client
+from prkit.core.model_clients.factory import ProviderRule
+
+def _load_my_provider(model: str, logger):
+    from my_package import MyClient
+    return MyClient(model, logger)
+
+register_model_client(ProviderRule(
+    name="my_provider",
+    match=lambda model: model.startswith("my-"),
+    load=_load_my_provider,
+))
+```
+
 ### PRKitLogger
 
 Centralized logger for consistent logging across PRKit packages. Provides colored console output, optional file logging, and environment-based configuration via `PRKIT_LOG_LEVEL`, `PRKIT_LOG_FILE`, `PRKIT_LOG_CONSOLE`, `PRKIT_LOG_COLORS`. Default log file: `{cwd}/prkit_logs/prkit.log`.

DATASETS.md

@@ -545,3 +545,75 @@ To add a new dataset:
 5. Add dataset information to this documentation
 
 See existing loaders in `src/prkit/datasets/loaders/` for examples.
+
+## Extending DatasetHub from External Code
+
+`DatasetHub` supports external loaders and downloaders registered at runtime — no fork or
+subclass needed.
+
+### Registering an external loader
+
+```python
+from prkit.datasets import DatasetHub
+from prkit.datasets.loaders.base_loader import BaseDatasetLoader
+from prkit.core.domain import PhysicalDataset, PhysicsProblem
+
+class MyLoader(BaseDatasetLoader):
+    @property
+    def field_mapping(self):
+        return {}
+
+    def get_info(self):
+        return {
+            "name": "my_dataset",
+            "variants": ["full"],
+            "splits": ["full"],
+        }
+
+    def load(self, data_dir=None, **kwargs):
+        # Read from data_dir and return a PhysicalDataset
+        ...
+
+DatasetHub.register("my_dataset", MyLoader)
+```
+
+After registration all hub methods (`load`, `get_info`, `list_available`) recognise
+`"my_dataset"`. Built-in loaders are always present regardless of registration order.
+
+### Loading from a local directory (no downloader)
+
+A loader does **not** require a paired downloader. Pass `data_dir` to read from a local
+path directly, bypassing any download step:
+
+```python
+dataset = DatasetHub.load("my_dataset", data_dir="/path/to/data")
+```
+
+This works even when no `BaseDownloader` is registered for the name.
+
+### Registering an external downloader
+
+```python
+from prkit.datasets.downloaders.base_downloader import BaseDownloader
+
+class MyDownloader(BaseDownloader):
+    @property
+    def dataset_name(self):
+        return "my_dataset"
+
+    @property
+    def download_info(self):
+        return {"variants": ["full"], "splits": ["full"]}
+
+    def _do_download(self, download_dir, **kwargs):
+        # Download logic — return download_dir when done
+        return download_dir
+
+    def verify(self, data_dir):
+        return True
+
+DatasetHub.register_downloader("my_dataset", MyDownloader)
+```
+
+With a downloader registered, `DatasetHub.load("my_dataset", auto_download=True)` will
+trigger `MyDownloader` when the data directory is missing.

RELEASE_NOTES.md

@@ -1,3 +1,30 @@
+## Physical Reasoning Toolkit — Next Release
+
+### Highlights
+
+**Custom endpoint flexibility for model clients.** `OpenAIModel` now accepts `base_url`,
+`api_key`, and `api_key_env` keyword arguments, making it straightforward to route traffic
+to a proxy or gateway that fronts the OpenAI Responses API without subclassing. `OllamaModel`
+gains the same `api_key` / `api_key_env` params for cloud endpoints (e.g. `ollama.com`),
+and its startup connectivity check now treats remote hosts gracefully — a failed preflight
+warns instead of raising, so cloud usage no longer requires suppressing the connection check.
+
+**`DatasetHub` registration-ordering bug fixed.** Calling `DatasetHub.register(name, Loader)`
+before any built-in was touched previously caused all built-in loaders and downloaders to be
+silently omitted. Built-ins are now seeded idempotently at the start of every public method.
+External registrations can now happen in any order and are safe alongside built-in datasets.
+
+**Extending prkit — documented stable API.** `DATASETS.md` and `CORE.md` now document the
+supported extension points: registering a `DatasetHub` loader or downloader from outside the
+package, local-directory loading without a paired downloader, custom-endpoint construction for
+`OpenAIModel` and `OllamaModel`, and adding new providers via `register_model_client`.
+
+**JSON-extraction consolidation (internal).** Three near-duplicate "extract JSON from model
+text" implementations have been removed. All call sites delegate to the single tested
+canonical helper in `prkit.core.model_clients.structured_output`. No public API change.
+
+---
+
 ## Physical Reasoning Toolkit v0.1.0
 
 First release of **PRKit**—a unified toolkit for AI physical reasoning research. PRKit provides shared abstractions for physics problems, model inference, evaluation, and structured annotation workflows.

pyproject.toml

@@ -1,13 +1,13 @@
 [build-system]
-requires = ["setuptools>=61.0", "wheel"]
+requires = ["setuptools>=77.0", "wheel"]
 build-backend = "setuptools.build_meta"
 
 [project]
 name = "physical-reasoning-toolkit"
 version = "0.1.0.post22"
 description = "A toolkit for physical-reasoning datasets, multi-provider LLM inference, answer evaluation, and annotation."
 readme = {file = "README.md", content-type = "text/markdown"}
-license = {text = "MIT"}
+license = "MIT"
 authors = [
     {name = "Yinghuan Zhang", email = "yinghuan.flash@gmail.com"}
 ]
@@ -25,7 +25,6 @@ classifiers = [
     "Intended Audience :: Developers",
     "Intended Audience :: Science/Research",
     "Intended Audience :: Education",
-    "License :: OSI Approved :: MIT License",
     "Operating System :: OS Independent",
     "Programming Language :: Python :: 3",
     "Programming Language :: Python :: 3.10",

src/prkit/core/__init__.py

@@ -4,8 +4,20 @@
 This package provides core functionality for PRKit (physical-reasoning-toolkit).
 """
 
+from .exceptions import (
+    ConfigError,
+    DatasetError,
+    ModelClientError,
+    PRKitError,
+    UnknownModelError,
+)
 from .logging_config import PRKitLogger
 
 __all__ = [
+    "PRKitError",
+    "UnknownModelError",
+    "ModelClientError",
+    "ConfigError",
+    "DatasetError",
     "PRKitLogger",
 ]

src/prkit/core/exceptions.py

@@ -0,0 +1,31 @@
+"""
+PRKit exception hierarchy.
+
+All package-specific exceptions derive from PRKitError so callers can catch
+the entire family with a single ``except PRKitError`` clause while still
+catching individual subtypes for finer-grained handling.
+
+Subclasses dual-inherit the closest matching builtin so that existing call
+sites which assert on builtins (e.g. ``except ValueError``) continue to work
+without modification.
+"""
+
+
+class PRKitError(Exception):
+    """Base class for all PRKit-specific exceptions."""
+
+
+class UnknownModelError(PRKitError, ValueError):
+    """Raised when a model name does not match any registered provider."""
+
+
+class ModelClientError(PRKitError, RuntimeError):
+    """Raised when a provider API call fails in a way that has useful context."""
+
+
+class ConfigError(PRKitError, ValueError):
+    """Raised for misconfigured or missing environment / config values."""
+
+
+class DatasetError(PRKitError, RuntimeError):
+    """Raised for dataset loading or download failures."""

src/prkit/core/model_clients/factory.py

@@ -16,6 +16,7 @@
 from collections.abc import Callable
 from dataclasses import dataclass
 
+from ..exceptions import UnknownModelError
 from .base import BaseModelClient
 
 ModelMatcher = Callable[[str], bool]
@@ -76,7 +77,7 @@ def _load_gpt(model: str, logger: logging.Logger | None) -> BaseModelClient:
     from .openai import OpenAIModel, _is_supported_openai_model
 
     if not _is_supported_openai_model(model):
-        raise ValueError(
+        raise UnknownModelError(
             f"Unsupported OpenAI model: {model}. "
             "Supported OpenAI models: gpt-4.1, gpt-5xxxx (gpt-5.1, gpt-5.2, etc.), "
             "and o-family (o3, o4, o4-mini, etc.)"
@@ -145,7 +146,7 @@ def create_model_client(
     for rule in _PROVIDER_RULES:
         if rule.matches(model_lower):
             return rule.load(model, logger)
-    raise ValueError(
+    raise UnknownModelError(
         f"Unknown model: {model}. "
         "Supported models: OpenAI (gpt-4.1, gpt-5xxxx, o-family), "
         "Anthropic (claude-*), Google (gemini-*), DeepSeek (deepseek-*), "