Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 7 additions & 26 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,8 @@ jobs:
- name: Run tests with coverage
run: make coverage

lint:
name: Lint and Format Check
check:
name: Check
runs-on: ubuntu-latest
steps:
# Required to use the reusable action
Expand All @@ -59,33 +59,14 @@ jobs:
- name: Install dependencies
run: uv sync --group dev

- name: Run formatting check
- name: Check format and lint
run: make format-check

- name: Check copyright headers
run: make copyright-check
- name: Run type checks
run: make typecheck

- name: Check uv.lock is up to date
run: make lock-check

typecheck:
name: Type Check (advisory)
runs-on: ubuntu-latest
continue-on-error: true
steps:
# Required to use the reusable action
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: "0"

- uses: ./.github/actions/setup-python-env
with:
fetch-depth: "0"
python-version: ${{ env.DEFAULT_PYTHON_VERSION }}

- name: Install dependencies
run: uv sync --group dev

- name: Run type checks
run: uv run tools/codestyle/typecheck.sh || true
- name: Check copyright headers
run: make copyright-check
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,8 @@ AGENTS.local.md
CLAUDE.local.md
.claude/settings.local.json
.sandbox/
.astnav/
.worktrees/
ai/tmp/

# Claude worktrees
Expand Down
15 changes: 7 additions & 8 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,14 +27,6 @@ repos:
types: [python]
pass_filenames: true

# - id: typecheck
# name: Typecheck (advisory)
# language: system
# entry: bash -c 'uv run tools/codestyle/typecheck.sh || true'
# types: [python]
# pass_filenames: false
# verbose: true

- id: copyright-fix
name: Copyright Fix
language: system
Expand All @@ -50,6 +42,13 @@ repos:
stages: [pre-commit]
pass_filenames: false

- id: check
name: Check
language: system
entry: make check
pass_filenames: false
stages: [pre-commit]

- id: check-signoff
name: Check commit is signed off (DCO)
entry: "grep -qE '^Signed-off-by: .+ <.+>'"
Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ make test # run all tests
make bootstrap # install dev dependencies
make format # ruff format + sort imports
make format-check # read-only lint check (used in CI)
make typecheck # ty type check (advisory)
make typecheck # blocking ty type check (warnings fail too)
make docs-serve # local MkDocs server at http://127.0.0.1:8000
```

Expand Down
28 changes: 25 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ This document covers contribution policy and pull request expectations. For loca
- [Conventional Commits](#conventional-commits)
- [Release Tags](#release-tags)
- [Branch Protection](#branch-protection)
- [Local Commit Safety](#local-commit-safety)
- [Pull Request Expectations](#pull-request-expectations)
- [Agent-Assisted Development](#agent-assisted-development)
- [Developer Certificate of Origin](#developer-certificate-of-origin)
Expand All @@ -24,9 +25,11 @@ This document covers contribution policy and pull request expectations. For loca
3. Create a branch from the latest `main` using the [branch naming](#branch-naming) convention.
4. For non-trivial changes, write a plan document before implementation. See [Agent-Assisted Development](#agent-assisted-development).
5. Implement the change following [AGENTS.md](AGENTS.md), [STYLEGUIDE.md](STYLEGUIDE.md), and [DEVELOPMENT.md](DEVELOPMENT.md).
6. Run the relevant local checks from [DEVELOPMENT.md](DEVELOPMENT.md#validation-before-opening-a-pr).
7. Open a pull request with a conventional title, a linked issue, and a completed checklist.
8. Address review feedback. CODEOWNERS are assigned automatically.
6. Install the repository hooks once per clone with `make install-pre-commit`.
7. Run the relevant tests from [DEVELOPMENT.md](DEVELOPMENT.md#validation-before-opening-a-pr).
8. Commit with DCO signoff. The pre-commit hooks run the repository checks before accepting the commit.
9. Open a pull request with a conventional title, a linked issue, and a completed checklist.
10. Address review feedback. CODEOWNERS are assigned automatically.

## Repository Rules

Expand Down Expand Up @@ -147,6 +150,25 @@ The `main` branch has these protections:
| Deletions | Blocked |
| Merge strategy | Squash only |

### Local Commit Safety

Install the repository hooks once per clone:

```bash
make install-pre-commit
```

Before Git records a commit, the hooks check file hygiene, format and lint staged Python files, repair SPDX headers,
verify `uv.lock` when `pyproject.toml` changes, and run `make check`. The aggregate check covers four read-only stages:

- `make format-check`
- `make typecheck`
- `make lock-check`
- `make copyright-check`

The commit-message hook rejects commits without a `Signed-off-by` line. If a hook changes a file, review the change,
stage it again, and retry the commit. Do not bypass repository hooks with `git commit --no-verify`.

## Pull Request Expectations

Every PR should include:
Expand Down
15 changes: 10 additions & 5 deletions DEVELOPMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,13 +154,16 @@ Run all read-only checks:
make check
```

Run the advisory type checker:
`make check` runs `make format-check`, `make typecheck`, `make lock-check`, and `make copyright-check`. CI runs the
same four Make targets as separate steps in its `Check` job so failures identify the affected stage.

Run the blocking type checker:

```bash
make typecheck
```

`ty` is currently advisory. Treat new type-check findings in touched code as review-worthy even when CI does not block on them.
`ty` checks `src`, `tests`, `tests_e2e`, `docs`, `scripts`, and `tools`. Errors and warnings fail locally and in CI.

Check lockfile freshness:

Expand All @@ -183,10 +186,12 @@ Install hooks once:
make install-pre-commit
```

Hooks run Ruff format and lint, uv lock verification, DCO signoff checks, and basic file hygiene. `ty` is installed
for `make typecheck` and `make check`, but it is not currently run as a pre-commit hook.
Before Git records a commit, the hooks check file hygiene, format and lint staged Python files, repair SPDX headers,
verify `uv.lock` when `pyproject.toml` changes, and run the repository-wide `make check`. The commit-message hook
rejects commits without a DCO `Signed-off-by` line.

If `pyproject.toml` changes and `uv.lock` is stale, the uv-lock hook may regenerate `uv.lock` and fail the commit. Add the updated `uv.lock` and retry.
If a hook changes a file, review the change, stage it again, and retry the commit. In particular, the uv-lock hook may
regenerate `uv.lock` when `pyproject.toml` changes. Do not bypass repository hooks with `git commit --no-verify`.

## Secrets and Credentials

Expand Down
6 changes: 3 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ help:
@echo ""
@echo " format - Format and fix code"
@echo " format-check - Check format and lint (read-only)"
@echo " typecheck - Run type checks (advisory, non-blocking)"
@echo " typecheck - Run blocking type checks"
@echo " copyright - Add missing SPDX headers to source files"
@echo " copyright-check - Check all source files have SPDX headers (read-only)"
@echo " check - Run all read-only checks"
Expand Down Expand Up @@ -67,8 +67,8 @@ format-check:
uv run tools/codestyle/ruff_check.sh

typecheck:
@echo "Running type checks (advisory -- see issue tracking full compliance)..."
-uv run tools/codestyle/typecheck.sh
@echo "Running type checks..."
uv run --group docs tools/codestyle/typecheck.sh

copyright:
@echo "Adding missing SPDX headers..."
Expand Down
6 changes: 3 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ dev = [
"pytest>=9.0.3,<10",
"pytest-cov>=7.0,<8",
"ruff>=0.12.3,<1",
"ty>=0.0.23,<0.1",
"ty==0.0.58",
{include-group = "measurement"},
]
measurement = [
Expand Down Expand Up @@ -88,11 +88,11 @@ markers = [
]

[tool.ty.src]
root = "src"
exclude = ["tools/"]
include = ["src", "tests", "tests_e2e", "docs", "scripts", "tools"]

[tool.ty.environment]
python-version = "3.11"
root = ["src", ".", "tools/measurement"]

[tool.coverage.run]
omit = []
2 changes: 1 addition & 1 deletion scripts/benchmark_sync_vs_async.py
Original file line number Diff line number Diff line change
Expand Up @@ -268,7 +268,7 @@ def finalize(row: dict[str, Any]) -> dict[str, Any]:
artifact_storage=artifact_storage,
model_configs=builder.model_configs,
secret_resolver=CompositeResolver([EnvironmentResolver(), PlaintextResolver()]),
model_provider_registry=resolve_model_provider_registry([provider], default_provider_name=provider.name),
model_provider_registry=resolve_model_provider_registry([provider]),
seed_reader_registry=SeedReaderRegistry(readers=[]),
seed_dataset_source=None,
run_config=run_config,
Expand Down
87 changes: 0 additions & 87 deletions src/anonymizer/distributed.py

This file was deleted.

16 changes: 8 additions & 8 deletions src/anonymizer/engine/detection/chunked_validation.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,11 @@
``NddAdapter._detect_missing_records`` surfaces it as a ``FailedRecord``.
Raw text never silently leaks through as unscrubbed output.

Concurrency. Plugin columns dispatch chunks with ``asyncio.gather`` and
``facade.agenerate()``. The legacy custom-column wrapper still uses a
Concurrency. Async plugin execution dispatches chunks with ``asyncio.gather``
and ``facade.agenerate()``. The plugin's synchronous execution path uses a
``ThreadPoolExecutor`` and ``facade.generate()``. Per-alias concurrency is
enforced downstream by DataDesigner's request-admission layer, so the
optional row-level cap only bounds local fan-out pressure.
enforced downstream by DataDesigner's request-admission layer, so the optional
row-level cap only bounds local fan-out pressure.
"""

from __future__ import annotations
Expand Down Expand Up @@ -100,8 +100,8 @@ class ChunkedValidationParams(BaseModel):
excerpt_window_chars: Chars of surrounding raw text included in each
chunk's excerpt on either side of the chunk span.
max_parallel_chunks: Optional row-local cap on concurrently dispatched
chunks. Defaults to all chunks for the legacy custom-column path;
plugin generators derive a cap from validator model capacity.
chunks. Plugin generators derive a default cap from validator model
capacity.
single_chunk_full_text: If True, a row with one validation chunk sees
the full tagged document. If False, even a single chunk uses the
excerpt window. The default preserves production parity with the
Expand Down Expand Up @@ -578,8 +578,8 @@ def chunked_validate_row(
) -> dict[str, Any]:
"""Run chunked validation for a single row and write ``COL_VALIDATION_DECISIONS``.

This sync path is kept for the legacy DataDesigner custom-column wrapper.
Plugin columns should call :func:`chunked_validate_row_async`.
The plugin's synchronous generator uses this path. Its asynchronous generator
calls :func:`chunked_validate_row_async`.
"""
candidates, dispatch_kwargs_per_chunk = _build_dispatch_kwargs_per_chunk(row, params, models)

Expand Down
Loading
Loading