Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
root = true

[*]
charset = utf-8
end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true
indent_style = space
indent_size = 4

[*.{ts,tsx,js,jsx,mjs,json,jsonc,yaml,yml,md,css}]
indent_size = 2

[Makefile]
indent_style = tab
20 changes: 11 additions & 9 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
# Based on standards/template ci.yml; keeps mask-engine specifics (matrix +
# spaCy model). Run `uvx copier update` to pull baseline changes.
name: CI

on:
Expand All @@ -14,15 +16,17 @@ concurrency:

jobs:
lint:
name: Lint (pre-commit)
name: Lint (pre-commit + mypy)
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
- uses: astral-sh/setup-uv@v5
with:
python-version: "3.12"
- uses: pre-commit/action@v3.0.1
- run: uv sync --extra dev
- run: uv run pre-commit run --all-files
- run: uv run mypy .

test:
name: Tests (Python ${{ matrix.python-version }})
Expand All @@ -34,13 +38,11 @@ jobs:
python-version: ["3.12", "3.13"]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
- uses: astral-sh/setup-uv@v5
with:
python-version: ${{ matrix.python-version }}
cache: "pip"
- name: Install baseline + dev deps
run: pip install -e ".[dev]"
- run: uv sync --extra dev
- name: Download spaCy model
run: python -m spacy download de_core_news_lg
run: uv run python -m spacy download de_core_news_lg
- name: Run tests
run: pytest -m "not slow" --tb=short
run: uv run pytest -m "not slow" --tb=short
56 changes: 29 additions & 27 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,33 +1,35 @@
# Managed by standards/template — run `uvx copier update` to sync.
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v6.0.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml
- id: debug-statements
- id: name-tests-test
args: [--pytest-test-first]
- repo: https://github.com/asottile/add-trailing-comma
rev: v4.0.0
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml
- id: check-merge-conflict
- id: check-added-large-files
- id: debug-statements

- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.15.14
hooks:
- id: add-trailing-comma
- repo: https://github.com/asottile/pyupgrade
rev: v3.21.2
- id: ruff-check
args: [--fix]
- id: ruff-format

- repo: local
hooks:
- id: pyupgrade
args: [--py312-plus]
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.19.1
- id: mypy
# Run via uv so mypy sees the project venv and all third-party type
# stubs / py.typed markers, instead of mirrors-mypy whose
# additional_dependencies drift from pyproject.toml.
name: mypy (strict)
entry: uv run mypy .
language: system
types: [python]
pass_filenames: false

- repo: https://github.com/gitleaks/gitleaks
rev: v8.30.1
hooks:
- id: mypy
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.15.2
hooks:
- id: ruff-check
args: [--fix]
- id: ruff-format
- repo: https://github.com/gitleaks/gitleaks
rev: v8.30.1
hooks:
- id: gitleaks
- id: gitleaks
34 changes: 34 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Standard dev commands (noirdoc engineering standard). Run `make help`.
.DEFAULT_GOAL := help

.PHONY: help install lint fmt fmt-check typecheck test test-slow check models

help: ## List available targets
@grep -hE '^[a-zA-Z_-]+:.*?## ' $(MAKEFILE_LIST) \
| awk 'BEGIN{FS=":.*?## "}{printf " \033[36m%-12s\033[0m %s\n", $$1, $$2}'

install: ## Set up the dev environment
uv sync --extra dev

lint: ## Lint (ruff)
uv run ruff check .

fmt: ## Auto-format (ruff)
uv run ruff format .

fmt-check: ## Check formatting (ruff)
uv run ruff format --check .

typecheck: ## Type-check (mypy, strict)
uv run mypy .

test: ## Run fast tests (excludes slow ML-model tests)
uv run python -m pytest -m "not slow"

test-slow: ## Run slow tests (loads ML models)
uv run python -m pytest -m slow

check: fmt-check lint typecheck test ## Run all gates (mirrors CI/pre-commit)

models: ## Download ML model weights
uv run noirdoc models pull
12 changes: 12 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,18 @@ The `encryption_key` must be identical across workers that need to read the same

Don't want to run this yourself? **[Noirdoc Cloud](https://noirdoc.de)** is the hosted API wrapper: a privacy-preserving reverse proxy for LLM calls that uses this exact pipeline, plus multi-tenancy, audit, and provider key management. Compliance story: what's on GitHub is what the cloud runs.

## Development

This repo uses the shared noirdoc tooling standard (`uv` + ruff/mypy). Common tasks go through `make`:

```bash
make install # set up the dev environment
make check # lint + format-check + typecheck + test — run before pushing
make test # run fast tests (excludes slow ML-model tests)
```

Run `make help` for the full list of targets (also: `make lint`, `make fmt`, `make typecheck`, `make test-slow`, `make models`).

## Contributing

Bug reports, detectors, and format support are all welcome. See [CONTRIBUTING.md](https://github.com/nextaim-de/noirdoc/blob/main/CONTRIBUTING.md) for dev setup, tests, and the recognizer pattern.
Expand Down
39 changes: 37 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -74,8 +74,9 @@ redis = [
dev = [
"pytest>=8.0",
"pytest-asyncio>=0.24.0",
"ruff>=0.15.0",
"ruff==0.15.14",
"mypy>=1.19.1",
"pre-commit>=4.0",
"fakeredis>=2.21.0",
]

Expand Down Expand Up @@ -116,11 +117,45 @@ target-version = "py312"
line-length = 100

[tool.ruff.lint]
select = ["E", "F", "I"]
select = ["E", "W", "F", "I", "B", "UP", "SIM", "TID", "RUF"]
ignore = [
"E501", # line length is enforced by the formatter
"RUF002", # en dashes etc. in docstrings are intentional prose punctuation
"RUF003", # en dashes etc. in comments are intentional prose punctuation
]

[tool.ruff.lint.per-file-ignores]
"tests/**" = ["B011"]
"**/__init__.py" = ["F401"]

[tool.ruff.lint.isort]
known-first-party = ["noirdoc"]

[tool.ruff.format]
quote-style = "double"
indent-style = "space"

[tool.mypy]
python_version = "3.12"
strict = true
warn_unused_ignores = true
warn_redundant_casts = true
warn_unreachable = true
disallow_untyped_defs = true
no_implicit_optional = true
plugins = ["pydantic.mypy"]

[[tool.mypy.overrides]]
module = "tests.*"
disallow_untyped_defs = false

# Third-party libs without type stubs / py.typed marker.
[[tool.mypy.overrides]]
module = [
"openpyxl.*",
"gliner.*",
"flair.*",
"pytesseract.*",
"pypdfium2.*",
]
ignore_missing_imports = true
2 changes: 1 addition & 1 deletion src/noirdoc/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
from noirdoc.sdk import RedactionResult, Redactor, redact

try:
from noirdoc._version import __version__ # type: ignore[import-not-found]
from noirdoc._version import __version__
except ImportError: # Source checkout without a build step (e.g. plain `pytest`).
__version__ = "0.0.0+unknown"

Expand Down
9 changes: 4 additions & 5 deletions src/noirdoc/daemon/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,11 @@
from __future__ import annotations

import asyncio
import contextlib
import json
import uuid
from pathlib import Path
from typing import Any
from typing import Any, cast

from noirdoc import __version__
from noirdoc.daemon import paths, spawn
Expand Down Expand Up @@ -96,7 +97,7 @@ async def _send_request(
result = response.get("result")
if result is None:
raise DaemonError("response missing both 'result' and 'error'")
return result
return cast("dict[str, Any]", result)


async def _wait_socket_gone(socket_path: Path, timeout: float) -> None:
Expand Down Expand Up @@ -146,10 +147,8 @@ async def call(method: str, params: dict[str, Any] | None = None) -> dict[str, A
# Ask the stale daemon to exit, wait for it to release the
# socket, then loop and let _spawn_and_connect bring up a
# fresh one at the current version.
try:
with contextlib.suppress(DaemonError):
await _send_request(reader, writer, "shutdown", {})
except DaemonError:
pass
await _close(writer)
await _wait_socket_gone(socket_path, SHUTDOWN_DRAIN_TIMEOUT)
continue
Expand Down
5 changes: 2 additions & 3 deletions src/noirdoc/daemon/paths.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

from __future__ import annotations

import contextlib
import os
from pathlib import Path

Expand Down Expand Up @@ -35,8 +36,6 @@ def ensure_root_dir() -> Path:
"""Create the daemon root with 0o700 perms, idempotent."""
d = root_dir()
d.mkdir(parents=True, exist_ok=True)
try:
with contextlib.suppress(OSError):
os.chmod(d, 0o700)
except OSError:
pass
return d
4 changes: 2 additions & 2 deletions src/noirdoc/daemon/protocol.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@

from __future__ import annotations

from typing import Annotated, Any, Literal, Union
from typing import Annotated, Any, Literal

from pydantic import BaseModel, Field

Expand Down Expand Up @@ -50,7 +50,7 @@ class RedactFileInput(BaseModel):


RedactInput = Annotated[
Union[RedactTextInput, RedactFileInput],
RedactTextInput | RedactFileInput,
Field(discriminator="type"),
]

Expand Down
21 changes: 8 additions & 13 deletions src/noirdoc/daemon/server.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
from __future__ import annotations

import asyncio
import contextlib
import json
import logging
import logging.handlers
Expand Down Expand Up @@ -286,7 +287,7 @@ async def handle_redact(
namespace=parsed.namespace,
namespace_root=parsed.namespace_root,
language=parsed.language,
detector=parsed.detector, # type: ignore[arg-type]
detector=parsed.detector,
score_threshold=parsed.score_threshold,
gliner_model=parsed.gliner_model,
)
Expand Down Expand Up @@ -495,10 +496,10 @@ async def _async_main() -> None:

loop = asyncio.get_running_loop()
for sig in (signal.SIGTERM, signal.SIGINT):
try:
# add_signal_handler is unsupported on Windows; we don't ship there
# but stay defensive.
with contextlib.suppress(NotImplementedError):
loop.add_signal_handler(sig, state.shutdown_event.set)
except NotImplementedError:
pass # Windows; we don't ship there but be defensive.

log.info("listening on %s", sock_path)
try:
Expand All @@ -509,23 +510,17 @@ async def _async_main() -> None:
idle_task.cancel()
warmup_task.cancel()
for task in (idle_task, warmup_task):
try:
with contextlib.suppress(asyncio.CancelledError, Exception):
await task
except (asyncio.CancelledError, Exception):
pass
try:
with contextlib.suppress(FileNotFoundError):
sock_path.unlink()
except FileNotFoundError:
pass
spawn.remove_pidfile()
log.info("daemon stopped")


def main() -> None:
try:
with contextlib.suppress(KeyboardInterrupt):
asyncio.run(_async_main())
except KeyboardInterrupt:
pass


if __name__ == "__main__":
Expand Down
Loading
Loading