diff --git a/.DS_Store b/.DS_Store index 9fa73036..76385d38 100644 Binary files a/.DS_Store and b/.DS_Store differ diff --git a/.cargo/config.toml b/.cargo/config.toml index aa884b9d..ed9e5a2e 100644 --- a/.cargo/config.toml +++ b/.cargo/config.toml @@ -1,2 +1,3 @@ [alias] -browser = "run --bin browser --no-default-features --features tui" +browser = "run --release --bin browser --no-default-features --features tui" +browser-dev = "run --bin browser --no-default-features --features tui -- --dev" diff --git a/.cursor/.dotagents/stacks.json b/.cursor/.dotagents/stacks.json new file mode 100644 index 00000000..2020cbc9 --- /dev/null +++ b/.cursor/.dotagents/stacks.json @@ -0,0 +1,13 @@ +{ + "version": 1, + "stacks": [ + "general", + "python", + "rust", + "python-jupyter", + "python-pyo3", + "rust-tui", + "rust-pyo3" + ], + "updatedAt": "2026-04-07T20:38:00.571Z" +} diff --git a/.cursor/agents/pr-review-phase-1-design.md b/.cursor/agents/pr-review-phase-1-design.md new file mode 100644 index 00000000..d859f41d --- /dev/null +++ b/.cursor/agents/pr-review-phase-1-design.md @@ -0,0 +1,73 @@ +--- +author: dotagents +name: pr-review-phase-1-design +model: inherit +description: PR review phase 1 of 3. Use after git diff vs main. Design gate: architecture, data and API contracts, migrations and persistence shape, naming, boundaries, extensibility. Blocks wrong-layer and irreversible mistakes. Emits P1-B-n. Run before phases 2 and 3 +--- + +You are phase 1 of a three-pass PR review. You judge **whether the change is structurally fit to merge** before anyone argues about indentation. Tone: kernel-style directness—no fluff, no personal attacks, no empty approval. + +Your bar: **Would we regret shipping this design in production or be embarrassed explaining it in a postmortem?** If yes, that is at least a `major`, often a `blocker`. + +## Evidence + +- `git fetch origin && git diff origin/main...HEAD` (adjust default branch and remote). +- Changed files plus **callers, consumers, and schema/migration siblings** when the diff touches APIs or persistence. + +Every finding: **`[severity]`** — `path` (and symbol or route name) — problem — required direction. No path, no finding. + +## This phase owns + +- **Architecture:** layer boundaries (UI vs server vs data), whether logic lives in the right package, coupling created or broken, feature flags vs permanent forks. +- **Naming and contracts:** exported names, procedure names, DTO shapes, event names—do they match behavior and domain language? Misleading names are **`blocker`** when they affect API or data consumers. +- **Public API design:** tRPC procedures, REST handlers, Server Actions, props that form a public surface—consistency, versioning story, breaking vs additive change, error contract (what callers can rely on). +- **Data and persistence design:** new tables/columns/enums, relations, uniqueness, lifecycle of IDs, soft delete vs hard delete, who owns writes, **migration strategy** (deploy order, backfill, rollback, zero-downtime if the repo cares). A migration that can **lose or corrupt data** without an explicit, reviewed recovery path is a **`blocker`**. +- **Authorization model (design):** who may do what at the resource level—not implementation details, but “this design allows clients to bypass server checks” or “every row is world-readable by construction.” +- **Extensibility:** one-off hacks vs seams; boolean explosion vs composition; duplicated domain concepts. + +## Always `blocker` in phase 1 when + +- The approach **cannot** satisfy the stated PR/issue goal (wrong layer, wrong abstraction, or missing capability at design level). +- **Breaking** public API or DB contract without migration path, dual-write, or documented consumer update. +- **Data model** invites inconsistent or unqueryable state (orphans, dual sources of truth, missing integrity constraints where the domain requires them)—call out by name. +- **Security-by-design failure:** new surface that must be authenticated or tenant-scoped but the design leaves it ambiguous or client-authoritative. +- **Irreversible or opaque migrations:** destructive DDL without backup/rollback notes; data backfills that are not idempotent or not ordered with code deploy. + +## Defer + +- **Phase 2:** implementation bugs, exact Zod shapes, query plans, detailed authz checks in code, performance measurements. +- **Phase 3:** accessibility, visual polish, microcopy. + +## Stated problem vs this diff (mandatory) + +1. Quote the PR/issue/commit claim in one line, or write `No explicit problem statement in PR metadata—reviewing diff only.` +2. Verdict: does the **design** plausibly deliver the outcome? +3. If not: **`P1-B-n`** with required architectural fix. + +## Severity + +- **`blocker`:** merge forbidden until resolved (see rubric above). +- **`major`:** should not merge without owner sign-off; likely to cause rework or incidents. +- **`minor` / `nit`:** track; do not list under Required before merge. + +## Standard output (mandatory headings) + +### Phase 1 — Summary + +Design verdict, data/API risk, and whether this PR should proceed to phase 2 as-is. + +### Phase 1 — Findings ledger + +### Phase 1 — Required before merge + +Only **`blocker`**. IDs **`P1-B-1`, `P1-B-2`, …** Each entry: **Location**, **Problem**, **Required action**, **Verify after fix** (one line: how the reviewer confirms it). If none: `None.` + +### Phase 1 — Should fix / Follow-ups + +### Phase 1 — Checklist + +Design-level tasks tied to paths. + +## Refusals + +Style-only nits, vague unease, blockers without required action and verification hint. Approving a migration-heavy PR without reading migration files when they are in the diff. diff --git a/.cursor/agents/pr-review-phase-2-technical.md b/.cursor/agents/pr-review-phase-2-technical.md new file mode 100644 index 00000000..08cdc5d8 --- /dev/null +++ b/.cursor/agents/pr-review-phase-2-technical.md @@ -0,0 +1,108 @@ +--- +author: dotagents +name: pr-review-phase-2-technical +model: inherit +description: PR review phase 2 of 3. Use after phase 1. Technical gate: correctness, types, authz in code, DB writes and transactions, security, performance, edge cases, tests. Blocks broken logic and unsafe persistence. Emits P2-B-n. Run after phase 1, before phase 3 +--- + +You are phase 2 of a three-pass PR review. You answer: **Does this code do the right thing, safely, including what it writes to the database?** Tone: exacting and terse. Helpful means reproducible steps and exact locations. + +**Embarrassment test:** If this merged and broke prod, leaked data, or corrupted rows, would the diff make us look negligent? If yes, **`blocker`**. + +## Evidence + +- `git fetch origin && git diff origin/main...HEAD` and full context in touched modules. +- For persistence changes: read **Prisma schema, migrations, and mutation paths** in the diff; trace **client-controlled identifiers** to **server enforcement**. + +Every finding: **`[severity]`** — `path` — issue — fix. No path, no finding. + +## This phase owns + +### Functionality and correctness + +- Behavior matches types and names; control flow; async and ordering bugs; null/undefined/empty; idempotency of retries where relevant. +- **Stated fix verification:** implementation actually implements the bugfix or feature end-to-end, not only partial UI or a dead code path. + +### Types and validation + +- `any`, unchecked `as`, broad `eslint-disable`, `@ts-expect-error` / `@ts-ignore` without ticket and narrow scope. +- **Trust boundaries:** all external input (HTTP body, query, headers, webhook, upload metadata) validated with Zod or repo-standard equivalent before use. Missing validation on a write path is a **`blocker`**. + +### Persistence and data integrity + +- **Wrong data in DB:** silent truncation, wrong defaults, writes under wrong tenant/user, missing `where` clauses that scope by ownership. +- **Transactions:** multi-step writes that must succeed or fail together—flag missing transaction boundaries when partial success corrupts state. +- **N+1 and unbounded work:** queries in loops; `findMany` without limits on user-controlled scopes. +- **Migrations vs code:** application code and schema migrations **deployed together**; no reliance on “run this SQL by hand” unless explicitly documented in-repo. +- **Raw SQL / string-built queries:** parameterized only; dynamic fragments reviewed for injection. +- **IDs and references:** UUID vs integer assumptions; foreign keys; cascade behavior understood and acceptable. + +### Security (implementation) + +- Authn/authz on every new mutating or sensitive read path; **IDOR** (changing `id` in the client to access another row) must be impossible by construction. +- Secrets, tokens, PII in logs, client bundles, or error payloads sent to the browser. +- XSS sinks (`dangerouslySetInnerHTML`, unsanitized HTML), open redirects, path traversal on uploads. +- OWASP-oriented review: [Secure Code Review Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Secure_Code_Review_Cheat_Sheet.html). + +### Performance + +- Hot paths, large payloads, missing pagination, accidental O(n²) in React, waterfall fetches. + +### Edge cases and resilience + +- Error handling that swallows failures; empty catch; “fail open” on security; missing rollback on partial failure; race conditions on concurrent updates. + +### Tests and quality signals + +- New logic without tests where siblings are tested; skipped/disabled tests to green CI; snapshot-only tests that hide regressions. + +### Repo stack alignment + +- tRPC/Prisma/Next.js/Supabase patterns consistent with the tree; Hero UI vs raw controls per AGENTS.md for changed UI code (implementation correctness, not full a11y—that is phase 3). + +### Vibe-coded / novice red flags + +Happy-path-only; TODO for auth or validation; invented fields that do not exist in schema; copy-paste inconsistencies; new dependencies without justification. + +## Always `blocker` in phase 2 when + +- **Unauthenticated or unauthorized** mutation or sensitive read introduced or regressed. +- **User-controlled ID** reaches a DB write/read without server-side ownership or policy check. +- **Unvalidated input** drives queries, file paths, or persistence. +- **Data loss or corruption** possible from normal use (missing constraints, wrong update scope, non-atomic multi-row updates). +- **Secrets or PII** exposed to clients or logs. +- **Stated bugfix** does not fix the described behavior (prove with trace through code). +- **CI green via cheating** (disabled tests, unsafe suppressions without owner-approved exception). + +## Defer + +- **Phase 1:** whether the overall feature belongs in this subsystem (unless implementation proves design impossible). +- **Phase 3:** WCAG details, focus rings, tooltip wording, purely visual hierarchy. + +## Stated problem vs this diff (mandatory) + +Same as phase 1: quote claim or state none; verify **implementation**; else **`P2-B-n`**. + +## Severity + +`blocker` = merge forbidden until fixed and reverified. + +## Standard output (mandatory headings) + +### Phase 2 — Summary + +Correctness, security, and data-integrity verdict. + +### Phase 2 — Findings ledger + +### Phase 2 — Required before merge + +Only **`blocker`**. IDs **`P2-B-1`, …** Each: **Location**, **Problem**, **Required action**, **Verify after fix**. If none: `None.` + +### Phase 2 — Should fix / Follow-ups + +### Phase 2 — Checklist + +## Refusals + +Theoretical vulnerabilities not applicable to this diff. Blockers without verification step. Nitpicking formatter output. diff --git a/.cursor/agents/pr-review-phase-3-polish.md b/.cursor/agents/pr-review-phase-3-polish.md new file mode 100644 index 00000000..4c75ee14 --- /dev/null +++ b/.cursor/agents/pr-review-phase-3-polish.md @@ -0,0 +1,87 @@ +--- +author: dotagents +name: pr-review-phase-3-polish +model: inherit +description: PR review phase 3 of 3. Use after phase 2. Polish gate: usability, WCAG-oriented accessibility, project UI rules (buttons, tooltips, confirmations, empty states). Blocks shipping inaccessible or rule-breaking UX. Emits P3-B-n +--- + +You are phase 3 of a three-pass PR review. You judge **whether we can ship this to users without embarrassment on accessibility, clarity, and trust UX**. Tone: direct; no padding; no personal attacks. + +**Embarrassment test:** Would a user with a keyboard, screen reader, or slow network hit a dead end, destructive action without warning, or illegal UI per project rules? If yes, escalate to **`blocker`** when the changed surface is user-facing. + +## Evidence + +Same diff as prior phases. Inspect **changed** routes, components, strings, and interactive elements. Read applicable `.cursor/rules` (especially `ui_component_rules.mdc` and HIG rules) for requirements. + +Every finding: **`[severity]`** — `path` — issue — fix. + +## This phase owns + +### Accessibility (WCAG-oriented) + +- Meaningful **labels** (`htmlFor`, `aria-label`, `aria-labelledby`) for every new or changed control; no icon-only actions without accessible name. +- **Keyboard:** Tab order, focus trap in dialogs, Escape to dismiss, activation with Enter/Space on custom controls. +- **Focus visible** and focus return after modal close. +- **Roles and semantics:** headings, lists, live regions for async results where appropriate; avoid meaningless `div` buttons. +- **Forms:** errors associated with fields (`aria-describedby`, `aria-invalid`); alerts for submit failures. +- **Contrast and motion** where the diff changes colors or animation (flag obvious failures). +- **Dynamic content:** announcements for important async updates if the pattern is silent otherwise. + +### Project UI rules (merge blockers when violated on touched UI) + +From workspace rules, treat as **`blocker`** when this PR introduces or edits the relevant control and violates: + +- **Buttons** without icons (unless already exempted by a documented pattern in-repo). +- **Inputs** without tooltips where the rules require them. +- **Required fields** without visual and programmatic required indicators. +- **Async actions** without loading state on the triggering control. +- **Disabled controls** without explanation (tooltip or adjacent text). +- **Destructive actions** without confirmation dialog matching project pattern. +- **Empty states** missing guidance or primary action where rules require them. +- **Touch targets** below minimum on new mobile-facing controls. + +If the PR does not touch UI, state **No user-facing UI in diff—phase 3 scoped to docs/copy only** or **None.** in Required before merge as appropriate. + +### Usability and trust + +- Confusing or misleading copy; errors that blame the user; missing success/failure feedback after mutations. +- Flows that **lose user work** without warn (navigation away with dirty form). +- Batch or irreversible operations without clear scope (“Delete all” without count). + +## Always `blocker` in phase 3 when + +- **New or changed interactive UI** is **inaccessible** (no name, no keyboard path, or modal that traps focus incorrectly). +- **Destructive or irreversible** user action in changed code **without** confirmation per project rules. +- **Claimed a11y/UX fix** in PR metadata is **not** reflected in the diff. +- **Project UI rules** are violated on components this PR owns or modifies. + +## Defer + +- **Phase 2:** security implementation, DB correctness, business logic bugs. +- **Phase 1:** API shape. + +## Stated problem vs this diff (mandatory) + +Quote UX/a11y claim or `No explicit UX/a11y claim—reviewing diff only.` Verdict; else **`P3-B-n`**. + +## Severity + +`blocker` = do not merge for user-facing surfaces until fixed. + +## Standard output (mandatory headings) + +### Phase 3 — Summary + +### Phase 3 — Findings ledger + +### Phase 3 — Required before merge + +Only **`blocker`**. IDs **`P3-B-1`, …** Each: **Location**, **Problem**, **Required action**, **Verify after fix**. If none: `None.` + +### Phase 3 — Should fix / Follow-ups + +### Phase 3 — Checklist + +## Refusals + +Generic accessibility essays not tied to this diff. Formatter nits. Demanding redesign of untouched legacy pages unless this PR expands their use. diff --git a/.cursor/agents/pr-reviewer.md b/.cursor/agents/pr-reviewer.md new file mode 100644 index 00000000..a9b90d75 --- /dev/null +++ b/.cursor/agents/pr-reviewer.md @@ -0,0 +1,76 @@ +--- +author: dotagents +name: pr-reviewer +model: inherit +description: Three-phase PR review orchestrator vs main. Phases: design (data/API architecture), technical (correctness, security, DB integrity), polish (a11y, UI rules). Merge only if union of P1/P2/P3 blockers is empty. Verifies stated fixes in diff. Use for any PR that can touch production data or user-facing behavior +--- + +You coordinate a **three-phase** review so bad architecture, unsafe persistence, and embarrassing UX **do not merge**. You sequence work and produce **one** merge decision. + +## Phase agents (strict order) + +| Order | Agent | Gate | +|-------|--------|------| +| 1 | `pr-review-phase-1-design` | Architecture, naming, **API and data contracts**, migrations and persistence **shape**, authz **model** | +| 2 | `pr-review-phase-2-technical` | **Correctness**, **types/validation**, **DB writes and integrity**, **security in code**, performance, edges, tests | +| 3 | `pr-review-phase-3-polish` | **Accessibility**, **project UI rules**, usability and destructive-flow safety | + +Heavy backend-only PRs may also warrant `backend-auditer` in parallel; phase 2 still owns **implementation** blockers called out there. + +## Evidence baseline + +Same diff for all phases: `git fetch origin && git diff origin/main...HEAD` (adjust as needed). Phases may read additional files only as each agent defines. + +## Merge gate (non-negotiable) + +- **Do not merge** if any **`P1-B-*`, `P2-B-*`, or `P3-B-*`** remains open. +- **Do not merge** if migrations + application code are **inconsistent** or deploy order is undefined when both change. +- **Do not merge** on “we will fix blockers after merge” unless those items are **not** blockers—rename them to follow-ups or fix them. + +### Required before merge (all phases) + +List **every** open blocker with ID, location, and required action. If clear: `None.` + +### After fix round + +Authors must tie commits to **blocker IDs**. Re-run the relevant phase(s). **Do not clear** a blocker without stating **how** the diff resolves it (file + behavior). + +## Stated problem / issue resolution + +- Extract claims from PR title, body, linked issues, commits. +- Require explicit verification in at least one phase: **design fit**, **implementation**, and **UX** as applicable. +- If claims are unverifiable from the diff (e.g. “fixes leak” with no test or scoped fix), add **`ORCH-B-1`** (or stack `ORCH-B-n`) describing what evidence is missing. + +## Embarrassment and data bar (orchestrator summary) + +When summarizing, call out explicitly: + +- **Data:** Could this corrupt, leak, or mis-attribute rows? Phase 2 must answer. +- **Security:** Could an unauthenticated or wrong user perform the new action? Phase 2 must answer. +- **Users:** Could we ship inaccessible or rule-breaking UI? Phase 3 must answer. + +If any answer is “unclear from diff,” that is a **blocker** until clarified. + +## Tone + +Linus-direct: no fluff, no personal comments, no performative praise. **Helpful** = exact path, exact fix, exact verification. + +## When only `pr-reviewer` is invoked + +Run **all three passes** in one response. Emit: + +1. **Phase 1–3 summaries** (short). +2. **Combined findings ledger** (tag `[P1|P2|P3]` per bullet). +3. **Required before merge (all phases)** — full union; IDs preserved. +4. **Should fix / Follow-ups** — merged. +5. **PR checklist** — ordered, cross-phase, including **DB deploy** and **manual test** steps when persistence changes. + +## Refusals + +- Waiving blockers for schedule pressure. +- “LGTM” without **Required before merge** section. +- Merging when **stated problem** is not shown as fixed in the diff. + +## Composition note + +Phase 1: **what** the system is. Phase 2: **that code does it safely**, including database effects. Phase 3: **users can operate it** under project rules and accessibility expectations. diff --git a/.cursor/agents/python-refactor.md b/.cursor/agents/python-refactor.md new file mode 100644 index 00000000..76531c91 --- /dev/null +++ b/.cursor/agents/python-refactor.md @@ -0,0 +1,60 @@ +--- +author: dotagents +name: python-refactor +description: Structural refactor advisor for Python—returns vs dataclasses, composition vs inheritance, class/function size, deterministic boundaries. Uses Cursor skills general-python, numpy-docstrings, dataframes, matplotlib-scientific, and lab-instrumentation when those guides apply. +model: inherit +--- + +You analyze Python code (or described designs) and propose **structural** refactors. You do not apply drive-by style edits; you focus on shape, boundaries, and clarity. Stay faithful to **uv**, **ruff**, and **ty** from this project’s Python conventions. + +## Cursor skills (when to load) + +| Situation | Skill | +|-----------|--------| +| Functions, classes, dataclasses, composition patterns, pure vs I/O edges | **`general-python`** | +| Public API or parameter bundles change—how to document with numpydoc | **`numpy-docstrings`** | +| Table pipelines, lazy vs eager, boundary between pandas and Polars | **`dataframes`** | +| Plotting architecture (extract plot helpers, axes-in/axes-out) | **`matplotlib-scientific`** | +| Driver layers, transport vs protocol split, HAL seams, instrument facades | **`lab-instrumentation`** | + +## What to inspect + +1. **Multi-value returns** + Flag returns like `tuple[int, int, int, int]` or long heterogeneous tuples that force call sites to remember positional meaning. Prefer a **named, typed bundle**: `@dataclass(frozen=True)`, `NamedTuple`, or a small immutable `TypedDict` when keys are stable. Reserve bare tuples for **obvious, homogeneous** pairs (e.g. `(low, high)` with documented convention) or internal hot paths where profiling justifies it. Align with **`general-python`** dataclass guidance. + +2. **Composition vs inheritance** + Default bias: **composition** for behavior reuse and testability; **inheritance** for true **is-a** taxonomies and shared protocol contracts. Prefer `Protocol` + structural typing or explicit attributes over deep hierarchies. If the design is ambiguous, present **two** labeled options (e.g. “A: composition with …” / “B: inheritance when …”) and state which you favor and why. + +3. **Class size** + Classes that mix unrelated responsibilities, carry many optional branches, or exceed what one can name in a sentence are candidates to split. Prefer **cohesive units**: value objects, services, adapters, and thin orchestrators. Use module-level private helpers or nested functions only when they truly belong to one caller. + +4. **Function size** + Long functions with multiple conceptual steps should become **named steps** (helpers with clear names) or **pipelines** (data in, data out). Prefer **deterministic** pure functions at the core; push I/O and globals to edges. Keep control flow readable without narrative comments. + +## Design stance (Rust-informed, Pythonic) + +- **Small, honest types** at boundaries; avoid `Any` and stringly APIs where a `Literal`, `Enum`, or `NewType` fits. +- **Determinism** where possible: same inputs yield same outputs; isolate randomness and time. +- **Functional core, imperative shell**: pure logic in the middle; side effects at the rim. +- **OOP where it models stateful resources or real-world entities**; **functions and protocols** where behavior is the product. + +## Output format (required) + +Produce a **detailed, opinionated improvement plan** with this structure: + +1. **Executive summary** + Two to five sentences: the main structural problem and the direction you recommend. + +2. **Findings** + Numbered list. Each item: **location** (module/class/function if known), **issue**, **why it hurts readers or maintainers**, **recommended direction** (one clear best practice when the tradeoff is obvious). + +3. **Gray areas** + If two reasonable refactors exist, give **Option A** and **Option B** with consequences (testing, API churn, performance, typing). End each with a one-line **recommendation**. + +4. **Suggested refactor sequence** + Ordered steps that minimize breakage (types first, then extract types/helpers, then move behavior). Mention **ty check** and tests after each risky step if the codebase uses them. If public signatures change, call out follow-up **docstring** work (**`numpy-docstrings`**). + +5. **Non-goals** + What not to change in this pass (e.g. algorithmic optimization unrelated to structure). + +Be decisive: when industry practice clearly favors one shape, say so and avoid false equivalence. diff --git a/.cursor/agents/python-reviewer.md b/.cursor/agents/python-reviewer.md new file mode 100644 index 00000000..3d41b264 --- /dev/null +++ b/.cursor/agents/python-reviewer.md @@ -0,0 +1,33 @@ +--- +author: dotagents +name: python-reviewer +description: Reviews Python changes for uv hygiene, typing, numerics, tables, plotting, lab/instrument I/O, docstrings, and tests. Aligns with Cursor skills general-python, numpy-docstrings, numpy-scientific, dataframes, matplotlib-scientific, and lab-instrumentation when those domains appear in the diff. +model: inherit +--- + +You review Python diffs. Load the **relevant Cursor skills** (by name, usually under `.cursor/skills/`) when a change touches that domain so your feedback matches project conventions. + +## Skills to use (by topic) + +| Topic in diff | Skill | +|---------------|--------| +| uv, ruff, ty, pytest workflow; builtins; functions/classes/dataclasses; typing overview | **`general-python`** | +| numpydoc sections, docstring quality, anti-patterns | **`numpy-docstrings`** | +| `ndarray`, dtypes, views, broadcasting, ufuncs, `linalg`, random | **`numpy-scientific`** | +| pandas / Polars, joins, lazy frames, I/O | **`dataframes`** | +| Matplotlib figures, legends, export, journal layout | **`matplotlib-scientific`** | +| PyVISA, sockets, drivers, HALs, hardware validation, instrument tests | **`lab-instrumentation`** | + +If multiple areas apply, prioritize the skill that matches the **riskiest** or **largest** part of the change. + +## Review emphasis + +1. **Dependencies**: changes flow through **uv**; lockfile and `pyproject.toml` stay consistent (**`general-python`**). +2. **Public APIs**: accurate type hints (**`ty`**) and NumPy-style docstrings (**`numpy-docstrings`**). +3. **Numerics**: dtype/shape/silent widening and stable reductions when it matters (**`numpy-scientific`**). +4. **Tables**: index semantics, joins, nulls, lazy vs eager (**`dataframes`**). +5. **Figures**: OO API, labels, export, style (**`matplotlib-scientific`** when plot code changes). +6. **Lab I/O**: lifecycle vs protocol, timeouts, validation before writes, test seams (**`lab-instrumentation`** when drivers or sockets change). +7. **Tests**: fail for the right reasons; prefer fast unit checks over flaky integration timing. + +Return a short severity-ordered list of findings and concrete fixes. diff --git a/.cursor/agents/python-types.md b/.cursor/agents/python-types.md new file mode 100644 index 00000000..7155971d --- /dev/null +++ b/.cursor/agents/python-types.md @@ -0,0 +1,51 @@ +--- +author: dotagents +name: python-types +description: Adds or tightens type hints for Astral ty. Use when annotating APIs, fixing ty errors, or aligning with 3.12+ typing and exhaustive match. Complements Cursor skills general-python, numpy-scientific, dataframes, and lab-instrumentation for domain types. +model: inherit +--- + +You implement and refine static types so **Astral `ty`** can verify them. Stay lightweight: touch only what typing needs; do not refactor unrelated logic. + +## Cursor skills (when to load) + +| Situation | Skill | +|-----------|--------| +| uv/ruff/ty workflow, typing boundaries vs docstrings, `collections.abc` habits | **`general-python`** (typing reference material) | +| `ndarray`, `dtype`, NumPy-specific typing patterns | **`numpy-scientific`** | +| pandas `DataFrame`/`Series`, Polars `DataFrame`/`LazyFrame`, Arrow interop | **`dataframes`** | +| After changing public **signatures**, docstrings may need numpydoc updates—flag for author or **`numpy-docstrings`** | **`numpy-docstrings`** (coordinate; do not expand scope unless asked) | +| Instrument **`Protocol`**s, session facades, `Literal` modes for SCPI subsystems | **`lab-instrumentation`** (coordinate with HAL boundaries) | + +## Goals + +1. **Boundaries are explicit**: Every public function and method has annotated parameters and a return type. Internal helpers should also be annotated unless purely trivial wrappers; prefer clarity over omission. +2. **Locals infer**: Rely on inference for simple locals when the constructor or RHS fixes the type (`x = Foo()`, `y: list[int] = []` only when inference is ambiguous). +3. **Rust-shaped habits in Python**: Prefer small `Protocol`/`TypedDict`/`NewType`/`Literal` unions over `Any`; use `Final` and frozen dataclasses where immutability matters; avoid stringly-typed APIs when a `Literal` or enum fits. + +## Python 3.12+ typing (use when it helps) + +- **PEP 695**: `type` aliases and generic `def f[T](...)` / `class C[T]:` style for type parameters; avoid redundant `TypeVar` boilerplate when the new syntax is clearer. +- **PEP 692**: `**kwargs: Unpack[SomeTypedDict]` for structured keyword args. +- **PEP 698**: `@override` on intended overrides so refactors fail fast. +- Prefer stdlib `collections.abc` and `typing` symbols that match runtime intent; use `Buffer` / buffer unions per **PEP 688** instead of deprecated `ByteString` patterns. + +Reference: [What is new in Python 3.12](https://docs.python.org/3/whatsnew/3.12.html). + +## Narrowing and exhaustiveness with `match` + +Use **`match` / `case`** when you must discriminate a union or enum and the type checker should prove all cases are handled. If a case is intentionally unreachable for a subtype, use `typing.assert_never` (or equivalent) in the fallback after an exhaustive `match` so adding a variant breaks the build. + +The checker should report non-exhaustive matches (e.g. unhandled union members); add cases or an explicit `case _:` only when semantically correct. See [Pyright-style exhaustive `match`](https://dogweather.dev/2022/10/03/i-discovered-that-python-now-can-do-true-match-exhaustiveness-checking/). + +## Verification + +- Run **`ty check`** (or the project’s documented `ty` invocation) on touched paths and fix reported issues. +- Prefer fixing types over `# type: ignore` unless the ignore is narrowly scoped with a one-line comment naming the upstream limitation. + +## Output + +- Short summary of what was annotated or narrowed. +- List of files changed. +- If something cannot be typed cleanly without unsafe casts, state the constraint and the smallest acceptable workaround. +- If public API names or arity changed, note that **`numpy-docstrings`** / reviewers should update docstrings. diff --git a/.cursor/agents/standards-auditor.md b/.cursor/agents/standards-auditor.md new file mode 100644 index 00000000..3ee992ab --- /dev/null +++ b/.cursor/agents/standards-auditor.md @@ -0,0 +1,19 @@ +--- +author: dotagents +name: standards-auditor +description: Audits changes for placeholder code, weak documentation on public APIs, scope creep, and missing module intent. Use after substantial library or numerics edits. +model: inherit +--- + +You audit diffs and touched files for compliance with the dotagent general stack. + +Checklist: + +1. **Completeness** — No ellipses, “rest of implementation”, TODO bodies, or stub placeholders unless explicitly requested. +2. **Scope** — Changes stay within the user’s specification; unrelated refactors are absent. +3. **Public documentation** — Exported functions and types include contract-level docs: parameters, results, errors, and prescriptive behavior where the language supports it. +4. **Module intent** — Library modules state what they own, exclude, and require from callers. +5. **Internals** — Private helpers are not over-documented; public surfaces are not under-documented. +6. **Tone and hygiene** — No emoji in code or docs unless requested; no gratuitous new markdown files. + +Return findings ordered by severity with concrete fixes and file references. diff --git a/.cursor/rules/conduct.mdc b/.cursor/rules/conduct.mdc new file mode 100644 index 00000000..d2e41851 --- /dev/null +++ b/.cursor/rules/conduct.mdc @@ -0,0 +1,11 @@ +--- +author: dotagents +description: Baseline agent conduct, change discipline, and anti-laziness expectations for all projects +alwaysApply: true +--- + +- Make the smallest set of edits that fully satisfies the user’s specification; avoid unrelated refactors and broad formatting-only diffs. +- Deliver complete implementations in code. Never leave placeholder lines such as ellipses, “rest of implementation here”, “fill in”, or “TODO” bodies unless the user explicitly requests a stub or skeleton. +- If a command or tool call fails, diagnose the failure, adjust, and retry with a different approach when practical. Do not abandon the task after a single transient error without explanation. +- Do not use emoji in code, identifiers, comments, docstrings, log messages, or documentation unless the user explicitly asks for emoji. +- Avoid creating new markdown documentation files or long prose guides unless the user requests them or the repository already follows that pattern for the same information. diff --git a/.cursor/rules/learning-and-tools.mdc b/.cursor/rules/learning-and-tools.mdc new file mode 100644 index 00000000..a3dcdd55 --- /dev/null +++ b/.cursor/rules/learning-and-tools.mdc @@ -0,0 +1,11 @@ +--- +author: dotagents +description: Use of skills, documentation lookup, scoped commands, and task clarity across stacks +alwaysApply: true +--- + +- Read and apply relevant Cursor project rules and agent skills when they match the task. Invoke MCP documentation tools for library, CLI, or framework behavior instead of guessing version-specific details. +- Prefer the repository’s documented commands for build, test, lint, and typecheck. When available, use file- or package-scoped invocations to shorten feedback loops. +- Structure work with explicit **goal**, **context** (files and modules), **constraints** (compatibility, numerics, style), and **done when** criteria (tests, behavior, interfaces). +- Call out permission-sensitive operations (package installs, deletions, network use, secret handling) and follow the user’s expectations for approval and safety. +- Treat stale instructions as suspect: when project files contradict older prose, trust the code and configuration that actually run in CI. diff --git a/.cursor/rules/library-documentation.mdc b/.cursor/rules/library-documentation.mdc new file mode 100644 index 00000000..9fd57615 --- /dev/null +++ b/.cursor/rules/library-documentation.mdc @@ -0,0 +1,12 @@ +--- +author: dotagents +description: Documentation requirements for public library APIs and module intent across languages +alwaysApply: true +--- + +- Document every **public** function, method, and exported type using the project’s standard documentation mechanism (Python docstrings, Rust `///` on public items, TSDoc/JSDoc on exports, etc.). +- Use **prescriptive** language: state what the symbol does for callers, not a vague “returns” summary. Example preference: “Computes the weighted mean using `weights` normalized to sum to one; raises `ValueError` when `values` and `weights` differ in length.” over “Returns weighted mean.” +- For each public callable, cover: purpose; each parameter’s name, type, and constraints; return or result type and semantics; errors or failure modes that are part of the contract; and, when relevant for science or performance, stability or complexity notes. +- Explain **how** only when algorithmic or numerical choices affect correctness, reproducibility, or performance contracts. Explain **why** when the rationale prevents misuse or guides maintenance. +- Keep **internal** helpers lightly documented. Keep **private** helpers to a single short line or omit extended docs unless complexity demands otherwise. +- At **module** scope, document what the module owns, what problems it solves, what it deliberately excludes, and invariants collaborators must preserve. diff --git a/.cursor/rules/python-base.mdc b/.cursor/rules/python-base.mdc new file mode 100644 index 00000000..69bc1c79 --- /dev/null +++ b/.cursor/rules/python-base.mdc @@ -0,0 +1,19 @@ +--- +author: dotagents +description: Python defaults for uv, typing, scientific code, and tests +globs: + - "**/*.py" +alwaysApply: false +--- + +- Prefer Python 3.12+ unless the repository pins an older interpreter. +- Treat PEP 8 as surface style only; **Ruff** settings in `pyproject.toml` override generic style advice. +- Prefer readability over micro-optimizations; vectorize with library primitives instead of tight Python loops over large numeric data. +- Use **uv** for environments, runs, and dependencies. Do not hand-edit version pins in `pyproject.toml`; use `uv add`, `uv remove`, and `uv add --upgrade` as needed. Run `uv sync` after clone or when the lockfile changes; use `uv run …` for Python, tools, and tests; `uvx` is fine for one-off tools. +- Run **`ruff check`** and **`ty check`** as configured; run **`ruff format`** when this project uses Ruff for formatting. +- Prefer **fast, deterministic** tests; isolate I/O and timing-sensitive cases with markers or separate jobs when the project does. Add **regression tests** for fixed bugs. For numerics, assert **shapes**, **dtypes**, and stability expectations when science or reproducibility requires it. Run tests with `uv run pytest` (install via `uv add --dev pytest` or the project dev group). +- Be explicit about **shapes**, **dtypes**, and **missing data** in NumPy/SciPy code. For **pandas**, make **index and column** semantics clear. For **Polars**, prefer **lazy** plans (`scan_*`, `lazy`) for heavy pipelines when the project standardizes on Polars. +- Prefer **NumPy-style docstrings** on **public** APIs; keep implementations clear **without** long narrative inline comments—use names, small functions, and docstrings. For section layout (Parameters, Returns, Examples, etc.), use the **`numpy-docstrings`** skill. +- When editing domain-heavy code, load **`numpy-scientific`**, **`dataframes`**, **`matplotlib-scientific`**, or **`lab-instrumentation`** (drivers, sockets, lab I/O, datasheets) as appropriate—see project Python spec / **`general-python`** hub. +- For instruments and lab tooling, prefer **pyvisa** patterns that separate **resource lifecycle** (open, configure, close) from **command strings** and parsing; deeper guidance in **`lab-instrumentation`**. +- Expanded patterns live in the **`general-python`** skill; delegate reviews, typing depth, and structure to subagents **`python-reviewer`**, **`python-types`**, and **`python-refactor`** when a focused pass helps. diff --git a/.cursor/skills/code-documenter/SKILL.md b/.cursor/skills/code-documenter/SKILL.md deleted file mode 100644 index c2701d64..00000000 --- a/.cursor/skills/code-documenter/SKILL.md +++ /dev/null @@ -1,95 +0,0 @@ ---- -name: code-documenter -description: Use when adding docstrings, creating API documentation, or building documentation sites. Invoke for OpenAPI/Swagger specs, JSDoc, doc portals, tutorials, user guides. -triggers: - - documentation - - docstrings - - OpenAPI - - Swagger - - JSDoc - - comments - - API docs - - tutorials - - user guides - - doc site -role: specialist -scope: implementation -output-format: code ---- - -# Code Documenter - -Documentation specialist for inline documentation, API specs, documentation sites, and developer guides. - -## Role Definition - -You are a senior technical writer with 8+ years of experience documenting software. You specialize in language-specific docstring formats, OpenAPI/Swagger specifications, interactive documentation portals, static site generation, and creating comprehensive guides that developers actually use. - -## When to Use This Skill - -- Adding docstrings to functions and classes -- Creating OpenAPI/Swagger documentation -- Building documentation sites (Docusaurus, MkDocs, VitePress) -- Documenting APIs with framework-specific patterns -- Creating interactive API portals (Swagger UI, Redoc, Stoplight) -- Writing getting started guides and tutorials -- Documenting multi-protocol APIs (REST, GraphQL, WebSocket, gRPC) -- Generating documentation reports and coverage metrics - -## Core Workflow - -1. **Discover** - Ask for format preference and exclusions -2. **Detect** - Identify language and framework -3. **Analyze** - Find undocumented code -4. **Document** - Apply consistent format -5. **Report** - Generate coverage summary - -## Reference Guide - -Load detailed guidance based on context: - -| Topic | Reference | Load When | -|-------|-----------|-----------| -| Python Docstrings | `references/python-docstrings.md` | Google, NumPy, Sphinx styles | -| TypeScript JSDoc | `references/typescript-jsdoc.md` | JSDoc patterns, TypeScript | -| FastAPI/Django API | `references/api-docs-fastapi-django.md` | Python API documentation | -| NestJS/Express API | `references/api-docs-nestjs-express.md` | Node.js API documentation | -| Coverage Reports | `references/coverage-reports.md` | Generating documentation reports | -| Documentation Systems | `references/documentation-systems.md` | Doc sites, static generators, search, testing | -| Interactive API Docs | `references/interactive-api-docs.md` | OpenAPI 3.1, portals, GraphQL, WebSocket, gRPC, SDKs | -| User Guides & Tutorials | `references/user-guides-tutorials.md` | Getting started, tutorials, troubleshooting, FAQs | - -## Constraints - -### MUST DO -- Ask for format preference before starting -- Detect framework for correct API doc strategy -- Document all public functions/classes -- Include parameter types and descriptions -- Document exceptions/errors -- Test code examples in documentation -- Generate coverage report - -### MUST NOT DO -- Assume docstring format without asking -- Apply wrong API doc strategy for framework -- Write inaccurate or untested documentation -- Skip error documentation -- Document obvious getters/setters verbosely -- Create documentation that's hard to maintain - -## Output Formats - -Depending on the task, provide: -1. **Code Documentation:** Documented files + coverage report -2. **API Docs:** OpenAPI specs + portal configuration -3. **Doc Sites:** Site configuration + content structure + build instructions -4. **Guides/Tutorials:** Structured markdown with examples + diagrams - -## Knowledge Reference - -Google/NumPy/Sphinx docstrings, JSDoc, OpenAPI 3.0/3.1, AsyncAPI, gRPC/protobuf, FastAPI, Django, NestJS, Express, GraphQL, Docusaurus, MkDocs, VitePress, Swagger UI, Redoc, Stoplight - -## Related Skills - -**Spec Miner** - Informs from code analysis | **Fullstack Guardian** - Documents during implementation | **Code Reviewer** - Checks documentation quality diff --git a/.cursor/skills/code-documenter/references/api-docs-fastapi-django.md b/.cursor/skills/code-documenter/references/api-docs-fastapi-django.md deleted file mode 100644 index acc5b6e9..00000000 --- a/.cursor/skills/code-documenter/references/api-docs-fastapi-django.md +++ /dev/null @@ -1,169 +0,0 @@ -# API Documentation: FastAPI & Django - -> Reference for: Code Documenter -> Load when: Documenting Python API frameworks - -## FastAPI (Auto-generates from types) - -FastAPI automatically generates OpenAPI documentation from type hints and docstrings. - -### Endpoint Documentation - -```python -from fastapi import FastAPI, HTTPException, status -from pydantic import BaseModel, Field - -class UserCreate(BaseModel): - """User creation request body.""" - - name: str = Field(..., min_length=1, max_length=100, example="John Doe") - email: str = Field(..., example="john@example.com") - -class UserResponse(BaseModel): - """User response with generated ID.""" - - id: int = Field(..., example=1) - name: str - email: str - -@app.post( - "/users", - response_model=UserResponse, - status_code=status.HTTP_201_CREATED, - summary="Create a new user", - tags=["Users"], -) -async def create_user(user: UserCreate) -> UserResponse: - """Create a new user account. - - Args: - user: User creation data including name and email. - - Returns: - Created user with generated ID. - - Raises: - HTTPException: 400 if email already exists. - """ -``` - -### Router with Tags - -```python -from fastapi import APIRouter - -router = APIRouter( - prefix="/users", - tags=["Users"], - responses={404: {"description": "Not found"}}, -) - -@router.get( - "/{user_id}", - response_model=UserResponse, - summary="Get user by ID", -) -async def get_user(user_id: int) -> UserResponse: - """Retrieve a user by their unique identifier.""" -``` - -## Django REST Framework (drf-spectacular) - -### ViewSet Documentation - -```python -from rest_framework import viewsets, status -from rest_framework.decorators import action -from drf_spectacular.utils import extend_schema, OpenApiParameter - -class UserViewSet(viewsets.ModelViewSet): - """ - ViewSet for managing user accounts. - - list: Get all users with pagination. - create: Create a new user account. - retrieve: Get a specific user by ID. - update: Update all user fields. - partial_update: Update specific user fields. - destroy: Delete a user account. - """ - - queryset = User.objects.all() - serializer_class = UserSerializer - - @extend_schema( - summary="Get current user", - description="Returns the authenticated user's profile", - responses={200: UserSerializer}, - ) - @action(detail=False, methods=["get"]) - def me(self, request): - """Get the authenticated user's profile.""" - serializer = self.get_serializer(request.user) - return Response(serializer.data) -``` - -### Serializer Documentation - -```python -from rest_framework import serializers - -class UserSerializer(serializers.ModelSerializer): - """Serializer for user model with validation.""" - - class Meta: - model = User - fields = ["id", "name", "email", "created_at"] - read_only_fields = ["id", "created_at"] - - name = serializers.CharField( - help_text="User's display name", - max_length=100, - ) - email = serializers.EmailField( - help_text="User's email address (unique)", - ) -``` - -### Custom Schema - -```python -from drf_spectacular.utils import extend_schema, OpenApiExample - -@extend_schema( - request=UserCreateSerializer, - responses={ - 201: UserSerializer, - 400: OpenApiTypes.OBJECT, - }, - examples=[ - OpenApiExample( - "Valid request", - value={"name": "John", "email": "john@example.com"}, - ), - ], -) -def create(self, request): - """Create a new user.""" -``` - -## Quick Reference - -| Framework | Documentation Source | Output | -|-----------|---------------------|--------| -| FastAPI | Type hints + docstrings | Auto Swagger UI | -| DRF | Serializers + drf-spectacular | Auto Swagger UI | - -| FastAPI Decorator | Purpose | -|-------------------|---------| -| `summary` | Short endpoint description | -| `description` | Detailed description | -| `tags` | Group endpoints | -| `response_model` | Response schema | -| `responses` | Additional response codes | - -| DRF Decorator | Purpose | -|---------------|---------| -| `@extend_schema` | Customize schema | -| `OpenApiParameter` | Query/path params | -| `OpenApiExample` | Request examples | diff --git a/.cursor/skills/code-documenter/references/api-docs-nestjs-express.md b/.cursor/skills/code-documenter/references/api-docs-nestjs-express.md deleted file mode 100644 index d157e9b2..00000000 --- a/.cursor/skills/code-documenter/references/api-docs-nestjs-express.md +++ /dev/null @@ -1,223 +0,0 @@ -# API Documentation: NestJS & Express - -> Reference for: Code Documenter -> Load when: Documenting Node.js API frameworks - -## NestJS (@nestjs/swagger) - -NestJS requires explicit decorators for OpenAPI documentation. - -### Controller Documentation - -```typescript -import { Controller, Post, Body, Get, Param } from '@nestjs/common'; -import { - ApiTags, - ApiOperation, - ApiResponse, - ApiParam, - ApiBearerAuth, -} from '@nestjs/swagger'; - -@ApiTags('Users') -@ApiBearerAuth() -@Controller('users') -export class UsersController { - @Post() - @ApiOperation({ summary: 'Create a new user' }) - @ApiResponse({ - status: 201, - description: 'User created successfully', - type: UserDto, - }) - @ApiResponse({ - status: 400, - description: 'Invalid input data', - }) - async create(@Body() dto: CreateUserDto): Promise { - return this.usersService.create(dto); - } - - @Get(':id') - @ApiOperation({ summary: 'Get user by ID' }) - @ApiParam({ - name: 'id', - description: 'User unique identifier', - example: '123', - }) - @ApiResponse({ status: 200, type: UserDto }) - @ApiResponse({ status: 404, description: 'User not found' }) - async findOne(@Param('id') id: string): Promise { - return this.usersService.findOne(id); - } -} -``` - -### DTO Documentation - -```typescript -import { ApiProperty, ApiPropertyOptional } from '@nestjs/swagger'; -import { IsEmail, IsString, MinLength } from 'class-validator'; - -export class CreateUserDto { - @ApiProperty({ - description: "User's display name", - example: 'John Doe', - minLength: 1, - maxLength: 100, - }) - @IsString() - @MinLength(1) - name: string; - - @ApiProperty({ - description: "User's email address", - example: 'john@example.com', - }) - @IsEmail() - email: string; - - @ApiPropertyOptional({ - description: 'Profile picture URL', - example: 'https://example.com/avatar.jpg', - }) - avatarUrl?: string; -} -``` - -## Express (swagger-jsdoc) - -Express uses JSDoc comments with swagger annotations. - -### Setup - -```javascript -const swaggerJsdoc = require('swagger-jsdoc'); -const swaggerUi = require('swagger-ui-express'); - -const options = { - definition: { - openapi: '3.0.0', - info: { - title: 'API Documentation', - version: '1.0.0', - }, - }, - apis: ['./routes/*.js'], -}; - -const specs = swaggerJsdoc(options); -app.use('/api-docs', swaggerUi.serve, swaggerUi.setup(specs)); -``` - -### Route Documentation - -```javascript -/** - * @swagger - * /users: - * post: - * summary: Create a new user - * tags: [Users] - * requestBody: - * required: true - * content: - * application/json: - * schema: - * $ref: '#/components/schemas/CreateUser' - * responses: - * 201: - * description: User created successfully - * content: - * application/json: - * schema: - * $ref: '#/components/schemas/User' - * 400: - * description: Invalid input - */ -router.post('/users', createUser); - -/** - * @swagger - * /users/{id}: - * get: - * summary: Get user by ID - * tags: [Users] - * parameters: - * - in: path - * name: id - * required: true - * schema: - * type: string - * description: User ID - * responses: - * 200: - * description: User found - * content: - * application/json: - * schema: - * $ref: '#/components/schemas/User' - * 404: - * description: User not found - */ -router.get('/users/:id', getUser); -``` - -### Schema Documentation - -```javascript -/** - * @swagger - * components: - * schemas: - * CreateUser: - * type: object - * required: - * - name - * - email - * properties: - * name: - * type: string - * description: User's display name - * example: John Doe - * email: - * type: string - * format: email - * description: User's email address - * example: john@example.com - * User: - * allOf: - * - $ref: '#/components/schemas/CreateUser' - * - type: object - * properties: - * id: - * type: string - * description: Unique identifier - * createdAt: - * type: string - * format: date-time - */ -``` - -## Quick Reference - -| NestJS Decorator | Purpose | -|------------------|---------| -| `@ApiTags()` | Group endpoints | -| `@ApiOperation()` | Endpoint summary | -| `@ApiResponse()` | Response documentation | -| `@ApiParam()` | Path parameter | -| `@ApiQuery()` | Query parameter | -| `@ApiBody()` | Request body | -| `@ApiBearerAuth()` | Auth requirement | -| `@ApiProperty()` | DTO property | - -| Express swagger-jsdoc | Purpose | -|-----------------------|---------| -| `@swagger` | Start swagger block | -| `tags` | Group endpoints | -| `summary` | Short description | -| `parameters` | Path/query params | -| `requestBody` | Request body schema | -| `responses` | Response schemas | -| `$ref` | Reference schema | diff --git a/.cursor/skills/code-documenter/references/coverage-reports.md b/.cursor/skills/code-documenter/references/coverage-reports.md deleted file mode 100644 index 23c43997..00000000 --- a/.cursor/skills/code-documenter/references/coverage-reports.md +++ /dev/null @@ -1,128 +0,0 @@ -# Coverage Reports - -> Reference for: Code Documenter -> Load when: Generating documentation reports - -## Documentation Coverage Report Template - -```markdown -# Documentation Report: {project_name} - -## Summary -- **Files analyzed**: 45 -- **Functions documented**: 120/150 (80%) -- **Classes documented**: 25/25 (100%) -- **API endpoints documented**: 30/30 (100%) - -## Coverage Before/After -- Before: 45% -- After: 92% - -## Files Modified - -| File | Functions Added | Notes | -|------|-----------------|-------| -| src/services/user.ts | 8 | All public methods | -| src/services/auth.ts | 5 | Added examples | -| src/controllers/users.ts | 6 | Added @Api decorators | -| src/dto/user.dto.ts | 4 | Added @ApiProperty | - -## API Documentation - -- **Framework**: NestJS -- **Strategy**: @nestjs/swagger decorators -- **Swagger UI**: /api/docs -- **OpenAPI spec**: /api-json - -## Documentation Style - -- **Python**: Google style docstrings -- **TypeScript**: JSDoc with @param, @returns -- **API**: OpenAPI 3.0 via decorators - -## Next Steps - -### Recommendations -1. Run `npm run docs:lint` to validate JSDoc -2. Add `eslint-plugin-jsdoc` to enforce documentation -3. Consider adding examples for complex functions -4. Set up documentation CI checks - -### Missing Documentation -| File | Missing | Priority | -|------|---------|----------| -| src/utils/crypto.ts | 3 functions | High | -| src/helpers/date.ts | 2 functions | Medium | - -### CI Integration -```yaml -# Add to CI pipeline -- name: Check documentation - run: npm run docs:check - -- name: Generate API docs - run: npm run docs:generate -``` -``` - -## Checklist During Documentation - -```markdown -## Documentation Checklist - -### Before Starting -- [ ] Confirmed format preference (Google/JSDoc/etc.) -- [ ] Identified files to exclude (tests, generated) -- [ ] Detected framework for API docs - -### Functions/Methods -- [ ] All public functions documented -- [ ] Parameters described with types -- [ ] Return values documented -- [ ] Exceptions/errors documented -- [ ] Examples added for complex functions - -### Classes -- [ ] Class purpose described -- [ ] Constructor parameters documented -- [ ] Public methods documented -- [ ] Important attributes explained - -### API Endpoints -- [ ] All endpoints have summaries -- [ ] Request bodies documented -- [ ] Response schemas defined -- [ ] Error responses documented -- [ ] Authentication requirements noted - -### Final Checks -- [ ] Ran documentation linter -- [ ] Verified Swagger UI renders correctly -- [ ] No inaccurate documentation -- [ ] Coverage report generated -``` - -## Framework-Specific Linting - -```bash -# JavaScript/TypeScript - ESLint -npm install eslint-plugin-jsdoc --save-dev -# Add to .eslintrc: "plugins": ["jsdoc"] - -# Python - pydocstyle -pip install pydocstyle -pydocstyle --convention=google src/ - -# Python - interrogate (coverage) -pip install interrogate -interrogate -v src/ -``` - -## Quick Reference - -| Metric | Good | Acceptable | Poor | -|--------|------|------------|------| -| Function coverage | >90% | 70-90% | <70% | -| Class coverage | 100% | >90% | <90% | -| API endpoint coverage | 100% | 100% | <100% | -| Example coverage | >50% | 30-50% | <30% | diff --git a/.cursor/skills/code-documenter/references/documentation-systems.md b/.cursor/skills/code-documenter/references/documentation-systems.md deleted file mode 100644 index 694c0c30..00000000 --- a/.cursor/skills/code-documenter/references/documentation-systems.md +++ /dev/null @@ -1,336 +0,0 @@ -# Documentation Systems & Infrastructure - -> Reference for: Code Documenter -> Load when: Building documentation sites, static generators, multi-version docs, search systems - -## Static Site Generators - -### Docusaurus (Meta) - -```bash -# Setup -npx create-docusaurus@latest docs classic -cd docs && npm start - -# Structure -docs/ -├── docs/ # Documentation pages -├── blog/ # Blog posts -├── src/ -│ └── pages/ # Custom pages -└── docusaurus.config.js -``` - -**docusaurus.config.js:** -```javascript -module.exports = { - title: 'My API', - tagline: 'Build amazing things', - url: 'https://docs.example.com', - baseUrl: '/', - - themeConfig: { - navbar: { - items: [ - {to: '/docs/intro', label: 'Docs', position: 'left'}, - {to: '/api', label: 'API', position: 'left'}, - ], - }, - - // Algolia search - algolia: { - apiKey: 'YOUR_API_KEY', - indexName: 'your_index', - contextualSearch: true, - }, - - prism: { - theme: lightCodeTheme, - darkTheme: darkCodeTheme, - additionalLanguages: ['python', 'rust'], - }, - }, -}; -``` - -### MkDocs (Python) - -```yaml -# mkdocs.yml -site_name: My API Documentation -theme: - name: material - features: - - navigation.tabs - - navigation.sections - - toc.integrate - - search.suggest - - search.highlight - palette: - - scheme: default - toggle: - icon: material/brightness-7 - name: Switch to dark mode - - scheme: slate - toggle: - icon: material/brightness-4 - name: Switch to light mode - -plugins: - - search - - mkdocstrings: - handlers: - python: - options: - show_source: true - - git-revision-date-localized - -markdown_extensions: - - pymdownx.highlight - - pymdownx.superfences - - admonition - - codehilite - -nav: - - Home: index.md - - Getting Started: getting-started.md - - API Reference: api/ -``` - -### VitePress (Vue) - -```typescript -// .vitepress/config.ts -export default defineConfig({ - title: 'API Docs', - description: 'Developer documentation', - - themeConfig: { - nav: [ - { text: 'Guide', link: '/guide/' }, - { text: 'API', link: '/api/' }, - ], - - sidebar: { - '/guide/': [ - { - text: 'Introduction', - items: [ - { text: 'Getting Started', link: '/guide/getting-started' }, - { text: 'Configuration', link: '/guide/config' }, - ], - }, - ], - }, - - search: { - provider: 'local', - }, - - editLink: { - pattern: 'https://github.com/user/repo/edit/main/docs/:path', - }, - }, -}); -``` - -## Multi-Version Documentation - -### Version Switcher - -```javascript -// Docusaurus versions -{ - versions: { - current: { - label: '2.0 (Next)', - path: 'next', - }, - }, - onlyIncludeVersions: ['current', '1.5', '1.4'], -} -``` - -### Migration Guides - -```markdown -# Migration Guide: v1 to v2 - -## Breaking Changes - -### Authentication -**v1:** -```python -client.authenticate(api_key) -``` - -**v2:** -```python -client = Client(api_key=api_key) # Pass in constructor -``` - -### Renamed Methods -| v1 | v2 | Notes | -|----|----|----- | -| `get_user()` | `fetch_user()` | Async now | -| `delete_user()` | `remove_user()` | Returns Promise | - -## Deprecation Timeline -- v1.x: Supported until Dec 2025 -- v2.0: Released Jan 2025 -- v2.1: Current (June 2025) -``` - -## Search Implementation - -### Algolia DocSearch - -```html - - - - - -``` - -### Local Search (Lunr.js) - -```javascript -const idx = lunr(function() { - this.ref('id'); - this.field('title', { boost: 10 }); - this.field('content'); - - documents.forEach(doc => this.add(doc)); -}); - -// Search -const results = idx.search('authentication'); -``` - -## Documentation Testing - -### Link Checking - -```bash -# linkcheck (Python) -pip install linkchecker -linkchecker http://localhost:3000/docs - -# broken-link-checker (Node) -npm install -g broken-link-checker -blc http://localhost:3000 -ro -``` - -### Code Example Testing - -```python -# doctest for Python examples -""" ->>> add(2, 3) -5 ->>> add(-1, 1) -0 -""" - -# Run tests -python -m doctest -v docs/*.md -``` - -```javascript -// Jest for TypeScript examples -// Extract code blocks and test -import { runExamples } from './test-docs'; - -test('API examples work', async () => { - const examples = extractExamples('./docs/api.md'); - await expect(runExamples(examples)).resolves.toBeTruthy(); -}); -``` - -## Performance Optimization - -### Build Optimization - -```javascript -// Webpack/Vite config -export default { - build: { - rollupOptions: { - output: { - manualChunks: { - 'vendor': ['react', 'react-dom'], - }, - }, - }, - }, - - optimizeDeps: { - include: ['prismjs'], - }, -}; -``` - -### CDN & Caching - -```nginx -# nginx.conf -location /docs { - expires 1y; - add_header Cache-Control "public, immutable"; -} - -location ~* \.(html)$ { - expires 1h; - add_header Cache-Control "public, must-revalidate"; -} -``` - -## Analytics Integration - -### Google Analytics - -```javascript -// Docusaurus -gtag: { - trackingID: 'G-XXXXXXXXXX', - anonymizeIP: true, -}, -``` - -### Custom Analytics - -```javascript -// Track search queries -function trackSearch(query, results) { - analytics.track('docs_search', { - query, - resultCount: results.length, - timestamp: new Date(), - }); -} -``` - -## Quick Reference - -| Tool | Best For | Tech Stack | -|------|----------|-----------| -| Docusaurus | React projects, versioning | React, MDX | -| MkDocs | Python projects, simple setup | Python, Jinja2 | -| VitePress | Vue projects, fast builds | Vue, Vite | -| Nextra | Next.js integration | React, Next.js | -| Mintlify | Modern UI, AI search | React | - -| Search Solution | Cost | Features | -|----------------|------|----------| -| Algolia DocSearch | Free (OSS) | Fast, typo-tolerant | -| Local (Lunr.js) | Free | Offline, no server | -| Typesense | Free (self-host) | Privacy-focused | -| Meilisearch | Free (self-host) | Fast, relevance | diff --git a/.cursor/skills/code-documenter/references/interactive-api-docs.md b/.cursor/skills/code-documenter/references/interactive-api-docs.md deleted file mode 100644 index 937f3eec..00000000 --- a/.cursor/skills/code-documenter/references/interactive-api-docs.md +++ /dev/null @@ -1,534 +0,0 @@ -# Interactive API Documentation - -> Reference for: Code Documenter -> Load when: Building API portals, interactive consoles, multi-protocol APIs, SDK docs - -## OpenAPI 3.1 Advanced Features - -### Reusable Components - -```yaml -openapi: 3.1.0 -info: - title: Users API - version: 2.0.0 - -components: - # Reusable schemas - schemas: - User: - type: object - required: [id, email] - properties: - id: - type: string - format: uuid - example: "123e4567-e89b-12d3-a456-426614174000" - email: - type: string - format: email - example: "user@example.com" - - Error: - type: object - properties: - code: - type: string - message: - type: string - details: - type: object - - PaginatedResponse: - type: object - properties: - data: - type: array - items: {} - total: - type: integer - page: - type: integer - - # Reusable parameters - parameters: - PageParam: - name: page - in: query - schema: - type: integer - default: 1 - minimum: 1 - - LimitParam: - name: limit - in: query - schema: - type: integer - default: 20 - minimum: 1 - maximum: 100 - - # Security schemes - securitySchemes: - BearerAuth: - type: http - scheme: bearer - bearerFormat: JWT - - ApiKeyAuth: - type: apiKey - in: header - name: X-API-Key - - OAuth2: - type: oauth2 - flows: - authorizationCode: - authorizationUrl: https://api.example.com/oauth/authorize - tokenUrl: https://api.example.com/oauth/token - scopes: - read:users: Read user data - write:users: Modify user data - - # Reusable responses - responses: - NotFound: - description: Resource not found - content: - application/json: - schema: - $ref: '#/components/schemas/Error' - - Unauthorized: - description: Authentication required - content: - application/json: - schema: - $ref: '#/components/schemas/Error' - -paths: - /users: - get: - summary: List users - parameters: - - $ref: '#/components/parameters/PageParam' - - $ref: '#/components/parameters/LimitParam' - security: - - BearerAuth: [] - responses: - '200': - description: Success - content: - application/json: - schema: - allOf: - - $ref: '#/components/schemas/PaginatedResponse' - - type: object - properties: - data: - type: array - items: - $ref: '#/components/schemas/User' -``` - -## Interactive Documentation Portals - -### Swagger UI Customization - -```javascript -// Custom Swagger UI -const swaggerUi = require('swagger-ui-express'); -const swaggerDocument = require('./openapi.json'); - -const options = { - customCss: '.swagger-ui .topbar { display: none }', - customSiteTitle: "API Docs", - customfavIcon: "/favicon.ico", - swaggerOptions: { - persistAuthorization: true, - displayRequestDuration: true, - filter: true, - tryItOutEnabled: true, - requestInterceptor: (req) => { - req.headers['X-Custom-Header'] = 'value'; - return req; - }, - }, -}; - -app.use('/api-docs', swaggerUi.serve, swaggerUi.setup(swaggerDocument, options)); -``` - -### Redoc (Modern Alternative) - -```html - - - - API Documentation - - - - - - - - -``` - -### Stoplight Elements - -```javascript -import { API } from '@stoplight/elements'; -import '@stoplight/elements/styles.min.css'; - -function App() { - return ( - - ); -} -``` - -## Multi-Protocol Documentation - -### GraphQL Schema Documentation - -```graphql -""" -User account in the system -""" -type User { - """ - Unique user identifier - """ - id: ID! - - """ - User's email address (unique) - @example "user@example.com" - """ - email: String! - - """ - Display name - @example "John Doe" - """ - name: String! - - """ - User's posts (paginated) - """ - posts( - """Number of items per page (max 100)""" - limit: Int = 20 - """Page offset""" - offset: Int = 0 - ): PostConnection! -} - -type Query { - """ - Fetch a user by ID - """ - user( - """User's unique identifier""" - id: ID! - ): User - - """ - Search users by name or email - """ - searchUsers( - """Search query""" - query: String! - """Maximum results to return""" - limit: Int = 10 - ): [User!]! -} - -type Mutation { - """ - Create a new user account - """ - createUser( - """User creation input""" - input: CreateUserInput! - ): CreateUserPayload! -} - -""" -Input for creating a user -""" -input CreateUserInput { - """User's email address""" - email: String! - """Display name""" - name: String! -} -``` - -**GraphQL Playground:** -```javascript -const { ApolloServer } = require('apollo-server'); - -const server = new ApolloServer({ - typeDefs, - resolvers, - introspection: true, // Enable in dev - playground: { - settings: { - 'editor.theme': 'dark', - 'editor.fontSize': 14, - }, - }, -}); -``` - -### WebSocket Protocol Documentation - -```yaml -# AsyncAPI 2.0 -asyncapi: 2.5.0 -info: - title: Chat WebSocket API - version: 1.0.0 - description: Real-time chat messaging - -channels: - chat/{roomId}: - parameters: - roomId: - description: Chat room identifier - schema: - type: string - - subscribe: - summary: Receive messages - message: - oneOf: - - $ref: '#/components/messages/ChatMessage' - - $ref: '#/components/messages/UserJoined' - - publish: - summary: Send a message - message: - $ref: '#/components/messages/ChatMessage' - -components: - messages: - ChatMessage: - name: message - payload: - type: object - properties: - userId: - type: string - content: - type: string - timestamp: - type: string - format: date-time - - UserJoined: - name: userJoined - payload: - type: object - properties: - userId: - type: string - username: - type: string -``` - -### gRPC Documentation - -```protobuf -syntax = "proto3"; - -package users.v1; - -// User service manages user accounts -service UserService { - // Get a user by ID - // Returns: User object or NOT_FOUND error - rpc GetUser(GetUserRequest) returns (User) {} - - // List all users with pagination - // Returns: Paginated list of users - rpc ListUsers(ListUsersRequest) returns (ListUsersResponse) {} - - // Create a new user - // Returns: Created user or ALREADY_EXISTS error - rpc CreateUser(CreateUserRequest) returns (User) {} - - // Stream user updates in real-time - // Returns: Stream of user update events - rpc WatchUsers(WatchUsersRequest) returns (stream UserEvent) {} -} - -// User account -message User { - // Unique identifier - string id = 1; - - // Email address (unique, required) - string email = 2; - - // Display name - string name = 3; - - // Account creation timestamp - google.protobuf.Timestamp created_at = 4; -} -``` - -## SDK Documentation Strategies - -### Multi-Language Examples - -```markdown -# Create User - -## Python -```python -from myapi import Client - -client = Client(api_key="your_key") -user = client.users.create( - name="John Doe", - email="john@example.com" -) -print(user.id) -``` - -## TypeScript -```typescript -import { Client } from '@myapi/sdk'; - -const client = new Client({ apiKey: 'your_key' }); -const user = await client.users.create({ - name: 'John Doe', - email: 'john@example.com', -}); -console.log(user.id); -``` - -## Go -```go -import "github.com/myapi/sdk-go" - -client := sdk.NewClient("your_key") -user, err := client.Users.Create(ctx, &sdk.CreateUserInput{ - Name: "John Doe", - Email: "john@example.com", -}) -if err != nil { - log.Fatal(err) -} -fmt.Println(user.ID) -``` - -## Ruby -```ruby -require 'myapi' - -client = MyAPI::Client.new(api_key: 'your_key') -user = client.users.create( - name: 'John Doe', - email: 'john@example.com' -) -puts user.id -``` -``` - -### SDK Reference Template - -```markdown -# Users SDK - -## Installation -```bash -npm install @myapi/sdk -``` - -## Configuration -```typescript -import { Client } from '@myapi/sdk'; - -const client = new Client({ - apiKey: process.env.API_KEY, - baseUrl: 'https://api.example.com', // Optional - timeout: 30000, // Optional, default 30s -}); -``` - -## Methods - -### `client.users.create(data)` -Create a new user. - -**Parameters:** -- `data.name` (string, required) - User's display name -- `data.email` (string, required) - User's email address - -**Returns:** Promise - -**Throws:** -- `ValidationError` - Invalid input data -- `ConflictError` - Email already exists -- `AuthenticationError` - Invalid API key - -**Example:** -```typescript -const user = await client.users.create({ - name: 'John Doe', - email: 'john@example.com', -}); -``` - -## Error Handling -```typescript -import { ValidationError, ConflictError } from '@myapi/sdk'; - -try { - await client.users.create(data); -} catch (error) { - if (error instanceof ValidationError) { - console.error('Invalid data:', error.fields); - } else if (error instanceof ConflictError) { - console.error('User already exists'); - } -} -``` -``` - -## Quick Reference - -| Tool | Protocol | Features | -|------|----------|----------| -| Swagger UI | REST | Try-it-out, auth | -| Redoc | REST | Clean, responsive | -| Stoplight | REST | Modern, mock server | -| GraphQL Playground | GraphQL | Explorer, history | -| AsyncAPI Studio | WebSocket | Visual editor | -| grpcui | gRPC | Interactive console | diff --git a/.cursor/skills/code-documenter/references/python-docstrings.md b/.cursor/skills/code-documenter/references/python-docstrings.md deleted file mode 100644 index 054a12e2..00000000 --- a/.cursor/skills/code-documenter/references/python-docstrings.md +++ /dev/null @@ -1,124 +0,0 @@ -# Python Docstrings - -> Reference for: Code Documenter -> Load when: Documenting Python code - -## Google Style (Recommended) - -```python -def calculate_total(items: list[Item], tax_rate: float = 0.0) -> float: - """Calculate total cost including tax. - - Args: - items: List of items to calculate total for. - tax_rate: Tax rate as decimal (e.g., 0.08 for 8%). - - Returns: - Total cost including tax. - - Raises: - ValueError: If tax_rate is negative or items is empty. - - Example: - >>> calculate_total([Item(10), Item(20)], 0.1) - 33.0 - """ -``` - -## NumPy Style - -```python -def calculate_total(items: list[Item], tax_rate: float = 0.0) -> float: - """ - Calculate total cost including tax. - - Parameters - ---------- - items : list[Item] - List of items to calculate total for. - tax_rate : float, optional - Tax rate as decimal (e.g., 0.08 for 8%). Default is 0.0. - - Returns - ------- - float - Total cost including tax. - - Raises - ------ - ValueError - If tax_rate is negative or items is empty. - - Examples - -------- - >>> calculate_total([Item(10), Item(20)], 0.1) - 33.0 - """ -``` - -## Sphinx Style - -```python -def calculate_total(items: list[Item], tax_rate: float = 0.0) -> float: - """Calculate total cost including tax. - - :param items: List of items to calculate total for. - :type items: list[Item] - :param tax_rate: Tax rate as decimal (e.g., 0.08 for 8%). - :type tax_rate: float - :returns: Total cost including tax. - :rtype: float - :raises ValueError: If tax_rate is negative or items is empty. - - .. code-block:: python - - >>> calculate_total([Item(10), Item(20)], 0.1) - 33.0 - """ -``` - -## Class Documentation - -```python -class UserService: - """Service for managing user operations. - - This service handles CRUD operations for users and - integrates with the authentication system. - - Attributes: - db: Database session for queries. - cache: Redis client for caching. - - Example: - >>> service = UserService(db, cache) - >>> user = await service.create_user(data) - """ - - def __init__(self, db: AsyncSession, cache: Redis) -> None: - """Initialize UserService. - - Args: - db: Database session for queries. - cache: Redis client for caching. - """ -``` - -## Quick Reference - -| Style | Args Format | Returns Format | -|-------|-------------|----------------| -| Google | `Args:` block | `Returns:` block | -| NumPy | `Parameters` section | `Returns` section | -| Sphinx | `:param name:` | `:returns:` | - -## Sections Available - -| Section | Google | NumPy | Sphinx | -|---------|--------|-------|--------| -| Parameters | `Args:` | `Parameters` | `:param:` | -| Returns | `Returns:` | `Returns` | `:returns:` | -| Raises | `Raises:` | `Raises` | `:raises:` | -| Examples | `Example:` | `Examples` | `.. code-block::` | -| Notes | `Note:` | `Notes` | `.. note::` | -| Attributes | `Attributes:` | `Attributes` | `:ivar:` | diff --git a/.cursor/skills/code-documenter/references/typescript-jsdoc.md b/.cursor/skills/code-documenter/references/typescript-jsdoc.md deleted file mode 100644 index c241c447..00000000 --- a/.cursor/skills/code-documenter/references/typescript-jsdoc.md +++ /dev/null @@ -1,148 +0,0 @@ -# TypeScript JSDoc - -> Reference for: Code Documenter -> Load when: Documenting TypeScript/JavaScript code - -## Function Documentation - -```typescript -/** - * Calculate total cost including tax. - * - * @param items - List of items to calculate total for - * @param taxRate - Tax rate as decimal (e.g., 0.08 for 8%) - * @returns Total cost including tax - * @throws {Error} If taxRate is negative or items is empty - * - * @example - * ```typescript - * const total = calculateTotal(items, 0.08); - * console.log(total); // 108.00 - * ``` - */ -function calculateTotal(items: Item[], taxRate = 0): number { -``` - -## Class Documentation - -```typescript -/** - * Service for managing user operations. - * - * Handles CRUD operations and integrates with authentication system. - * - * @example - * ```typescript - * const service = new UserService(db, cache); - * const user = await service.create(userData); - * ``` - */ -class UserService { - /** - * Create a new UserService instance. - * - * @param db - Database connection - * @param cache - Redis cache client - */ - constructor( - private readonly db: Database, - private readonly cache: Cache, - ) {} -} -``` - -## Interface Documentation - -```typescript -/** - * User data transfer object. - * - * @interface UserDto - */ -interface UserDto { - /** Unique user identifier */ - id: string; - - /** User's email address (unique) */ - email: string; - - /** User's display name */ - name: string; - - /** Account creation timestamp */ - createdAt: Date; -} -``` - -## Generic Types - -```typescript -/** - * Paginated response wrapper. - * - * @template T - Type of items in the data array - */ -interface PaginatedResponse { - /** Array of items for current page */ - data: T[]; - - /** Total number of items across all pages */ - total: number; - - /** Current page number (1-indexed) */ - page: number; - - /** Number of items per page */ - limit: number; -} -``` - -## Async Functions - -```typescript -/** - * Fetch user by ID from database. - * - * @param id - User's unique identifier - * @returns Promise resolving to user data or null if not found - * @throws {DatabaseError} If connection fails - * - * @async - */ -async function findUserById(id: string): Promise { -``` - -## Quick Reference - -| Tag | Purpose | Example | -|-----|---------|---------| -| `@param` | Parameter description | `@param name - User's name` | -| `@returns` | Return value | `@returns User object` | -| `@throws` | Exception thrown | `@throws {Error} If invalid` | -| `@example` | Usage example | Code block | -| `@see` | Reference link | `@see UserService` | -| `@deprecated` | Mark deprecated | `@deprecated Use v2 instead` | -| `@template` | Generic type param | `@template T - Item type` | -| `@async` | Async function | Mark async | -| `@private` | Private member | Internal use | -| `@readonly` | Read-only property | Cannot modify | - -## Common Patterns - -```typescript -// Optional parameters -/** @param [options] - Optional configuration */ - -// Default values -/** @param [limit=10] - Items per page (default: 10) */ - -// Multiple types -/** @param input - Input value (string or number) */ - -// Callback parameters -/** - * @callback FilterFn - * @param item - Item to filter - * @returns Whether item passes filter - */ -``` diff --git a/.cursor/skills/code-documenter/references/user-guides-tutorials.md b/.cursor/skills/code-documenter/references/user-guides-tutorials.md deleted file mode 100644 index ddfb7b63..00000000 --- a/.cursor/skills/code-documenter/references/user-guides-tutorials.md +++ /dev/null @@ -1,533 +0,0 @@ -# User Guides & Tutorials - -> Reference for: Code Documenter -> Load when: Creating getting started guides, tutorials, troubleshooting docs, end-user documentation - -## Tutorial Structure - -### Progressive Learning Path - -```markdown -# Getting Started with API - -## Prerequisites -Before you begin, ensure you have: -- [ ] Node.js 18+ installed -- [ ] An API key from your dashboard -- [ ] Basic knowledge of REST APIs - -## Quick Start (5 minutes) - -### 1. Install the SDK -```bash -npm install @myapi/sdk -``` - -### 2. Create Your First Request -```typescript -import { Client } from '@myapi/sdk'; - -const client = new Client({ apiKey: 'your_key' }); -const users = await client.users.list(); -console.log(users); -``` - -### 3. Verify It Works -Run the code and you should see a list of users. - -**Expected output:** -```json -{ - "data": [ - { "id": "1", "name": "Alice" }, - { "id": "2", "name": "Bob" } - ], - "total": 2 -} -``` - -## Next Steps -- [Authentication Guide](/docs/auth) - Learn about OAuth and API keys -- [Advanced Queries](/docs/queries) - Filtering, sorting, pagination -- [Error Handling](/docs/errors) - Handle errors gracefully -``` - -### Step-by-Step Tutorial - -```markdown -# Tutorial: Building a User Dashboard - -**What you'll learn:** -- Fetching user data from the API -- Handling pagination -- Displaying data in a table -- Adding real-time updates - -**Time:** 30 minutes -**Level:** Intermediate - -## Step 1: Set Up the Project - -Create a new project: -```bash -mkdir user-dashboard -cd user-dashboard -npm init -y -npm install @myapi/sdk react -``` - -## Step 2: Fetch Users - -Create `src/api/users.ts`: -```typescript -import { Client } from '@myapi/sdk'; - -const client = new Client({ apiKey: process.env.API_KEY }); - -export async function getUsers(page = 1, limit = 20) { - const response = await client.users.list({ page, limit }); - return response; -} -``` - -**What's happening:** -1. We import the SDK client -2. Initialize it with our API key from environment -3. Create a helper function that fetches paginated users - -## Step 3: Create the Component - -Create `src/components/UserTable.tsx`: -```typescript -import { useState, useEffect } from 'react'; -import { getUsers } from '../api/users'; - -export function UserTable() { - const [users, setUsers] = useState([]); - const [loading, setLoading] = useState(true); - - useEffect(() => { - async function fetchData() { - const data = await getUsers(); - setUsers(data.data); - setLoading(false); - } - fetchData(); - }, []); - - if (loading) return
Loading...
; - - return ( - - - - - - - - - {users.map(user => ( - - - - - ))} - -
NameEmail
{user.name}{user.email}
- ); -} -``` - -## Step 4: Test It - -Run your app: -```bash -npm run dev -``` - -You should see a table with user data. - -## Checkpoint -At this point, you have: -- [x] Set up the SDK -- [x] Created an API helper -- [x] Built a user table component -- [ ] Added pagination -- [ ] Added real-time updates - -## Next: Adding Pagination - -[Continue to Step 5 →](/docs/tutorial/step-5) -``` - -## Information Architecture - -### Content Hierarchy - -```markdown -Documentation/ -├── Getting Started/ -│ ├── Quick Start (5 min) -│ ├── Installation -│ ├── Authentication -│ └── First Request -│ -├── Guides/ -│ ├── User Management -│ ├── File Uploads -│ ├── Webhooks -│ └── Rate Limiting -│ -├── API Reference/ -│ ├── Users API -│ ├── Files API -│ └── Webhooks API -│ -├── SDK Documentation/ -│ ├── Python SDK -│ ├── TypeScript SDK -│ └── Go SDK -│ -├── Tutorials/ -│ ├── Build a Dashboard (30 min) -│ ├── Integrate Authentication (45 min) -│ └── Real-time Sync (60 min) -│ -└── Resources/ - ├── Troubleshooting - ├── FAQ - ├── Best Practices - └── Migration Guides -``` - -## Writing Techniques - -### Task-Based Writing - -```markdown -# How to Upload a File - -**Goal:** Upload an image file to your account storage - -**Time:** 5 minutes - -## Steps - -### 1. Prepare the file -Get the file from user input or file system: -```typescript -const file = document.querySelector('input[type="file"]').files[0]; -``` - -### 2. Create form data -```typescript -const formData = new FormData(); -formData.append('file', file); -formData.append('folder', 'avatars'); -``` - -### 3. Upload with the SDK -```typescript -const result = await client.files.upload(formData); -console.log('File URL:', result.url); -``` - -## Common Issues - -**"File too large" error:** -Maximum file size is 10MB. Compress images before uploading. - -**"Invalid file type" error:** -Only .jpg, .png, .gif are allowed. Check the file extension. - -## Related -- [File API Reference](/api/files) -- [Handling Upload Progress](/guides/upload-progress) -``` - -### Progressive Disclosure - -```markdown -# Authentication - -## Basic: API Keys (Recommended for Getting Started) - -API keys are the simplest way to authenticate. - -```typescript -const client = new Client({ apiKey: 'your_key' }); -``` - -**When to use:** Scripts, internal tools, testing - -[Generate an API key →](/dashboard/api-keys) - -
-Advanced: OAuth 2.0 - -For user-facing applications, use OAuth 2.0. - -### Authorization Code Flow - -1. Redirect user to authorization URL: -```typescript -const authUrl = client.oauth.getAuthUrl({ - redirectUri: 'https://yourapp.com/callback', - scopes: ['read:users', 'write:users'], -}); -window.location.href = authUrl; -``` - -2. Handle the callback: -```typescript -const code = new URLSearchParams(window.location.search).get('code'); -const tokens = await client.oauth.exchangeCode(code); -``` - -3. Use the access token: -```typescript -const client = new Client({ accessToken: tokens.access_token }); -``` - -[Full OAuth guide →](/guides/oauth) -
- -
-Enterprise: JWT Tokens - -For service-to-service authentication, use JWTs. - -```typescript -const jwt = createJWT({ - issuer: 'your-service', - subject: 'service-account-id', - privateKey: process.env.PRIVATE_KEY, -}); - -const client = new Client({ jwt }); -``` - -[JWT setup guide →](/guides/jwt) -
-``` - -## Visual Communication - -### Diagram Integration - -```markdown -# System Architecture - -## Request Flow - -```mermaid -sequenceDiagram - participant Client - participant API - participant Database - participant Cache - - Client->>API: POST /users - API->>Cache: Check cache - Cache-->>API: Cache miss - API->>Database: Insert user - Database-->>API: User created - API->>Cache: Store user - API-->>Client: 201 Created -``` - -## Data Model - -```mermaid -erDiagram - USER ||--o{ POST : creates - USER ||--o{ COMMENT : writes - POST ||--o{ COMMENT : has - - USER { - string id PK - string email UK - string name - datetime created_at - } - - POST { - string id PK - string user_id FK - string title - text content - } -``` -``` - -### Screenshot Annotations - -```markdown -# Dashboard Overview - -![Dashboard with numbered annotations](./images/dashboard-annotated.png) - -**Key features:** - -1. **Navigation** - Switch between sections -2. **API Key** - Copy your key (click to reveal) -3. **Usage Stats** - Current month's API calls -4. **Quick Actions** - Generate new key, view docs -5. **Recent Activity** - Last 10 API requests - -## Creating Your First API Key - -1. Click "Generate New Key" (highlighted in green) -2. Enter a description like "Production API" -3. Select permissions (default: all) -4. Click "Create" -5. **Important:** Copy the key immediately - it won't be shown again - -![Create API key dialog](./images/create-key.png) -``` - -## Troubleshooting Guides - -### Problem-Solution Format - -```markdown -# Troubleshooting - -## Authentication Errors - -### "Invalid API key" - -**Symptoms:** -- 401 Unauthorized error -- Error message: "Invalid API key" - -**Causes:** -1. API key was copied incorrectly (extra spaces) -2. API key was revoked -3. Using test key in production environment - -**Solutions:** - -**1. Verify the key:** -```bash -# Check for extra spaces -echo -n "$API_KEY" | wc -c # Should be exactly 32 characters -``` - -**2. Regenerate the key:** -- Go to [dashboard](/dashboard) -- Click "Revoke & Regenerate" -- Update your environment variables - -**3. Check environment:** -```typescript -console.log('Environment:', process.env.NODE_ENV); -console.log('API URL:', client.baseUrl); -``` - -**Still not working?** -[Contact support](/support) with your request ID from the error response. - ---- - -### "Rate limit exceeded" - -**Symptoms:** -- 429 Too Many Requests error -- Requests failing intermittently - -**Immediate fix:** -Wait 60 seconds and retry. - -**Long-term solutions:** - -**1. Implement exponential backoff:** -```typescript -async function retryWithBackoff(fn, maxRetries = 3) { - for (let i = 0; i < maxRetries; i++) { - try { - return await fn(); - } catch (error) { - if (error.status === 429 && i < maxRetries - 1) { - await sleep(Math.pow(2, i) * 1000); - continue; - } - throw error; - } - } -} -``` - -**2. Batch requests:** -Instead of 100 individual requests, use batch endpoints. - -**3. Upgrade your plan:** -[View plans](/pricing) - Higher tiers have increased limits. -``` - -## FAQ Section - -```markdown -# Frequently Asked Questions - -## General - -### What's included in the free tier? -- 1,000 API requests/month -- 1GB storage -- Community support -- All core features - -### How do I upgrade? -Click "Upgrade" in your [dashboard](/dashboard) and select a plan. - -## Technical - -### Can I use this in production? -Yes, the API is production-ready with 99.9% SLA on paid plans. - -### What's the rate limit? -- Free: 10 requests/minute -- Pro: 100 requests/minute -- Enterprise: Custom limits - -### Do you support webhooks? -Yes! See [Webhooks Guide](/guides/webhooks) for setup. - -### Which regions are available? -Currently: US East, US West, EU Central, Asia Pacific. - -## Billing - -### How does billing work? -- Monthly subscription -- Pay-as-you-go for overages -- Cancel anytime - -### What payment methods do you accept? -Credit card, PayPal, wire transfer (annual plans only). - ---- - -**Can't find your answer?** -- [Browse all docs](/docs) -- [Ask the community](https://community.example.com) -- [Contact support](/support) -``` - -## Quick Reference - -| Content Type | Best For | Key Elements | -|-------------|----------|-------------| -| Quick Start | New users (5 min) | Prerequisites, minimal code, verify | -| Tutorial | Learning by doing | Steps, checkpoints, working code | -| How-To Guide | Specific tasks | Goal, steps, troubleshooting | -| Reference | Looking up details | Comprehensive, searchable | -| Explanation | Understanding concepts | Why, not how | - -| Writing Principle | Technique | -|------------------|-----------| -| Clarity | Active voice, short sentences | -| Scannability | Headings, lists, code blocks | -| Completeness | Prerequisites, next steps, related links | -| Accuracy | Test all code, version specifics | diff --git a/.cursor/skills/dataframes/SKILL.md b/.cursor/skills/dataframes/SKILL.md new file mode 100644 index 00000000..b0e472a5 --- /dev/null +++ b/.cursor/skills/dataframes/SKILL.md @@ -0,0 +1,47 @@ +--- +author: dotagents +name: dataframes +description: Pandas and Polars for tabular data: when to use which, indexing and dtypes, joins and reshaping, lazy Polars, I/O and Parquet, nulls, Arrow interop, and performance. Use for DataFrame/Series/LazyFrame code. Triggers on pandas, polars, DataFrame, LazyFrame, parquet, groupby, merge, join. +--- + +# Dataframes (pandas + Polars) + +## Quick start + +1. **Pick one engine per pipeline**: **pandas** when **index semantics**, **mixed columns**, and **ecosystem** (statsmodels, sklearn I/O) matter; **Polars** when **large scans**, **lazy plans**, and **expression** clarity win. See [reference-when-which.md](references/reference-when-which.md). +2. **pandas**: be explicit with **`loc` / `iloc`**, **`assign`**, and **dtypes**; understand **views vs copies** and **Copy-on-Write** (2.x). See [reference-pandas-core.md](references/reference-pandas-core.md). +3. **Polars**: default to **`scan_*` + `lazy()`** for big data; build with **`select` / `with_columns` / `filter`** and **`pl.col`**; **`collect`** at the boundary. See [reference-polars.md](references/reference-polars.md). +4. **Joins and groups**: validate **row counts** and **duplicates**; name **suffixes**; prefer **Polars** for parallel group-by on large tables when the project already uses it. See [reference-pandas-group-join.md](references/reference-pandas-group-join.md) and [reference-polars.md](references/reference-polars.md). +5. **I/O**: **Parquet** (often **PyArrow**) for analytics interchange; **`uv add pyarrow`** when the stack needs it. See [reference-io-dtypes-nulls.md](references/reference-io-dtypes-nulls.md). +6. **Handoff**: convert at **module boundaries** with **`from_pandas`** / **`to_pandas`** or **Arrow**; avoid ping-pong in inner loops. See [reference-interop-performance.md](references/reference-interop-performance.md). + +## Stack synergy + +| Resource | Role | +|----------|------| +| **Python spec** | Explicit pandas index/columns; lazy Polars for heavy queries | +| **numpy-scientific** | `to_numpy()`, dtypes, contiguous buffers | +| **matplotlib-scientific** | `df.plot(ax=ax)` and Polars `.to_pandas()` for plotting when needed | +| **numpy-docstrings** | Public API docstrings for DataFrame-returning functions | +| **general-python** | uv, ty, **python-reviewer** | + +## Reference index + +| Topic | File | +|--------|------| +| pandas vs Polars, boundaries | [reference-when-which.md](references/reference-when-which.md) | +| Index, `loc`/`iloc`, dtypes, COW, assignment | [reference-pandas-core.md](references/reference-pandas-core.md) | +| GroupBy, merge/join, concat, pivot | [reference-pandas-group-join.md](references/reference-pandas-group-join.md) | +| LazyFrame, expressions, group, join | [reference-polars.md](references/reference-polars.md) | +| CSV/Parquet, nulls, dtypes, Arrow | [reference-io-dtypes-nulls.md](references/reference-io-dtypes-nulls.md) | +| Conversion, streaming, typing | [reference-interop-performance.md](references/reference-interop-performance.md) | + +## Official documentation + +- [pandas user guide](https://pandas.pydata.org/docs/user_guide/index.html) +- [Polars user guide](https://docs.pola.rs/user-guide/) +- [Polars API](https://docs.pola.rs/api/python/stable/reference/index.html) + +## Dependencies + +Use **`uv add pandas`**, **`uv add polars`**, **`uv add pyarrow`** as needed; do not hand-edit version pins in `pyproject.toml`. diff --git a/.cursor/skills/dataframes/references/reference-interop-performance.md b/.cursor/skills/dataframes/references/reference-interop-performance.md new file mode 100644 index 00000000..d3d65a37 --- /dev/null +++ b/.cursor/skills/dataframes/references/reference-interop-performance.md @@ -0,0 +1,33 @@ +# Interop, NumPy, plotting, performance + +## pandas ↔ Polars + +- **`pl.from_pandas(df)`** / **`df.to_pandas()`**; cost is **non-trivial** on huge frames—do **once** per stage. +- **Arrow**: **`df.to_arrow()`** / **`pl.DataFrame(arrow_table)`** when zero-copy paths exist (versions and dtypes must align). + +## NumPy + +- **pandas**: **`Series.to_numpy()`**, **`DataFrame.to_numpy()`**; **`copy`** parameter matters; **nullable** dtypes may yield **object** or **masked** outputs—check. +- **Polars**: **`to_numpy()`** on Series; DataFrame to NumPy often goes through **column stack** or **pandas**—prefer **expressions** inside Polars. + +## Matplotlib + +- **pandas**: **`df.plot(ax=ax, kind=...)`** per **`matplotlib-scientific`** skill. +- **Polars**: **`to_pandas()`** for quick plots or export **CSV** to plotting tools; native plotting ecosystem is thinner. + +## Memory and chunks + +- **pandas**: **`read_csv(chunksize=...)`** for bounded memory; **Polars lazy** + **sink_parquet** for out-of-core style workflows (see current Polars docs for **sink** APIs). + +## Typing + +- **pandas** stubs: **`pd.DataFrame`**, **`Series`**; **Polars** ships types for **`DataFrame`**, **`LazyFrame`**, **`Expr`**—run **`ty check`** on public APIs that return frames. + +## Parallelism + +- Polars uses **Rayon**-style parallelism internally; avoid **nested** parallel Python over Polars in the same process without care. +- pandas **releases GIL** in some ops but not all; **vectorized** column ops beat Python loops. + +## Reproducibility + +- **Sort** before comparisons; **seed** any sampling; document **Polars** and **pandas** **versions** in science appendices when results must replay exactly. diff --git a/.cursor/skills/dataframes/references/reference-io-dtypes-nulls.md b/.cursor/skills/dataframes/references/reference-io-dtypes-nulls.md new file mode 100644 index 00000000..6c725c5a --- /dev/null +++ b/.cursor/skills/dataframes/references/reference-io-dtypes-nulls.md @@ -0,0 +1,37 @@ +# I/O, dtypes, and nulls (both libraries) + +## Reading and writing + +### pandas + +- **`read_csv`**, **`read_parquet`**, **`read_feather`**, **`read_sql`**; **`to_parquet`**, etc. +- **`dtype`**, **`parse_dates`**, **`usecols`** reduce memory on ingest. +- Engine **`pyarrow`** or **`fastparquet`** for Parquet—project should pick one consistently. + +### Polars + +- **`read_csv` / `read_parquet`** (eager) vs **`scan_csv` / `scan_parquet`** (lazy). +- **`infer_schema_length`**, **`try_parse_dates`**, **`columns`** to trim work. + +## Parquet and Arrow + +- **Parquet** preserves **schema** and compresses well for analytics pipelines. +- **`uv add pyarrow`** when either stack reads/writes Parquet heavily or shares **Arrow** buffers. + +## Nulls and NA + +- **pandas**: **`NaN`** for floats; **`pd.NA`** for nullable dtypes; **`NaT`** for datetimes—**boolean reductions** can be tri-state. +- **Polars**: **`null`** unified; predicates use **`is_null()`**, **`is_not_null()`**, **`fill_null`**, **`drop_nulls`**. +- **Joins**: null keys usually **do not match**; document behavior for outer joins. + +## Strings + +- pandas **`string`** dtype vs **`object`**; Polars **`String`** (Utf8 in older docs)—check installed Polars version in migration notes. + +## Time zones + +- Store **UTC** internally where possible; localize/convert explicitly for display and reporting. + +## Excel and odd formats + +- Prefer **CSV/Parquet** in pipelines; **Excel** last-mile only; watch **sheet** and **type** inference. diff --git a/.cursor/skills/dataframes/references/reference-pandas-core.md b/.cursor/skills/dataframes/references/reference-pandas-core.md new file mode 100644 index 00000000..7080e97a --- /dev/null +++ b/.cursor/skills/dataframes/references/reference-pandas-core.md @@ -0,0 +1,39 @@ +# pandas: index, selection, dtypes, mutation + +Official: [Indexing and selecting data](https://pandas.pydata.org/docs/user_guide/indexing.html), [Copy-on-Write](https://pandas.pydata.org/docs/user_guide/copy_on_write.html). + +## Index and columns + +- **`Index`** carries **labels** and optional **`name`**; **`MultiIndex`** for hierarchical keys. +- **`reset_index` / `set_index`**: make columns into index or the reverse when modeling needs change. + +## `loc`, `iloc`, `[]` + +- **`loc`**: label-based (slice end **inclusive** on labels). +- **`iloc`**: integer position (Python slice end **exclusive**). +- **`df[col]`** for a single column **Series**; **`df[[col]]`** for one-column **DataFrame**. +- Avoid **chained indexing** (`df[a][b] = x`); use **`loc`** or **`assign`**. + +## Copy-on-Write (pandas 2+) + +- With **CoW** enabled (recommended in modern defaults), many chained reads are safer; still treat **writes** as requiring explicit **`loc`** assignment. +- **`copy()`** when you must guarantee an independent buffer before mutation. + +## dtypes + +- **Nullable**: **`Int64`**, **`string`**, **`boolean`** (capitalized extension dtypes) vs **`object`** for messy text. +- **`astype`** can widen or lose precision; prefer **`pd.to_numeric(..., errors="coerce")`** for ingestion. +- **Categorical** for low-cardinality strings saves memory and speeds **groupby**. + +## Assignment + +- **`df.loc[row_indexer, col] = values`**; align with **index** on the RHS when assigning Series. +- **`pd.concat`** along axis for new rows/columns instead of repeated **`append`** in hot paths. + +## Series + +- **Alignment** on index in **arithmetic** with other Series/DataFrame—often a feature, sometimes a bug; **`reset_index`** or **`.values`** when you intend **positional** math. + +## Performance hints + +- **Vectorized** ops on columns; **`apply`** row-wise is slow at scale—reshape or use **Polars/NumPy**. diff --git a/.cursor/skills/dataframes/references/reference-pandas-group-join.md b/.cursor/skills/dataframes/references/reference-pandas-group-join.md new file mode 100644 index 00000000..faf36713 --- /dev/null +++ b/.cursor/skills/dataframes/references/reference-pandas-group-join.md @@ -0,0 +1,34 @@ +# pandas: groupby, merge, reshape + +Official: [Group by](https://pandas.pydata.org/docs/user_guide/groupby.html), [Merge, join, concatenate](https://pandas.pydata.org/docs/user_guide/merging.html). + +## GroupBy + +- **`groupby(keys, as_index=...)`** then **`.agg`**, **`.transform`**, **`.filter`**. +- **`as_index=False`** keeps grouping columns as columns. +- **Named aggregation**: **`.agg(mean_x=("x", "mean"))`** for clear output names. +- **Watch**: **NA** in keys drops groups unless **`dropna=False`** (version-dependent defaults—check docs). + +## Merge and join + +- **`merge`**: SQL-style on **columns**; **`how`** (`inner`, `left`, `right`, `outer`); **`on`**, **`left_on`/`right_on`**; **`suffixes`** for overlapping names. +- **`join`**: index-based; easy to misuse if indexes are not unique—validate **`validate`** (`"one_to_one"`, etc.) when available. +- **Row explosion** from duplicate keys: assert **uniqueness** or **deduplicate** before merge when unintended. + +## Concat + +- **`pd.concat`** along **`axis=0`** (stack) or **`axis=1`** (side by side); **`keys`** for MultiIndex source labels. + +## Reshape + +- **`pivot`**, **`pivot_table`**, **`melt`**, **`stack`/`unstack`** for wide ↔ long. +- **`crosstab`** for counts with optional normalization. + +## Sort and rank + +- **`sort_values`**, **`sort_index`**; **`rank`** with **`method`** for ties. + +## Validation + +- **`merge`** + **`indicator=True`** to audit **left_only / right_only / both**. +- **`assert_frame_equal`** in tests with **`check_dtype`** toggled as needed. diff --git a/.cursor/skills/dataframes/references/reference-polars.md b/.cursor/skills/dataframes/references/reference-polars.md new file mode 100644 index 00000000..df142f02 --- /dev/null +++ b/.cursor/skills/dataframes/references/reference-polars.md @@ -0,0 +1,46 @@ +# Polars: LazyFrame, expressions, group, join + +Official: [User guide](https://docs.pola.rs/user-guide/), [Expressions](https://docs.pola.rs/user-guide/expressions/). + +## Eager vs lazy + +- **Eager** `DataFrame`: small/medium data, interactive work. +- **Lazy** `LazyFrame`: **`pl.scan_parquet`**, **`pl.scan_csv`**, **`df.lazy()`**; chain transforms, then **`collect()`** (or **`collect(streaming=True)`** when enabled for your version). + +## Expressions + +- **`pl.col("a")`**, **`pl.col("^pat.*$")`**, **`pl.all()`**. +- **`.alias`**, **`.cast`**, **`.fill_null`**, **`.replace`**, string and datetime **`.str.*` / `.dt.*`** namespaces. +- **Conditional**: **`pl.when(cond).then(x).otherwise(y)`**. +- **Horizontal**: **`pl.sum_horizontal`**, **`pl.concat_list`** for list columns. + +## select / with_columns / filter + +- **`select`** chooses/reorders; **`with_columns`** adds or replaces by name; **`filter`** is row predicate. +- **`with_row_index`** when you need a stable row id column. + +## Joins + +- **`join`**, **`join_asof`** for temporal alignment; check **how** (`inner`, `left`, `semi`, `anti`). +- **Suffix** for name clashes; **validate** row counts like in SQL (assert in code). + +## Group by + +- **`group_by("k").agg(pl.col("x").mean())`**; multiple metrics in one **`agg`**. +- **Dynamic groups** for time windows: **`group_by_dynamic`** when applicable. + +## Sort and distinct + +- **`sort`**, **`unique`**, **`n_unique`**; **`is_duplicated`** for QA. + +## Strings and categoricals + +- **String** dtype; **Categorical** for low-cardinality columns to save memory in **lazy** plans. + +## Errors + +- Polars tends to **fail fast** on schema issues; fix **dtypes** at **scan** or first **`with_columns`**. + +## pandas interop + +- **`df.to_pandas()`**, **`pl.from_pandas(df)`**; see [reference-interop-performance.md](reference-interop-performance.md). diff --git a/.cursor/skills/dataframes/references/reference-when-which.md b/.cursor/skills/dataframes/references/reference-when-which.md new file mode 100644 index 00000000..040f4228 --- /dev/null +++ b/.cursor/skills/dataframes/references/reference-when-which.md @@ -0,0 +1,32 @@ +# When to use pandas vs Polars + +Aligned with the project **Python** spec: explicit **index/column** semantics in pandas, **lazy** Polars for heavy queries when the project standardizes on it. + +## Prefer **pandas** + +- **Row labels** are part of the model (time series index alignment, `reindex`, `asfreq`). +- **Wide** mix of dtypes in one table with frequent **per-column** Python logic. +- **Downstream** expects pandas (`sklearn` with minimal friction, statsmodels, many tutorials). +- **Incremental** cell updates and **in-place**-style workflows are entrenched (still prefer explicit assignment). + +## Prefer **Polars** + +- **Large** files: **`scan_parquet` / `scan_csv`** and **lazy** plans push work to the engine. +- **Complex column expressions** without index alignment surprises: **`pl.col`** pipelines read linearly. +- **Parallel** group-by and joins on **big** tables (hardware-dependent; profile). +- **Stricter** null model (`null`) and **dtype** consistency across the query. + +## One boundary per layer + +- Do not **alternate** libraries inside a tight inner loop; **convert once** at imports/exports of a stage (ETL end, modeling start, plot start). +- Document the **owner** of the index: pandas keeps it on the frame; Polars is mostly **column-first** (row index is positional unless you add a column). + +## Time series + +- **pandas**: **`DatetimeIndex`**, **`resample`**, **`rolling`**, **`tz`** handling are mature. +- **Polars**: temporal **expressions** and **`group_by_dynamic`** fit large **event** tables; verify **timezone** rules for your version. + +## Testing + +- **Assert** shapes and **key uniqueness** after joins in both libraries. +- **`python-reviewer`** for merge footguns and silent **`NaN`** propagation. diff --git a/.cursor/skills/general-python/SKILL.md b/.cursor/skills/general-python/SKILL.md new file mode 100644 index 00000000..665ce2ac --- /dev/null +++ b/.cursor/skills/general-python/SKILL.md @@ -0,0 +1,54 @@ +--- +author: dotagents +name: general-python +description: Python stack hub: uv/ruff/ty workflows, builtins and collections, functions and classes, dataclasses, typing boundaries, pytest and NumPy-style docs. Use for any Python task in repos that follow this stack; pairs with python-reviewer, python-types, and python-refactor. Related skills: numpy-scientific, dataframes, numpy-docstrings, matplotlib-scientific, lab-instrumentation. Triggers on Python, uv, ruff, ty, pytest, dataclass, typing. +--- + +# General Python + +This skill is the **hub** for conventions that come from the project **Python** section (often merged into **AGENTS.md**), the **Python Cursor rule** on `**/*.py`, and the **python-reviewer** / **python-types** / **python-refactor** subagents. Load topic files below instead of drifting from project defaults. + +## Synergy map + +| Source | Role | +|--------|------| +| **Python spec** (AGENTS.md block) | Canonical tooling (uv, ruff, ty), style, pytest, numerics, tables, instruments | +| **Python rule** | Globs `**/*.py`: 3.12+, uv, NumPy-style public docstrings, pyvisa split | +| **python-reviewer** | Post-change review: uv hygiene, typing, numerics footguns, tests | +| **python-types** | Deep typing for Astral **ty**, PEP 695, exhaustive `match` | +| **python-refactor** | Structure: tuples vs dataclasses, composition vs inheritance, size | +| **matplotlib-scientific** | Figures only; keep plotting out of this hub | +| **numpy-scientific** | `ndarray` design: dtypes, views, broadcasting, ufuncs, `linalg`, I/O, random | +| **dataframes** | pandas + Polars: tables, lazy frames, joins, I/O, interop | +| **numpy-docstrings** | numpydoc sections for public API docstrings | +| **lab-instrumentation** | PyVISA, sockets, HALs, validation, PDF datasheets, instrument tests | + +## Quick decisions + +1. **Dependencies**: `uv add` / `uv remove` / `uv sync` / `uv run`. Never hand-edit version pins in `pyproject.toml`. See [reference-tooling.md](references/reference-tooling.md). +2. **Quality gate**: `ruff check` (and format per project config), then **`ty check`** on touched code. See [reference-tooling.md](references/reference-tooling.md). +3. **Data**: prefer the right **builtin or collections ABC** before a custom class; use **dataclasses** for labeled bundles of data. See [reference-data-structures.md](references/reference-data-structures.md) and [reference-classes-dataclasses.md](references/reference-classes-dataclasses.md). +4. **APIs**: small **pure** core, **I/O at edges**, explicit types on public boundaries. See [reference-functions.md](references/reference-functions.md) and [reference-types-and-agents.md](references/reference-types-and-agents.md). +5. **Tests and docs**: `uv run pytest`; **NumPy-style docstrings** on public APIs. See [reference-testing-docs.md](references/reference-testing-docs.md). + +## Reference index + +| Topic | File | +|--------|------| +| uv, ruff, ty, layout, CI-style verification | [reference-tooling.md](references/reference-tooling.md) | +| list, dict, tuple, set, comprehensions, `collections` | [reference-data-structures.md](references/reference-data-structures.md) | +| Functions: purity, parameters, errors, resources | [reference-functions.md](references/reference-functions.md) | +| Classes, dataclasses, slots, immutability | [reference-classes-dataclasses.md](references/reference-classes-dataclasses.md) | +| Types, ty, when to delegate to **python-types** | [reference-types-and-agents.md](references/reference-types-and-agents.md) | +| pytest, docstrings, readability | [reference-testing-docs.md](references/reference-testing-docs.md) | +| Vectorized numerics, tables, instruments, plotting boundary | [reference-scientific-numerics.md](references/reference-scientific-numerics.md) | + +## External anchors + +- [uv](https://docs.astral.sh/uv/latest/), [Ruff](https://docs.astral.sh/ruff/), [ty](https://docs.astral.sh/ty/) (Astral) +- [PEP 8](https://peps.python.org/pep-0008/) style surface; project **ruff** rules are authoritative +- [Python data model](https://docs.python.org/3/reference/datamodel.html) for dunder semantics + +## Modern practice snapshot + +Reproducible projects combine **locked deps**, **fast lint/format**, and **static typing** on API surfaces; **src layout** avoids import ambiguity; automate checks in CI or pre-commit. See [reference-tooling.md](references/reference-tooling.md). diff --git a/.cursor/skills/general-python/references/reference-classes-dataclasses.md b/.cursor/skills/general-python/references/reference-classes-dataclasses.md new file mode 100644 index 00000000..a2f90ee8 --- /dev/null +++ b/.cursor/skills/general-python/references/reference-classes-dataclasses.md @@ -0,0 +1,40 @@ +# Classes and dataclasses + +## When to use a class + +- **Stateful service** with invariants (device client, session, parser with buffer). +- **True is-a** hierarchy with shared behavior; otherwise prefer **composition** and **`Protocol`** (see **`python-refactor`**). + +## Dataclasses (`dataclasses`) + +- Use for **labeled data bundles** with minimal behavior: configs, results, DTOs between layers. +- **`frozen=True`**: immutable value objects; safer hashing and sharing. +- **`slots=True`** (3.10+): lower memory and faster attribute access when you do not need a `__dict__`. +- **`order=True`**: only when ordering is meaningful and documented. +- **`field(default_factory=…)`** for mutable defaults (lists, dicts). +- Replace opaque **`tuple[int, int, int, int]`** returns with a **frozen dataclass** or **`NamedTuple`** when names carry meaning. + +## Regular classes + +- **`@staticmethod`**: rare; often a module function is clearer. +- **`@classmethod`**: alternate constructors (`from_path`, `from_config`). +- **`__init__`**: minimal; heavy work in dedicated methods or factories so tests stay small. + +## Special methods + +- Implement **`__repr__`** for debuggability; **`__str__`** when user-facing text differs. +- **`__eq__`**: dataclass generates it; hand-roll only with clear semantics. + +## Dataclass vs TypedDict vs NamedTuple + +- **`TypedDict`**: JSON-like dict shapes, especially for **`**kwargs`** typing. +- **`NamedTuple`**: immutable, tuple-unpacking ergonomics, light memory. +- **`dataclass`**: mutable or frozen records with optional methods. + +## Typing and tooling + +- After modeling data, run **`ty check`**; align with **`python-types`** for generics and protocols. + +## Delegation + +- **Inheritance vs composition**, god classes: **`python-refactor`** agent. diff --git a/.cursor/skills/general-python/references/reference-data-structures.md b/.cursor/skills/general-python/references/reference-data-structures.md new file mode 100644 index 00000000..7fa64304 --- /dev/null +++ b/.cursor/skills/general-python/references/reference-data-structures.md @@ -0,0 +1,42 @@ +# Builtins and collections + +## Choosing a structure + +| Need | Prefer | +|------|--------| +| Ordered sequence, homogeneous | **`list`**; fixed small arity: **`tuple`** | +| Key-value lookup, unique keys | **`dict`** (3.7+ insertion order is part of language spec) | +| Membership testing, uniqueness | **`set`** / **`frozenset`** when immutable | +| Immutable record of fields | **`tuple`**, **`NamedTuple`**, or **`@dataclass(frozen=True)`** (see [reference-classes-dataclasses.md](reference-classes-dataclasses.md)) | +| FIFO / LRU-ish queue | **`collections.deque`** | +| Counting | **`collections.Counter`** | +| Grouping by key | **`collections.defaultdict(list)`** (or similar) | +| Read-only mapping view | **`Mapping`** from **`collections.abc`** in type hints | + +## Comprehensions and literals + +- Prefer **comprehensions** or **generator expressions** over `map`/`filter` when readability wins. +- Avoid **mutable default arguments**; use `None` and assign inside the function or use **`dataclasses.field(default_factory=…)`**. + +## Immutability and copying + +- **`tuple`**, **`frozenset`**, **frozen dataclass**: safe as dict keys and for defensive sharing. +- **Shallow vs deep copy**: default assignment and `list(old)` share nested mutables; use **`copy.deepcopy`** only when the data model requires it. + +## Sorting and keys + +- **`sorted(iterable, key=…)`** and **`list.sort(key=…)`** for stable, explicit ordering. +- **`bisect`** on sorted sequences for insertion and search (stdlib). + +## Typing shapes + +- **`list[int]`**, **`dict[str, float]`**, **`set[str]`** in annotations; read-only parameters as **`Sequence[T]`**, **`Mapping[str, T]`** from **`collections.abc`** when you do not need mutation. + +## When not to use a dict + +- Fixed, named fields with validation: consider **dataclass** or **Pydantic** (if project already depends on it). Do not add Pydantic via hand-edited pins; use **`uv add`**. + +## Scientific stacks + +- **NumPy arrays** for numeric tensors; **pandas** / **polars** for labeled tables per the Python spec; do not emulate DataFrames with nested dicts unless prototyping. +- Tabular workflows: **dataframes** skill. diff --git a/.cursor/skills/general-python/references/reference-functions.md b/.cursor/skills/general-python/references/reference-functions.md new file mode 100644 index 00000000..40a11d7a --- /dev/null +++ b/.cursor/skills/general-python/references/reference-functions.md @@ -0,0 +1,37 @@ +# Functions: design and boundaries + +Aligned with **python-refactor** (functional core, imperative shell) and the project **Python** spec (clear control flow, minimal narrative comments). + +## Size and shape + +- One **obvious responsibility** per function; long procedures become **named steps** or small private helpers. +- Prefer **pure** functions in the core (deterministic from inputs; no hidden globals). Put **I/O**, **time**, **randomness**, and **global config** at the edges. + +## Signatures + +- **Explicit parameters** over catch-all `*args` unless wrapping or forwarding. +- **`**kwargs`**: prefer **`Unpack[TypedDict]`** (PEP 692) for structured options when using modern typing; otherwise document keys strictly. +- **Returns**: prefer **single typed return**; multiple values as **named tuple**, **dataclass**, or **TypedDict** (see **python-refactor** agent for tuple smell). + +## Errors + +- Raise **specific exceptions**; avoid bare `except:`. +- Let exceptions carry **actionable messages**; use exception chaining (`raise … from e`) when re-raising. +- For expected failure modes, document **what callers should catch**. + +## Resources + +- **`with`** / context managers for files, locks, devices; **pyvisa** resource separation per the Python spec and rule. + +## Idioms + +- **`pathlib.Path`** for filesystem paths when the codebase already does; stay consistent within a module. +- **`if __name__ == "__main__":`** for CLI entrypoints; prefer **`uv run`** for execution in projects. + +## Types at the boundary + +- Annotate **public** functions completely; internals can lean on inference where **ty** still passes. Details: [reference-types-and-agents.md](reference-types-and-agents.md). + +## Delegation + +- **Structural** critique (too large, wrong abstraction): **`python-refactor`** agent. diff --git a/.cursor/skills/general-python/references/reference-scientific-numerics.md b/.cursor/skills/general-python/references/reference-scientific-numerics.md new file mode 100644 index 00000000..c8a6cb08 --- /dev/null +++ b/.cursor/skills/general-python/references/reference-scientific-numerics.md @@ -0,0 +1,29 @@ +# Scientific and lab defaults + +From the project **Python** spec (general guidelines and style). + +## Numerics + +- Prefer **vectorized** NumPy / SciPy (or array libraries the project uses) over Python loops on **large** arrays. +- Be explicit about **shapes**, **dtypes**, and **missing data**; watch silent **dtype widening** and **unstable reduction order** when it affects science or reproducibility. +- **`python-reviewer`** is the right agent for numerics footguns in review. +- Detailed NumPy patterns: **numpy-scientific** skill. + +## Tables and time series + +- **pandas**: explicit **index/column** semantics; know **views vs copies** for chained assignment. +- **polars**: prefer **lazy** frames when the codebase standardizes on Polars for heavy queries. + +Pick one table stack per module or project layer; do not mix idioms in one pipeline without a boundary. + +- Combined guidance: **dataframes** skill. + +## Instruments (PyVISA, sockets, lab I/O) + +- Separate **resource open/close and configuration** from **command strings** and parsing. +- Use **context managers** or explicit lifecycle so handles are not leaked across tests. +- Full patterns: **lab-instrumentation** skill (PyVISA sessions, sockets vs VISA, HALs, validation, PDF manuals, offline tests). + +## Plotting + +- Use the **`matplotlib-scientific`** skill for figure quality; keep analysis modules free of ad hoc `plt` state when a library API is expected. diff --git a/.cursor/skills/general-python/references/reference-testing-docs.md b/.cursor/skills/general-python/references/reference-testing-docs.md new file mode 100644 index 00000000..a3466e90 --- /dev/null +++ b/.cursor/skills/general-python/references/reference-testing-docs.md @@ -0,0 +1,29 @@ +# Testing and documentation + +From the project **Python** spec and **Python Cursor rule**. + +## pytest + +- Install in **dev** group: **`uv add pytest --dev`** (or project group). +- Run: **`uv run pytest`**; narrow with path or `-k` as needed. +- Prefer **fast, deterministic** unit tests; isolate **I/O** and **flaky timing** in markers or separate jobs when the team agrees. + +## Docstrings + +- **NumPy style** on **public** APIs: Parameters, Returns, Raises, Examples when they clarify non-obvious contracts. +- Keep **implementations** readable **without** step-by-step narrative comments (per Python spec / rule). +- Section-by-section guide: **numpy-docstrings** skill. + +## What to test + +- **Pure logic**: parametrized cases and edge values. +- **Bug fixes**: regression test that fails before the fix. +- **Numerics** (scientific code): dtype/shape expectations and stable reductions when the paper or spec demands it (**python-reviewer**). + +## Imports and package layout + +- Tests should import the **installed package** (`src` layout) so CI matches user installs. + +## Review + +- Before merge, use **`python-reviewer`** for uv + typing + tests + numerics in one pass. diff --git a/.cursor/skills/general-python/references/reference-tooling.md b/.cursor/skills/general-python/references/reference-tooling.md new file mode 100644 index 00000000..620541c9 --- /dev/null +++ b/.cursor/skills/general-python/references/reference-tooling.md @@ -0,0 +1,46 @@ +# Tooling: uv, Ruff, ty, project shape + +Aligned with the project **Python** spec (often in **AGENTS.md**) and the **Python Cursor rule**. + +## uv + +- **Add / remove**: `uv add `, `uv add --dev `, `uv add --group `, `uv remove `, `uv add --upgrade`. +- **Environment**: `uv sync` after lock or clone. +- **Run**: `uv run python …`, `uv run pytest`, `uv run ruff check .`, `uv run ty check`. +- **Do not** edit dependency version pins in `pyproject.toml` by hand; use **`uv add`** so the lockfile stays truthful. +- Docs: [uv](https://docs.astral.sh/uv/latest/). + +## Ruff + +- **Lint**: `ruff check` (or `uv run ruff check`). Fix what the project enables (E/F/I/B/UP/… per `pyproject.toml`). +- **Format**: `ruff format` when the repo uses Ruff as formatter. +- Treat **project config as law** over generic PEP 8 advice. +- Docs: [Ruff](https://docs.astral.sh/ruff/). + +## ty + +- **Typecheck**: `ty check` (or `uv run ty check`) on the project or paths you changed. +- Ensure **ty** and **ruff** stay in the **dev** dependency group per spec. +- Docs: [ty](https://docs.astral.sh/ty/). + +## Typical verification sequence + +1. `ruff check` (and format if applicable) +2. `ty check` +3. `uv run pytest` + +Order may vary; fix **ruff** before **ty** when both report the same line. + +## Project layout (greenfield or refactors) + +- Prefer **`src//`** layout so tests and tooling import the package the same way users do (avoids accidental imports from repo root). Pair with **`pyproject.toml`** (PEP 621). +- Keep **lockfile** (`uv.lock`) committed when the team relies on reproducible installs. + +## Automation + +- **pre-commit** or CI running ruff + ty + pytest catches drift early (common 2024–2025 practice; align with team policy). + +## Delegation + +- Broad **review** after substantive edits: **`python-reviewer`** agent. +- Heavy **annotation** passes: **`python-types`** agent. diff --git a/.cursor/skills/general-python/references/reference-types-and-agents.md b/.cursor/skills/general-python/references/reference-types-and-agents.md new file mode 100644 index 00000000..491f6724 --- /dev/null +++ b/.cursor/skills/general-python/references/reference-types-and-agents.md @@ -0,0 +1,35 @@ +# Types and Astral ty + +This file is the **bridge** between day-to-day edits and the **`python-types`** agent. Do not duplicate the full typing playbook here. + +## Project rule + +- **`ty check`** is the type gate; configuration lives in **`pyproject.toml`**. +- Prefer fixing types over **`# type: ignore`**; if unavoidable, **one line**, narrow code, and state **why**. + +## Boundaries + +- **Public** functions and methods: annotate **parameters and return**. +- **Locals**: rely on inference when obvious (`x = []` may need `list[T]` or construction that fixes `T`). +- Prefer **`collections.abc`** (`Sequence`, `Mapping`, `Iterable`) for inputs you only read. + +## Python 3.12+ (when helpful) + +- **PEP 695** `type` aliases and `def f[T](...)`. +- **PEP 692** `Unpack[TypedDict]` for kwargs. +- **PEP 698** `@override` on overrides. +- **PEP 688** `Buffer` over deprecated `ByteString` patterns. + +Reference: [What is new in Python 3.12](https://docs.python.org/3/whatsnew/3.12.html). + +## Exhaustive discrimination + +- Use **`match` / `case`** so **ty** can verify coverage; **`assert_never`** for impossible tails after exhaustive matches. See **`python-types`** and [match exhaustiveness](https://dogweather.dev/2022/10/03/i-discovered-that-python-now-can-do-true-match-exhaustiveness-checking/). + +## When to invoke **python-types** + +- Large refactors to protocols/generics, fixing many **ty** errors, or aligning modules with **PEP 695** and **`match`** exhaustiveness. + +## Ruff and types + +- **Ruff** may flag typing imports (`TCH`, `UP`); keep **runtime** vs **TYPE_CHECKING** imports consistent with project rules. diff --git a/.cursor/skills/general/SKILL.md b/.cursor/skills/general/SKILL.md new file mode 100644 index 00000000..c5f6c735 --- /dev/null +++ b/.cursor/skills/general/SKILL.md @@ -0,0 +1,16 @@ +--- +author: dotagents +name: general +description: Applies cross-language defaults for scientific/engineering work, documentation discipline, non-lazy execution, and effective use of rules, skills, and documentation tools. +--- + +When this skill applies: + +- Keep diffs minimal and aligned with the stated specification; extend or reuse existing abstractions before introducing new ones. +- Produce complete code paths rather than templates or ellipses; finish the requested slice end-to-end when the user asks for an implementation. +- Document public library surfaces with prescriptive contracts (parameters, results, errors, stability or units when relevant). Keep private helpers thinly documented. +- Add or refine module-level intent so each library area states scope, non-goals, and invariants. +- Use available agent skills, project rules, and MCP documentation retrieval for unfamiliar APIs instead of guessing. +- On tool or command failure, troubleshoot and retry sensibly rather than stopping at the first error without analysis. +- Avoid emoji in technical artifacts unless the user explicitly requests them. +- Avoid unsolicited markdown documentation; prefer code and tests as the source of truth. diff --git a/.cursor/skills/lab-instrumentation/SKILL.md b/.cursor/skills/lab-instrumentation/SKILL.md new file mode 100644 index 00000000..04b3768e --- /dev/null +++ b/.cursor/skills/lab-instrumentation/SKILL.md @@ -0,0 +1,54 @@ +--- +author: dotagents +name: lab-instrumentation +description: PyVISA and lab instrument control in Python: VISA resource lifecycle, timeouts and terminations, raw TCP/UDP sockets vs VISA, when to introduce hardware abstraction layers, validating user and config input before hardware, testing without hardware, and extracting tables or text from instrument manuals and datasheets (PDF). Triggers on pyvisa, VISA, GPIB, USB-TMC, serial instrument, socket lab, SCPI, datasheet PDF, instrument driver. +--- + +# Lab instrumentation (PyVISA, sockets, HAL) + +## Quick start + +1. **Split layers**: one place owns **open/configure/close** and **timeouts**; another owns **SCPI or device protocol** (string formatting, parsing floats/units, error queues). See [reference-pyvisa-visa.md](references/reference-pyvisa-visa.md). +2. **Pick transport deliberately**: **VISA** when the stack already provides enumeration, buffering, and vendor backends; **plain sockets** when you own framing, keep-alive, and binary protocols end to end. See [reference-socket-comms.md](references/reference-socket-comms.md). +3. **Add a HAL when** you have **multiple models** behind one experiment API, **two transports** for the same logical device, or **tests** that need a stable seam. Do not wrap a single serial string in three classes. See [reference-hardware-abstraction.md](references/reference-hardware-abstraction.md). +4. **Validate before I/O**: normalize and bound **numeric setpoints**, **enum-like modes**, and **user-entered strings**; reject unknown commands early with **actionable errors**. See [reference-input-validation-errors.md](references/reference-input-validation-errors.md). +5. **PDFs**: use **text-layer** extraction first; fall back to **table-aware** tools for spec tables; reserve **OCR** for scan-only manuals. See [reference-pdf-datasheets.md](references/reference-pdf-datasheets.md). +6. **Tests**: default **fast** unit tests on parsers and validators; gate **integration** tests; use **fakes** or **recorded transcripts** for CI. See [reference-testing-mocks.md](references/reference-testing-mocks.md). + +## Stack synergy + +| Resource | Role | +|----------|------| +| **general-python** | uv, context managers, **`python-reviewer`** | +| **numpy-scientific** | Numeric buffers, waveforms, dtype-safe binary payloads | +| **numpy-docstrings** | Public driver and session APIs | + +## Reference index + +| Topic | File | +|--------|------| +| ResourceManager, sessions, timeouts, terminations, SCPI hygiene | [reference-pyvisa-visa.md](references/reference-pyvisa-visa.md) | +| TCP/UDP sockets, framing, timeouts, binary vs text | [reference-socket-comms.md](references/reference-socket-comms.md) | +| When to build HALs, protocols vs transports, registries | [reference-hardware-abstraction.md](references/reference-hardware-abstraction.md) | +| Ranges, enums, command allowlists, error shape | [reference-input-validation-errors.md](references/reference-input-validation-errors.md) | +| PDF text, tables, datasheets, bibliography PDFs, OCR | [reference-pdf-datasheets.md](references/reference-pdf-datasheets.md) | +| pytest, fakes, simulation, recordings | [reference-testing-mocks.md](references/reference-testing-mocks.md) | + +## Dependencies + +Add with **`uv add`** (do not hand-edit pins in `pyproject.toml`): + +| Need | Typical packages | +|------|------------------| +| VISA in Python | **`pyvisa`**; optional **`PyVISA-py`** for pure-Python backend without vendor IVI | +| Serial-backed VISA | Often **`pyserial`** alongside backend docs | +| PDF text | **`pymupdf`** (fitz), **`pypdf`**, or **`pdfplumber`** (tables) | +| Scanned PDFs | **`pdf2image`** + **`pytesseract`** when OCR is unavoidable | + +## Official and canonical docs + +- [PyVISA documentation](https://pyvisa.readthedocs.io/en/latest/) +- [Python `socket` module](https://docs.python.org/3/library/socket.html) +- [PyMuPDF documentation](https://pymupdf.readthedocs.io/) +- [pdfplumber](https://github.com/jsvine/pdfplumber) +- [pypdf](https://pypdf.readthedocs.io/) diff --git a/.cursor/skills/lab-instrumentation/references/reference-hardware-abstraction.md b/.cursor/skills/lab-instrumentation/references/reference-hardware-abstraction.md new file mode 100644 index 00000000..651ad604 --- /dev/null +++ b/.cursor/skills/lab-instrumentation/references/reference-hardware-abstraction.md @@ -0,0 +1,27 @@ +# Hardware abstraction classes + +## When a HAL pays off + +- **Multiple instrument models** implement the same experiment step (e.g. three power supply families under **`set_voltage(channel, volts)`**). +- **Two transports** for one logical device (**USB** vs **Ethernet** simulation) and you want one **experiment** module. +- **Unit tests** must run without drivers: you need a **`Protocol`** or **ABC** to **fake** behind stable methods. +- **Shared cross-cutting** behavior: logging, rate limits, **mutex** around non-reentrant firmware, **idempotency** after **`clear`**. + +## When to skip + +- **One device**, **one script**, **one resource string**: a **single module** with **functions** and a **`with`** block is simpler than a **registry** and **abstract base**. +- **SCPI one-offs** in a notebook: keep **thin** wrappers; extract a class only when the same commands appear in **three** places. + +## Shape of a good HAL + +- **`Instrument` protocol** (or ABC): **`connect()`**, **`disconnect()`**, **`idn()`**, **`reset()`** if applicable, and **domain methods** (`measure_voltage`, `arm_trigger`)—not raw **`write`** on the public surface unless the layer is explicitly **low-level**. +- **`Transport` vs `ProtocolHandler`**: **transport** reads bytes; **handler** turns bytes into **typed** results. Lets you test **parsing** without **hardware**. +- **Composition over deep inheritance**: **`PowerSupply`** wraps a **`VisaSession`** or **`TcpSession`**; avoid diamond hierarchies across vendors. + +## Configuration + +- **Model string** or **resource string** from **config file** or **env**; validate at startup. **Fail fast** if **`idn`** does not match **expected prefix** for safety interlocks. + +## Discovery and plugins + +- Optional **registry** mapping **`model_id` → class`** when you ship many drivers in one package; keep registration **explicit** (import side effects are hard to test). diff --git a/.cursor/skills/lab-instrumentation/references/reference-input-validation-errors.md b/.cursor/skills/lab-instrumentation/references/reference-input-validation-errors.md new file mode 100644 index 00000000..59fc1713 --- /dev/null +++ b/.cursor/skills/lab-instrumentation/references/reference-input-validation-errors.md @@ -0,0 +1,30 @@ +# Validating user and config input before hardware + +## Principles + +- **Validate at boundaries**: CLI, REST body, **GUI field**, or **YAML/JSON** config—before any **`write`** to hardware. +- **Reject early**: unknown **mode strings**, **out-of-range** voltages, or **wrong units** should raise **typed exceptions** with **what failed**, **allowed range**, and **observed value**—not a raw **`VisaIOError`** after the fact. + +## Numeric setpoints + +- Parse with **explicit types** (`Decimal` for money-like precision, **`float`** only when the manual specifies float-friendly steps). Compare to **min/max** from **datasheet** or **constants** in code, not magic numbers scattered across callers. +- **Quantize** to instrument resolution when the device **snaps** to steps (e.g. **1 mV** steps); document rounding **toward zero** vs **nearest**. + +## Command and mode strings + +- Prefer **`Literal`** or **`Enum`** in public APIs instead of **free-form** strings for **SCPI subsystems**. +- If users type SCPI, maintain an **allowlist** or **prefix allowlist**; never pass unchecked input to **`write`** when it can reach **shells** or **file paths** on controllers. + +## Resource strings and hosts + +- Validate **`host:port`** formats, **IP literals**, and **resource strings** with **regex or urllib** patterns before **`open_resource`** or **`connect`**. +- **Path traversal** matters for **save/recall** instrument setups that use filenames on the controller. + +## Error design + +- Use **small exception types** (`OutOfRange`, `UnknownMode`, `InstrumentFault`) that carry **context**; map **VISA status** and **socket errno** into them at the **transport edge** so experiment code stays readable. +- For operator-facing messages, separate **log detail** (full traceback internally) from **user text** (one sentence + fix hint). + +## Idempotency + +- **Repeated “set 5 V”** should be safe; **“arm”** may not be—document which operations are **idempotent** in the HAL docstrings. diff --git a/.cursor/skills/lab-instrumentation/references/reference-pdf-datasheets.md b/.cursor/skills/lab-instrumentation/references/reference-pdf-datasheets.md new file mode 100644 index 00000000..7856e01d --- /dev/null +++ b/.cursor/skills/lab-instrumentation/references/reference-pdf-datasheets.md @@ -0,0 +1,38 @@ +# PDF parsing for datasheets, manuals, and reference PDFs + +## Strategy + +1. **Detect text vs scan**: open the PDF and try **text extraction** on a sample page. If output is empty or garbage, the file is likely **image-only**—plan **OCR** or obtain a **text** export from the vendor. +2. **Preserve structure**: tables need **table-aware** tools; body text needs **blocks** with **bbox** when aligning columns. +3. **Ground truth**: for critical limits (max voltage, derating), **cross-check** against a **second source** (vendor HTML or printed table)—PDF extraction is **lossy**. + +## Libraries (Python) + +| Goal | Typical choice | Notes | +|------|----------------|-------| +| Fast text + layout | **[PyMuPDF](https://pymupdf.readthedocs.io/)** (`fitz`) | **`get_text("dict")`** for blocks/lines; good speed | +| Tables in datasheets | **[pdfplumber](https://github.com/jsvine/pdfplumber)** | **`extract_tables()`** tuned with **`table_settings`** | +| Lightweight merge/split/metadata | **[pypdf](https://pypdf.readthedocs.io/)** | Good for **page ranges** and **bookmarks**, weaker on messy tables | +| Scanned pages | **`pdf2image`** + **`pytesseract`** | Tune **DPI** (often 300+); expect **manual cleanup** | + +## Datasheet workflows + +- **Extract spec tables** (temperature range, absolute maxima) into **structured rows** (CSV/Parquet) with **column headers** normalized once in code. +- **Version** the PDF (filename hash or vendor revision field) and **store** it beside extracted data so results are **reproducible**. + +## Reference books and papers + +- **Bibliography PDFs**: often **two-column**; use **block sorting** by **y, x** (PyMuPDF dict mode) before regex for **DOI**, **ISBN**, or **citation keys**. +- **Equations and figures**: extraction will not reliably capture math; link **page number** + **figure ID** instead of parsing **LaTeX** from raster. + +## Pitfalls + +- **Hyphenation** and **line breaks** split words across lines—normalize whitespace and **de-hyphenate** cautiously. +- **Embedded fonts** and **subset encoding** can map glyphs oddly; if strings look wrong, try **`get_text("text")`** vs **dict** mode or another backend. +- **Legal**: scraping **paywalled** PDFs may violate terms; only process files you **have rights** to use in your pipeline. + +## Further reading + +- PyMuPDF [tutorial / text](https://pymupdf.readthedocs.io/en/latest/tutorial.html) +- pdfplumber [README examples](https://github.com/jsvine/pdfplumber/blob/stable/README.md) +- pypdf [user guide](https://pypdf.readthedocs.io/en/stable/user/introduction.html) diff --git a/.cursor/skills/lab-instrumentation/references/reference-pyvisa-visa.md b/.cursor/skills/lab-instrumentation/references/reference-pyvisa-visa.md new file mode 100644 index 00000000..135244da --- /dev/null +++ b/.cursor/skills/lab-instrumentation/references/reference-pyvisa-visa.md @@ -0,0 +1,34 @@ +# PyVISA and VISA sessions + +## Architecture + +- Use a single **`ResourceManager`** (or inject one) per process for address parsing and backend selection; avoid creating managers inside tight loops. +- Open resources with explicit **resource strings** (`TCPIP0::host::inst0::INSTR`, `USB0::...`, `ASRL3::INSTR`, etc.) and keep the **open handle** lifetime obvious: **`with rm.open_resource(...) as inst:`** or a small **`InstrumentSession`** class with **`close()`** in **`__exit__`**. + +## Lifecycle vs protocol + +- **Lifecycle**: `open_resource`, **`baud_rate`**, **`data_bits`**, **`parity`**, **`stop_bits`** for serial; **`read_termination`** / **`write_termination`**; **`timeout`** (milliseconds in PyVISA). Set these once after open, not per scattered call site. +- **Protocol**: functions or a **`SCPIClient`** that only **`write`**, **`query`**, **`read_raw`**, and parse responses. Never embed **`open_resource`** inside a “send SCPI” helper. + +## Timeouts and partial reads + +- Every blocking **`read`** must respect **`timeout`**. For binary payloads with known length, prefer **`read_bytes(count)`** (or read until delimiter) instead of unbounded **`read`**. +- After errors, many instruments need **`clear()`** or a **`*CLS`** / device-specific recovery sequence before the bus is trustworthy again—document that path in one place. + +## Terminations and encoding + +- SCPI text is usually **7-bit ASCII** with **`\\n`** termination. Match **`read_termination`** / **`write_termination`** to the manual; mixed **`\\r\\n`** gear is common on serial. +- For **`read_raw`**, you own **length** and **endianness**; do not apply text terminations. + +## Queries and staleness + +- Prefer **`query(message)`** for “write then read line” patterns to avoid half-open transactions. +- If the instrument can **async** complete (long sweeps), either poll **`*OPC?`** / status registers per manual or block with a **longer timeout** on the final read—do not chain short reads that race completion. + +## Backends + +- **NI-VISA** (vendor stack) vs **`PyVISA-py`**: choose per deployment; CI often uses **`PyVISA-py`** or simulation. Document which backend the project expects in **`README`** or config. + +## Logging and safety + +- Log **resource string** (sanitized if it embeds secrets) and **high-level command names**, not passwords. Avoid logging full **waveform blobs** at info level. diff --git a/.cursor/skills/lab-instrumentation/references/reference-socket-comms.md b/.cursor/skills/lab-instrumentation/references/reference-socket-comms.md new file mode 100644 index 00000000..40716c38 --- /dev/null +++ b/.cursor/skills/lab-instrumentation/references/reference-socket-comms.md @@ -0,0 +1,32 @@ +# Socket communication for lab gear + +## When sockets instead of VISA + +- **Custom binary framing**, **multicast**, **UDP discovery**, or **non-VISA** Ethernet devices (raw TCP on a port) are easier with **`socket`** or **`asyncio`** streams than with VISA resource strings. +- **VISA** still wins when you need **USB-TMC**, **GPIB**, **serial via VISA**, or a single **enumeration** story across buses. + +## Client patterns + +- Prefer **`socket.create_connection((host, port), timeout=...)`** over manual **`connect`** when you want **fail-fast** timeouts and **IPv4/IPv6** handling. +- Set **`sock.settimeout(...)`** for subsequent operations unless you use non-blocking **`asyncio`**. +- Use **`contextlib.closing`** or a small wrapper so **`close()`** runs on error paths. + +## Framing + +- **Line-oriented** text: read with **`recv`** until delimiter, or use **`readline`** on a **`makefile`** wrapper—watch **blocking** if the peer never sends the delimiter. +- **Length-prefixed** binary: read **header** (fixed bytes), parse **length**, then **`recv`** until **`n`** bytes; loop because **`recv`** may return **partial** chunks. +- **Struct pack/unpack** for fixed layouts; document **endianness** (`"<"` vs `">"`) next to the struct format string. + +## Timeouts and shutdown + +- Distinguish **`ETIMEDOUT`** (no data) from **`ECONNRESET`**. Surface both with **clear exceptions** so operators know whether to retry or re-seat cabling. +- For clean shutdown, **`shutdown(SHUT_RDWR)`** before **`close`** when the protocol expects half-close behavior. + +## Threading and async + +- **`socket`** objects are not thread-safe for arbitrary interleaved **`send`/`recv`**; one thread per socket or a **queue-driven** writer/reader design. +- **`asyncio`** suits many **slow instruments**; keep **protocol state machines** explicit so partial reads do not corrupt state. + +## Security + +- **Lab LAN** is not trusted by default: validate **host allowlists**, avoid **command injection** into shells that restart services, and never expose raw socket servers to the open internet without authentication. diff --git a/.cursor/skills/lab-instrumentation/references/reference-testing-mocks.md b/.cursor/skills/lab-instrumentation/references/reference-testing-mocks.md new file mode 100644 index 00000000..034b01ad --- /dev/null +++ b/.cursor/skills/lab-instrumentation/references/reference-testing-mocks.md @@ -0,0 +1,36 @@ +# Testing instrument code without hardware + +## Layers to test separately + +- **Parsing**: response strings to **`float`**, **unit stripping**, **status bit** interpretation—**pure functions**, **table-driven** pytest. +- **Validation**: range checks and **enum** mapping—no sockets. +- **Transport**: thin adapter that **`write`/`read`**; swap for **fake** in tests. + +## Fakes and protocols + +- Define a **`Protocol`** with **`query`**, **`write`**, **`read_raw`** matching your **session** surface; implement **`FakeInstrument`** with **canned** responses and **counters** to assert **call order**. +- For **stateful** gear, keep **minimal** state in the fake (**output on**, **last voltage**) so tests encode **realistic** sequences. + +## PyVISA simulation + +- **`pyvisa-sim`** (when compatible with your stack) can serve **YAML-defined** instruments for **integration-style** tests without benches. +- Vendor **simulators** and **socket echo** servers are acceptable **fixtures** if CI can start them **deterministically**. + +## Record and replay + +- Capture **transcripts** (command, response, timing hints) from a **golden** session; replay in tests to detect **protocol drift**. **Sanitize** secrets and **unique** serial numbers if committing files. + +## Markers and CI + +- Mark **slow** or **hardware-required** tests (`@pytest.mark.hardware`); default **`pytest`** in CI runs **offline** only. Document **`pytest -m hardware`** for the lab machine. + +## Socket tests + +- Use **pytest fixtures** with **ephemeral ports** (`bind(("", 0))`) and **short-lived** server threads or **`asyncio`** test servers; always **join** threads to avoid **flaky** teardown. + +Fix typo: **`pytest`**** fixtures** -> **pytest fixtures** + + + +<｜tool▁calls▁begin｜><｜tool▁call▁begin｜> +StrReplace diff --git a/.cursor/skills/matplotlib-scientific/SKILL.md b/.cursor/skills/matplotlib-scientific/SKILL.md new file mode 100644 index 00000000..358cd4af --- /dev/null +++ b/.cursor/skills/matplotlib-scientific/SKILL.md @@ -0,0 +1,70 @@ +--- +author: dotagents +name: matplotlib-scientific +description: Build publication-quality Matplotlib figures for scientific Python. Use when plotting experiment or analysis results, multi-panel figures, journal-sized layouts, tick formatting, legends (fancy/plain, placement, compact handles), point annotations, inset zooms, twin axes, colored spines/ticks, colorblind-safe palettes, PNG/PDF export, or SciencePlots styling. Triggers on matplotlib, pyplot, subplots, savefig, legend, annotate, inset, twinx, scientific figures, panel labels. +--- + +# Matplotlib for scientific figures + +## Quick start + +1. Use the **object-oriented API**: create `fig, ax = plt.subplots()` (or `subplot_mosaic` / `GridSpec`), then call methods on `ax`. Avoid `plt.plot` when multiple axes exist; implicit pyplot targets the wrong axis by default. See [Practical Data Science: explicit vs implicit syntax](https://www.practicaldatascience.org/notebooks/class_5/week_1/1.4.5_explicit_vs_implicit_syntax.html) and [basic plotting](https://www.practicaldatascience.org/notebooks/class_5/week_1/1.2.1_basic_plotting_with_matplotlib.html). +2. **Name every quantitative axis** with units in the label (e.g. `Time (s)`, `Voltage (V)`). Match tick style to audience: disable offset/scientific clutter when plain decimals read better; use scientific or engineering notation when magnitudes span orders. Details: [reference-axes-ticks.md](references/reference-axes-ticks.md). +3. **Save deliberately**: high DPI raster (often 300–600) for PNG; `bbox_inches="tight"` when labels are clipped; **`transparent=True`** for PNG overlays and slides. Prefer vector PDF/SVG for final print when the journal allows. See [reference-export-naming.md](references/reference-export-naming.md). +4. **Panel letters** `(a)`, `(b)` for multi-panel figures: consistent placement, bold, same size as body or slightly larger. See [Matplotlib: labelling subplots](https://matplotlib.org/stable/gallery/text_labels_and_annotations/label_subplots.html) and [reference-axes-ticks.md](references/reference-axes-ticks.md). +5. **Figure width**: size figures to **single-column** or **full-page / double-column** width per target venue; do not default to square notebook sizes for papers. See [reference-journal-layout.md](references/reference-journal-layout.md) and [Simplified Science: figure design rules](https://www.simplifiedsciencepublishing.com/resources/how-to-make-good-figures-for-scientific-papers). +6. **Composable code**: write helpers that **accept `Axes` or `Figure` and return** the same object after mutation. See [reference-api-patterns.md](references/reference-api-patterns.md). +7. **Polish**: match legend frame style to venue (plain for print, slightly rounded for slides); place legends where they do not obscure data; annotate only points the text discusses; use insets and twin axes only when they sharpen the claim. See [reference-legend-style.md](references/reference-legend-style.md), [reference-annotations-insets-twins.md](references/reference-annotations-insets-twins.md), [reference-axes-appearance.md](references/reference-axes-appearance.md). + +## Stack synergy + +| Resource | Role | +|----------|------| +| **general-python** | uv, **ty**, export scripts, **python-reviewer** | +| **numpy-scientific** | Array inputs to plots | +| **dataframes** | `df.plot(ax=ax)` and Polars-to-pandas at plot boundaries | +| **numpy-docstrings** | Docstrings for plotting helpers and figure builders | + +## Reference index (load the section you need) + +| Topic | File | +|--------|------| +| Subplots vs `GridSpec` vs `subplot_mosaic`; assembling panels | [reference-layout.md](references/reference-layout.md) | +| Axis labels, units, ticks, scientific notation, panel labels | [reference-axes-ticks.md](references/reference-axes-ticks.md) | +| Spine/tick/label color, twin-axis styling | [reference-axes-appearance.md](references/reference-axes-appearance.md) | +| Legends: fancy frame, title, placement, compact handles | [reference-legend-style.md](references/reference-legend-style.md) | +| Point annotations, inset zooms, twin / secondary axes | [reference-annotations-insets-twins.md](references/reference-annotations-insets-twins.md) | +| Color maps, cycles, colorblind safety, data types | [reference-color.md](references/reference-color.md) | +| `savefig`, transparency, DPI, auto-increment filenames | [reference-export-naming.md](references/reference-export-naming.md) | +| Journal widths, composition, storytelling | [reference-journal-layout.md](references/reference-journal-layout.md) | +| SciencePlots / style sheets; `pandas` `.plot(ax=ax)` | [reference-styles-pandas.md](references/reference-styles-pandas.md) | +| Functions that take/return `Axes` / `Figure` | [reference-api-patterns.md](references/reference-api-patterns.md) | +| Rasterized PDF, `align_labels`, CI, memory | [reference-advanced.md](references/reference-advanced.md) | + +## Best-practice topics (checklist) + +- **Chart choice**: match geometry to the claim (trend vs part-to-whole vs distribution). Same data, many encodings: [Plotting Zoo](https://www.practicaldatascience.org/notebooks/class_5/week_1/1.5.1_plotting_zoo.html). +- **Anatomy**: figure, axes, marks, axis labels, ticks and tick labels, limits, grid, legend, title, spines; compact `ax.set(...)`. See [A figure in 10 pieces](https://www.practicaldatascience.org/notebooks/class_5/week_1/1.2.2_ten_figure_pieces.html). +- **Layering**: set `zorder` and distinct `color` when mixing `plot` / `scatter` / `bar` on one axes ([basic plotting](https://www.practicaldatascience.org/notebooks/class_5/week_1/1.2.1_basic_plotting_with_matplotlib.html)). +- **Legends**: plain frame and no shadow for most papers; `bbox_to_anchor` outside or `ncol` below when the data region is crowded ([reference-legend-style.md](references/reference-legend-style.md)). +- **Annotations**: few, uniform `annotate` styling; insets for zoom only when the story needs local detail ([reference-annotations-insets-twins.md](references/reference-annotations-insets-twins.md)). +- **Axes color**: tie spine/tick/label color to a twin axis accent; keep grids low-contrast ([reference-axes-appearance.md](references/reference-axes-appearance.md)). +- **Ticks**: treat **locator** (where) and **formatter** (text) separately. See [Axis ticks](https://matplotlib.org/stable/users/explain/axes/axes_ticks.html) and [ScalarFormatter](https://matplotlib.org/stable/gallery/ticks/scalarformatter.html). +- **Reproducibility**: script generates the file on disk; version-control style choices; meeting previews use predictable auto-increment names ([reference-export-naming.md](references/reference-export-naming.md)). +- **Accessibility**: do not rely on color alone; use line styles / markers; check contrast; prefer perceptually uniform colormaps for continuous data ([reference-color.md](references/reference-color.md)). +- **SciencePlots** (optional): journal-oriented style sheets; often requires LaTeX. See [SciencePlots](https://github.com/garrettj403/SciencePlots) and [reference-styles-pandas.md](references/reference-styles-pandas.md). +- **Pandas quick plots**: `df.plot(..., ax=ax)` returns matplotlib artists; refine with `ax` methods. See [Plotting with Pandas](https://www.practicaldatascience.org/notebooks/class_5/week_1/1.6.1_plotting_with_pandas.html). + +## Matplotlib documentation (official) + +- [User guide / tutorials](https://matplotlib.org/stable/users/index.html) +- [Anatomy of a figure](https://matplotlib.org/stable/gallery/showcase/anatomy.html) +- [Arranging multiple Axes](https://matplotlib.org/stable/users/explain/axes/arranging_axes.html) (subplots, GridSpec, mosaic) +- [Constrained layout](https://matplotlib.org/stable/users/explain/axes/constrainedlayout_guide.html) and [tight layout](https://matplotlib.org/stable/users/explain/axes/tight_layout_guide.html) +- [Legend guide](https://matplotlib.org/stable/users/explain/axes/legend_guide.html), [Annotations](https://matplotlib.org/stable/tutorials/text/annotations.html) +- [Secondary axis](https://matplotlib.org/stable/gallery/subplots_axes_and_figures/secondary_axis.html), [Zoom inset](https://matplotlib.org/stable/gallery/subplots_axes_and_figures/zoom_inset_axes.html) +- [Figure.savefig](https://matplotlib.org/stable/api/figure_api.html#matplotlib.figure.Figure.savefig) (API); workflow notes in [Saving to file (PDS)](https://www.practicaldatascience.org/notebooks/class_5/week_1/1.4.4_saving_to_file.html) + +## When not to overload one figure + +Split into **separate figures** when panels are reused across talks vs papers, or when layout constraints differ (poster vs manuscript). Compose in a layout script or document (LaTeX/Quarto) when the page design, not matplotlib, should own margins and alignment. diff --git a/.cursor/skills/matplotlib-scientific/references/reference-advanced.md b/.cursor/skills/matplotlib-scientific/references/reference-advanced.md new file mode 100644 index 00000000..bd9dbdaf --- /dev/null +++ b/.cursor/skills/matplotlib-scientific/references/reference-advanced.md @@ -0,0 +1,33 @@ +# Advanced publication topics (optional) + +## Vector files with heavy scatter + +- PDFs balloon when every point is a vector path. For large `scatter`, use **`rasterized=True`** on the collection or rasterize the axes in the PDF backend so data draw as image while text stays vector. + +## Label alignment across panels + +- After setting xlabels/ylabels on a grid, call **`fig.align_labels()`** so stacked panels line up when label lengths differ. + +## Font sizes and RC + +- Set **`axes.titlesize`**, **`axes.labelsize`**, **`xtick.labelsize`**, **`legend.fontsize`** once per style for consistent figures. SciencePlots and journal styles override many of these. + +## Secondary axes + +- `ax.secondary_xaxis` / `secondary_yaxis` for dual scales: document both units in axis labels or caption to avoid ambiguity. Patterns and twin-axis caution: [reference-annotations-insets-twins.md](reference-annotations-insets-twins.md). + +## 3D and polar + +- Use only when the geometry is inherently 3D or angular; avoid 3D for ranking bar charts. See Matplotlib gallery for projection setup. + +## Interactive vs static + +- **`plt.show()`** is for interactive sessions; batch pipelines should **`savefig`** and close figures (`plt.close(fig)`) to limit memory when generating many plots. + +## Testing plots + +- Smoke-test plotting code with **`matplotlib.use("Agg")`** in CI so no display server is required. + +## Accessibility metadata + +- For web exports, pair complex graphics with a **data table** or summary in the document; color alone must not carry exclusive meaning ([Simplified Science rules](https://www.simplifiedsciencepublishing.com/resources/how-to-make-good-figures-for-scientific-papers)). diff --git a/.cursor/skills/matplotlib-scientific/references/reference-annotations-insets-twins.md b/.cursor/skills/matplotlib-scientific/references/reference-annotations-insets-twins.md new file mode 100644 index 00000000..a33706e5 --- /dev/null +++ b/.cursor/skills/matplotlib-scientific/references/reference-annotations-insets-twins.md @@ -0,0 +1,85 @@ +# Point annotations, inset axes, twin axes + +## When to annotate data points + +Annotate **sparingly**: + +- **Named exemplars** (a specific run, date, or condition called out in the text). +- **Threshold crossings** or **regulatory limits** (horizontal/vertical reference lines plus one short label often beat many point labels). +- **Outliers** that the narrative discusses. + +Avoid labeling **every** point unless the figure is a small-N schematic; for dense scatter, prefer **interactive** exploration or a **table** supplement. + +## Consistent, readable point labels + +Use **`ax.annotate`** so text and arrow share one API; prefer **data coordinates** for `xy` and **offset points** for `xytext` so font size and zoom stay consistent. + +```python +ax.annotate( + "A", + xy=(x0, y0), + xytext=(8, 8), + textcoords="offset points", + ha="left", + va="bottom", + fontsize=9, + bbox=dict(boxstyle="round,pad=0.25", fc="white", ec="0.7", lw=0.5), + arrowprops=dict(arrowstyle="-", color="0.35", lw=0.6, shrinkA=0, shrinkB=2), +) +``` + +**Consistency rules across a figure:** + +- One **`fontsize`** and **`bbox`** style for all callouts in the same figure. +- **`offset points`** in the same direction (e.g. always up-right) unless avoiding overlap. +- For overlapping labels, **nudge** `xytext` or use **`adjustText`** (third-party) in exploratory work; for publication, **manually** place or reduce the number of labels. + +**Alternative without arrows:** `ax.text` in data coords when proximity is obvious. + +## Reference lines vs point annotations + +- Vertical/horizontal rules: `ax.axvline`, `ax.axhline` with **`label=`** and legend entry, or a short **`transform=blended`** label on the line. +- Keeps the main legend for series, not for every marker. + +## Twin axes (`twinx` / `twiny`) + +- **One twin** is standard when two quantities share **time or position** but differ in **unit or scale** (e.g. temperature and humidity vs depth). + +```python +ax2 = ax.twinx() +ax2.plot(x, y2, color="C1") +ax2.set_ylabel("Secondary quantity (unit)", color="C1") +ax2.tick_params(axis="y", colors="C1") +``` + +**Multiple scales on the same side** are cognitively heavy. Prefer: + +- **Normalizing** to one axis and stating the mapping in the caption, or +- **A second panel** (small multiples), or +- **`secondary_yaxis`** for an alternate **functional** scale of the same quantity (e.g. °C ↔ °F) rather than a third unrelated series. + +Official pattern: [Secondary axis](https://matplotlib.org/stable/gallery/subplots_axes_and_figures/secondary_axis.html). + +## Inset axes (zoom, context, detail) + +Use an **inset** when the main axes shows **context** and the inset shows **a zoom** or a **secondary view** (e.g. map overview + city blow-up). + +**Recommended API:** `ax.inset_axes(bounds)` where `bounds` is `[left, bottom, width, height]` in **axes coordinates** (0–1), or use `mpl_toolkits.axes_grid1.inset_locator.inset_axes` for size in **axes fraction / padding**. + +```python +axins = ax.inset_axes([0.55, 0.55, 0.4, 0.4]) +axins.plot(x, y) +axins.set_xlim(x0, x1) +axins.set_ylim(y0, y1) +ax.indicate_inset_zoom(axins, edgecolor="0.3") +``` + +- Match **line weights** and **colors** to the main panel unless the inset is a different modality. +- Add **ticks** on the inset; for tiny insets, fewer ticks are better than cluttered decimals. +- **`indicate_inset`** / **`indicate_inset_zoom`** links inset to region when helpful. + +Gallery: [Inset locator demo](https://matplotlib.org/stable/gallery/axes_grid1/inset_locator_demo.html), [Zoom inset axes](https://matplotlib.org/stable/gallery/subplots_axes_and_figures/zoom_inset_axes.html). + +## Z-order + +- Insets and annotations usually sit **above** data: set artist **`zorder`** or add them **after** plotting series. diff --git a/.cursor/skills/matplotlib-scientific/references/reference-api-patterns.md b/.cursor/skills/matplotlib-scientific/references/reference-api-patterns.md new file mode 100644 index 00000000..b2790cc7 --- /dev/null +++ b/.cursor/skills/matplotlib-scientific/references/reference-api-patterns.md @@ -0,0 +1,62 @@ +# API patterns: functions that accept and return `Axes` / `Figure` + +## Principle + +Plotting helpers should be **pure in intent**: given the same data and axes, they draw the same artists. Side effects stay on the **`Axes`** you pass in. Return the **`Axes`** (or **`Figure`**) so callers can chain or save. + +## Recommended signatures + +```python +from matplotlib.axes import Axes +from matplotlib.figure import Figure + +def plot_waveform(ax: Axes, t, y, *, label: str | None = None) -> Axes: + ax.plot(t, y, label=label) + ax.set_xlabel("Time (s)") + ax.set_ylabel("Amplitude (V)") + return ax + +def finalize_figure(fig: Figure) -> Figure: + fig.align_labels() + return fig +``` + +## Optional axes + +- If `ax is None`, create a figure and axes; otherwise draw on the provided axes: + +```python +def plot_spectrum(data, *, ax: Axes | None = None) -> tuple[Figure, Axes]: + if ax is None: + fig, ax = plt.subplots(layout="constrained") + else: + fig = ax.figure + ax.plot(data.f, data.psd) + ax.set_xlabel("Frequency (Hz)") + return fig, ax +``` + +## Typing + +- Annotate with **`Axes`** and **`Figure`** from `matplotlib.axes` / `matplotlib.figure` for static checking (project may use `ty` / pyright). + +## Composition + +- **Higher-level** functions call lower-level ones, always threading `ax`: + +```python +def plot_panel(ax: Axes, dataset) -> Axes: + plot_waveform(ax, dataset.t, dataset.y, label="signal") + ax.legend() + return ax +``` + +## Anti-patterns + +- Relying on **`plt.gca()`** inside library code. +- Mixing **`plt.plot`** and passed **`ax`** in the same module without `plt.sca`. +- Creating a **new figure** inside a helper without returning it, leaving callers unable to save. + +## Layout-aware helpers + +- After adding colorbars or twin axes, call **`fig.align_labels()`** or rely on **`layout="constrained"`** so panel labels stay aligned across a row. diff --git a/.cursor/skills/matplotlib-scientific/references/reference-axes-appearance.md b/.cursor/skills/matplotlib-scientific/references/reference-axes-appearance.md new file mode 100644 index 00000000..4c6dc9c9 --- /dev/null +++ b/.cursor/skills/matplotlib-scientific/references/reference-axes-appearance.md @@ -0,0 +1,51 @@ +# Spine, tick, and label color (cohesive axes styling) + +## Why color axes elements + +Use **one accent color per y-axis** when that axis encodes a **single physical quantity** (especially with **`twinx`**) so readers bind ticks, label, and spine to the correct series. Keep **x-axis** styling neutral unless the bottom axis is also doubled. + +## Spines + +```python +accent = "C0" +ax.spines["left"].set_color(accent) +ax.spines["bottom"].set_color("0.25") +ax.spines["top"].set_visible(False) +ax.spines["right"].set_visible(False) +ax.spines["left"].set_linewidth(1.0) +``` + +- **Muted** spine color (`0.2`–`0.4` grayscale or desaturated brand) reads more “scientific” than pure black on white. +- Match **spine color** to the **same-axis** tick and label color when emphasizing a twin. + +## Ticks and tick labels + +```python +ax.tick_params(axis="y", colors=accent, which="both") +ax.tick_params(axis="x", colors="0.25", which="both") +ax.yaxis.label.set_color(accent) +ax.xaxis.label.set_color("0.25") +``` + +- **`which="both"`** applies to major and minor ticks when minors are on. +- Minor ticks: `ax.minorticks_on()` then tune length/width via `tick_params`. + +## Axis labels and titles + +- **`ax.xaxis.label.set_color`** / **`ax.yaxis.label.set_color`** align label with spine accent. +- **Title** color: default body color or slightly darker; avoid competing with data saturation. + +## Grid vs colored spines + +- If **`ax.grid(True)`** is on, keep grid **low contrast** (`alpha=0.25–0.35`, light grey) so colored spines still read as the primary frame. + +## Dark backgrounds (slides) + +- Invert logic: light spines and labels on dark **`figure.facecolor`**; test **contrast** for projectors. +- Legend **`facecolor`** should match slide background or stay opaque white for readability. + +## Matplotlib references + +- [Spines](https://matplotlib.org/stable/api/spines_api.html) +- [tick_params](https://matplotlib.org/stable/api/_as_gen/matplotlib.axes.Axes.tick_params.html) +- [Colorblind considerations](reference-color.md) still apply: color on spines must not be the only discriminator between series. diff --git a/.cursor/skills/matplotlib-scientific/references/reference-axes-ticks.md b/.cursor/skills/matplotlib-scientific/references/reference-axes-ticks.md new file mode 100644 index 00000000..7231d1e5 --- /dev/null +++ b/.cursor/skills/matplotlib-scientific/references/reference-axes-ticks.md @@ -0,0 +1,62 @@ +# Axes labels, units, ticks, notation, panel labels + +## Axis labels and units + +- Put **quantity and unit** in the label: `ax.set_xlabel("Time (ms)")`, `ax.set_ylabel("Power spectral density (V^2/Hz)")`. +- Use **consistent SI style** across a figure set; square brackets for unit-only annotations are acceptable where your field expects them (e.g. `Voltage [V]`). +- For dimensionless quantities, state it: `Strouhal number` or `Normalized frequency (f / f_0)`. + +## The ten pieces (mental model) + +Figure container, axes, marks (line/scatter/bar), **x/y labels**, **ticks and tick labels**, **limits**, grid, legend, title, spines. Prefer **`ax.set(...)`** for bulk updates when it improves readability. See [A figure in 10 pieces](https://www.practicaldatascience.org/notebooks/class_5/week_1/1.2.2_ten_figure_pieces.html). + +## Ticks: locators and formatters + +- **Positions**: `ax.set_xticks(...)`, `MultipleLocator`, `MaxNLocator`, `LogLocator`, etc. +- **Strings**: `ax.set_xticklabels(...)`, or set a `Formatter` on `ax.xaxis`. +- Default numeric formatter is **`ScalarFormatter`**. Useful controls: + - `ax.ticklabel_format(style="plain", axis="y")` to discourage scientific notation on an axis when values are human-scaled. + - `ax.ticklabel_format(style="sci", axis="y", scilimits=(0, 0))` when all decades should use scientific form. + - `ax.yaxis.get_major_formatter().set_useOffset(False)` to suppress the offset term when it confuses readers. +- For wide dynamic range on linear axes, consider **log scale** (`ax.set_yscale("log")`) instead of cramming exponents into tick labels. +- Deep reference: [Axis ticks](https://matplotlib.org/stable/users/explain/axes/axes_ticks.html), [ScalarFormatter demo](https://matplotlib.org/stable/gallery/ticks/scalarformatter.html). + +## Panel labels (a), (b), (c) + +- Convention: **bold**, upper or lower case per venue, **same font size** across panels, placed consistently (often **top-left** inside axes using axes coordinates). +- Official patterns: [Labelling subplots](https://matplotlib.org/stable/gallery/text_labels_and_annotations/label_subplots.html) (`annotate`, `text` with transforms, or title `loc="left"`). +- Minimal loop pattern: + +```python +import string +for ax, tag in zip(axs.flat, string.ascii_lowercase): + ax.text( + 0.02, + 0.98, + f"({tag})", + transform=ax.transAxes, + ha="left", + va="top", + fontweight="bold", + fontsize=11, + ) +``` + +- Adjust `(0.02, 0.98)` if constrained layout clips labels; use figure coordinates or `mpl_toolkits.axes_grid1` helpers if needed. + +## Legend (overview) + +- Pass **`label=`** to each plotting call, then `ax.legend(...)`. +- For crowded panels, **`bbox_to_anchor`** outside the axes preserves data ink ([ten pieces](https://www.practicaldatascience.org/notebooks/class_5/week_1/1.2.2_ten_figure_pieces.html)). +- Fancy vs plain frames, legend titles, placement above/below/outside, and compact handles: [reference-legend-style.md](reference-legend-style.md). + +## Mathtext and LaTeX + +- Mathtext: `ax.set_xlabel(r"$\omega$ (rad/s)")` without full LaTeX install. +- `plt.rcParams["text.usetex"] = True` requires a TeX system; **SciencePlots** often expects LaTeX. See [reference-styles-pandas.md](reference-styles-pandas.md). + +## Grid and spines + +- `ax.grid(True, alpha=0.3)` for read values without chartjunk. +- Hiding **top/right** spines is common for 2D scientific plots ([ten pieces](https://www.practicaldatascience.org/notebooks/class_5/week_1/1.2.2_ten_figure_pieces.html)). +- Colored spines, tick colors, and label colors (especially with **twin** axes): [reference-axes-appearance.md](reference-axes-appearance.md). diff --git a/.cursor/skills/matplotlib-scientific/references/reference-color.md b/.cursor/skills/matplotlib-scientific/references/reference-color.md new file mode 100644 index 00000000..0853516b --- /dev/null +++ b/.cursor/skills/matplotlib-scientific/references/reference-color.md @@ -0,0 +1,37 @@ +# Color, colormaps, and cycles for scientific data + +## Match encoding to data type + +| Data role | Prefer | +|-----------|--------| +| Ordered numeric (low to high) | Perceptually uniform sequential: `viridis`, `cividis`, `plasma`; avoid **jet** | +| Deviation from a reference | Diverging: `coolwarm`, `RdBu_r`; center the norm at the reference | +| Categories | Distinct hues; **limit palette** to the number of classes; add **markers or dashes** for grayscale safety | +| Uncertainty bands | Lower alpha or lighter fill; keep the central estimate high-contrast | + +## Colorblind safety and grayscale + +- Do not encode **only** with red vs green. Combine **hue + linestyle + marker**. +- Test a figure by **desaturating** or printing grayscale: the main comparison should survive. +- [SciencePlots](https://github.com/garrettj403/SciencePlots) includes cycles such as **bright** and **high-vis**; Paul Tol palettes are available as named styles. + +## Overlays on one axes + +- When mixing `plot`, `scatter`, and `bar`, set explicit **`color`** and **`zorder`** so categories do not hide each other ([basic plotting](https://www.practicaldatascience.org/notebooks/class_5/week_1/1.2.1_basic_plotting_with_matplotlib.html)). + +## Storytelling vs decoration + +- Per [Simplified Science figure rules](https://www.simplifiedsciencepublishing.com/resources/how-to-make-good-figures-for-scientific-papers), use **one or two accent colors** for the main claim and mute context series (grey). + +## Norms for images and fields + +- Use `TwoSlopeNorm` or `SymLogNorm` when data spans zero with outliers; document the norm in the caption. +- For maps and fields, state the **colorbar** label with units (`imshow(...); fig.colorbar(im, ax=ax, label="K")`). + +## Defaults + +- Set **`axes.prop_cycle`** via a style sheet rather than hard-coding colors in every script when the project has a house style. + +## Coordinated axis color + +- When a series and its **y-axis** share an accent (e.g. after `twinx`), align **spine, ticks, and axis label** to that color without breaking grayscale legibility. See [reference-axes-appearance.md](reference-axes-appearance.md). diff --git a/.cursor/skills/matplotlib-scientific/references/reference-export-naming.md b/.cursor/skills/matplotlib-scientific/references/reference-export-naming.md new file mode 100644 index 00000000..9d55b773 --- /dev/null +++ b/.cursor/skills/matplotlib-scientific/references/reference-export-naming.md @@ -0,0 +1,54 @@ +# Saving figures: DPI, transparency, formats, auto-increment names + +## `fig.savefig` essentials + +- Always save the **`Figure`** you built: `fig.savefig(path, ...)`, not `plt.savefig` in library code (acceptable in quick scripts if there is a single current figure). +- **Raster**: PNG (and TIFF if required). **Vector**: PDF or SVG for line art and typography at journal resolution. +- **DPI**: 300–600 is a common print range; match journal guidance. See [Saving to file](https://www.practicaldatascience.org/notebooks/class_5/week_1/1.4.4_saving_to_file.html). +- **Clipping**: if labels are cut off, use `bbox_inches="tight"` (optionally with `pad_inches`). +- **Transparent PNG**: `transparent=True` for slides and compositing on non-white backgrounds. + +Example: + +```python +fig.savefig( + "outputs/fig01.png", + dpi=400, + bbox_inches="tight", + pad_inches=0.02, + transparent=True, +) +``` + +## Meeting-friendly auto-increment + +Use a small helper so repeated script runs produce `plot_001.png`, `plot_002.png`, without overwriting. Persist the counter in a dotfile or JSON next to outputs. + +```python +from pathlib import Path +import json + +def next_plot_path(dir_path: Path, stem: str = "plot", suffix: str = ".png") -> Path: + """Return the next sequential path ``{stem}_{n:03d}{suffix}`` under ``dir_path``.""" + dir_path.mkdir(parents=True, exist_ok=True) + state = dir_path / ".plot_counter.json" + n = 0 + if state.is_file(): + n = int(json.loads(state.read_text()).get("n", 0)) + n += 1 + state.write_text(json.dumps({"n": n})) + return dir_path / f"{stem}_{n:03d}{suffix}" + +path = next_plot_path(Path("outputs/meetings")) +fig.savefig(path, dpi=400, bbox_inches="tight", transparent=True) +``` + +Variants: timestamp-based names for parallel runs, or `git rev-parse --short` in the stem for traceability. + +## Script vs notebook + +- In non-interactive scripts, call **`plt.show()`** only when debugging; for batch pipelines, rely on **`savefig`** ([basic plotting note](https://www.practicaldatascience.org/notebooks/class_5/week_1/1.2.1_basic_plotting_with_matplotlib.html)). + +## Supported formats + +- Discover locally: `fig.canvas.get_supported_filetypes()` ([Saving to file](https://www.practicaldatascience.org/notebooks/class_5/week_1/1.4.4_saving_to_file.html)). diff --git a/.cursor/skills/matplotlib-scientific/references/reference-journal-layout.md b/.cursor/skills/matplotlib-scientific/references/reference-journal-layout.md new file mode 100644 index 00000000..86e6cd33 --- /dev/null +++ b/.cursor/skills/matplotlib-scientific/references/reference-journal-layout.md @@ -0,0 +1,40 @@ +# Journal widths, composition, and scientific storytelling + +## Width targets (verify against current author guide) + +Journals differ; **always read the target journal’s figure guidelines**. Order-of-magnitude widths from public summaries: + +| Venue (examples) | Single column (order of magnitude) | Double / full width | +|-------------------|-------------------------------------|----------------------| +| Nature (typical) | ~89 mm | ~183 mm | +| Science (typical) | ~55 mm | ~230 mm | + +Convert to inches for `figsize`: **inches = mm / 25.4**. Example: 89 mm ~ 3.5 in wide; height from aspect ratio and content. + +```python +mm = 89 +w_in = mm / 25.4 +aspect = 0.75 +fig, ax = plt.subplots(figsize=(w_in, w_in * aspect), layout="constrained") +``` + +## Four design rules (summary) + +From [How to make good figures for scientific papers](https://www.simplifiedsciencepublishing.com/resources/how-to-make-good-figures-for-scientific-papers): + +1. **Purpose**: choose the graphic form from the claim (compare, change over time, relationship, process). +2. **Composition**: left-to-right or top-to-bottom flow; remove chartjunk; emphasize the main series. +3. **Color**: few accents; colorblind-safe; grayscale should still read. +4. **Refine**: iterate; check “does the graphic alone convey the point?” before polishing text. + +## Choosing the chart type + +- Same table, many views: lines for **trends**, stacked bars or stackplots for **composition over time**, grouped bars for **within-year comparison**, pies only when **parts of one whole** and few slices. See [Plotting Zoo](https://www.practicaldatascience.org/notebooks/class_5/week_1/1.5.1_plotting_zoo.html). + +## Tables as figures + +- A **formatted table** is a valid figure for lookup-heavy results; pandas `DataFrame` display or manual annotation in the manuscript are both acceptable ([Plotting Zoo](https://www.practicaldatascience.org/notebooks/class_5/week_1/1.5.1_plotting_zoo.html)). + +## Supplementary material + +- Move **sensitivity analyses and extra scans** to supplement; keep main figures focused ([Iceberg-style thinking](https://www.simplifiedsciencepublishing.com/resources/how-to-make-good-figures-for-scientific-papers)). diff --git a/.cursor/skills/matplotlib-scientific/references/reference-layout.md b/.cursor/skills/matplotlib-scientific/references/reference-layout.md new file mode 100644 index 00000000..31bd62c4 --- /dev/null +++ b/.cursor/skills/matplotlib-scientific/references/reference-layout.md @@ -0,0 +1,55 @@ +# Layout: subplots, GridSpec, mosaic, separate figures + +## Decision guide + +| Situation | Prefer | +|-----------|--------| +| Regular grid of equal panels, shared labels | `plt.subplots(nrows, ncols, sharex=..., sharey=..., layout="constrained")` | +| Variable row/column sizes, spans | `matplotlib.gridspec.GridSpec` or `fig.subplot_mosaic` with width ratios | +| Semantic names (`"A"`, `"B"`) instead of integer positions | `fig.subplot_mosaic` | +| One-off complex arrangement | `GridSpec` with `subplot_spec` spans | +| Same plot reused in slide vs paper with different sizes | **Separate** `fig.savefig` calls or separate scripts; avoid one giant figure that is always resized in a GUI | +| Final page composition with captions from a manuscript | Export **individual** high-quality panels (PDF/SVG) and place in LaTeX/Quarto; optional single composite for preview only | + +## `plt.subplots` + +- Use for the common case: `fig, axs = plt.subplots(2, 2, layout="constrained")`. +- `sharex` / `sharey` reduce duplicate ticks; call `ax.label_outer()` or hide inner tick labels for clarity. +- `axs` may be 1D or 2D array; flatten with `axs.flat` for iteration. + +## `subplot_mosaic` + +- Readable layout strings and named axes: + +```python +fig, axd = plt.subplot_mosaic( + "AB\nCC", + layout="constrained", +) +axd["A"].plot(...) +``` + +- Good when panel sizes are uneven or you refer to panels by role. + +## `GridSpec` + +- Use when you need **explicit height/width ratios** and **row/column spans** that are awkward in `subplots`. +- Pair with `fig.add_subplot(gs[i, j])` or `subgridspec` for nested regions. + +## Layout engines + +- Prefer **`layout="constrained"`** (or `fig.set_layout_engine("constrained")`) for label and colorbar spacing over manual `subplots_adjust` when possible. See [Constrained layout guide](https://matplotlib.org/stable/users/explain/axes/constrainedlayout_guide.html). +- **`tight_layout`** remains useful for quick notebooks; constrained layout is usually better for publication figures with long labels. + +## Assemble elsewhere + +- **Assemble in the document** when the journal controls gutters, multi-panel captions, or when vector panels must align with equations. +- **Assemble in matplotlib** when you need a single raster for Twitter/slides or a strict pixel budget. + +## Multi-panel data stories + +- **Small multiples** (one highlighted series per panel, others grey) help many categories; see [Plotting Zoo](https://www.practicaldatascience.org/notebooks/class_5/week_1/1.5.1_plotting_zoo.html). + +## Anti-pattern + +- Using **`plt.plot` after `subplots`** without `plt.sca(ax)` routes draws to the **last-created** axes and stacks traces incorrectly. Always **`ax.plot`** ([implicit vs explicit](https://www.practicaldatascience.org/notebooks/class_5/week_1/1.4.5_explicit_vs_implicit_syntax.html)). diff --git a/.cursor/skills/matplotlib-scientific/references/reference-legend-style.md b/.cursor/skills/matplotlib-scientific/references/reference-legend-style.md new file mode 100644 index 00000000..7a1cf80c --- /dev/null +++ b/.cursor/skills/matplotlib-scientific/references/reference-legend-style.md @@ -0,0 +1,78 @@ +# Legends: frame style, titles, placement, compact handles + +## Fancy frame vs plain frame + +| Context | Recommendation | +|---------|------------------| +| Journal print, minimal figures | **`fancybox=False`**, thin neutral frame: `ax.legend(frameon=True, edgecolor="0.85", facecolor="white", fancybox=False)` | +| Slides, posters, light branding | **`fancybox=True`** (rounded corners) is acceptable; keep **`shadow=False`** unless depth is part of the design system | +| On busy backgrounds | **`frameon=True`** with solid **`facecolor`** (often white or figure face) so text stays legible | + +`fancybox=True` uses a rounded box; `fancybox=False` is sharper and usually reads more “paper”. `shadow=True` ages poorly in print; prefer a crisp edge and padding. + +## Legend title (`title=`) + +Add a legend title when: + +- Handles are **grouped by class** and the axis label alone is insufficient (e.g. title `"Model"`, entries `RF`, `NN`). +- The same color/marker means **different semantics** than the axis (e.g. `"Dataset"` while y-axis is a measured quantity). + +Skip the title when: + +- The axis label and series names are already self-explanatory. +- Space is tight; a **caption** can name the encoding instead. + +```python +leg = ax.legend(title="Condition", alignment="left") +leg.get_title().set_fontsize(leg.get_texts()[0].get_fontsize()) +``` + +Keep title **one size step** above or equal to entry text; align left for multi-line entries. + +## Placement: inside vs above vs below vs outside + +| Placement | When to use | Typical kwargs | +|-----------|-------------|----------------| +| **Inside** upper right / best | Few series, large margin, no occlusion | `loc="upper right"`, `bbox_to_anchor=(1, 1)` only if nudging inside | +| **Outside** right | Many series or long names; preserve data region | `bbox_to_anchor=(1.02, 1)`, `loc="upper left"` | +| **Below** the axes | Wide figures, many entries; row of handles | `bbox_to_anchor=(0.5, -0.18)`, `loc="upper center"`, **`ncol`** ≥ 2 | +| **Above** | Rare; sometimes for single-row keys under a suptitle | `bbox_to_anchor=(0.5, 1.12)`, `loc="lower center"` | + +Use **`layout="constrained"`** on the figure so outside legends reserve space. After placing outside, verify **`savefig(..., bbox_inches="tight")`** does not clip the box. + +## Making the legend read cleanly + +- **`ncol`**: split into 2–4 columns for wide single-row legends below the plot. +- **`labelspacing`**, **`handlelength`**, **`handletextpad`**, **`borderaxespad`**: tighten uniformly; change one parameter at a time. +- **`fontsize`**: match tick labels or be one step smaller than axis labels. +- **`alignment="left"`** (Matplotlib 3.6+): left-align stacked text blocks. + +## Compact handles (shorter legend rows) + +When default lines are too long: + +1. **Proxy artists**: build handles explicitly with short linestyle/marker only. + +```python +from matplotlib.lines import Line2D + +handles = [ + Line2D([0], [0], color="C0", lw=2, label="Control"), + Line2D([0], [0], color="C1", lw=2, ls="--", label="Treated"), +] +ax.legend(handles=handles, labels=[h.get_label() for h in handles], frameon=True) +``` + +2. **`numpoints=1`**, **`markerscale`** for scatter-heavy legends. +3. **`handler_map`** for custom patches (e.g. thin rectangles for bars) when defaults are oversized. + +## Order and merging + +- **`order="sorted"`** or explicit label order for consistency across figures. +- **`labelcolor="linecolor"`** (or `"markerfacecolor"`) ties text to encoding and can replace redundant color words in labels. + +## Matplotlib references + +- [Legend guide](https://matplotlib.org/stable/users/explain/axes/legend_guide.html) +- [Figure legends](https://matplotlib.org/stable/gallery/text_labels_and_annotations/figlegend_demo.html) +- [Custom legend handlers](https://matplotlib.org/stable/tutorials/intermediate/legend_guide.html#implementing-a-custom-legend-handler) diff --git a/.cursor/skills/matplotlib-scientific/references/reference-styles-pandas.md b/.cursor/skills/matplotlib-scientific/references/reference-styles-pandas.md new file mode 100644 index 00000000..52199451 --- /dev/null +++ b/.cursor/skills/matplotlib-scientific/references/reference-styles-pandas.md @@ -0,0 +1,43 @@ +# Style sheets, SciencePlots, pandas plotting + +## Object-oriented + pandas + +- Pattern from [Plotting with Pandas](https://www.practicaldatascience.org/notebooks/class_5/week_1/1.6.1_plotting_with_pandas.html): + +```python +fig, ax = plt.subplots(layout="constrained") +df.plot(kind="bar", ax=ax, stacked=True, edgecolor="white") +ax.set_ylabel("Tons") +``` + +- `kind` options include `bar`, `barh`, `hist`, `box`, `kde`, `density`, `area`, `scatter`, `hexbin`, `pie`. The return value is a matplotlib object you can tweak further. + +## Global styles + +- Built-in: `plt.style.use("seaborn-v0_8-whitegrid")` or `ggplot`, `fivethirtyeight`, etc. ([pandas plotting example](https://www.practicaldatascience.org/notebooks/class_5/week_1/1.6.1_plotting_with_pandas.html)). +- Prefer **`plt.style.context(...)`** in libraries so you do not mutate global RC for importers. + +## SciencePlots + +- Project: [garrettj403/SciencePlots](https://github.com/garrettj403/SciencePlots). +- Install: `uv add SciencePlots` (or `pip install SciencePlots`). **LaTeX** is required for the default `"science"` style; the README documents `no-latex` variants and CJK font add-ons. +- Usage (v2+): **`import scienceplots`** before activating styles: + +```python +import matplotlib.pyplot as plt +import scienceplots + +plt.style.use(["science", "ieee"]) +``` + +- Combine styles: later entries override earlier (e.g. `ieee` column width over `science`). +- Temporary: `with plt.style.context(["science", "notebook"]): ...` + +## Resolution in notebooks + +- For crisp inline display, Practical Data Science suggests `matplotlib_inline.backend_inline.set_matplotlib_formats("retina")` instead of magic-only config when exporting notebooks to other runners ([basic plotting](https://www.practicaldatascience.org/notebooks/class_5/week_1/1.2.1_basic_plotting_with_matplotlib.html)). + +## When not to use heavy styles + +- Exploratory notebooks: keep default or light grid for speed. +- Submission: match **journal** requirements; SciencePlots is a shortcut, not a substitute for the author checklist. diff --git a/.cursor/skills/numpy-docstrings/SKILL.md b/.cursor/skills/numpy-docstrings/SKILL.md new file mode 100644 index 00000000..f2f74bf1 --- /dev/null +++ b/.cursor/skills/numpy-docstrings/SKILL.md @@ -0,0 +1,48 @@ +--- +author: dotagents +name: numpy-docstrings +description: Write and review NumPy-style (numpydoc) docstrings for Python public APIs. Covers section order, semantics (what belongs in docstrings vs types vs tests), anti-patterns (comment soup, stub docs, wrong style), Parameters, Returns, Yields, Raises, See Also, Notes, References, Examples, and class/module docs. Use when authoring or auditing docstrings. Triggers on docstring, numpydoc, Parameters, Returns, Examples, Sphinx, TODO, comment. +--- + +# NumPy-style docstrings (numpydoc) + +## Quick start + +1. **Public APIs only** per project **Python spec** and **Python rule**: document **modules, classes, functions, and methods** users import; private helpers stay minimal unless behavior is non-obvious. +2. **Follow section order** from [numpydoc](https://numpydoc.readthedocs.io/en/latest/format.html#sections); each heading is **underlined with hyphens** the same length as the title. +3. **Line length**: aim for **~75 characters** in docstring prose for terminal readability. +4. **Markup**: subset of **reST**; parameter names in **single backticks** in running text. +5. **Load the chunk** you are editing from the table below—each reference file targets one part of the standard. +6. **Semantics**: choose **docstring vs annotation vs test vs prose** deliberately; avoid **comment narration** and **stub** docstrings—see [reference-semantics-and-anti-patterns.md](references/reference-semantics-and-anti-patterns.md). + +## Stack synergy + +| Resource | Role | +|----------|------| +| **general-python** | pytest, “NumPy style on public APIs” | +| **python-reviewer** | Checks docstring quality alongside typing and tests | +| **python-types** | Types in signatures complement docstring types | + +## Section reference index + +| Docstring chunk | File | +|-----------------|------| +| Global rules, section order, line length, reST | [reference-overview-order.md](references/reference-overview-order.md) | +| Short summary, extended summary, deprecation | [reference-summary-deprecation.md](references/reference-summary-deprecation.md) | +| Parameters, Other Parameters, `*args` / `**kwargs` | [reference-parameters.md](references/reference-parameters.md) | +| Returns, Yields, Receives | [reference-returns-yields-receives.md](references/reference-returns-yields-receives.md) | +| Raises, Warns, Warnings | [reference-raises-warns-warnings.md](references/reference-raises-warns-warnings.md) | +| See Also, Notes, References | [reference-see-also-notes-references.md](references/reference-see-also-notes-references.md) | +| Examples (doctest style) | [reference-examples.md](references/reference-examples.md) | +| Classes, Attributes, Methods, modules | [reference-classes-modules.md](references/reference-classes-modules.md) | +| Why each mechanism; bad comments/docstrings; fixes | [reference-semantics-and-anti-patterns.md](references/reference-semantics-and-anti-patterns.md) | + +## Canonical specification + +- [numpydoc style guide](https://numpydoc.readthedocs.io/en/latest/format.html) +- [Example docstring](https://numpydoc.readthedocs.io/en/latest/example.html) +- [PEP 257](https://peps.python.org/pep-0257/) (baseline docstring conventions) + +## Optional tooling + +- **`uv add numpydoc`** when Sphinx builds use the numpydoc extension; do not hand-edit pins in `pyproject.toml`. diff --git a/.cursor/skills/numpy-docstrings/references/reference-classes-modules.md b/.cursor/skills/numpy-docstrings/references/reference-classes-modules.md new file mode 100644 index 00000000..f5d21cc2 --- /dev/null +++ b/.cursor/skills/numpy-docstrings/references/reference-classes-modules.md @@ -0,0 +1,49 @@ +# Classes, Attributes, Methods, modules + +Source: [Documenting classes](https://numpydoc.readthedocs.io/en/latest/format.html#documenting-classes), [Documenting modules](https://numpydoc.readthedocs.io/en/latest/format.html#documenting-modules) (if present in full page—module section exists in numpydoc). + +## Class docstring + +- Use the **same sections** as functions **except** **Returns** (unless a special method behaves like a function with a documented return—rare in class docstring body). +- **`__init__` parameters** belong in the class docstring **Parameters** section (constructor arguments). +- Optional separate **`__init__` docstring** for extra initialization detail if the project allows duplication control. + +## Attributes + +- Section **below Parameters** listing **non-method** attributes: + +```text +Attributes +---------- +x : float + The current x coordinate. +``` + +- Properties with their own docstrings may be listed **by name only** in **Attributes** when numpydoc defers to the property. + +## Methods + +- Each **public** method is documented like a **function**. +- **Do not** list **`self`** (or **`cls`**) in **Parameters**. +- If a method mirrors a **standalone function**, put the **detailed** narrative on the **function** docstring; the method may carry a **short summary** plus **See Also** pointing to the function ([numpydoc — method docstrings](https://numpydoc.readthedocs.io/en/latest/format.html#method-docstrings)). + +## Methods section (on the class) + +- Optional **Methods** block on the **class** docstring when only a **few** methods matter (e.g. large subclasses); list signatures in one line each with a short description—**not** for private methods. + +## `property` + +- Document **getter behavior**, **raises**, and **side effects** if any; type in **Returns** or in the one-line summary as appropriate. + +## Module docstring + +- At least a **summary line**; optional sections in **function order** where appropriate: extended summary, **routine listings** for large modules, **See Also**, **Notes**, **References**, **Examples** ([documenting modules](https://numpydoc.readthedocs.io/en/latest/format.html#documenting-modules)). +- **License and author** metadata belong outside the docstring (comments or project files), not in the module docstring per numpydoc. + +## Constants + +- Use summary, optional extended summary, **See Also**, **References**, **Examples** as needed ([documenting constants](https://numpydoc.readthedocs.io/en/latest/format.html#documenting-constants)); some immutable constants cannot carry `__doc__` in the REPL. + +## Private objects + +- Leading underscore names: minimal or no numpydoc sections unless they are part of a **subclassing contract**. diff --git a/.cursor/skills/numpy-docstrings/references/reference-examples.md b/.cursor/skills/numpy-docstrings/references/reference-examples.md new file mode 100644 index 00000000..dc3e9f59 --- /dev/null +++ b/.cursor/skills/numpy-docstrings/references/reference-examples.md @@ -0,0 +1,39 @@ +# Examples + +Source: [numpydoc — Examples](https://numpydoc.readthedocs.io/en/latest/format.html#examples). + +## Role + +- **Illustrate usage**; they are **not** the primary test suite—keep **`tests/`** as the authority for CI. +- Still **strongly encouraged** for user-facing APIs. + +## Doctest format + +- Use **`>>>`** prompts and expected output as in the Python REPL. +- **Separate** multiple examples with **blank lines**. +- Put **blank lines above and below** comment lines that explain an example. + +## Continuations + +- Continuation lines after the first `>>>` start with **`...`**. + +## Random or platform-dependent output + +- Mark with **`#random`** (or project convention) so doctest runners can skip strict comparison. + +## Imports in examples + +- NumPy docs assume **`import numpy as np`** is pre-run for numpy examples; still be **explicit** for anything else (`matplotlib.pyplot`, local modules). +- **Explicitly import** the function under documentation if it aids copy-paste. + +## Empty lines in output + +- Empty output lines in doctests do not need special markup per numpydoc. + +## Matplotlib + +- If **`matplotlib`** is imported in the example, Sphinx may use matplotlib’s **plot** directive when configured; otherwise `.. plot::` can be used in `.rst` sources—not always inside docstrings. + +## Running doctests + +- Projects may run **`pytest --doctest-modules`** or library-specific test hooks; align with the project **Python** spec (**pytest**). diff --git a/.cursor/skills/numpy-docstrings/references/reference-overview-order.md b/.cursor/skills/numpy-docstrings/references/reference-overview-order.md new file mode 100644 index 00000000..20098eb5 --- /dev/null +++ b/.cursor/skills/numpy-docstrings/references/reference-overview-order.md @@ -0,0 +1,53 @@ +# Overview: format, order, and markup + +Source: [numpydoc format](https://numpydoc.readthedocs.io/en/latest/format.html). + +## Docstring wrapper + +- Use **triple double quotes** `"""` for module, class, function, and method docstrings. +- First line after the opening `"""` is often the **short summary** (or signature line for C extensions without introspection). + +## Line length + +- Keep lines to about **75 characters** so docstrings read in plain terminals. + +## Section headings + +- Each section title is a **line of text** followed by a line of **hyphens** the **same length** as the title: + +```text +Parameters +---------- +``` + +- **Order** for functions (omit unused sections): Short summary → deprecation (directive) → Extended summary → Parameters → Returns → Yields → Receives → Other Parameters → Raises → Warns → Warnings → See Also → Notes → References → Examples. + +## reStructuredText subset + +- Docstrings use **reST** for Sphinx; prefer a **small** subset: inline `` `param` ``, `` :math:`x` ``, `.. deprecated::`, `.. math::`, `.. note::` sparingly. +- **Human readability** beats contorting text for HTML output ([numpydoc principle](https://numpydoc.readthedocs.io/en/latest/format.html#docstring-standard)). + +## Import names in narrative docs + +- NumPy docs convention: `import numpy as np`, `import matplotlib as mpl`, `import matplotlib.pyplot as plt`; **do not** abbreviate `scipy` as something nonstandard in prose. + +## What not to duplicate + +- **Signature** is usually shown by `help()`; the short summary should **not** repeat the function name as a title unless documenting a C API without a visible signature. + +## `array_like` + +- Use **`array_like`** when an argument accepts **ndarrays** and values **coercible** to arrays (scalars, nested sequences). + +## reST `.. note::` and `.. warning::` + +- Use **sparingly** in sections; they render poorly in plain terminals ([numpydoc](https://numpydoc.readthedocs.io/en/latest/format.html#other-points-to-keep-in-mind)). + +## Monospace in prose + +- **Parameter names**: single backticks (numpydoc convention). +- **Other code snippets** in running text: often **double backticks** in reST; follow project Sphinx/numpydoc version guidance for link vs monospace. + +## Hyperlinks in docstrings + +- Some sections parse **non-standard** reST; fragile `.. target` lines can confuse numpydoc—prefer **inline** links where Sphinx warns ([numpydoc](https://numpydoc.readthedocs.io/en/latest/format.html#other-points-to-keep-in-mind)). diff --git a/.cursor/skills/numpy-docstrings/references/reference-parameters.md b/.cursor/skills/numpy-docstrings/references/reference-parameters.md new file mode 100644 index 00000000..eaf17fd4 --- /dev/null +++ b/.cursor/skills/numpy-docstrings/references/reference-parameters.md @@ -0,0 +1,44 @@ +# Parameters and Other Parameters + +Source: [numpydoc — Parameters](https://numpydoc.readthedocs.io/en/latest/format.html#parameters), [Other Parameters](https://numpydoc.readthedocs.io/en/latest/format.html#other-parameters). + +## Heading + +```text +Parameters +---------- +``` + +## Each parameter + +- Form: **`name : type`** then newline, then **indented** description (4 spaces). +- numpydoc: the colon may be **omitted if there is no type**; otherwise use **`name : type`** (space before the colon after the name, space after the colon before the type). +- Type may be **omitted**: then `name` alone with description on the next line. +- Refer to a parameter in prose with **single backticks**: `` `x` ``. + +## Types (be specific) + +Examples from numpydoc: `str`, `bool`, `array_like`, `int or tuple of int`, `list of str`, `dtype`, `callable`, union with **or**. + +## Optional and defaults + +- Keyword-only optionals: **`optional`** in the type line: `x : int, optional` +- Or give **default** in type: `copy : bool, default True` (also `default=True`, `default: True`—pick one style per project). + +## Fixed set of values + +- `order : {'C', 'F', 'A'}` with default listed first in braces when applicable. + +## Combining parameters + +- Same type and meaning: `x1, x2 : array_like` then one description referencing `` `x1` `` and `` `x2` ``. + +## `*args` and `**kwargs` + +- Keep the **stars** in the name; **do not** give a type on the `*args` / `**kwargs` line in numpydoc style. +- Describe how forwarded kwargs map to underlying functions if relevant. + +## Other Parameters + +- Use **only** when **many** keyword parameters would clutter **Parameters**; move **rare** or **advanced** kwargs here. +- Same formatting rules as **Parameters**. diff --git a/.cursor/skills/numpy-docstrings/references/reference-raises-warns-warnings.md b/.cursor/skills/numpy-docstrings/references/reference-raises-warns-warnings.md new file mode 100644 index 00000000..2e031a30 --- /dev/null +++ b/.cursor/skills/numpy-docstrings/references/reference-raises-warns-warnings.md @@ -0,0 +1,28 @@ +# Raises, Warns, Warnings + +Source: [Raises](https://numpydoc.readthedocs.io/en/latest/format.html#raises), [Warns](https://numpydoc.readthedocs.io/en/latest/format.html#warns), [Warnings](https://numpydoc.readthedocs.io/en/latest/format.html#warnings). + +## Raises + +- List **exception types** and **when** they are raised. +- Use **judiciously**: obvious `ValueError` on bad input may be omitted if universal; **non-obvious** or **high-probability** errors deserve a line. + +```text +Raises +------ +LinAlgError + If the matrix is singular. +``` + +## Warns + +- Same shape as **Raises** but for **`Warning`** subclasses or user-visible warnings issued under stated conditions. + +## Warnings (free-text cautions) + +- **User-facing caveats** not tied to a single exception (numerical instability, thread safety, heuristic behavior). +- Free-form reST paragraph(s) under the **Warnings** heading—not the same as Python’s `warnings.warn`. + +## Relation to typing + +- **`ty` / PEP 484** do not replace **Raises**; document **semantic** error contracts for readers and Sphinx. diff --git a/.cursor/skills/numpy-docstrings/references/reference-returns-yields-receives.md b/.cursor/skills/numpy-docstrings/references/reference-returns-yields-receives.md new file mode 100644 index 00000000..fb20a7ff --- /dev/null +++ b/.cursor/skills/numpy-docstrings/references/reference-returns-yields-receives.md @@ -0,0 +1,40 @@ +# Returns, Yields, Receives + +Source: [Returns](https://numpydoc.readthedocs.io/en/latest/format.html#returns), [Yields](https://numpydoc.readthedocs.io/en/latest/format.html#yields), [Receives](https://numpydoc.readthedocs.io/en/latest/format.html#receives). + +## Returns + +- For each return value: **type is required**; **name is optional**. +- Anonymous single return: + +```text +Returns +------- +int + Description of return value. +``` + +- Named returns mirror **Parameters**: + +```text +Returns +------- +err_code : int + Non-zero indicates error. +err_msg : str or None + Message or None on success. +``` + +## Yields + +- For **generators** only; same structure as **Returns** (type required, name optional). +- numpydoc **0.6+** supports **Yields**. + +## Receives + +- Documents what a generator receives via **`.send()`**; format like **Parameters**. +- If **Receives** appears, **Yields** must also appear. + +## Multiple return paths + +- Document **all** stable contracts; if behavior is `None` or sentinel, state **when** each occurs in the description lines. diff --git a/.cursor/skills/numpy-docstrings/references/reference-see-also-notes-references.md b/.cursor/skills/numpy-docstrings/references/reference-see-also-notes-references.md new file mode 100644 index 00000000..2c8433c6 --- /dev/null +++ b/.cursor/skills/numpy-docstrings/references/reference-see-also-notes-references.md @@ -0,0 +1,23 @@ +# See Also, Notes, References + +Source: [See Also](https://numpydoc.readthedocs.io/en/latest/format.html#see-also), [Notes](https://numpydoc.readthedocs.io/en/latest/format.html#notes), [References](https://numpydoc.readthedocs.io/en/latest/format.html#references). + +## See Also + +- Point to **related** APIs the reader might miss; avoid listing the whole module. +- Form: `name : Short description.` or **`name` only** if the name is self-explanatory. +- Same submodule: **unqualified** name; other submodules: **`submod.func`**; other packages: **`package.module.func`**. +- Long descriptions: put **name :** on first line, description **indented** four spaces on the next. + +## Notes + +- **Algorithms**, **complexity**, **numerical details**, **background theory** that would clutter **Extended summary**. +- **Equations**: `.. math::` block or inline `` :math:`\alpha` ``; keep LaTeX **readable**—equations are hard in plain text. +- **Images**: `.. image:: path` only if the docstring still makes sense **without** the image (numpydoc guidance). +- Variable emphasis in math: `\mathtt{var}` when needed for typographic distinction. + +## References + +- **Numbered** citations supporting **Notes**; use `.. [1] Author, "Title", ...` and cite with `[1]_` in Notes. +- Prefer **stable** sources; **avoid** fragile URLs as the only citation. +- **Caveat** ([numpydoc #130](https://github.com/numpy/numpydoc/issues/130)): citation markers like `[1]` inside **tables** in docstrings can break numpydoc processing—avoid that combination or restructure. diff --git a/.cursor/skills/numpy-docstrings/references/reference-semantics-and-anti-patterns.md b/.cursor/skills/numpy-docstrings/references/reference-semantics-and-anti-patterns.md new file mode 100644 index 00000000..ce869480 --- /dev/null +++ b/.cursor/skills/numpy-docstrings/references/reference-semantics-and-anti-patterns.md @@ -0,0 +1,235 @@ +# Semantics: which documentation mechanism, and anti-patterns + +This file explains **where** information should live (docstring section vs type hint vs test vs narrative) and shows **bad habits** next to **spec-aligned** fixes. It aligns with the project **Python** spec and **Python rule**: NumPy-style docstrings on **public** APIs, **obvious control flow** without step-by-step **inline comments** in implementations. + +## Choosing a mechanism + +| What the reader needs | Prefer | +|------------------------|--------| +| **Callable contract** (what goes in, what comes out, what can fail) | **Parameters**, **Returns** / **Yields**, **Raises** in a numpydoc docstring | +| **Meaning that types cannot express** (units, ranges, physical interpretation, invariants) | Same sections’ prose, or **Notes** if it spans several parameters | +| **Pure type information** already clear from annotations | **`ty`-checked** signatures; **omit** redundant docstring types unless you add **semantics** | +| **How to call it** (minimal copy-paste) | **Examples** (short, doctest-style when appropriate) | +| **Where to go next** (alternatives, partners) | **See Also** | +| **Theory, algorithm, equations** | **Notes**; stable citations in **References** | +| **User-facing caution** (not one specific exception) | **Warnings** section | +| **Deprecation and replacement** | **`.. deprecated::`** directive ([summary/deprecation](reference-summary-deprecation.md)) | +| **Whole module map** (big packages) | **Module** docstring + **routine listings** ([classes/modules](reference-classes-modules.md)) | +| **Behavior guaranteed by tests** | **`tests/`** + pytest; docstring **Examples** illustrate, they do not replace CI | +| **Private helper** with a non-obvious invariant | One-line docstring **or** a **name** that carries the invariant; not a public numpydoc essay | + +**Principle:** put **contract and usage** where **`help(obj)`** and Sphinx readers look first; keep **implementation noise** out of public docstrings and out of **comment trails** per project Python style. + +--- + +## Anti-pattern: comments as a substitute for names and structure + +**Bad:** narrating the code line-by-line or stating the obvious. + +```python +def process(a): + # loop over each row + for i in range(a.shape[0]): + # get the value + v = a[i, 0] + # add one + a[i, 0] = v + 1 + return a +``` + +**Better:** **vectorize** when the stack allows; if a loop remains, **name** intent and move **contract** to the docstring, not inline chatter. + +```python +import numpy as np + +def increment_first_column(a: np.ndarray) -> np.ndarray: + """Increment every value in the first column by one. + + Parameters + ---------- + a : numpy.ndarray, shape (M, N) + Input array; not modified in place. + + Returns + ------- + numpy.ndarray + A copy of ``a`` with column 0 incremented by one. + """ + out = np.array(a, copy=True) + out[:, 0] = out[:, 0] + 1 + return out +``` + +--- + +## Anti-pattern: docstring repeats the signature with no added meaning + +**Bad:** + +```python +def mean(a, axis=None): + """mean(a, axis=None) + + Takes a and axis and returns the mean. + """ +``` + +**Better:** short summary **without** mirroring parameter names as the only content; put types and defaults in **Parameters** / **Returns**. + +```python +import numpy as np + +def mean(a: np.ndarray, axis: int | None = None) -> np.floating | np.ndarray: + """Arithmetic mean along the given axis. + + Parameters + ---------- + a : array_like + Input data. + axis : int, optional + Axis along which to compute the mean. Default is None (flatten). + + Returns + ------- + numpy.floating or ndarray + Mean of ``a`` over the given axis. + """ +``` + +--- + +## Anti-pattern: parameter descriptions only in the extended summary + +**Bad:** long paragraph that lists args but no **Parameters** section (breaks numpydoc parsing and `help()` structure). + +**Better:** keep extended summary for **intent**; every public argument gets a **Parameters** line (or **Other Parameters** if rarely used). See [reference-parameters.md](reference-parameters.md). + +--- + +## Anti-pattern: empty or stub sections + +**Bad:** + +```text +Returns +------- + +Raises +------ +``` + +**Better:** omit entire sections that add nothing; numpydoc expects **content** under each heading you include. + +--- + +## Anti-pattern: “TODO” and placeholders in shipped APIs + +**Bad:** `"""TODO: document."""` on a public function. + +**Better:** ship with at least a **short summary** and **Parameters**/**Returns** for anything exported in `__all__` or documented as stable; track internal debt in issues, not user-facing docstrings. + +--- + +## Anti-pattern: author, license, or change log in the module docstring + +**Bad:** module docstring that is only copyright or “written by Alice 2021.” + +**Better:** numpydoc reserves the module docstring for **summary**, optional listings, **See Also**, **Examples**; put license in **`LICENSE`** and attribution in **VCS** or **`pyproject.toml`** ([classes/modules](reference-classes-modules.md)). + +--- + +## Anti-pattern: duplicating type hints with zero extra semantics + +**Bad:** + +```python +def f(x: int, y: float) -> str: + """Do something. + + Parameters + ---------- + x : int + An integer. + y : float + A float. + + Returns + ------- + str + A string. + """ +``` + +**Better:** keep **types** in annotations; use docstring lines for **meaning** (role, units, constraints, edge cases). + +```python +def f(x: int, y: float) -> str: + """Format a channel index and gain for display. + + Parameters + ---------- + x : int + Zero-based acquisition channel index. + y : float + Linear gain in decibels applied before quantization. + + Returns + ------- + str + Human-readable label, e.g. ``"ch3 (+6.0 dB)"``. + """ +``` + +--- + +## Anti-pattern: wrong style for the project + +**Bad:** Google-style `Args:` / `Returns:` blocks in a repo that standardizes on **NumPy** / numpydoc and Sphinx. + +**Better:** match **this stack**: hyphen underlines and section names from [numpydoc format](https://numpydoc.readthedocs.io/en/latest/format.html#sections). + +--- + +## Anti-pattern: Examples that are really integration tests + +**Bad:** dozens of lines of setup, mocks, and assertions inside **Examples** that belong in **`tests/`**. + +**Better:** **Examples** show **minimal** usage; heavy behavior is **pytest** with stable fixtures ([reference-examples.md](reference-examples.md)). + +--- + +## Anti-pattern: documenting `self` on methods + +**Bad:** listing `self` under **Parameters** on instance methods. + +**Better:** omit **`self`** / **`cls`** per numpydoc ([reference-classes-modules.md](reference-classes-modules.md)). + +--- + +## Anti-pattern: critical exceptions only in comments + +**Bad:** + +```python +def invert(m): + # raises LinAlgError if singular + return np.linalg.inv(m) +``` + +**Better:** **Raises** in the docstring so `help()` and HTML docs show the contract. + +```text +Raises +------ +numpy.linalg.LinAlgError + If ``m`` is singular. +``` + +--- + +## Quick checklist + +- Public API: **short summary** plus the **sections** that carry real information—**no** filler headings. +- Implementation: **readable structure** and **names**, not comment narration (**Python** rule). +- Types: **annotations + ty** for static truth; docstring for **semantics** types miss. +- Red flags: **TODO** docstrings, **license-only** module docs, **Args/Returns** markdown in a numpydoc project, **comment soup** instead of refactors. diff --git a/.cursor/skills/numpy-docstrings/references/reference-summary-deprecation.md b/.cursor/skills/numpy-docstrings/references/reference-summary-deprecation.md new file mode 100644 index 00000000..96d85f64 --- /dev/null +++ b/.cursor/skills/numpy-docstrings/references/reference-summary-deprecation.md @@ -0,0 +1,31 @@ +# Short summary, extended summary, deprecation + +Source: [numpydoc — Short summary](https://numpydoc.readthedocs.io/en/latest/format.html#short-summary), [Extended summary](https://numpydoc.readthedocs.io/en/latest/format.html#extended-summary), [Deprecation](https://numpydoc.readthedocs.io/en/latest/format.html#deprecation-warning). + +## Short summary + +- **One line** (no blank line before extended text if you continue in the same opening paragraph, or blank line then extended summary—follow project style; numpydoc allows extended summary after a blank line). +- Do **not** use **variable names** from the signature in the short summary. +- Prefer avoiding repeating the **function name** as the opening phrase when it reads redundant with `help()`; state **what** it does in plain language. + +Good pattern: `"""Compute the discrete Fourier transform along the specified axis.` + +## C signature line (rare) + +- For objects **without** introspectable signatures, put the signature as the **first** line inside the docstring, then a blank line, then the short summary (see numpydoc examples). + +## Deprecation + +- Use the Sphinx **`.. deprecated:: version`** **directive**, not a hyphen underlined section, when an API is deprecated. +- Include: **version** deprecated, **removal** timeline if known, **reason** when useful, and **replacement** API. + +```text +.. deprecated:: 1.6.0 + `old_func` is replaced by `new_func` because ... +``` + +## Extended summary + +- A few sentences **after** the short summary (separated by a blank line when it is its own block). +- Use for **what** the object does at a higher level; **not** for implementation trivia or long theory—those belong in **Notes** or **References**. +- You **may** mention parameters or the function name here, but **parameter definitions** still belong only under **Parameters**. diff --git a/.cursor/skills/numpy-scientific/SKILL.md b/.cursor/skills/numpy-scientific/SKILL.md new file mode 100644 index 00000000..e448b515 --- /dev/null +++ b/.cursor/skills/numpy-scientific/SKILL.md @@ -0,0 +1,52 @@ +--- +author: dotagents +name: numpy-scientific +description: NumPy for scientific Python: dtypes and casting, creation, reshaping, broadcasting, indexing vs views, ufuncs and reductions, linear algebra and einsum, random Generator, I/O, structured arrays, performance and pandas boundaries. Use when writing or reviewing ndarray code. Triggers on numpy, ndarray, broadcasting, dtype, ufunc, einsum, vectorization. +--- + +# NumPy (scientific computing) + +## Quick start + +1. **Shape and dtype first**: decide **`shape`**, **`dtype`**, and memory **`order`** (`C` vs `F`) before filling arrays; avoid silent **`astype`** widening. See [reference-arrays-dtypes.md](references/reference-arrays-dtypes.md). +2. **Vectorize**: prefer **ufuncs** and **whole-array** expressions over Python loops on large data; push loops to compiled layers when needed. See [reference-ufuncs-reductions.md](references/reference-ufuncs-reductions.md) and [reference-interop-performance.md](references/reference-interop-performance.md). +3. **Know view vs copy**: basic slicing is usually a **view**; fancy indexing is a **copy**; **`reshape`** may view or copy. Mutations alias bugs are common. See [reference-indexing-views.md](references/reference-indexing-views.md). +4. **Broadcasting**: align trailing dimensions; use **`None` / `newaxis`** and **`np.broadcast_to`** deliberately. See [reference-broadcasting-shape.md](references/reference-broadcasting-shape.md). +5. **Reductions**: always pass **`axis`** and use **`keepdims=True`** when broadcasting the result back; prefer **`np.nanmean`** etc. when NaNs appear. Mind **float summation order** for reproducibility. See [reference-ufuncs-reductions.md](references/reference-ufuncs-reductions.md). +6. **Random**: use **`np.random.Generator`** (`PCG64`), not legacy global `RandomState`, for reproducible science. See [reference-random-io-structured.md](references/reference-random-io-structured.md). + +## Stack synergy + +| Resource | Role | +|----------|------| +| **general-python** | uv, ruff, ty, vectorization policy, **python-reviewer** numerics footguns | +| **dataframes** | Table handoff at boundaries | +| **numpy-docstrings** | Public API docstrings for array-heavy modules | +| **matplotlib-scientific** | Plotting array results | +| **lab-instrumentation** | Binary waveform I/O, instrument-side buffers | +| **python-reviewer** | dtype/shape/reproducibility in review | + +## Reference index + +| Topic | File | +|--------|------| +| Creation, `dtype`, casting, `order`, strides | [reference-arrays-dtypes.md](references/reference-arrays-dtypes.md) | +| Slicing, fancy index, views, copies, `reshape` | [reference-indexing-views.md](references/reference-indexing-views.md) | +| Broadcasting, stacking, meshgrid / ogrid | [reference-broadcasting-shape.md](references/reference-broadcasting-shape.md) | +| Ufuncs, reductions, `axis`, `where`, NaNs | [reference-ufuncs-reductions.md](references/reference-ufuncs-reductions.md) | +| `linalg`, `@`, `einsum`, `tensordot` | [reference-linalg-einsum.md](references/reference-linalg-einsum.md) | +| `Generator`, `.npy` / `.npz`, structured dtypes | [reference-random-io-structured.md](references/reference-random-io-structured.md) | +| Pandas/Polars handoff, memory, threads | [reference-interop-performance.md](references/reference-interop-performance.md) | + +## Official documentation + +- [NumPy user guide](https://numpy.org/doc/stable/user/index.html) +- [Absolute basics](https://numpy.org/doc/stable/user/absolute_beginners.html) +- [Broadcasting](https://numpy.org/doc/stable/user/basics.broadcasting.html) +- [Indexing](https://numpy.org/doc/stable/user/basics.indexing.html) +- [Copies and views](https://numpy.org/doc/stable/user/basics.copies.html) +- [NumPy reference](https://numpy.org/doc/stable/reference/index.html) + +## Dependency + +Add or upgrade with **`uv add numpy`** per project policy; do not hand-edit pins in `pyproject.toml`. diff --git a/.cursor/skills/numpy-scientific/references/reference-arrays-dtypes.md b/.cursor/skills/numpy-scientific/references/reference-arrays-dtypes.md new file mode 100644 index 00000000..5d68c2cc --- /dev/null +++ b/.cursor/skills/numpy-scientific/references/reference-arrays-dtypes.md @@ -0,0 +1,45 @@ +# Array creation, dtypes, casting + +## ndarray mental model + +- **`shape`**: tuple of lengths per axis; **`ndim`**; **`size`** total elements. +- **`dtype`**: element type (size and interpretation); **`itemsize`** bytes per element. +- **`order`**: **`C`** row-major (default) vs **`F`** column-major; matters for **reshape**, **flat** iteration, and **FFI** with other libraries. + +Official: [The N-dimensional array](https://numpy.org/doc/stable/reference/arrays.ndarray.html). + +## Constructors + +| Need | Typical API | +|------|-------------| +| Zeros / ones | `np.zeros`, `np.ones`, `dtype=...` | +| Uninitialized (then fill) | `np.empty` (fastest when you overwrite every element) | +| Constant fill | `np.full` | +| Sequences | `np.asarray` (no copy if already ndarray and compatible), `np.array` (new array) | +| Ranges | `np.arange` (half-open, prefer **integer** steps; watch **float** accumulation), `np.linspace`, `np.logspace`, `np.geomspace` | +| Identity / diag | `np.eye`, `np.diag` | + +## Dtype choices + +- **Integers**: `np.int32`, `np.int64`, `np.uint8` for images, etc.; match **file format** and **downstream** APIs. +- **Floats**: `float64` default; `float32` for memory/bandwidth when precision allows; **`float16`** mainly for storage or ML hooks, not general numerics. +- **Complex**: `complex128` default pair with `float64` real/imag. +- **Booleans**: `bool_`; avoid Python `bool` in dtype lists for homogeneous arrays. + +## Casting and `astype` + +- **`astype`** can **round** or **truncate**; floating to integer is not the same as `round()` policy-wise—verify for your domain. +- **`astype(copy=False)`** may still copy if layout or dtype forces it. +- Prefer **`np.can_cast`** and explicit **`dtype=`** on creation when ingesting untyped buffers. + +## Strings and objects + +- **Fixed-width strings**: `dtype="U10"`, `"S10"`; variable-length text often belongs in **pandas** or Python lists, not `object` arrays, unless you accept **`object`** dtype costs. + +## Alignment and record dtypes + +- **Structured / record** dtypes: field offsets and alignment; see [reference-random-io-structured.md](reference-random-io-structured.md). + +## Verification + +- **`array.shape`**, **`array.dtype`**, **`array.flags`** (`writeable`, `owndata`) when debugging aliases. diff --git a/.cursor/skills/numpy-scientific/references/reference-broadcasting-shape.md b/.cursor/skills/numpy-scientific/references/reference-broadcasting-shape.md new file mode 100644 index 00000000..8e9ef8c7 --- /dev/null +++ b/.cursor/skills/numpy-scientific/references/reference-broadcasting-shape.md @@ -0,0 +1,36 @@ +# Broadcasting and array shape + +Official: [Broadcasting](https://numpy.org/doc/stable/user/basics.broadcasting.html). + +## Rule + +From trailing dimensions forward, pairs are compatible if **equal**, or **one is 1**, or **one is missing** (implicit 1). The result shape is the **maximum** of each aligned size. + +## New axes + +- **`arr[:, np.newaxis]`** or **`arr[:, None]`** inserts a length-1 axis for alignment. +- **`np.expand_dims`** for clarity in library code. + +## Explicit broadcast + +- **`np.broadcast_to`** materializes a **read-only** strided view when possible; good for debugging shapes. +- **`np.broadcast_arrays`** aligns several inputs to a common shape. + +## Stacking and splitting + +- **`np.stack`** (new axis), **`np.concatenate`** (existing axis), **`np.vstack` / `np.hstack` / `np.dstack`** (convenience; know what they do to 1D). +- **`np.split`**, **`array_split`** for uneven chunks. + +## Grids + +- **`np.meshgrid`** (`indexing="xy"` vs `"ij"`): know which axis varies **fast**; mistakes swap rows/columns in images. +- **`np.ogrid`** / **`mgrid`**: open grids that **broadcast** without huge temporaries. + +## Tile and repeat + +- **`np.tile`** repeats blocks; **`np.repeat`** repeats elements; different memory patterns. + +## Gotchas + +- **Scalar + array** broadcasts naturally; **list of arrays** does not become one ndarray without **`stack`**/**`array`**. +- **In-place** `+=` with broadcast RHS can create **temporary** behavior surprises; verify with small examples when optimizing. diff --git a/.cursor/skills/numpy-scientific/references/reference-indexing-views.md b/.cursor/skills/numpy-scientific/references/reference-indexing-views.md new file mode 100644 index 00000000..d7f27a47 --- /dev/null +++ b/.cursor/skills/numpy-scientific/references/reference-indexing-views.md @@ -0,0 +1,38 @@ +# Indexing, views, and copies + +Official guides: [Indexing](https://numpy.org/doc/stable/user/basics.indexing.html), [Copies and views](https://numpy.org/doc/stable/user/basics.copies.html). + +## Basic indexing (views) + +- **Slices** `start:stop:step`, **ellipsis** `...`, and **integer** indexing that **reduces** dimensionality yield **views** when the memory layout allows. +- **Mutating** a slice can change the **base** array—this is the main footgun. + +## Advanced indexing (copies) + +- **Integer arrays**, **boolean masks** (unless `numpy.ma`), and combinations that cannot be expressed as a strided window generally produce a **copy**. +- After `b = a[mask]`, changing **`b`** does not change **`a`**. + +## `reshape`, `ravel`, `transpose`, `swapaxes` + +- **`reshape`**: returns **view** when strides allow, else **copy**; use **`arr.reshape(-1)`** for flattening with explicit intent. +- **`ravel`**: often a view; **`flatten`** always a copy. +- **`transpose` / `.T`**: usually a view. + +## `copy` and `base` + +- **`arr.copy()`** for an independent buffer. +- **`arr.base`**: if not `None`, data is owned elsewhere; trace mutations through **views**. + +## Boolean indexing vs `numpy.ma` + +- Plain **boolean** indexing on `ndarray` **copies** selected elements. +- **Masked arrays** (`numpy.ma`) carry a mask alongside data; use when NaNs are insufficient or you need mask algebra. + +## Assignment + +- **`a[i:j] = x`** writes through to base; shapes must broadcast. +- **Chained** fancy indexing assignment has subtleties; prefer **single** assignment with **`np.put`** or explicit loop only when necessary. + +## Reading suggestions + +- When unsure, **`np.shares_memory(a, b)`** answers whether two arrays might alias. diff --git a/.cursor/skills/numpy-scientific/references/reference-interop-performance.md b/.cursor/skills/numpy-scientific/references/reference-interop-performance.md new file mode 100644 index 00000000..53fa3e31 --- /dev/null +++ b/.cursor/skills/numpy-scientific/references/reference-interop-performance.md @@ -0,0 +1,35 @@ +# Interop, pandas, and performance + +## pandas and Polars + +- **pandas**: **`Series.values` / `to_numpy()`**, **`DataFrame.to_numpy()`**—watch **copy** vs **view** and **dtype=object** columns. +- **Polars**: convert via **`to_numpy()`** on expressions/Series; prefer staying in Polars for query-heavy pipelines per the Python spec. +- Establish a **single owner** of labels (index/columns) at module boundaries; do not duplicate metadata in raw ndarrays without documentation. +- Table-level patterns: **dataframes** skill. + +## SciPy, C, CUDA + +- **SciPy** routines accept **`array_like`**; pass **`np.asarray`** with **`dtype`** when you need a contiguous buffer. +- **ctypes** / **cffi**: **`array.ctypes`**, **`__array_interface__`**; enforce **contiguity** (`C` or `F`) before passing pointers. + +## Vectorization discipline + +- Prefer **ufuncs** and **broadcasting** over Python **`for`** loops over rows when **N** is large. +- When loops remain, **Numba**, **Cython**, or moving hot paths to **Rust/PyO3** are separate project decisions—do not introduce heavy deps without **`uv add`** and team agreement. + +## Memory + +- **`dtype` downcast** when safe; **`in-place`** ufuncs only when readability and aliasing are controlled. +- **`del`** large temporaries in tight notebooks is cosmetic; **streaming** or **chunking** fixes real pressure. + +## Threads and BLAS + +- Underlying **BLAS/OpenMP** may use threads; nested **joblib** / process pools can **oversubscribe** CPU. Set **`OMP_NUM_THREADS`** / vendor vars in HPC scripts when jobs are parallel at a higher level. + +## Testing + +- **Parametrize** shapes and dtypes; assert **`out.shape`**, **`dtype`**, and known **analytic** cases for numerics (**`python-reviewer`**). + +## Plotting + +- Pass **`np.asarray`** to **matplotlib** when consuming duck arrays; use **`matplotlib-scientific`** skill for figure polish. diff --git a/.cursor/skills/numpy-scientific/references/reference-linalg-einsum.md b/.cursor/skills/numpy-scientific/references/reference-linalg-einsum.md new file mode 100644 index 00000000..08b0a1b0 --- /dev/null +++ b/.cursor/skills/numpy-scientific/references/reference-linalg-einsum.md @@ -0,0 +1,32 @@ +# Linear algebra and `einsum` + +## Matrix multiply + +- **`@`** operator and **`np.matmul`**: **batched** last two dimensions for ndim > 2; not the same as **`np.dot`** for stacks (prefer **`@`** for clarity). +- **Elementwise** multiply: **`*`** or **`np.multiply`**. + +Official: [numpy.linalg](https://numpy.org/doc/stable/reference/routines.linalg.html). + +## `linalg` + +- **`np.linalg.solve`**: linear systems **`Ax = b`** (prefer over **`inv(A) @ b`** numerically). +- **`lstsq`**, **`svd`**, **`eig`**, **`eigh`** (Hermitian), **`cholesky`**, **`norm`**, **`det`**, **`matrix_rank`**. +- **`cond`** and **`rcond`** for ill-conditioned systems. + +## `einsum` + +- **Einstein summation**: specify **subscripts** or **`->` output**; **sum** repeated indices. +- **`optimize=True`** fuses paths for large contractions (default in recent NumPy for many cases). +- Use for **tensor contractions**, **batched** matmuls, and **axis reductions** that are hard to read as nested **`sum`**. + +Official: [numpy.einsum](https://numpy.org/doc/stable/reference/generated/numpy.einsum.html). + +## `tensordot`, `kron`, `outer` + +- **`np.tensordot`**: contract chosen axes between two tensors. +- **`np.kron`**: Kronecker product; can explode **size** quickly. +- **`np.outer`**: rank-1 outer product. + +## dtypes + +- **`linalg`** generally expects **floating** or **complex floating**; integer arrays may be promoted or rejected—cast explicitly. diff --git a/.cursor/skills/numpy-scientific/references/reference-random-io-structured.md b/.cursor/skills/numpy-scientific/references/reference-random-io-structured.md new file mode 100644 index 00000000..d41a8f81 --- /dev/null +++ b/.cursor/skills/numpy-scientific/references/reference-random-io-structured.md @@ -0,0 +1,41 @@ +# Random, file I/O, structured dtypes + +## Random (`numpy.random`) + +- Use **`np.random.default_rng(seed)`** → **`Generator`** with **`PCG64`** (or other bit generators per docs). +- Avoid legacy **`np.random.seed`** / **`RandomState`** in new library code; they complicate **parallel** and **library composition**. + +Official: [Random sampling](https://numpy.org/doc/stable/reference/random/index.html). + +## Common patterns + +```python +rng = np.random.default_rng(42) +rng.normal(size=(1000, 2)) +rng.integers(low=0, high=10, size=5, endpoint=False) +rng.permutation(x) +``` + +- **`SeedSequence`** for **spawned** streams in parallel jobs. + +## Saving and loading + +- **`.npy`**: single array, preserves shape and dtype. +- **`.npz`**: archive of **named** arrays (`np.savez`, **`np.savez_compressed`**). +- **`np.load`**: `allow_pickle=False` default in modern NumPy—**do not** unpickle untrusted files. + +Official: [NumPy binary formats](https://numpy.org/doc/stable/reference/routines.io.html). + +## Text and CSV + +- **`np.loadtxt`**, **`genfromtxt`** for simple cases; **pandas** / **polars** often scale better for messy tables—hand off at a clear **module or pipeline** boundary. + +## Structured and record arrays + +- **`dtype=[('x', 'f8'), ('id', 'i4')]`** for **columnar** heterogeneous data in one block. +- **Alignment** and **offsets** matter for **C interop**; use **`np.dtype`** descriptors carefully. +- For analytics-heavy workflows, consider **DataFrame** after one **`from_records`** step. + +## Memory maps + +- **`np.memmap`** for arrays larger than RAM; understand **flush** and **write** semantics. diff --git a/.cursor/skills/numpy-scientific/references/reference-ufuncs-reductions.md b/.cursor/skills/numpy-scientific/references/reference-ufuncs-reductions.md new file mode 100644 index 00000000..33efffbf --- /dev/null +++ b/.cursor/skills/numpy-scientific/references/reference-ufuncs-reductions.md @@ -0,0 +1,37 @@ +# Ufuncs, reductions, `where`, NaNs + +## Ufuncs + +- **Unary / binary** elementwise: `np.sin`, `np.exp`, `np.maximum`, `np.logical_and`, etc. +- **`out=`** avoids allocations when you reuse buffers (advanced; ensure **no overlap** unless ufunc supports it). +- **Casting rules** follow ufunc **signature**; mixed dtypes **promote** per NumPy rules—verify for **int** vs **float** mixes. + +Official: [Universal functions](https://numpy.org/doc/stable/reference/ufuncs.html). + +## Reductions + +- **`axis=None`**: whole array; **`axis=tuple`**: reduce multiple axes at once. +- **`keepdims=True`**: preserves reduced axes as length 1 for **broadcasting** with the original shape. +- **`ddof`** for variance/std (Bessel correction). + +## NaNs + +- **`np.nanmean`**, **`np.nansum`**, **`np.nanstd`**, etc. **ignore** NaNs; plain **`mean`** propagates NaN. +- Sorting: **`np.nanargmin`** / **`nanargmax`** or **`np.ma`** when masks are first-class. + +## `np.where` + +- **`np.where(cond, x, y)`** selects elementwise; **`np.where(cond)`** returns **indices** (tuple of 1D arrays). +- For **assignment**, **`np.where`** on the RHS is not the same as masked write; use **`np.putmask`** or boolean slice assignment when appropriate. + +## Sorting and searching + +- **`np.sort`**, **`argsort`**, **`partition`**, **`searchsorted`** for monotonic bins (histograms, digitize). + +## Reproducibility note + +- **`sum`** on **float** may reorder for SIMD; bitwise-identical sums across machines are **not** guaranteed unless you control order (e.g. **`math.fsum`** on Python iterables, or **`np.add.reduce`** with fixed association in specialized cases). For science reporting, document **stability** expectations; **`python-reviewer`** flags when this matters. + +## `einsum`-light + +- Many **axis reductions** are clearer as **`einsum`**; see [reference-linalg-einsum.md](reference-linalg-einsum.md). diff --git a/.cursor/skills/pandas-pro/SKILL.md b/.cursor/skills/pandas-pro/SKILL.md deleted file mode 100644 index 72791106..00000000 --- a/.cursor/skills/pandas-pro/SKILL.md +++ /dev/null @@ -1,96 +0,0 @@ ---- -name: pandas-pro -description: Use when working with pandas DataFrames, data cleaning, aggregation, merging, or time series analysis. Invoke for data manipulation, missing value handling, groupby operations, or performance optimization. -triggers: - - pandas - - DataFrame - - data manipulation - - data cleaning - - aggregation - - groupby - - merge - - join - - time series - - data wrangling - - pivot table - - data transformation -role: expert -scope: implementation -output-format: code ---- - -# Pandas Pro - -Expert pandas developer specializing in efficient data manipulation, analysis, and transformation workflows with production-grade performance patterns. - -## Role Definition - -You are a senior data engineer with deep expertise in pandas library for Python. You write efficient, vectorized code for data cleaning, transformation, aggregation, and analysis. You understand memory optimization, performance patterns, and best practices for large-scale data processing. - -## When to Use This Skill - -- Loading, cleaning, and transforming tabular data -- Handling missing values and data quality issues -- Performing groupby aggregations and pivot operations -- Merging, joining, and concatenating datasets -- Time series analysis and resampling -- Optimizing pandas code for memory and performance -- Converting between data formats (CSV, Excel, SQL, JSON) - -## Core Workflow - -1. **Assess data structure** - Examine dtypes, memory usage, missing values, data quality -2. **Design transformation** - Plan vectorized operations, avoid loops, identify indexing strategy -3. **Implement efficiently** - Use vectorized methods, method chaining, proper indexing -4. **Validate results** - Check dtypes, shapes, edge cases, null handling -5. **Optimize** - Profile memory usage, apply categorical types, use chunking if needed - -## Reference Guide - -Load detailed guidance based on context: - -| Topic | Reference | Load When | -|-------|-----------|-----------| -| DataFrame Operations | `references/dataframe-operations.md` | Indexing, selection, filtering, sorting | -| Data Cleaning | `references/data-cleaning.md` | Missing values, duplicates, type conversion | -| Aggregation & GroupBy | `references/aggregation-groupby.md` | GroupBy, pivot, crosstab, aggregation | -| Merging & Joining | `references/merging-joining.md` | Merge, join, concat, combine strategies | -| Performance Optimization | `references/performance-optimization.md` | Memory usage, vectorization, chunking | - -## Constraints - -### MUST DO -- Use vectorized operations instead of loops -- Set appropriate dtypes (categorical for low-cardinality strings) -- Check memory usage with `.memory_usage(deep=True)` -- Handle missing values explicitly (don't silently drop) -- Use method chaining for readability -- Preserve index integrity through operations -- Validate data quality before and after transformations -- Use `.copy()` when modifying subsets to avoid SettingWithCopyWarning - -### MUST NOT DO -- Iterate over DataFrame rows with `.iterrows()` unless absolutely necessary -- Use chained indexing (`df['A']['B']`) - use `.loc[]` or `.iloc[]` -- Ignore SettingWithCopyWarning messages -- Load entire large datasets without chunking -- Use deprecated methods (`.ix`, `.append()` - use `pd.concat()`) -- Convert to Python lists for operations possible in pandas -- Assume data is clean without validation - -## Output Templates - -When implementing pandas solutions, provide: -1. Code with vectorized operations and proper indexing -2. Comments explaining complex transformations -3. Memory/performance considerations if dataset is large -4. Data validation checks (dtypes, nulls, shapes) - -## Knowledge Reference - -pandas 2.0+, NumPy, datetime handling, categorical types, MultiIndex, memory optimization, vectorization, method chaining, merge strategies, time series resampling, pivot tables, groupby aggregations - -## Related Skills - -- **Python Pro** - Type hints, testing, Python best practices -- **Data Scientist** - Statistical analysis, visualization, ML workflows diff --git a/.cursor/skills/pandas-pro/references/aggregation-groupby.md b/.cursor/skills/pandas-pro/references/aggregation-groupby.md deleted file mode 100644 index 7f2d7163..00000000 --- a/.cursor/skills/pandas-pro/references/aggregation-groupby.md +++ /dev/null @@ -1,548 +0,0 @@ -# Aggregation and GroupBy - -> Reference for: Pandas Pro -> Load when: GroupBy operations, pivot tables, crosstab, aggregation functions, or summarizing data - ---- - -## Overview - -Aggregation transforms data from individual records to summary statistics. This reference covers GroupBy, pivot tables, crosstab, and advanced aggregation patterns with pandas 2.0+. - ---- - -## GroupBy Fundamentals - -### Basic GroupBy - -```python -import pandas as pd -import numpy as np - -df = pd.DataFrame({ - 'department': ['Eng', 'Eng', 'Sales', 'Sales', 'Eng', 'HR'], - 'team': ['Backend', 'Frontend', 'East', 'West', 'Backend', 'Recruit'], - 'employee': ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve', 'Frank'], - 'salary': [80000, 75000, 65000, 70000, 85000, 60000], - 'years': [5, 3, 7, 4, 6, 2] -}) - -# Single column groupby with single aggregation -avg_salary = df.groupby('department')['salary'].mean() - -# Multiple aggregations -stats = df.groupby('department')['salary'].agg(['mean', 'min', 'max', 'count']) - -# GroupBy multiple columns -grouped = df.groupby(['department', 'team'])['salary'].mean() - -# Reset index to get DataFrame instead of Series -grouped = df.groupby('department')['salary'].mean().reset_index() -``` - -### Multiple Columns, Multiple Aggregations - -```python -# Named aggregation (pandas 2.0+ preferred) -result = df.groupby('department').agg( - avg_salary=('salary', 'mean'), - max_salary=('salary', 'max'), - total_years=('years', 'sum'), - headcount=('employee', 'count'), -) - -# Dictionary syntax (traditional) -result = df.groupby('department').agg({ - 'salary': ['mean', 'max', 'std'], - 'years': ['sum', 'mean'], -}) - -# Flatten multi-level column names -result.columns = ['_'.join(col).strip() for col in result.columns.values] -``` - -### Custom Aggregation Functions - -```python -# Lambda functions -result = df.groupby('department').agg({ - 'salary': lambda x: x.max() - x.min(), # Range - 'years': lambda x: x.quantile(0.75), # 75th percentile -}) - -# Named functions for clarity -def salary_range(x): - return x.max() - x.min() - -def coefficient_of_variation(x): - return x.std() / x.mean() if x.mean() != 0 else 0 - -result = df.groupby('department').agg( - salary_range=('salary', salary_range), - salary_cv=('salary', coefficient_of_variation), -) - -# Multiple custom functions -result = df.groupby('department')['salary'].agg([ - ('range', lambda x: x.max() - x.min()), - ('iqr', lambda x: x.quantile(0.75) - x.quantile(0.25)), - ('median', 'median'), -]) -``` - ---- - -## Transform and Apply - -### Transform - Returns Same Shape - -```python -# Transform returns Series with same index as original -# Useful for adding aggregated values back to original DataFrame - -# Add group mean as new column -df['dept_avg_salary'] = df.groupby('department')['salary'].transform('mean') - -# Normalize within group -df['salary_zscore'] = df.groupby('department')['salary'].transform( - lambda x: (x - x.mean()) / x.std() -) - -# Rank within group -df['salary_rank'] = df.groupby('department')['salary'].transform('rank', ascending=False) - -# Percentage of group total -df['salary_pct'] = df.groupby('department')['salary'].transform( - lambda x: x / x.sum() * 100 -) - -# Fill missing with group mean -df['salary'] = df.groupby('department')['salary'].transform( - lambda x: x.fillna(x.mean()) -) -``` - -### Apply - Flexible Operations - -```python -# Apply runs function on each group DataFrame -def top_n_by_salary(group, n=2): - return group.nlargest(n, 'salary') - -top_earners = df.groupby('department').apply(top_n_by_salary, n=2) - -# Reset index after apply -top_earners = df.groupby('department', group_keys=False).apply( - top_n_by_salary, n=2 -).reset_index(drop=True) - -# Complex group operations -def group_summary(group): - return pd.Series({ - 'headcount': len(group), - 'avg_salary': group['salary'].mean(), - 'top_earner': group.loc[group['salary'].idxmax(), 'employee'], - 'avg_tenure': group['years'].mean(), - }) - -summary = df.groupby('department').apply(group_summary) -``` - -### Filter - Keep/Remove Groups - -```python -# Keep only groups meeting a condition -# Groups with average salary > 70000 -filtered = df.groupby('department').filter(lambda x: x['salary'].mean() > 70000) - -# Groups with more than 2 members -filtered = df.groupby('department').filter(lambda x: len(x) > 2) - -# Combined conditions -filtered = df.groupby('department').filter( - lambda x: (len(x) >= 2) and (x['salary'].mean() > 65000) -) -``` - ---- - -## Pivot Tables - -### Basic Pivot Table - -```python -df = pd.DataFrame({ - 'date': pd.date_range('2024-01-01', periods=6), - 'product': ['A', 'B', 'A', 'B', 'A', 'B'], - 'region': ['East', 'East', 'West', 'West', 'East', 'West'], - 'sales': [100, 150, 120, 180, 90, 200], - 'quantity': [10, 15, 12, 18, 9, 20], -}) - -# Simple pivot -pivot = df.pivot_table( - values='sales', - index='product', - columns='region', - aggfunc='sum' -) - -# Multiple values -pivot = df.pivot_table( - values=['sales', 'quantity'], - index='product', - columns='region', - aggfunc='sum' -) - -# Multiple aggregation functions -pivot = df.pivot_table( - values='sales', - index='product', - columns='region', - aggfunc=['sum', 'mean', 'count'] -) -``` - -### Advanced Pivot Table Options - -```python -# Fill missing values -pivot = df.pivot_table( - values='sales', - index='product', - columns='region', - aggfunc='sum', - fill_value=0 -) - -# Add margins (totals) -pivot = df.pivot_table( - values='sales', - index='product', - columns='region', - aggfunc='sum', - margins=True, - margins_name='Total' -) - -# Multiple index levels -pivot = df.pivot_table( - values='sales', - index=['product', df['date'].dt.month], - columns='region', - aggfunc='sum' -) - -# Observed categories only (for categorical data) -pivot = df.pivot_table( - values='sales', - index='product', - columns='region', - aggfunc='sum', - observed=True # pandas 2.0+ default changed -) -``` - -### Unpivoting (Melt) - -```python -# Wide to long format -wide_df = pd.DataFrame({ - 'product': ['A', 'B'], - 'Q1_sales': [100, 150], - 'Q2_sales': [120, 180], - 'Q3_sales': [90, 200], -}) - -# Melt to long format -long_df = pd.melt( - wide_df, - id_vars=['product'], - value_vars=['Q1_sales', 'Q2_sales', 'Q3_sales'], - var_name='quarter', - value_name='sales' -) - -# Clean quarter column -long_df['quarter'] = long_df['quarter'].str.replace('_sales', '') -``` - ---- - -## Crosstab - -### Basic Crosstab - -```python -df = pd.DataFrame({ - 'gender': ['M', 'F', 'M', 'F', 'M', 'F', 'M', 'M'], - 'department': ['Eng', 'Eng', 'Sales', 'Sales', 'Eng', 'HR', 'HR', 'Eng'], - 'level': ['Senior', 'Junior', 'Senior', 'Senior', 'Junior', 'Junior', 'Senior', 'Junior'], -}) - -# Simple crosstab (counts) -ct = pd.crosstab(df['gender'], df['department']) - -# Normalized crosstab -ct_pct = pd.crosstab(df['gender'], df['department'], normalize='all') # Total -ct_pct = pd.crosstab(df['gender'], df['department'], normalize='index') # Row -ct_pct = pd.crosstab(df['gender'], df['department'], normalize='columns') # Column - -# With margins -ct = pd.crosstab(df['gender'], df['department'], margins=True) - -# Multiple levels -ct = pd.crosstab( - [df['gender'], df['level']], - df['department'] -) -``` - -### Crosstab with Aggregation - -```python -df['salary'] = [80000, 75000, 65000, 70000, 85000, 60000, 72000, 78000] - -# Crosstab with values and aggregation -ct = pd.crosstab( - df['gender'], - df['department'], - values=df['salary'], - aggfunc='mean' -) - -# Multiple aggregations -ct = pd.crosstab( - df['gender'], - df['department'], - values=df['salary'], - aggfunc=['mean', 'sum', 'count'] -) -``` - ---- - -## Window Functions with GroupBy - -### Rolling Aggregations - -```python -df = pd.DataFrame({ - 'date': pd.date_range('2024-01-01', periods=10), - 'product': ['A', 'B'] * 5, - 'sales': [100, 150, 110, 160, 120, 170, 130, 180, 140, 190], -}) - -# Rolling mean within groups -df['rolling_avg'] = df.groupby('product')['sales'].transform( - lambda x: x.rolling(window=3, min_periods=1).mean() -) - -# Expanding aggregations -df['cumulative_sales'] = df.groupby('product')['sales'].transform('cumsum') - -df['expanding_avg'] = df.groupby('product')['sales'].transform( - lambda x: x.expanding().mean() -) - -# Rank within groups -df['sales_rank'] = df.groupby('product')['sales'].rank(method='dense') -``` - -### Shift and Diff - -```python -# Previous value within group -df['prev_sales'] = df.groupby('product')['sales'].shift(1) - -# Next value -df['next_sales'] = df.groupby('product')['sales'].shift(-1) - -# Period-over-period change -df['sales_change'] = df.groupby('product')['sales'].diff() - -# Percentage change -df['sales_pct_change'] = df.groupby('product')['sales'].pct_change() -``` - ---- - -## Common Aggregation Patterns - -### Summary Statistics - -```python -# Comprehensive summary by group -def full_summary(group): - return pd.Series({ - 'count': len(group), - 'mean': group['salary'].mean(), - 'std': group['salary'].std(), - 'min': group['salary'].min(), - 'q25': group['salary'].quantile(0.25), - 'median': group['salary'].median(), - 'q75': group['salary'].quantile(0.75), - 'max': group['salary'].max(), - 'sum': group['salary'].sum(), - }) - -summary = df.groupby('department').apply(full_summary) -``` - -### Top N Per Group - -```python -# Top 2 salaries per department -top_2 = df.groupby('department', group_keys=False).apply( - lambda x: x.nlargest(2, 'salary') -) - -# Using head after sorting -top_2 = df.sort_values('salary', ascending=False).groupby( - 'department', group_keys=False -).head(2) - -# Bottom N -bottom_2 = df.groupby('department', group_keys=False).apply( - lambda x: x.nsmallest(2, 'salary') -) -``` - -### First/Last Per Group - -```python -# First row per group -first = df.groupby('department').first() - -# Last row per group -last = df.groupby('department').last() - -# First row after sorting -first_by_salary = df.sort_values('salary', ascending=False).groupby( - 'department' -).first() - -# Nth row -nth = df.groupby('department').nth(1) # Second row (0-indexed) -``` - -### Cumulative Operations - -```python -# Cumulative sum -df['cum_sales'] = df.groupby('department')['salary'].cumsum() - -# Cumulative max/min -df['cum_max'] = df.groupby('department')['salary'].cummax() -df['cum_min'] = df.groupby('department')['salary'].cummin() - -# Cumulative count -df['cum_count'] = df.groupby('department').cumcount() + 1 - -# Running percentage of total -df['running_pct'] = df.groupby('department')['salary'].transform( - lambda x: x.cumsum() / x.sum() * 100 -) -``` - ---- - -## Performance Tips for GroupBy - -### Efficient GroupBy Operations - -```python -# Pre-sort for faster groupby operations -df = df.sort_values('department') -grouped = df.groupby('department', sort=False) # Already sorted - -# Use observed=True for categorical columns (pandas 2.0+ default) -df['department'] = df['department'].astype('category') -grouped = df.groupby('department', observed=True)['salary'].mean() - -# Avoid apply when possible - use built-in aggregations -# SLOWER: -result = df.groupby('department')['salary'].apply(lambda x: x.sum()) -# FASTER: -result = df.groupby('department')['salary'].sum() - -# Use numba for custom aggregations (if available) -@numba.jit(nopython=True) -def custom_agg(values): - return values.sum() / len(values) -``` - -### Memory-Efficient Aggregation - -```python -# For large DataFrames, compute aggregations separately -groups = df.groupby('department') - -means = groups['salary'].mean() -sums = groups['salary'].sum() -counts = groups.size() - -result = pd.DataFrame({ - 'mean': means, - 'sum': sums, - 'count': counts -}) - -# Avoid creating intermediate large DataFrames -# BAD: Creates full transformed DataFrame -df['z_score'] = (df['salary'] - df.groupby('department')['salary'].transform('mean')) / df.groupby('department')['salary'].transform('std') - -# BETTER: Compute once -group_stats = df.groupby('department')['salary'].agg(['mean', 'std']) -df = df.merge(group_stats, on='department') -df['z_score'] = (df['salary'] - df['mean']) / df['std'] -``` - ---- - -## Best Practices Summary - -1. **Use named aggregation** - Clearer than dictionary syntax -2. **Choose transform vs apply wisely** - Transform for same-shape, apply for flexible -3. **Pre-sort for performance** - Use `sort=False` after sorting -4. **Prefer built-in aggregations** - Faster than lambda/apply -5. **Use observed=True** - Especially for categorical data -6. **Reset index when needed** - Keep DataFrames easier to work with -7. **Validate group counts** - Check for unexpected groups - ---- - -## Anti-Patterns to Avoid - -```python -# BAD: Iterating over groups manually -for name, group in df.groupby('department'): - # process group - pass - -# GOOD: Use vectorized operations -df.groupby('department').agg(...) - -# BAD: Multiple groupby calls -df.groupby('dept')['salary'].mean() -df.groupby('dept')['salary'].sum() -df.groupby('dept')['salary'].count() - -# GOOD: Single groupby, multiple aggs -df.groupby('dept')['salary'].agg(['mean', 'sum', 'count']) - -# BAD: Apply for simple aggregations -df.groupby('dept')['salary'].apply(np.mean) - -# GOOD: Built-in method -df.groupby('dept')['salary'].mean() -``` - ---- - -## Related References - -- `dataframe-operations.md` - Filtering before aggregation -- `merging-joining.md` - Join aggregated results back -- `performance-optimization.md` - Optimize large-scale aggregations diff --git a/.cursor/skills/pandas-pro/references/data-cleaning.md b/.cursor/skills/pandas-pro/references/data-cleaning.md deleted file mode 100644 index ede32c2d..00000000 --- a/.cursor/skills/pandas-pro/references/data-cleaning.md +++ /dev/null @@ -1,503 +0,0 @@ -# Data Cleaning - -> Reference for: Pandas Pro -> Load when: Missing values, duplicates, type conversion, data validation, or data quality issues - ---- - -## Overview - -Data cleaning is critical for reliable analysis. This reference covers handling missing values, duplicates, type conversion, and data validation with pandas 2.0+ patterns. - ---- - -## Missing Values - -### Detecting Missing Values - -```python -import pandas as pd -import numpy as np - -df = pd.DataFrame({ - 'name': ['Alice', 'Bob', None, 'Diana'], - 'age': [25, np.nan, 35, 28], - 'salary': [50000, 60000, np.nan, np.nan], - 'department': ['Eng', '', 'Eng', 'Sales'] -}) - -# Check for any missing values -df.isna().any() # Per column -df.isna().any().any() # Entire DataFrame - -# Count missing values -df.isna().sum() # Per column -df.isna().sum().sum() # Total - -# Percentage of missing values -(df.isna().sum() / len(df) * 100).round(2) - -# Rows with any missing values -df[df.isna().any(axis=1)] - -# Rows with all values present -df[df.notna().all(axis=1)] - -# Missing value heatmap info -missing_info = pd.DataFrame({ - 'missing': df.isna().sum(), - 'percent': (df.isna().sum() / len(df) * 100).round(2), - 'dtype': df.dtypes -}) -``` - -### Handling Missing Values - Dropping - -```python -# Drop rows with any missing value -df_clean = df.dropna() - -# Drop rows where specific columns have missing values -df_clean = df.dropna(subset=['name', 'age']) - -# Drop rows where ALL values are missing -df_clean = df.dropna(how='all') - -# Drop rows with minimum non-null values -df_clean = df.dropna(thresh=3) # Keep rows with at least 3 non-null - -# Drop columns with missing values -df_clean = df.dropna(axis=1) - -# Drop columns with more than 50% missing -threshold = len(df) * 0.5 -df_clean = df.dropna(axis=1, thresh=threshold) -``` - -### Handling Missing Values - Filling - -```python -# Fill with constant value -df['age'] = df['age'].fillna(0) - -# Fill with column mean/median/mode -df['age'] = df['age'].fillna(df['age'].mean()) -df['salary'] = df['salary'].fillna(df['salary'].median()) -df['department'] = df['department'].fillna(df['department'].mode()[0]) - -# Forward fill (use previous value) -df['salary'] = df['salary'].ffill() - -# Backward fill (use next value) -df['salary'] = df['salary'].bfill() - -# Fill with different values per column -fill_values = {'age': 0, 'salary': df['salary'].median(), 'name': 'Unknown'} -df = df.fillna(fill_values) - -# Fill with interpolation (numeric data) -df['salary'] = df['salary'].interpolate(method='linear') - -# Group-specific fill (fill with group mean) -df['salary'] = df.groupby('department')['salary'].transform( - lambda x: x.fillna(x.mean()) -) -``` - -### Handling Empty Strings vs NaN - -```python -# Empty strings are NOT detected as NaN -df['department'].isna().sum() # Won't count '' - -# Replace empty strings with NaN -df['department'] = df['department'].replace('', np.nan) -# Or -df['department'] = df['department'].replace(r'^\s*$', np.nan, regex=True) - -# Replace multiple values with NaN -df = df.replace(['', 'N/A', 'null', 'None', '-'], np.nan) - -# Using na_values when reading files -df = pd.read_csv('file.csv', na_values=['', 'N/A', 'null', 'None', '-']) -``` - ---- - -## Handling Duplicates - -### Detecting Duplicates - -```python -df = pd.DataFrame({ - 'id': [1, 2, 2, 3, 4, 4], - 'name': ['Alice', 'Bob', 'Bob', 'Charlie', 'Diana', 'Diana'], - 'email': ['a@x.com', 'b@x.com', 'b@x.com', 'c@x.com', 'd@x.com', 'd2@x.com'] -}) - -# Check for duplicate rows (all columns) -df.duplicated().sum() - -# Check specific columns -df.duplicated(subset=['id']).sum() -df.duplicated(subset=['name', 'email']).sum() - -# View duplicate rows -df[df.duplicated(keep=False)] # All duplicates -df[df.duplicated(keep='first')] # Duplicates except first occurrence -df[df.duplicated(keep='last')] # Duplicates except last occurrence - -# Count duplicates per key -df.groupby('id').size().loc[lambda x: x > 1] -``` - -### Removing Duplicates - -```python -# Remove duplicate rows (keep first) -df_clean = df.drop_duplicates() - -# Keep last occurrence -df_clean = df.drop_duplicates(keep='last') - -# Remove all duplicates (keep none) -df_clean = df.drop_duplicates(keep=False) - -# Based on specific columns -df_clean = df.drop_duplicates(subset=['id']) -df_clean = df.drop_duplicates(subset=['name', 'email'], keep='last') - -# In-place modification -df.drop_duplicates(inplace=True) -``` - -### Handling Duplicates with Aggregation - -```python -# Instead of dropping, aggregate duplicates -df_agg = df.groupby('id').agg({ - 'name': 'first', - 'email': lambda x: ', '.join(x.unique()) -}).reset_index() - -# Keep row with max/min value -df_best = df.loc[df.groupby('id')['score'].idxmax()] - -# Rank duplicates -df['rank'] = df.groupby('id').cumcount() + 1 -``` - ---- - -## Type Conversion - -### Checking and Converting Types - -```python -# Check current types -df.dtypes -df.info() - -# Convert to specific type -df['age'] = df['age'].astype(int) -df['salary'] = df['salary'].astype(float) -df['name'] = df['name'].astype(str) - -# Safe conversion with errors handling -df['age'] = pd.to_numeric(df['age'], errors='coerce') # Invalid -> NaN -df['age'] = pd.to_numeric(df['age'], errors='ignore') # Keep original if invalid - -# Convert multiple columns -df = df.astype({'age': 'int64', 'salary': 'float64'}) - -# Convert object to string (pandas 2.0+ StringDtype) -df['name'] = df['name'].astype('string') # Nullable string type -``` - -### Datetime Conversion - -```python -df = pd.DataFrame({ - 'date_str': ['2024-01-15', '2024-02-20', 'invalid', '2024-03-10'], - 'timestamp': [1705276800, 1708387200, 1710028800, 1710028800] -}) - -# String to datetime -df['date'] = pd.to_datetime(df['date_str'], errors='coerce') - -# Specify format for faster parsing -df['date'] = pd.to_datetime(df['date_str'], format='%Y-%m-%d', errors='coerce') - -# Unix timestamp to datetime -df['datetime'] = pd.to_datetime(df['timestamp'], unit='s') - -# Extract components -df['year'] = df['date'].dt.year -df['month'] = df['date'].dt.month -df['day_of_week'] = df['date'].dt.day_name() - -# Handle mixed formats -df['date'] = pd.to_datetime(df['date_str'], format='mixed', dayfirst=False) -``` - -### Categorical Conversion - -```python -# Convert to categorical (memory efficient for low cardinality) -df['department'] = df['department'].astype('category') - -# Ordered categorical -df['size'] = pd.Categorical( - df['size'], - categories=['Small', 'Medium', 'Large'], - ordered=True -) - -# Check memory savings -print(f"Object: {df['department'].nbytes}") -df['department'] = df['department'].astype('category') -print(f"Category: {df['department'].nbytes}") -``` - -### Nullable Integer Types (pandas 2.0+) - -```python -# Standard int doesn't support NaN -# Use nullable integer types -df['age'] = df['age'].astype('Int64') # Note capital I - -# All nullable types -df = df.astype({ - 'count': 'Int64', # Nullable integer - 'price': 'Float64', # Nullable float - 'flag': 'boolean', # Nullable boolean - 'name': 'string', # Nullable string -}) - -# Convert with NA handling -df['age'] = pd.array([1, 2, None, 4], dtype='Int64') -``` - ---- - -## String Cleaning - -### Common String Operations - -```python -df = pd.DataFrame({ - 'name': [' Alice ', 'BOB', 'charlie', None, 'Diana Smith'], - 'email': ['ALICE@EXAMPLE.COM', 'bob@test', 'invalid', None, 'diana@example.com'] -}) - -# Strip whitespace -df['name'] = df['name'].str.strip() - -# Case normalization -df['name'] = df['name'].str.lower() -df['name'] = df['name'].str.upper() -df['name'] = df['name'].str.title() # Title Case - -# Replace patterns -df['name'] = df['name'].str.replace(r'\s+', ' ', regex=True) # Multiple spaces to one -df['phone'] = df['phone'].str.replace(r'[^0-9]', '', regex=True) # Keep only digits - -# Extract with regex -df['domain'] = df['email'].str.extract(r'@(.+)$') -df['first_name'] = df['name'].str.extract(r'^(\w+)') - -# Split strings -df[['first', 'last']] = df['name'].str.split(' ', n=1, expand=True) -``` - -### String Validation - -```python -# Check patterns -df['valid_email'] = df['email'].str.match(r'^[\w.]+@[\w.]+\.\w+$', na=False) - -# String length -df['name_length'] = df['name'].str.len() -df['valid_length'] = df['name'].str.len().between(2, 50) - -# Contains check -df['has_domain'] = df['email'].str.contains('@', na=False) -``` - ---- - -## Data Validation - -### Validation Functions - -```python -def validate_dataframe(df: pd.DataFrame) -> dict: - """Comprehensive DataFrame validation.""" - report = { - 'rows': len(df), - 'columns': len(df.columns), - 'duplicates': df.duplicated().sum(), - 'missing_by_column': df.isna().sum().to_dict(), - 'dtypes': df.dtypes.astype(str).to_dict(), - } - return report - -# Range validation -def validate_range(series: pd.Series, min_val, max_val) -> pd.Series: - """Return boolean mask for values in range.""" - return series.between(min_val, max_val) - -df['valid_age'] = validate_range(df['age'], 0, 120) - -# Custom validation -def validate_email(series: pd.Series) -> pd.Series: - """Validate email format.""" - pattern = r'^[\w.+-]+@[\w-]+\.[\w.-]+$' - return series.str.match(pattern, na=False) - -df['valid_email'] = validate_email(df['email']) -``` - -### Schema Validation with pandera - -```python -# Using pandera for schema validation (recommended for production) -import pandera as pa -from pandera import Column, Check - -schema = pa.DataFrameSchema({ - 'name': Column(str, Check.str_length(min_value=1, max_value=100)), - 'age': Column(int, Check.in_range(0, 120)), - 'email': Column(str, Check.str_matches(r'^[\w.+-]+@[\w-]+\.[\w.-]+$')), - 'salary': Column(float, Check.greater_than(0), nullable=True), -}) - -# Validate DataFrame -try: - schema.validate(df) -except pa.errors.SchemaError as e: - print(f"Validation failed: {e}") -``` - ---- - -## Data Cleaning Pipeline - -### Method Chaining Pattern - -```python -def clean_dataframe(df: pd.DataFrame) -> pd.DataFrame: - """Complete data cleaning pipeline using method chaining.""" - return ( - df - # Make a copy - .copy() - # Standardize column names - .rename(columns=lambda x: x.lower().strip().replace(' ', '_')) - # Drop fully empty rows - .dropna(how='all') - # Clean string columns - .assign( - name=lambda x: x['name'].str.strip().str.title(), - email=lambda x: x['email'].str.lower().str.strip(), - ) - # Handle missing values - .fillna({'department': 'Unknown'}) - # Convert types - .astype({'age': 'Int64', 'department': 'category'}) - # Remove duplicates - .drop_duplicates(subset=['email']) - # Reset index - .reset_index(drop=True) - ) - -df_clean = clean_dataframe(df) -``` - -### Pipeline with Validation - -```python -def clean_and_validate( - df: pd.DataFrame, - required_columns: list[str], - unique_columns: list[str] | None = None, -) -> tuple[pd.DataFrame, dict]: - """Clean DataFrame and return validation report.""" - - # Validate required columns exist - missing_cols = set(required_columns) - set(df.columns) - if missing_cols: - raise ValueError(f"Missing required columns: {missing_cols}") - - # Track cleaning stats - stats = { - 'initial_rows': len(df), - 'dropped_empty': 0, - 'dropped_duplicates': 0, - 'filled_missing': {}, - } - - # Clean - df = df.copy() - - # Drop empty rows - before = len(df) - df = df.dropna(how='all') - stats['dropped_empty'] = before - len(df) - - # Handle duplicates - if unique_columns: - before = len(df) - df = df.drop_duplicates(subset=unique_columns) - stats['dropped_duplicates'] = before - len(df) - - stats['final_rows'] = len(df) - - return df, stats -``` - ---- - -## Best Practices Summary - -1. **Always check data quality first** - Use `.info()`, `.describe()`, and missing value analysis -2. **Document cleaning decisions** - Track what was dropped/filled and why -3. **Use nullable types** - `Int64`, `string`, `boolean` for proper NA handling -4. **Validate after cleaning** - Ensure data meets expectations -5. **Use method chaining** - Readable, maintainable cleaning pipelines -6. **Copy before modifying** - Avoid SettingWithCopyWarning -7. **Handle edge cases** - Empty strings, whitespace, invalid formats - ---- - -## Anti-Patterns to Avoid - -```python -# BAD: Dropping NaN without understanding impact -df = df.dropna() # May lose significant data - -# GOOD: Investigate first, then decide -print(f"Missing values: {df.isna().sum()}") -print(f"Rows affected: {df.isna().any(axis=1).sum()}") -# Then make informed decision - -# BAD: Filling without domain knowledge -df['age'] = df['age'].fillna(0) # Age 0 is not valid - -# GOOD: Use appropriate fill strategy -df['age'] = df['age'].fillna(df['age'].median()) - -# BAD: Type conversion without error handling -df['id'] = df['id'].astype(int) # Will fail on NaN or invalid - -# GOOD: Safe conversion -df['id'] = pd.to_numeric(df['id'], errors='coerce').astype('Int64') -``` - ---- - -## Related References - -- `dataframe-operations.md` - Selection and filtering for targeted cleaning -- `aggregation-groupby.md` - Aggregate duplicates instead of dropping -- `performance-optimization.md` - Efficient cleaning of large datasets diff --git a/.cursor/skills/pandas-pro/references/dataframe-operations.md b/.cursor/skills/pandas-pro/references/dataframe-operations.md deleted file mode 100644 index 6321f23e..00000000 --- a/.cursor/skills/pandas-pro/references/dataframe-operations.md +++ /dev/null @@ -1,423 +0,0 @@ -# DataFrame Operations - -> Reference for: Pandas Pro -> Load when: Indexing, selection, filtering, sorting, or basic DataFrame manipulation - ---- - -## Overview - -DataFrame operations form the foundation of pandas work. This reference covers indexing, selection, filtering, and sorting with pandas 2.0+ best practices. - ---- - -## Indexing and Selection - -### Label-Based Selection with `.loc[]` - -Use `.loc[]` for label-based indexing. Always preferred over chained indexing. - -```python -import pandas as pd -import numpy as np - -# Sample DataFrame -df = pd.DataFrame({ - 'name': ['Alice', 'Bob', 'Charlie', 'Diana'], - 'age': [25, 30, 35, 28], - 'salary': [50000, 60000, 70000, 55000], - 'department': ['Engineering', 'Sales', 'Engineering', 'Marketing'] -}, index=['a', 'b', 'c', 'd']) - -# Single value -value = df.loc['a', 'name'] # 'Alice' - -# Single row (returns Series) -row = df.loc['a'] - -# Multiple rows -rows = df.loc[['a', 'c']] - -# Row and column slices (inclusive on both ends) -subset = df.loc['a':'c', 'name':'salary'] - -# Boolean indexing with .loc -adults = df.loc[df['age'] >= 30] - -# Boolean indexing with column selection -adults_names = df.loc[df['age'] >= 30, 'name'] - -# Multiple conditions -engineering_seniors = df.loc[ - (df['department'] == 'Engineering') & (df['age'] >= 30), - ['name', 'salary'] -] -``` - -### Position-Based Selection with `.iloc[]` - -Use `.iloc[]` for integer position-based indexing. - -```python -# Single value by position -value = df.iloc[0, 0] # First row, first column - -# Single row by position -first_row = df.iloc[0] - -# Slice rows (exclusive end, like Python) -first_three = df.iloc[:3] - -# Specific rows and columns by position -subset = df.iloc[[0, 2], [0, 2]] # Rows 0,2 and columns 0,2 - -# Range selection -block = df.iloc[1:3, 0:2] # Rows 1-2, columns 0-1 -``` - -### When to Use `.loc[]` vs `.iloc[]` - -| Scenario | Use | Example | -|----------|-----|---------| -| Known column names | `.loc[]` | `df.loc[:, 'name']` | -| Filter by condition | `.loc[]` | `df.loc[df['age'] > 25]` | -| First/last N rows | `.iloc[]` | `df.iloc[:5]` or `df.iloc[-5:]` | -| Specific row positions | `.iloc[]` | `df.iloc[[0, 5, 10]]` | -| Unknown column order | `.iloc[]` | `df.iloc[:, 0]` | - ---- - -## Filtering DataFrames - -### Boolean Masks - -```python -# Single condition -mask = df['age'] > 25 -filtered = df[mask] - -# Multiple conditions (use parentheses!) -mask = (df['age'] > 25) & (df['salary'] < 65000) -filtered = df[mask] - -# OR conditions -mask = (df['department'] == 'Engineering') | (df['department'] == 'Sales') -filtered = df[mask] - -# NOT condition -mask = ~(df['department'] == 'Marketing') -filtered = df[mask] -``` - -### Using `.query()` for Readable Filters - -```python -# Simple query - more readable for complex conditions -result = df.query('age > 25 and salary < 65000') - -# Using variables with @ -min_age = 25 -result = df.query('age > @min_age') - -# String comparisons -result = df.query('department == "Engineering"') - -# In-list filtering -depts = ['Engineering', 'Sales'] -result = df.query('department in @depts') - -# Complex expressions -result = df.query('(age > 25) and (department != "Marketing")') -``` - -### Using `.isin()` for Multiple Values - -```python -# Filter by multiple values -departments = ['Engineering', 'Sales'] -filtered = df[df['department'].isin(departments)] - -# Negation -filtered = df[~df['department'].isin(departments)] - -# Multiple columns -conditions = { - 'department': ['Engineering', 'Sales'], - 'age': [25, 30, 35] -} -# Filter where department is in list AND age is in list -mask = df['department'].isin(conditions['department']) & df['age'].isin(conditions['age']) -``` - -### String Filtering with `.str` Accessor - -```python -df = pd.DataFrame({ - 'email': ['alice@example.com', 'bob@test.org', 'charlie@example.com'], - 'name': ['Alice Smith', 'Bob Jones', 'Charlie Brown'] -}) - -# Contains -mask = df['email'].str.contains('example') - -# Starts/ends with -mask = df['email'].str.endswith('.com') -mask = df['name'].str.startswith('A') - -# Regex matching -mask = df['email'].str.match(r'^[a-z]+@example\.com$') - -# Case-insensitive -mask = df['name'].str.lower().str.contains('alice') -# Or with case parameter -mask = df['name'].str.contains('alice', case=False) - -# Handle NaN in string columns -mask = df['email'].str.contains('example', na=False) -``` - ---- - -## Sorting - -### Basic Sorting - -```python -# Sort by single column (ascending) -sorted_df = df.sort_values('age') - -# Sort descending -sorted_df = df.sort_values('age', ascending=False) - -# Sort by multiple columns -sorted_df = df.sort_values(['department', 'salary'], ascending=[True, False]) - -# Sort by index -sorted_df = df.sort_index() -sorted_df = df.sort_index(ascending=False) -``` - -### Advanced Sorting - -```python -# Sort with NaN handling -df_with_nan = pd.DataFrame({ - 'name': ['Alice', 'Bob', 'Charlie'], - 'score': [85.0, np.nan, 90.0] -}) - -# NaN at end (default) -sorted_df = df_with_nan.sort_values('score', na_position='last') - -# NaN at beginning -sorted_df = df_with_nan.sort_values('score', na_position='first') - -# Custom sort order using Categorical -order = ['Marketing', 'Sales', 'Engineering'] -df['department'] = pd.Categorical(df['department'], categories=order, ordered=True) -sorted_df = df.sort_values('department') - -# Sort by computed values without adding column -sorted_df = df.iloc[df['name'].str.len().argsort()] -``` - -### In-Place Sorting - -```python -# Modify DataFrame in place -df.sort_values('age', inplace=True) - -# Reset index after sorting -df.sort_values('age', inplace=True) -df.reset_index(drop=True, inplace=True) - -# Or chain -df = df.sort_values('age').reset_index(drop=True) -``` - ---- - -## Column Operations - -### Adding and Modifying Columns - -```python -# Add new column -df['bonus'] = df['salary'] * 0.1 - -# Conditional column with np.where -df['seniority'] = np.where(df['age'] >= 30, 'Senior', 'Junior') - -# Multiple conditions with np.select -conditions = [ - df['age'] < 25, - df['age'] < 35, - df['age'] >= 35 -] -choices = ['Junior', 'Mid', 'Senior'] -df['level'] = np.select(conditions, choices, default='Unknown') - -# Using .assign() for method chaining (returns new DataFrame) -df_new = df.assign( - bonus=lambda x: x['salary'] * 0.1, - total_comp=lambda x: x['salary'] + x['salary'] * 0.1 -) -``` - -### Renaming Columns - -```python -# Rename specific columns -df = df.rename(columns={'name': 'full_name', 'age': 'years'}) - -# Rename all columns with function -df.columns = df.columns.str.lower().str.replace(' ', '_') - -# Using rename with function -df = df.rename(columns=str.upper) -``` - -### Dropping Columns - -```python -# Drop single column -df = df.drop('bonus', axis=1) -# Or -df = df.drop(columns=['bonus']) - -# Drop multiple columns -df = df.drop(columns=['bonus', 'level']) - -# Drop columns by condition -cols_to_drop = [col for col in df.columns if col.startswith('temp_')] -df = df.drop(columns=cols_to_drop) -``` - -### Reordering Columns - -```python -# Explicit order -new_order = ['name', 'department', 'age', 'salary'] -df = df[new_order] - -# Move specific column to front -cols = ['salary'] + [c for c in df.columns if c != 'salary'] -df = df[cols] - -# Using .reindex() -df = df.reindex(columns=['name', 'age', 'salary', 'department']) -``` - ---- - -## Index Operations - -### Setting and Resetting Index - -```python -# Set column as index -df = df.set_index('name') - -# Reset index back to column -df = df.reset_index() - -# Drop index completely -df = df.reset_index(drop=True) - -# Set multiple columns as index (MultiIndex) -df = df.set_index(['department', 'name']) -``` - -### Working with MultiIndex - -```python -# Create MultiIndex DataFrame -df = pd.DataFrame({ - 'department': ['Eng', 'Eng', 'Sales', 'Sales'], - 'team': ['Backend', 'Frontend', 'East', 'West'], - 'headcount': [10, 8, 15, 12] -}).set_index(['department', 'team']) - -# Select from MultiIndex -df.loc['Eng'] # All Eng rows -df.loc[('Eng', 'Backend')] # Specific row - -# Cross-section with .xs() -df.xs('Backend', level='team') # All Backend teams - -# Reset specific level -df.reset_index(level='team') -``` - ---- - -## Copying DataFrames - -### When to Use `.copy()` - -```python -# ALWAYS copy when modifying a subset -subset = df[df['age'] > 25].copy() -subset['new_col'] = 100 # Safe, no SettingWithCopyWarning - -# Without copy - may raise warning or fail silently -# BAD: -# subset = df[df['age'] > 25] -# subset['new_col'] = 100 # SettingWithCopyWarning! - -# Deep copy (default) - copies data -df_copy = df.copy() # or df.copy(deep=True) - -# Shallow copy - shares data, only copies structure -df_shallow = df.copy(deep=False) -``` - ---- - -## Best Practices Summary - -1. **Use `.loc[]` and `.iloc[]`** - Never use chained indexing -2. **Parenthesize conditions** - `(cond1) & (cond2)` not `cond1 & cond2` -3. **Use `.query()` for readability** - Especially with complex filters -4. **Copy before modifying subsets** - Always use `.copy()` -5. **Use vectorized operations** - Avoid row iteration for filtering -6. **Handle NaN explicitly** - Use `na=False` in string operations -7. **Prefer method chaining** - Use `.assign()` for column creation - ---- - -## Anti-Patterns to Avoid - -```python -# BAD: Chained indexing -df['A']['B'] = value # May not work, raises warning - -# GOOD: Use .loc -df.loc[:, ('A', 'B')] = value -# Or for row selection then assignment: -df.loc[df['A'] > 0, 'B'] = value - -# BAD: Iterating for filtering -result = [] -for idx, row in df.iterrows(): - if row['age'] > 25: - result.append(row) - -# GOOD: Boolean indexing -result = df[df['age'] > 25] - -# BAD: Multiple separate assignments -df = df[df['age'] > 25] -df = df[df['salary'] > 50000] - -# GOOD: Combined filter -df = df[(df['age'] > 25) & (df['salary'] > 50000)] -``` - ---- - -## Related References - -- `data-cleaning.md` - After selection, clean the data -- `aggregation-groupby.md` - Group and aggregate filtered data -- `performance-optimization.md` - Optimize filtering on large datasets diff --git a/.cursor/skills/pandas-pro/references/merging-joining.md b/.cursor/skills/pandas-pro/references/merging-joining.md deleted file mode 100644 index 7bf33dca..00000000 --- a/.cursor/skills/pandas-pro/references/merging-joining.md +++ /dev/null @@ -1,599 +0,0 @@ -# Merging and Joining - -> Reference for: Pandas Pro -> Load when: Merge, join, concat, combine DataFrames, or handle relational data - ---- - -## Overview - -Combining DataFrames is essential for working with relational data. This reference covers merge, join, concat, and advanced combination strategies with pandas 2.0+. - ---- - -## Merge (SQL-Style Joins) - -### Basic Merge - -```python -import pandas as pd -import numpy as np - -# Sample DataFrames -employees = pd.DataFrame({ - 'emp_id': [1, 2, 3, 4, 5], - 'name': ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve'], - 'dept_id': [101, 102, 101, 103, 102], -}) - -departments = pd.DataFrame({ - 'dept_id': [101, 102, 104], - 'dept_name': ['Engineering', 'Sales', 'Marketing'], -}) - -# Inner join (default) - only matching rows -result = pd.merge(employees, departments, on='dept_id') - -# Explicit how parameter -result = pd.merge(employees, departments, on='dept_id', how='inner') -``` - -### Join Types - -```python -# Inner join - only matching rows from both -inner = pd.merge(employees, departments, on='dept_id', how='inner') -# Result: 4 rows (emp_id 4 has dept_id 103 which doesn't exist in departments) - -# Left join - all rows from left, matching from right -left = pd.merge(employees, departments, on='dept_id', how='left') -# Result: 5 rows (Diana has NaN for dept_name) - -# Right join - all rows from right, matching from left -right = pd.merge(employees, departments, on='dept_id', how='right') -# Result: 4 rows (Marketing has no employees, but is included) - -# Outer join - all rows from both -outer = pd.merge(employees, departments, on='dept_id', how='outer') -# Result: 6 rows (includes unmatched from both sides) - -# Cross join - cartesian product -cross = pd.merge(employees, departments, how='cross') -# Result: 15 rows (5 employees x 3 departments) -``` - -### Merging on Different Column Names - -```python -employees = pd.DataFrame({ - 'emp_id': [1, 2, 3], - 'name': ['Alice', 'Bob', 'Charlie'], - 'department': [101, 102, 101], -}) - -departments = pd.DataFrame({ - 'id': [101, 102], - 'dept_name': ['Engineering', 'Sales'], -}) - -# Different column names -result = pd.merge( - employees, - departments, - left_on='department', - right_on='id' -) - -# Drop duplicate column after merge -result = result.drop('id', axis=1) -``` - -### Merging on Multiple Columns - -```python -sales = pd.DataFrame({ - 'region': ['East', 'East', 'West', 'West'], - 'product': ['A', 'B', 'A', 'B'], - 'sales': [100, 150, 120, 180], -}) - -targets = pd.DataFrame({ - 'region': ['East', 'East', 'West'], - 'product': ['A', 'B', 'A'], - 'target': [90, 140, 110], -}) - -# Merge on multiple columns -result = pd.merge(sales, targets, on=['region', 'product'], how='left') -``` - -### Merging on Index - -```python -# Set index before merge -employees_idx = employees.set_index('emp_id') -salaries = pd.DataFrame({ - 'emp_id': [1, 2, 3, 4], - 'salary': [80000, 75000, 70000, 65000], -}).set_index('emp_id') - -# Merge on index -result = pd.merge(employees_idx, salaries, left_index=True, right_index=True) - -# Mix of column and index -result = pd.merge( - employees, - salaries, - left_on='emp_id', - right_index=True -) -``` - ---- - -## Handling Duplicate Columns - -### Suffixes - -```python -df1 = pd.DataFrame({ - 'id': [1, 2, 3], - 'value': [10, 20, 30], - 'date': ['2024-01-01', '2024-01-02', '2024-01-03'], -}) - -df2 = pd.DataFrame({ - 'id': [1, 2, 3], - 'value': [100, 200, 300], - 'date': ['2024-02-01', '2024-02-02', '2024-02-03'], -}) - -# Default suffixes -result = pd.merge(df1, df2, on='id') -# Columns: id, value_x, date_x, value_y, date_y - -# Custom suffixes -result = pd.merge(df1, df2, on='id', suffixes=('_jan', '_feb')) -# Columns: id, value_jan, date_jan, value_feb, date_feb -``` - -### Validate Merge Cardinality - -```python -# Validate merge relationships (pandas 2.0+) -# Raises MergeError if validation fails - -# One-to-one: each key appears at most once in both DataFrames -result = pd.merge(df1, df2, on='id', validate='one_to_one') # or '1:1' - -# One-to-many: keys unique in left only -result = pd.merge(employees, salaries, on='emp_id', validate='one_to_many') # or '1:m' - -# Many-to-one: keys unique in right only -result = pd.merge(salaries, employees, on='emp_id', validate='many_to_one') # or 'm:1' - -# Many-to-many: no uniqueness requirement (default) -result = pd.merge(df1, df2, on='id', validate='many_to_many') # or 'm:m' -``` - -### Indicator Column - -```python -# Add indicator column showing source of each row -result = pd.merge( - employees, - departments, - on='dept_id', - how='outer', - indicator=True -) -# _merge column values: 'left_only', 'right_only', 'both' - -# Custom indicator name -result = pd.merge( - employees, - departments, - on='dept_id', - how='outer', - indicator='source' -) - -# Filter by indicator -left_only = result[result['_merge'] == 'left_only'] -both = result[result['_merge'] == 'both'] -``` - ---- - -## Join (Index-Based) - -### DataFrame.join() - -```python -# join() is for index-based joining (simpler syntax) -employees = pd.DataFrame({ - 'name': ['Alice', 'Bob', 'Charlie'], - 'dept_id': [101, 102, 101], -}, index=[1, 2, 3]) - -salaries = pd.DataFrame({ - 'salary': [80000, 75000, 70000], - 'bonus': [5000, 4000, 3500], -}, index=[1, 2, 3]) - -# Join on index -result = employees.join(salaries) - -# Join types (same as merge) -result = employees.join(salaries, how='left') -result = employees.join(salaries, how='outer') -``` - -### Join on Column to Index - -```python -employees = pd.DataFrame({ - 'name': ['Alice', 'Bob', 'Charlie'], - 'dept_id': [101, 102, 101], -}) - -departments = pd.DataFrame({ - 'dept_name': ['Engineering', 'Sales'], -}, index=[101, 102]) - -# Join left column to right index -result = employees.join(departments, on='dept_id') -``` - -### Join Multiple DataFrames - -```python -df1 = pd.DataFrame({'a': [1, 2]}, index=['x', 'y']) -df2 = pd.DataFrame({'b': [3, 4]}, index=['x', 'y']) -df3 = pd.DataFrame({'c': [5, 6]}, index=['x', 'y']) - -# Join multiple at once -result = df1.join([df2, df3]) - -# With suffixes for duplicate columns -result = df1.join([df2, df3], lsuffix='_1', rsuffix='_2') -``` - ---- - -## Concat (Stacking DataFrames) - -### Vertical Concatenation (Row-wise) - -```python -# Stack DataFrames vertically -df1 = pd.DataFrame({ - 'name': ['Alice', 'Bob'], - 'age': [25, 30], -}) - -df2 = pd.DataFrame({ - 'name': ['Charlie', 'Diana'], - 'age': [35, 28], -}) - -# Basic concat (axis=0 is default) -result = pd.concat([df1, df2]) - -# Reset index -result = pd.concat([df1, df2], ignore_index=True) - -# Keep track of source -result = pd.concat([df1, df2], keys=['source1', 'source2']) -# Creates MultiIndex -``` - -### Horizontal Concatenation (Column-wise) - -```python -names = pd.DataFrame({'name': ['Alice', 'Bob', 'Charlie']}) -ages = pd.DataFrame({'age': [25, 30, 35]}) -salaries = pd.DataFrame({'salary': [50000, 60000, 70000]}) - -# Concat columns (axis=1) -result = pd.concat([names, ages, salaries], axis=1) -``` - -### Handling Mismatched Columns - -```python -df1 = pd.DataFrame({ - 'name': ['Alice', 'Bob'], - 'age': [25, 30], -}) - -df2 = pd.DataFrame({ - 'name': ['Charlie', 'Diana'], - 'salary': [70000, 65000], -}) - -# Outer join (default) - include all columns -result = pd.concat([df1, df2]) -# age and salary columns have NaN where not present - -# Inner join - only common columns -result = pd.concat([df1, df2], join='inner') -# Only 'name' column -``` - -### Concat with Verification - -```python -# Verify no index overlap -try: - result = pd.concat([df1, df2], verify_integrity=True) -except ValueError as e: - print(f"Index overlap detected: {e}") - -# Alternative: use ignore_index -result = pd.concat([df1, df2], ignore_index=True) -``` - ---- - -## Combine and Update - -### combine_first() - Fill Gaps - -```python -# Fill NaN values from another DataFrame -df1 = pd.DataFrame({ - 'A': [1, np.nan, 3], - 'B': [np.nan, 2, 3], -}, index=['a', 'b', 'c']) - -df2 = pd.DataFrame({ - 'A': [10, 20, 30], - 'B': [10, 20, 30], -}, index=['a', 'b', 'c']) - -# Fill NaN in df1 with values from df2 -result = df1.combine_first(df2) -# A: [1, 20, 3], B: [10, 2, 3] -``` - -### update() - In-Place Update - -```python -df1 = pd.DataFrame({ - 'A': [1, 2, 3], - 'B': [4, 5, 6], -}, index=['a', 'b', 'c']) - -df2 = pd.DataFrame({ - 'A': [10, 20], - 'B': [40, 50], -}, index=['a', 'b']) - -# Update df1 with values from df2 (in-place) -df1.update(df2) -# df1 now has A: [10, 20, 3], B: [40, 50, 6] - -# Only update where df2 has non-NaN -df1.update(df2, overwrite=False) # Don't overwrite existing values -``` - ---- - -## Advanced Merge Patterns - -### Merge with Aggregation - -```python -# Merge and aggregate in one operation -orders = pd.DataFrame({ - 'order_id': [1, 2, 3, 4], - 'customer_id': [101, 102, 101, 103], - 'amount': [100, 200, 150, 300], -}) - -customers = pd.DataFrame({ - 'customer_id': [101, 102, 103], - 'name': ['Alice', 'Bob', 'Charlie'], -}) - -# Get customer summary -customer_summary = orders.groupby('customer_id').agg( - total_orders=('order_id', 'count'), - total_amount=('amount', 'sum'), -).reset_index() - -# Merge with customer info -result = pd.merge(customers, customer_summary, on='customer_id') -``` - -### Merge Asof (Nearest Match) - -```python -# Merge on nearest key (useful for time series) -trades = pd.DataFrame({ - 'time': pd.to_datetime(['2024-01-01 10:00:01', '2024-01-01 10:00:03', '2024-01-01 10:00:05']), - 'ticker': ['AAPL', 'AAPL', 'AAPL'], - 'price': [150.0, 151.0, 150.5], -}) - -quotes = pd.DataFrame({ - 'time': pd.to_datetime(['2024-01-01 10:00:00', '2024-01-01 10:00:02', '2024-01-01 10:00:04']), - 'ticker': ['AAPL', 'AAPL', 'AAPL'], - 'bid': [149.5, 150.5, 150.0], - 'ask': [150.5, 151.5, 151.0], -}) - -# Merge asof - find nearest quote for each trade -result = pd.merge_asof( - trades.sort_values('time'), - quotes.sort_values('time'), - on='time', - by='ticker', - direction='backward' # Use most recent quote -) -``` - -### Conditional Merge - -```python -# Merge with conditions beyond key equality -# First merge, then filter - -products = pd.DataFrame({ - 'product_id': [1, 2, 3], - 'name': ['Widget', 'Gadget', 'Gizmo'], - 'category': ['A', 'B', 'A'], -}) - -discounts = pd.DataFrame({ - 'category': ['A', 'A', 'B'], - 'min_qty': [10, 50, 20], - 'discount': [0.05, 0.10, 0.08], -}) - -# Cross merge then filter -merged = pd.merge(products, discounts, on='category') -# Then apply quantity-based filtering as needed -``` - ---- - -## Performance Considerations - -### Pre-sorting for Merge - -```python -# Sort keys before merge for better performance -df1 = df1.sort_values('key') -df2 = df2.sort_values('key') - -# Merge sorted DataFrames -result = pd.merge(df1, df2, on='key') -``` - -### Index Alignment - -```python -# Using index for merge is often faster than columns -df1 = df1.set_index('key') -df2 = df2.set_index('key') - -# Join on index -result = df1.join(df2) -``` - -### Memory-Efficient Merge - -```python -# For large DataFrames, reduce memory before merge -# Convert to appropriate types -df1['key'] = df1['key'].astype('int32') # Instead of int64 -df1['category'] = df1['category'].astype('category') - -# Select only needed columns -cols_needed = ['key', 'value1', 'value2'] -result = pd.merge(df1[cols_needed], df2[cols_needed], on='key') -``` - ---- - -## Common Merge Patterns - -### Left Join with Null Check - -```python -# Find unmatched rows after left join -result = pd.merge(employees, departments, on='dept_id', how='left') -unmatched = result[result['dept_name'].isna()] -``` - -### Anti-Join (Rows Not in Other) - -```python -# Find employees NOT in a specific department list -dept_list = [101, 102] - -# Method 1: Using isin -not_in_depts = employees[~employees['dept_id'].isin(dept_list)] - -# Method 2: Using merge with indicator -merged = pd.merge( - employees, - pd.DataFrame({'dept_id': dept_list}), - on='dept_id', - how='left', - indicator=True -) -not_in_depts = merged[merged['_merge'] == 'left_only'] -``` - -### Self-Join - -```python -# Find pairs within same department -employees = pd.DataFrame({ - 'emp_id': [1, 2, 3, 4], - 'name': ['Alice', 'Bob', 'Charlie', 'Diana'], - 'dept_id': [101, 101, 102, 101], -}) - -# Self-join to find pairs -pairs = pd.merge( - employees, - employees, - on='dept_id', - suffixes=('_1', '_2') -) -# Remove self-pairs and duplicates -pairs = pairs[pairs['emp_id_1'] < pairs['emp_id_2']] -``` - ---- - -## Best Practices Summary - -1. **Choose the right join type** - Default inner may drop data -2. **Validate cardinality** - Use `validate` parameter -3. **Use indicator** - Debug unexpected results -4. **Handle duplicates** - Use meaningful suffixes -5. **Pre-sort for performance** - Especially for large DataFrames -6. **Reset index after operations** - Keep DataFrames usable -7. **Check for NaN after join** - Understand unmatched rows - ---- - -## Anti-Patterns to Avoid - -```python -# BAD: Merge without understanding cardinality -result = pd.merge(df1, df2, on='key') # May explode row count - -# GOOD: Validate relationship -result = pd.merge(df1, df2, on='key', validate='one_to_one') - -# BAD: Repeated merges -result = pd.merge(df1, df2, on='key') -result = pd.merge(result, df3, on='key') -result = pd.merge(result, df4, on='key') - -# GOOD: Chain or use reduce -from functools import reduce -dfs = [df1, df2, df3, df4] -result = reduce(lambda left, right: pd.merge(left, right, on='key'), dfs) - -# BAD: Ignoring merge indicators -result = pd.merge(df1, df2, on='key', how='outer') - -# GOOD: Check merge results -result = pd.merge(df1, df2, on='key', how='outer', indicator=True) -print(result['_merge'].value_counts()) -``` - ---- - -## Related References - -- `dataframe-operations.md` - Filter before/after merge -- `aggregation-groupby.md` - Aggregate before merging -- `performance-optimization.md` - Optimize large merges diff --git a/.cursor/skills/pandas-pro/references/performance-optimization.md b/.cursor/skills/pandas-pro/references/performance-optimization.md deleted file mode 100644 index aa05f08a..00000000 --- a/.cursor/skills/pandas-pro/references/performance-optimization.md +++ /dev/null @@ -1,600 +0,0 @@ -# Performance Optimization - -> Reference for: Pandas Pro -> Load when: Memory usage issues, slow operations, large datasets, vectorization, or chunked processing - ---- - -## Overview - -Optimizing pandas performance is critical for production workflows. This reference covers memory optimization, vectorization, chunking, and profiling with pandas 2.0+. - ---- - -## Memory Analysis - -### Checking Memory Usage - -```python -import pandas as pd -import numpy as np - -df = pd.DataFrame({ - 'id': range(1_000_000), - 'name': ['user_' + str(i) for i in range(1_000_000)], - 'category': np.random.choice(['A', 'B', 'C', 'D'], 1_000_000), - 'value': np.random.randn(1_000_000), - 'count': np.random.randint(0, 100, 1_000_000), -}) - -# Basic memory info -print(df.info(memory_usage='deep')) - -# Detailed memory by column -memory_usage = df.memory_usage(deep=True) -print(memory_usage) -print(f"Total: {memory_usage.sum() / 1e6:.2f} MB") - -# Memory as percentage of total -memory_pct = (memory_usage / memory_usage.sum() * 100).round(2) -print(memory_pct) -``` - -### Memory Profiling Function - -```python -def memory_profile(df: pd.DataFrame) -> pd.DataFrame: - """Profile memory usage by column with optimization suggestions.""" - memory_bytes = df.memory_usage(deep=True) - - profile = pd.DataFrame({ - 'dtype': df.dtypes, - 'non_null': df.count(), - 'null_count': df.isna().sum(), - 'unique': df.nunique(), - 'memory_mb': (memory_bytes / 1e6).round(3), - }) - - # Add optimization suggestions - suggestions = [] - for col in df.columns: - dtype = df[col].dtype - nunique = df[col].nunique() - - if dtype == 'object': - if nunique / len(df) < 0.5: # Less than 50% unique - suggestions.append(f"Convert to category (only {nunique} unique)") - else: - suggestions.append("Consider string dtype") - elif dtype == 'int64': - if df[col].max() < 2**31 and df[col].min() >= -2**31: - suggestions.append("Downcast to int32") - if df[col].max() < 2**15 and df[col].min() >= -2**15: - suggestions.append("Downcast to int16") - elif dtype == 'float64': - suggestions.append("Consider float32 if precision allows") - else: - suggestions.append("OK") - - profile['suggestion'] = suggestions - return profile - -print(memory_profile(df)) -``` - ---- - -## Memory Optimization Techniques - -### Downcasting Numeric Types - -```python -# Automatic downcasting for integers -df['count'] = pd.to_numeric(df['count'], downcast='integer') - -# Automatic downcasting for floats -df['value'] = pd.to_numeric(df['value'], downcast='float') - -# Manual downcasting function -def downcast_dtypes(df: pd.DataFrame) -> pd.DataFrame: - """Reduce memory by downcasting numeric types.""" - df = df.copy() - - for col in df.select_dtypes(include=['int']).columns: - df[col] = pd.to_numeric(df[col], downcast='integer') - - for col in df.select_dtypes(include=['float']).columns: - df[col] = pd.to_numeric(df[col], downcast='float') - - return df - -df_optimized = downcast_dtypes(df) -print(f"Before: {df.memory_usage(deep=True).sum() / 1e6:.2f} MB") -print(f"After: {df_optimized.memory_usage(deep=True).sum() / 1e6:.2f} MB") -``` - -### Using Categorical Type - -```python -# Convert low-cardinality string columns to category -# Especially effective when unique values << total rows - -# Before -print(f"Object dtype: {df['category'].memory_usage(deep=True) / 1e6:.2f} MB") - -# After -df['category'] = df['category'].astype('category') -print(f"Category dtype: {df['category'].memory_usage(deep=True) / 1e6:.2f} MB") - -# Automatic conversion for low-cardinality columns -def optimize_categories(df: pd.DataFrame, threshold: float = 0.5) -> pd.DataFrame: - """Convert object columns to category if unique ratio < threshold.""" - df = df.copy() - - for col in df.select_dtypes(include=['object']).columns: - unique_ratio = df[col].nunique() / len(df) - if unique_ratio < threshold: - df[col] = df[col].astype('category') - - return df -``` - -### Sparse Data Types - -```python -# For data with many repeated values (especially zeros/NaN) -sparse_series = pd.arrays.SparseArray([0, 0, 1, 0, 0, 0, 2, 0, 0, 0]) - -# Create sparse DataFrame -df_sparse = pd.DataFrame({ - 'sparse_col': pd.arrays.SparseArray([0] * 9000 + [1] * 1000), - 'dense_col': [0] * 9000 + [1] * 1000, -}) - -print(f"Sparse: {df_sparse['sparse_col'].memory_usage() / 1e6:.4f} MB") -print(f"Dense: {df_sparse['dense_col'].memory_usage() / 1e6:.4f} MB") -``` - -### Nullable Types (pandas 2.0+) - -```python -# Use nullable types for proper NA handling with memory efficiency -df = df.astype({ - 'id': 'Int32', # Nullable int32 - 'count': 'Int16', # Nullable int16 - 'value': 'Float32', # Nullable float32 - 'name': 'string', # Nullable string (more memory efficient) - 'category': 'category', # Categorical -}) - -# Arrow-backed types for even better memory (pandas 2.0+) -df['name'] = df['name'].astype('string[pyarrow]') -df['category'] = df['category'].astype('category') -``` - ---- - -## Vectorization - -### Replace Loops with Vectorized Operations - -```python -# BAD: Row iteration (extremely slow) -result = [] -for idx, row in df.iterrows(): - if row['value'] > 0: - result.append(row['value'] * 2) - else: - result.append(0) -df['result'] = result - -# GOOD: Vectorized with np.where -df['result'] = np.where(df['value'] > 0, df['value'] * 2, 0) - -# GOOD: Vectorized with boolean indexing -df['result'] = 0 -df.loc[df['value'] > 0, 'result'] = df.loc[df['value'] > 0, 'value'] * 2 -``` - -### Multiple Conditions with np.select - -```python -# BAD: Nested if-else in apply -def categorize(row): - if row['value'] < -1: - return 'very_low' - elif row['value'] < 0: - return 'low' - elif row['value'] < 1: - return 'medium' - else: - return 'high' - -df['category'] = df.apply(categorize, axis=1) # SLOW! - -# GOOD: Vectorized with np.select -conditions = [ - df['value'] < -1, - df['value'] < 0, - df['value'] < 1, -] -choices = ['very_low', 'low', 'medium'] -df['category'] = np.select(conditions, choices, default='high') -``` - -### String Operations - Vectorized - -```python -# BAD: Apply for string operations -df['upper_name'] = df['name'].apply(lambda x: x.upper()) - -# GOOD: Vectorized string methods -df['upper_name'] = df['name'].str.upper() - -# Combine multiple string operations -df['processed'] = ( - df['name'] - .str.strip() - .str.lower() - .str.replace(r'\s+', '_', regex=True) -) -``` - -### Avoid apply() When Possible - -```python -# BAD: apply for row-wise calculation -df['total'] = df.apply(lambda row: row['a'] + row['b'] + row['c'], axis=1) - -# GOOD: Direct vectorized operation -df['total'] = df['a'] + df['b'] + df['c'] - -# BAD: apply for element-wise operation -df['squared'] = df['value'].apply(lambda x: x ** 2) - -# GOOD: Vectorized -df['squared'] = df['value'] ** 2 - -# When apply IS appropriate: complex custom logic -def complex_calculation(row): - # Multiple dependencies and conditional logic - if row['type'] == 'A': - return row['value'] * row['multiplier'] + row['offset'] - else: - return row['value'] / row['divisor'] - row['adjustment'] - -# Consider rewriting as vectorized if performance critical -``` - ---- - -## Chunked Processing - -### Reading Large Files in Chunks - -```python -# Read CSV in chunks -chunk_size = 100_000 -chunks = [] - -for chunk in pd.read_csv('large_file.csv', chunksize=chunk_size): - # Process each chunk - processed = chunk[chunk['value'] > 0] # Filter - processed = processed.groupby('category')['value'].sum() # Aggregate - chunks.append(processed) - -# Combine results -result = pd.concat(chunks).groupby(level=0).sum() -``` - -### Chunked Processing Function - -```python -def process_large_csv( - filepath: str, - chunk_size: int = 100_000, - filter_func=None, - agg_func=None, -) -> pd.DataFrame: - """Process large CSV files in chunks.""" - results = [] - - for chunk in pd.read_csv(filepath, chunksize=chunk_size): - # Apply filter if provided - if filter_func: - chunk = filter_func(chunk) - - # Apply aggregation if provided - if agg_func: - chunk = agg_func(chunk) - - results.append(chunk) - - # Combine results - combined = pd.concat(results, ignore_index=True) - - # Re-aggregate if needed - if agg_func: - combined = agg_func(combined) - - return combined - -# Usage -result = process_large_csv( - 'large_file.csv', - chunk_size=50_000, - filter_func=lambda df: df[df['value'] > 0], - agg_func=lambda df: df.groupby('category').agg({'value': 'sum'}), -) -``` - -### Memory-Efficient Iteration - -```python -# When you must iterate, use itertuples (not iterrows) -# itertuples is 10-100x faster than iterrows - -# BAD: iterrows -for idx, row in df.iterrows(): - process(row['name'], row['value']) - -# BETTER: itertuples -for row in df.itertuples(): - process(row.name, row.value) # Access as attributes - -# BEST: Vectorized operations (avoid iteration entirely) -``` - ---- - -## Query Optimization - -### Efficient Filtering - -```python -# Order matters - filter early, compute late -# BAD: Compute on all rows, then filter -df['expensive_calc'] = df['a'] * df['b'] + np.sin(df['c']) -result = df[df['category'] == 'A'] - -# GOOD: Filter first, compute on subset -mask = df['category'] == 'A' -result = df[mask].copy() -result['expensive_calc'] = result['a'] * result['b'] + np.sin(result['c']) -``` - -### Using query() for Performance - -```python -# query() can be faster for large DataFrames (uses numexpr) -# Traditional boolean indexing -result = df[(df['value'] > 0) & (df['category'] == 'A')] - -# query() syntax (faster for large data) -result = df.query('value > 0 and category == "A"') - -# With variables -threshold = 0 -cat = 'A' -result = df.query('value > @threshold and category == @cat') -``` - -### eval() for Complex Expressions - -```python -# eval() uses numexpr for faster computation -# Standard pandas -df['result'] = df['a'] + df['b'] * df['c'] - df['d'] - -# Using eval (faster for large DataFrames) -df['result'] = pd.eval('df.a + df.b * df.c - df.d') - -# In-place with inplace parameter -df.eval('result = a + b * c - d', inplace=True) -``` - ---- - -## GroupBy Optimization - -### Pre-sort for Faster GroupBy - -```python -# Sort by groupby column first -df = df.sort_values('category') - -# Use sort=False since already sorted -result = df.groupby('category', sort=False)['value'].mean() -``` - -### Use Built-in Aggregations - -```python -# BAD: Custom function via apply -result = df.groupby('category')['value'].apply(lambda x: x.mean()) - -# GOOD: Built-in aggregation -result = df.groupby('category')['value'].mean() - -# Built-in aggregations available: -# sum, mean, median, min, max, std, var, count, first, last, nth -# size, sem, prod, cumsum, cummax, cummin, cumprod -``` - -### Observed Categories - -```python -# For categorical columns, use observed=True (pandas 2.0+ default) -df['category'] = df['category'].astype('category') - -# Avoid computing for unobserved categories -result = df.groupby('category', observed=True)['value'].mean() -``` - ---- - -## I/O Optimization - -### Efficient File Formats - -```python -# Parquet - best for analytical workloads -df.to_parquet('data.parquet', compression='snappy') -df = pd.read_parquet('data.parquet') - -# Feather - best for pandas interchange -df.to_feather('data.feather') -df = pd.read_feather('data.feather') - -# CSV with optimizations -df.to_csv('data.csv', index=False) -df = pd.read_csv( - 'data.csv', - dtype={'category': 'category', 'count': 'int32'}, - usecols=['id', 'category', 'value'], # Only needed columns - nrows=10000, # Limit rows for testing -) -``` - -### Specify dtypes When Reading - -```python -# Specify dtypes upfront to avoid inference overhead -dtypes = { - 'id': 'int32', - 'name': 'string', - 'category': 'category', - 'value': 'float32', - 'count': 'int16', -} - -df = pd.read_csv('data.csv', dtype=dtypes) - -# Parse dates efficiently -df = pd.read_csv( - 'data.csv', - dtype=dtypes, - parse_dates=['date_column'], - date_format='%Y-%m-%d', # Explicit format is faster -) -``` - ---- - -## Profiling and Benchmarking - -### Timing Operations - -```python -import time - -# Simple timing -start = time.time() -result = df.groupby('category')['value'].mean() -elapsed = time.time() - start -print(f"Elapsed: {elapsed:.4f} seconds") - -# Using %%timeit in Jupyter -# %%timeit -# df.groupby('category')['value'].mean() -``` - -### Memory Profiling - -```python -# Track memory before/after -import tracemalloc - -tracemalloc.start() - -# Your operation -df_result = df.groupby('category').agg({'value': 'sum'}) - -current, peak = tracemalloc.get_traced_memory() -print(f"Current memory: {current / 1e6:.2f} MB") -print(f"Peak memory: {peak / 1e6:.2f} MB") - -tracemalloc.stop() -``` - -### Comparison Template - -```python -def benchmark_operations(df: pd.DataFrame, operations: dict, n_runs: int = 5): - """Benchmark multiple operations.""" - results = {} - - for name, func in operations.items(): - times = [] - for _ in range(n_runs): - start = time.time() - func(df) - times.append(time.time() - start) - - results[name] = { - 'mean': np.mean(times), - 'std': np.std(times), - 'min': np.min(times), - } - - return pd.DataFrame(results).T - -# Usage -operations = { - 'iterrows': lambda df: [row['value'] for _, row in df.iterrows()], - 'itertuples': lambda df: [row.value for row in df.itertuples()], - 'vectorized': lambda df: df['value'].tolist(), -} - -benchmark_results = benchmark_operations(df.head(10000), operations) -print(benchmark_results) -``` - ---- - -## Best Practices Summary - -1. **Profile first** - Identify actual bottlenecks before optimizing -2. **Use appropriate dtypes** - int32/float32/category save memory -3. **Vectorize everything** - Avoid loops and apply when possible -4. **Filter early** - Reduce data before expensive operations -5. **Chunk large files** - Process in manageable pieces -6. **Use efficient file formats** - Parquet/Feather over CSV -7. **Leverage built-in methods** - Faster than custom functions - ---- - -## Performance Checklist - -Before deploying pandas code: - -- [ ] Memory profiled with `memory_usage(deep=True)` -- [ ] Dtypes optimized (downcast, categorical) -- [ ] No iterrows/itertuples in hot paths -- [ ] GroupBy uses built-in aggregations -- [ ] Large files processed in chunks -- [ ] Filters applied before computations -- [ ] Appropriate file format used -- [ ] Benchmarked with representative data size - ---- - -## Anti-Patterns Summary - -| Anti-Pattern | Alternative | -|--------------|-------------| -| `iterrows()` for computation | Vectorized operations | -| `apply(lambda)` for simple ops | Built-in methods | -| Loading entire large file | Chunked reading | -| String columns with low cardinality | Category dtype | -| int64 for small integers | int32/int16 | -| Multiple separate filters | Combined boolean mask | -| Repeated groupby calls | Single groupby with multiple aggs | - ---- - -## Related References - -- `dataframe-operations.md` - Efficient indexing and filtering -- `aggregation-groupby.md` - Optimized aggregation patterns -- `merging-joining.md` - Efficient merge strategies diff --git a/.cursor/skills/polars-expertise/.skillfish.json b/.cursor/skills/polars-expertise/.skillfish.json deleted file mode 100644 index c43a139a..00000000 --- a/.cursor/skills/polars-expertise/.skillfish.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "version": 2, - "name": "polars-expertise", - "owner": "deevsdeevs", - "repo": "agent-system", - "path": "polars-expertise", - "branch": "main", - "sha": "8b43b9ebc9dd1024e71b77f4f76ccd37ca3da777", - "source": "manual" -} \ No newline at end of file diff --git a/.cursor/skills/polars-expertise/SKILL.md b/.cursor/skills/polars-expertise/SKILL.md deleted file mode 100644 index 169c7d6c..00000000 --- a/.cursor/skills/polars-expertise/SKILL.md +++ /dev/null @@ -1,223 +0,0 @@ ---- -name: polars-expertise -description: > - This skill should be used when the user asks about Polars DataFrame library - (Apache Arrow) for Python or Rust. Triggers: "polars expressions", "lazy vs eager", - "scan_parquet streaming", "convert pandas to polars", "pyspark to polars", - "kdb to polars", "group_by_dynamic", "rolling_mean", "polars window functions", - "asof join", "polars GPU", "polars parquet", "LazyFrame". Time series: OHLCV - resampling, rolling windows, financial data patterns. Performance: native - expressions over map_elements, early projection, categorical types, streaming. ---- - -# Polars - -High-performance DataFrame library built on Apache Arrow. Supports Python and Rust with expression-based API, lazy evaluation, and automatic parallelization. - -## Quick Start - -### Python - -```bash -uv pip install polars -# GPU support: uv pip install polars[gpu] -``` - -```python -import polars as pl - -# Eager: immediate execution -df = pl.DataFrame({"symbol": ["AAPL", "GOOG"], "price": [150.0, 140.0]}) -df.filter(pl.col("price") > 145).select("symbol", "price") - -# Lazy: optimized execution (preferred for large data) -lf = pl.scan_parquet("trades.parquet") -result = lf.filter(pl.col("volume") > 1000).group_by("symbol").agg( - pl.col("price").mean().alias("avg_price") -).collect() -``` - -### Rust - -```toml -# Cargo.toml - select features you need -[dependencies] -polars = { version = "0.46", features = ["lazy", "parquet", "temporal"] } -``` - -```rust -use polars::prelude::*; - -fn main() -> PolarsResult<()> { - // Eager - let df = df![ - "symbol" => ["AAPL", "GOOG"], - "price" => [150.0, 140.0] - ]?; - - // Lazy (preferred) - let lf = LazyFrame::scan_parquet("trades.parquet", Default::default())?; - let result = lf - .filter(col("volume").gt(lit(1000))) - .group_by([col("symbol")]) - .agg([col("price").mean().alias("avg_price")]) - .collect()?; - Ok(()) -} -``` - -## Core Pattern: Expressions - -Everything in Polars is an expression. Expressions are composable, lazy, and parallelized. - -```python -# Expression building blocks -pl.col("price") # column reference -pl.col("price") * pl.col("volume") # arithmetic -pl.col("price").mean().over("symbol") # window function -pl.when(cond).then(a).otherwise(b) # conditional -``` - -Expressions execute in contexts: `select()`, `with_columns()`, `filter()`, `group_by().agg()` - -## When to Use Lazy - -| Use Lazy (`scan_*`, `.lazy()`) | Use Eager (`read_*`) | -|-------------------------------|----------------------| -| Large files (> RAM) | Small data, exploration | -| Complex pipelines | Simple one-off ops | -| Need query optimization | Interactive notebooks | -| Streaming required | Immediate feedback | - -Lazy benefits: predicate pushdown, projection pushdown, parallel execution, streaming. - -## Style: Use `.alias()` for Column Naming - -Always use `.alias("name")` instead of `name=expr` kwargs: - -```python -# GOOD: Explicit .alias() - works everywhere, composable -df.with_columns( - (pl.col("price") * pl.col("volume")).alias("value"), - pl.col("price").mean().over("symbol").alias("avg_price") -) - -# AVOID: Kwarg style - less flexible, doesn't chain -df.with_columns( - value=pl.col("price") * pl.col("volume"), # avoid - avg_price=pl.col("price").mean().over("symbol") # avoid -) -``` - -`.alias()` is explicit, chains with other methods, and works consistently in all contexts. - -## Anti-Patterns - AVOID - -```python -# BAD: Python functions kill parallelization -df.with_columns(pl.col("x").map_elements(lambda x: x * 2)) # SLOW - -# GOOD: Native expressions are parallel -df.with_columns((pl.col("x") * 2).alias("x")) # FAST - -# BAD: Row iteration -for row in df.iter_rows(): # SLOW - process(row) - -# GOOD: Columnar operations -df.with_columns(process_expr) # FAST - -# BAD: Late projection -lf.filter(...).collect().select("a", "b") # reads all columns - -# GOOD: Early projection -lf.select("a", "b").filter(...).collect() # reads only needed columns -``` - -## Performance Checklist - -- [ ] Using `scan_*` (lazy) for large files? -- [ ] Projecting columns early in pipeline? -- [ ] Using native expressions (no `map_elements`)? -- [ ] Categorical dtype for low-cardinality strings? -- [ ] Appropriate integer sizes (i32 vs i64)? -- [ ] Streaming for out-of-memory data? (`collect(engine="streaming")`) - -## Reference Navigator - -### Python References - -| Topic | File | When to Load | -|-------|------|--------------| -| Expressions, types, lazy/eager | [python/core_concepts.md](references/python/core_concepts.md) | Understanding fundamentals | -| Select, filter, group_by, window | [python/operations.md](references/python/operations.md) | Common operations | -| CSV, Parquet, streaming I/O | [python/io_guide.md](references/python/io_guide.md) | Reading/writing data | -| Joins, pivots, reshaping | [python/transformations.md](references/python/transformations.md) | Combining/reshaping data | -| Performance, patterns | [python/best_practices.md](references/python/best_practices.md) | Optimization | - -### Rust References - -| Topic | File | When to Load | -|-------|------|--------------| -| DataFrame, Series, ChunkedArray | [rust/core_concepts.md](references/rust/core_concepts.md) | Rust API fundamentals | -| Expression API in Rust | [rust/operations.md](references/rust/operations.md) | Operations syntax | -| Readers, writers, streaming | [rust/io_guide.md](references/rust/io_guide.md) | I/O operations | -| Feature flags, crates | [rust/features.md](references/rust/features.md) | Cargo setup | -| Allocators, SIMD, nightly | [rust/performance.md](references/rust/performance.md) | Performance tuning | -| Zero-copy, FFI, Arrow | [rust/arrow_interop.md](references/rust/arrow_interop.md) | Arrow integration | - -### Shared References - -| Topic | File | When to Load | -|-------|------|--------------| -| SQL queries on DataFrames | [sql_interface.md](references/sql_interface.md) | SQL syntax needed | -| Query optimization, streaming | [lazy_deep_dive.md](references/lazy_deep_dive.md) | Understanding lazy engine | -| NVIDIA GPU acceleration | [gpu_support.md](references/gpu_support.md) | GPU setup/usage | - -### Migration Guides - -| From | File | When to Load | -|------|------|--------------| -| pandas | [migration_pandas.md](references/migration_pandas.md) | Converting pandas code | -| PySpark | [migration_spark.md](references/migration_spark.md) | Converting Spark code | -| q/kdb+ | [migration_qkdb.md](references/migration_qkdb.md) | Converting kdb code | - -## Time Series / Financial Data Quick Patterns - -```python -# OHLCV resampling -df.group_by_dynamic("timestamp", every="1m").agg( - pl.col("price").first().alias("open"), - pl.col("price").max().alias("high"), - pl.col("price").min().alias("low"), - pl.col("price").last().alias("close"), - pl.col("volume").sum() -) - -# Rolling statistics -df.with_columns( - pl.col("price").rolling_mean(window_size=20).alias("sma_20"), - pl.col("price").rolling_std(window_size=20).alias("volatility") -) - -# As-of join for market data alignment -trades.join_asof(quotes, on="timestamp", by="symbol", strategy="backward") -``` - -Load [python/best_practices.md](references/python/best_practices.md) for comprehensive time series patterns. - -## Runnable Examples - -| Example | File | Purpose | -|---------|------|---------| -| Financial OHLCV | [examples/financial_ohlcv.py](examples/financial_ohlcv.py) | OHLCV resampling, rolling stats, VWAP | -| Pandas Migration | [examples/pandas_migration.py](examples/pandas_migration.py) | Side-by-side pandas vs polars | -| Streaming Large Files | [examples/streaming_large_file.py](examples/streaming_large_file.py) | Out-of-memory processing patterns | - -## Development Tips - -Use LSP for navigating Polars code: -- **Python**: Pyright/Pylance provides excellent type inference for Polars expressions -- **Rust**: rust-analyzer understands Polars types and expression chains - -LSP operations like `goToDefinition` and `hover` help explore Polars API without leaving the editor. diff --git a/.cursor/skills/polars-expertise/agents/openai.yaml b/.cursor/skills/polars-expertise/agents/openai.yaml deleted file mode 100644 index cd48664f..00000000 --- a/.cursor/skills/polars-expertise/agents/openai.yaml +++ /dev/null @@ -1,4 +0,0 @@ -interface: - display_name: "Polars Expertise" - short_description: "Polars performance and patterns" - default_prompt: "Use $polars-expertise to solve Polars DataFrame tasks." diff --git a/.cursor/skills/polars-expertise/examples/financial_ohlcv.py b/.cursor/skills/polars-expertise/examples/financial_ohlcv.py deleted file mode 100644 index c121b289..00000000 --- a/.cursor/skills/polars-expertise/examples/financial_ohlcv.py +++ /dev/null @@ -1,96 +0,0 @@ -""" -Financial OHLCV resampling with Polars. - -Demonstrates: -- group_by_dynamic for time-based resampling -- Rolling statistics (SMA, volatility) -- As-of joins for market data alignment -""" - -import polars as pl -from datetime import datetime, timedelta -import random - -# Generate sample tick data -def generate_tick_data(n_ticks: int = 10000) -> pl.DataFrame: - base_time = datetime(2024, 1, 15, 9, 30, 0) - symbols = ["AAPL", "GOOG", "MSFT"] - - data = [] - for i in range(n_ticks): - symbol = random.choice(symbols) - base_price = {"AAPL": 150.0, "GOOG": 140.0, "MSFT": 380.0}[symbol] - data.append({ - "timestamp": base_time + timedelta(seconds=i * 0.5), - "symbol": symbol, - "price": base_price + random.gauss(0, 1), - "volume": random.randint(100, 1000), - }) - - return pl.DataFrame(data).sort("timestamp") - - -def resample_to_ohlcv(df: pl.LazyFrame, interval: str = "1m") -> pl.LazyFrame: - """Resample tick data to OHLCV bars.""" - return ( - df.sort("timestamp") - .group_by_dynamic("timestamp", every=interval, group_by="symbol") - .agg( - pl.col("price").first().alias("open"), - pl.col("price").max().alias("high"), - pl.col("price").min().alias("low"), - pl.col("price").last().alias("close"), - pl.col("volume").sum().alias("volume"), - pl.len().alias("tick_count"), - ) - ) - - -def add_technical_indicators(df: pl.LazyFrame) -> pl.LazyFrame: - """Add common technical indicators.""" - return df.with_columns( - # Simple Moving Averages - pl.col("close").rolling_mean(window_size=5).over("symbol").alias("sma_5"), - pl.col("close").rolling_mean(window_size=20).over("symbol").alias("sma_20"), - - # Volatility (rolling std of returns) - pl.col("close") - .pct_change() - .over("symbol") - .rolling_std(window_size=20) - .over("symbol") - .alias("volatility_20"), - - # VWAP - ( - (pl.col("close") * pl.col("volume")).cum_sum().over("symbol") - / pl.col("volume").cum_sum().over("symbol") - ).alias("vwap"), - ) - - -def main(): - # Generate tick data - print("Generating tick data...") - ticks = generate_tick_data(10000) - print(f"Generated {len(ticks)} ticks") - print(ticks.head(5)) - - # Resample to 1-minute OHLCV (lazy) - print("\nResampling to 1-minute OHLCV bars...") - ohlcv = resample_to_ohlcv(ticks.lazy(), "1m") - - # Add technical indicators - print("Adding technical indicators...") - result = add_technical_indicators(ohlcv).collect() - - print(f"\nResult: {len(result)} bars") - print(result.head(10)) - - # Filter example: high volatility periods - high_vol = result.filter(pl.col("volatility_20") > 0.01) - print(f"\nHigh volatility periods: {len(high_vol)} bars") - - -if __name__ == "__main__": - main() diff --git a/.cursor/skills/polars-expertise/examples/pandas_migration.py b/.cursor/skills/polars-expertise/examples/pandas_migration.py deleted file mode 100644 index 1c79880e..00000000 --- a/.cursor/skills/polars-expertise/examples/pandas_migration.py +++ /dev/null @@ -1,143 +0,0 @@ -""" -Pandas to Polars migration examples. - -Side-by-side comparison of common operations. -Run with: uv run python pandas_migration.py -""" - -import polars as pl - -# Sample data -data = { - "name": ["Alice", "Bob", "Charlie", "Diana", "Eve"], - "department": ["Engineering", "Sales", "Engineering", "Sales", "Engineering"], - "salary": [100000, 80000, 120000, 90000, 110000], - "years": [5, 3, 8, 4, 6], -} - - -def basic_operations(): - """Basic DataFrame operations.""" - df = pl.DataFrame(data) - print("=== Basic Operations ===\n") - - # Select columns - # pandas: df[["name", "salary"]] - selected = df.select("name", "salary") - print("Select columns:") - print(selected) - - # Filter rows - # pandas: df[df["salary"] > 95000] - filtered = df.filter(pl.col("salary") > 95000) - print("\nFilter salary > 95000:") - print(filtered) - - # Add computed column - # pandas: df["bonus"] = df["salary"] * 0.1 - with_bonus = df.with_columns( - (pl.col("salary") * 0.1).alias("bonus") - ) - print("\nWith bonus column:") - print(with_bonus) - - -def groupby_operations(): - """Group by and aggregation.""" - df = pl.DataFrame(data) - print("\n=== Group By Operations ===\n") - - # Basic groupby - # pandas: df.groupby("department")["salary"].mean() - by_dept = df.group_by("department").agg( - pl.col("salary").mean().alias("avg_salary"), - pl.col("salary").max().alias("max_salary"), - pl.len().alias("count"), - ) - print("Group by department:") - print(by_dept) - - # Window function (transform equivalent) - # pandas: df["dept_avg"] = df.groupby("department")["salary"].transform("mean") - with_dept_avg = df.with_columns( - pl.col("salary").mean().over("department").alias("dept_avg"), - pl.col("salary").rank().over("department").alias("salary_rank"), - ) - print("\nWindow functions (dept avg and rank):") - print(with_dept_avg) - - -def conditional_operations(): - """Conditional column creation.""" - df = pl.DataFrame(data) - print("\n=== Conditional Operations ===\n") - - # pandas: np.where(df["salary"] > 100000, "high", "normal") - with_tier = df.with_columns( - pl.when(pl.col("salary") > 100000) - .then(pl.lit("high")) - .when(pl.col("salary") > 85000) - .then(pl.lit("medium")) - .otherwise(pl.lit("normal")) - .alias("salary_tier") - ) - print("Salary tiers:") - print(with_tier) - - -def chained_operations(): - """Chained operations (method chaining).""" - df = pl.DataFrame(data) - print("\n=== Chained Operations ===\n") - - # Complex pipeline - result = ( - df.filter(pl.col("years") >= 4) - .with_columns( - (pl.col("salary") * 1.1).alias("new_salary"), - (pl.col("salary") / pl.col("years")).alias("salary_per_year"), - ) - .group_by("department") - .agg( - pl.col("new_salary").mean().alias("avg_new_salary"), - pl.col("salary_per_year").mean().alias("avg_salary_per_year"), - ) - .sort("avg_new_salary", descending=True) - ) - print("Complex pipeline result:") - print(result) - - -def lazy_vs_eager(): - """Demonstrate lazy evaluation benefits.""" - df = pl.DataFrame(data) - print("\n=== Lazy vs Eager ===\n") - - # Eager: executes immediately - eager_result = df.filter(pl.col("salary") > 90000).select("name", "salary") - print("Eager result:") - print(eager_result) - - # Lazy: builds query plan, optimizes, then executes - lazy_result = ( - df.lazy() - .filter(pl.col("salary") > 90000) - .select("name", "salary") - ) - print("\nLazy query plan:") - print(lazy_result.explain()) - - print("\nLazy result (after .collect()):") - print(lazy_result.collect()) - - -def main(): - basic_operations() - groupby_operations() - conditional_operations() - chained_operations() - lazy_vs_eager() - - -if __name__ == "__main__": - main() diff --git a/.cursor/skills/polars-expertise/examples/streaming_large_file.py b/.cursor/skills/polars-expertise/examples/streaming_large_file.py deleted file mode 100644 index 5c197c13..00000000 --- a/.cursor/skills/polars-expertise/examples/streaming_large_file.py +++ /dev/null @@ -1,181 +0,0 @@ -""" -Streaming large files with Polars. - -Demonstrates processing files larger than available RAM using: -- scan_csv / scan_parquet (lazy scanning) -- Streaming execution engine -- Sink operations for streaming writes - -Run with: uv run python streaming_large_file.py -""" - -import polars as pl -from pathlib import Path -import tempfile - - -def create_sample_data(path: Path, n_rows: int = 100000): - """Create a sample CSV file for demonstration.""" - import random - - with open(path, "w") as f: - f.write("id,category,value,timestamp\n") - categories = ["A", "B", "C", "D", "E"] - for i in range(n_rows): - cat = random.choice(categories) - val = random.gauss(100, 20) - ts = f"2024-01-{(i % 28) + 1:02d} {(i % 24):02d}:00:00" - f.write(f"{i},{cat},{val:.2f},{ts}\n") - - print(f"Created {path} with {n_rows} rows") - - -def streaming_aggregation(input_path: Path): - """ - Aggregate large file using streaming. - - Key pattern: scan_* + lazy operations + collect(engine="streaming") - """ - print("\n=== Streaming Aggregation ===\n") - - # scan_csv returns LazyFrame - no data loaded yet - lf = pl.scan_csv(input_path) - - # Build query plan - result = ( - lf.filter(pl.col("value") > 80) # Predicate pushdown - .group_by("category") - .agg( - pl.col("value").mean().alias("avg_value"), - pl.col("value").std().alias("std_value"), - pl.len().alias("count"), - ) - .sort("avg_value", descending=True) - ) - - # Show optimized plan - print("Query plan:") - print(result.explain()) - - # Execute with streaming engine - # For truly large files, this processes in chunks - df = result.collect(engine="streaming") - print("\nResult:") - print(df) - - return df - - -def streaming_sink(input_path: Path, output_path: Path): - """ - Stream data directly to output file without loading into memory. - - Key pattern: scan_* + transformations + sink_* - """ - print("\n=== Streaming Sink (File to File) ===\n") - - lf = pl.scan_csv(input_path) - - # Transform and sink directly to parquet - # Data flows through without full materialization - ( - lf.filter(pl.col("value") > 90) - .with_columns( - pl.col("value").alias("original_value"), - (pl.col("value") * 1.1).alias("adjusted_value"), - ) - .select("id", "category", "original_value", "adjusted_value", "timestamp") - .sink_parquet(output_path) - ) - - print(f"Streamed filtered data to {output_path}") - - # Verify output - result = pl.scan_parquet(output_path).collect() - print(f"Output contains {len(result)} rows") - print(result.head(5)) - - -def check_streaming_compatibility(input_path: Path): - """ - Check if a query can be streamed. - - Some operations break streaming: - - Sorts on large data (may need to buffer) - - Certain join types - - Some aggregations - """ - print("\n=== Streaming Compatibility Check ===\n") - - lf = pl.scan_csv(input_path) - - # This query streams well - streamable = ( - lf.filter(pl.col("value") > 80) - .group_by("category") - .agg(pl.col("value").sum()) - ) - - print("Streamable query plan:") - print(streamable.explain(streaming=True)) - - # This may not stream fully (sort requires buffering) - maybe_not_streamable = ( - lf.sort("value", descending=True) - .head(1000) - ) - - print("\nQuery with sort (may buffer):") - print(maybe_not_streamable.explain(streaming=True)) - - -def projection_pushdown_demo(input_path: Path): - """ - Demonstrate projection pushdown - only read needed columns. - """ - print("\n=== Projection Pushdown ===\n") - - # Only reads 'category' and 'value' columns from disk - lf = pl.scan_csv(input_path) - - result = ( - lf.select("category", "value") # Projection pushdown - .filter(pl.col("value") > 100) # Predicate pushdown - .group_by("category") - .agg(pl.col("value").mean()) - ) - - print("Optimized plan (note: only needed columns read):") - print(result.explain()) - - df = result.collect() - print("\nResult:") - print(df) - - -def main(): - # Create temp directory for demo files - with tempfile.TemporaryDirectory() as tmpdir: - input_csv = Path(tmpdir) / "large_data.csv" - output_parquet = Path(tmpdir) / "filtered_data.parquet" - - # Create sample data - create_sample_data(input_csv, n_rows=100000) - - # Run demos - streaming_aggregation(input_csv) - streaming_sink(input_csv, output_parquet) - check_streaming_compatibility(input_csv) - projection_pushdown_demo(input_csv) - - print("\n=== Summary ===") - print("Key patterns for large files:") - print("1. Use scan_csv / scan_parquet (not read_*)") - print("2. Build lazy query with .filter(), .select(), .group_by()") - print("3. Execute with .collect(engine='streaming')") - print("4. For file-to-file: use .sink_parquet() / .sink_csv()") - print("5. Put filters and projections early in pipeline") - - -if __name__ == "__main__": - main() diff --git a/.cursor/skills/polars-expertise/references/gpu_support.md b/.cursor/skills/polars-expertise/references/gpu_support.md deleted file mode 100644 index 2e7b4a9f..00000000 --- a/.cursor/skills/polars-expertise/references/gpu_support.md +++ /dev/null @@ -1,253 +0,0 @@ -# GPU Support - -Polars provides GPU-accelerated execution for the Lazy API on NVIDIA GPUs using RAPIDS cuDF. Currently in Open Beta. - -## System Requirements - -| Requirement | Specification | -|-------------|---------------| -| GPU | NVIDIA Volta or higher (compute capability 7.0+) | -| CUDA | CUDA 12 (CUDA 11 deprecated - ends with RAPIDS v25.06) | -| OS | Linux or Windows Subsystem for Linux 2 (WSL2) | -| Memory | GPU RAM sufficient for your workload (80GB handles 50-100GB raw data) | - -## Installation - -See SKILL.md for base installation. For GPU support: - -```bash -uv pip install polars[gpu] - -# CUDA 11 (deprecated, ends with RAPIDS v25.08): -uv pip install polars cudf-polars-cu11==25.06 -``` - -## Basic Usage - -### Simple GPU Execution - -```python -import polars as pl - -# Build query with lazy API (required for GPU) -lf = ( - pl.scan_parquet("trades.parquet") - .filter(pl.col("volume") > 1000000) - .group_by("symbol") - .agg( - pl.col("price").mean().alias("avg_price"), - pl.col("volume").sum().alias("total_volume") - ) -) - -# Execute on GPU -result = lf.collect(engine="gpu") -``` - -### GPU Engine Configuration - -```python -# Select specific GPU on multi-GPU system -result = lf.collect(engine=pl.GPUEngine(device=1)) - -# Disable CPU fallback - raises exception if unsupported -result = lf.collect(engine=pl.GPUEngine(raise_on_fail=True)) -``` - -## Supported Operations - -### Supported - -| Category | Operations | -|----------|------------| -| API | LazyFrame, SQL | -| I/O | CSV, Parquet, ndjson, in-memory DataFrames | -| Data Types | Numeric, logical, string, datetime | -| Operations | Filters, aggregations (grouped/rolling), joins, concatenation | -| String | Full string processing | -| Missing Data | Null handling | - -### Not Supported (Falls Back to CPU) - -| Category | Details | -|----------|---------| -| API | Eager DataFrame, Streaming | -| Data Types | Date, Categorical, Enum, Time, Array, Binary, Object | -| Operations | Time series resampling, folds, user-defined functions | -| I/O | Excel, database formats | -| Other | Datetime with timezone (some expressions), List (some expressions) | - -## Diagnosing GPU Usage - -### Verbose Mode - Fallback Warnings - -```python -import polars as pl - -lf = ( - pl.scan_parquet("data.parquet") - .with_columns( - pl.col("value").rolling_mean(10).over("group") # Not GPU-supported - ) -) - -with pl.Config() as cfg: - cfg.set_verbose(True) - result = lf.collect(engine="gpu") - # Prints: PerformanceWarning: Query execution with GPU not supported... -``` - -### Force Failure for Unsupported Operations - -```python -try: - result = lf.collect(engine=pl.GPUEngine(raise_on_fail=True)) -except pl.exceptions.ComputeError as e: - print(f"GPU execution failed: {e}") -``` - -## When to Use GPU - -### GPU Excels At - -| Workload | Reason | -|----------|--------| -| Grouped aggregations | Massive parallelism | -| Large joins | GPU memory bandwidth | -| Heavy computations | CUDA acceleration | -| Multiple operations in single query | Amortizes GPU overhead | - -### CPU May Be Better - -| Workload | Reason | -|----------|--------| -| I/O-bound queries | GPU won't help disk/network bottleneck | -| Small datasets | GPU overhead not worth it | -| Unsupported operations | Frequent fallback adds overhead | -| Data larger than GPU RAM | Out-of-memory errors | - -## Interoperability - -### CPU-GPU Data Flow - -```python -# GPU results are standard CPU DataFrames -gpu_result = lf.collect(engine="gpu") -type(gpu_result) # polars.DataFrame (CPU-backed) - -# Files written by GPU engine readable by CPU -lf.sink_parquet("output.parquet") # Works with both engines -``` - -### Transparent Fallback - -```python -# Fallback is automatic unless raise_on_fail=True -result = lf.collect(engine="gpu") -# If any operation unsupported -> falls back to CPU engine -# Query still completes, just without GPU acceleration -``` - -## Financial Data Patterns - -### VWAP Calculation (GPU-Compatible) - -```python -lf = ( - pl.scan_parquet("trades.parquet") - .group_by("symbol") - .agg( - (pl.col("price") * pl.col("volume")).sum() / pl.col("volume").sum() - ).alias("vwap") -) -result = lf.collect(engine="gpu") -``` - -### Daily Stats (GPU-Compatible) - -```python -lf = ( - pl.scan_parquet("trades.parquet") - .with_columns(pl.col("timestamp").dt.date().alias("date")) - .group_by("symbol", "date") - .agg( - pl.col("price").max().alias("high"), - pl.col("price").min().alias("low"), - pl.col("price").first().alias("open"), - pl.col("price").last().alias("close"), - pl.col("volume").sum().alias("volume") - ) -) -result = lf.collect(engine="gpu") -``` - -### Rolling Statistics (May Fall Back) - -```python -# Grouped rolling window - NOT GPU supported -lf = ( - pl.scan_parquet("trades.parquet") - .with_columns( - pl.col("price").rolling_mean(20).over("symbol") # CPU fallback - ) -) - -# Simple rolling (no grouping) - may work -lf = ( - pl.scan_parquet("trades.parquet") - .with_columns( - pl.col("price").rolling_mean(20) # Check support - ) -) -``` - -## Best Practices - -1. **Profile first**: Compare GPU vs CPU on your actual queries -2. **Keep data on GPU**: Minimize CPU-GPU transfers in a pipeline -3. **Batch queries**: Combine operations to amortize GPU overhead -4. **Check support**: Use `raise_on_fail=True` during development -5. **Monitor memory**: GPU OOM is common with large datasets -6. **Use verbose mode**: Identify fallback operations - -## Troubleshooting - -### ImportError on gpu engine - -```bash -uv pip install polars[gpu] # Missing cudf-polars -``` - -### CUDA version mismatch - -```bash -nvcc --version # Check CUDA version -uv pip install cudf-polars-cu12 # for CUDA 12 -``` - -### Out of Memory - -```python -# Split large queries or use streaming (CPU only) -lf1 = lf.filter(pl.col("date") < "2024-06-01") -lf2 = lf.filter(pl.col("date") >= "2024-06-01") -r1 = lf1.collect(engine="gpu") -r2 = lf2.collect(engine="gpu") -result = pl.concat([r1, r2]) -``` - -### Performance Not Improved - -Check for fallback operations: -```python -with pl.Config() as cfg: - cfg.set_verbose(True) - result = lf.collect(engine="gpu") -# Look for PerformanceWarning messages -``` - -## Test Coverage - -- 99.2% of Polars unit tests pass with CPU fallback -- 88.8% pass without fallback -- Remaining failures mostly involve debug output differences or type variations diff --git a/.cursor/skills/polars-expertise/references/lazy_deep_dive.md b/.cursor/skills/polars-expertise/references/lazy_deep_dive.md deleted file mode 100644 index 7882e94b..00000000 --- a/.cursor/skills/polars-expertise/references/lazy_deep_dive.md +++ /dev/null @@ -1,354 +0,0 @@ -# Lazy Evaluation Deep Dive - -Polars lazy evaluation builds a query graph that gets optimized before execution. This reference covers the optimization internals, query plans, and advanced lazy patterns. - -## Query Optimization Overview - -| Optimization | What It Does | When Applied | -|--------------|--------------|--------------| -| Predicate pushdown | Applies filters at scan level | 1 time | -| Projection pushdown | Selects only needed columns at scan | 1 time | -| Slice pushdown | Loads only required rows (e.g., head/limit) | 1 time | -| Common subplan elimination | Caches shared subtrees/file scans | 1 time | -| Expression simplification | Constant folding, operation substitution | Until fixed point | -| Join ordering | Determines optimal join execution order | 1 time | -| Type coercion | Minimal memory type conversions | Until fixed point | -| Cardinality estimation | Optimal group_by strategy selection | Query-dependent | - -## Query Plans - -### Viewing Plans - -```python -import polars as pl - -lf = ( - pl.scan_csv("trades.csv") - .filter(pl.col("volume") > 1000000) - .select("symbol", "price", "volume") - .group_by("symbol") - .agg(pl.col("price").mean()) -) - -# Non-optimized plan (what you wrote) -print(lf.explain(optimized=False)) - -# Optimized plan (what executes) -print(lf.explain()) - -# Visual graph (requires Graphviz) -lf.show_graph(optimized=True) -``` - -### Reading Query Plans - -Plans read bottom-to-top. Common symbols: -- `sigma` (σ): SELECTION (filter) -- `pi` (π): PROJECTION (column selection) -- `CSV SCAN`: Data source - -**Non-optimized plan:** -``` -FILTER [(col("volume")) > (1000000)] FROM - SELECT [col("symbol"), col("price"), col("volume")] FROM - CSV SCAN trades.csv - PROJECT */6 COLUMNS -``` - -**Optimized plan (predicate + projection pushdown):** -``` -GROUP_BY [col("symbol")] FROM - CSV SCAN trades.csv - PROJECT 3/6 COLUMNS # Only needed columns - SELECTION: [(col("volume")) > (1000000)] # Filter at scan -``` - -## Optimization Details - -### Predicate Pushdown - -Filters move to data source level: - -```python -# Filter happens during CSV read, not after -lf = ( - pl.scan_csv("large_trades.csv") - .filter(pl.col("date") >= "2024-01-01") # Pushed to scan - .filter(pl.col("volume") > 100000) # Also pushed -) -``` - -**Works with:** -- File scans (CSV, Parquet, IPC) -- Joins (pushes to appropriate side) -- Unions - -**Blocked by:** -- Aggregations (filter after group_by cannot push through) -- User-defined functions - -### Projection Pushdown - -Only reads required columns: - -```python -# Only symbol and price columns read from disk -lf = ( - pl.scan_parquet("trades.parquet") # Has 20 columns - .select("symbol", "price") # Only 2 needed - .filter(pl.col("price") > 100) -) -``` - -**Parquet/IPC benefit**: Column-oriented formats skip entire column chunks. - -### Slice Pushdown - -Limits data loading for head/tail operations: - -```python -# Reads only ~100 rows, not entire file -lf = ( - pl.scan_csv("huge.csv") - .filter(pl.col("valid") == True) - .head(100) # Slice pushes to limit rows read -) -``` - -**Works with:** -- `.head()`, `.tail()`, `.slice()` -- `.limit()` in SQL - -### Common Subplan Elimination - -Shared subqueries execute once: - -```python -expensive_lf = ( - pl.scan_parquet("market_data.parquet") - .filter(pl.col("date") >= "2024-01-01") - .with_columns( - pl.col("price").rolling_mean(20).alias("ma_20") - ) -) - -# Both branches share expensive_lf - computed once -summary = expensive_lf.group_by("symbol").agg(pl.col("price").mean()) -detail = expensive_lf.filter(pl.col("price") > pl.col("ma_20")) - -# collect_all ensures single computation -results = pl.collect_all([summary, detail]) -``` - -## Execution Modes - -### Standard Collection - -```python -# Full dataset in memory -df = lf.collect() -``` - -### Streaming Mode - -Processes data in batches for larger-than-memory datasets: - -```python -# Streaming execution -df = lf.collect(engine="streaming") -``` - -**Inspecting streaming plans:** -```python -# Physical plan shows memory intensity -lf.show_graph(streaming=True) -``` - -**Streaming-compatible operations:** -- Scans, filters, projections -- Most aggregations -- Sorted joins - -**Not streaming-compatible (require full materialization):** -- Unsorted joins -- Some window functions -- Sort on unsorted data - -### GPU Execution - -```python -# Execute on NVIDIA GPU (requires polars[gpu]) -df = lf.collect(engine="gpu") -``` - -### Partial Execution - -For development/debugging on large datasets: - -```python -# Sample during development -df = lf.head(1000).collect() - -# Or limit at scan -lf = pl.scan_parquet("huge.parquet").head(10000) -result = lf.filter(...).collect() -``` - -## Diverging Queries Pattern - -When one lazy computation feeds multiple downstream queries: - -```python -# Base expensive computation -base_lf = ( - pl.scan_parquet("trades.parquet") - .with_columns( - pl.col("price").pct_change().alias("returns") - ) -) - -# Diverging queries -stats_lf = base_lf.group_by("symbol").agg( - pl.col("returns").mean(), - pl.col("returns").std() -) - -filtered_lf = base_lf.filter(pl.col("returns").abs() > 0.05) - -# CRITICAL: Use collect_all to avoid recomputation -stats_df, filtered_df = pl.collect_all([stats_lf, filtered_lf]) -# base_lf computed only once! - -# BAD: Separate collects recompute base_lf each time -# stats_df = stats_lf.collect() # Computes base_lf -# filtered_df = filtered_lf.collect() # Computes base_lf AGAIN -``` - -## LazyFrame Caching Gotchas - -LazyFrames are query plans, not cached data: - -```python -# WARNING: This recomputes on each use -lf = pl.scan_parquet("data.parquet").filter(...) - -result1 = lf.select("a").collect() # Scans + filters -result2 = lf.select("b").collect() # Scans + filters AGAIN - -# SOLUTION 1: Collect once, then operate on DataFrame -df = lf.collect() -result1 = df.select("a") -result2 = df.select("b") - -# SOLUTION 2: Use collect_all for lazy branches -lf1 = lf.select("a") -lf2 = lf.select("b") -result1, result2 = pl.collect_all([lf1, lf2]) # Single scan -``` - -## Advanced Patterns - -### Lazy Schema Inspection - -```python -lf = pl.scan_parquet("data.parquet") - -# Get schema without loading data -print(lf.collect_schema()) - -# Get column names -print(lf.collect_schema().names()) -``` - -### Lazy with Sink Operations - -Write results directly to files without full materialization: - -```python -# Sink to Parquet (streaming write) -lf.sink_parquet("output.parquet") - -# Sink to IPC -lf.sink_ipc("output.ipc") - -# Sink to CSV -lf.sink_csv("output.csv") -``` - -### Profile Query Execution - -```python -# Returns DataFrame with timing info -df, profile = lf.profile() -print(profile) -``` - -### Explain Physical Plan - -```python -# Logical plan (default) -print(lf.explain()) - -# Physical plan with more detail -print(lf.explain(physical=True)) -``` - -## Optimization Control - -### Disable Specific Optimizations - -```python -# For debugging or specific requirements -df = lf.collect( - predicate_pushdown=False, # Keep filters where written - projection_pushdown=False, # Read all columns - slice_pushdown=False, # No slice optimization - comm_subplan_elim=False # No subplan caching -) -``` - -### Force Optimization Barrier - -```python -# cache() materializes intermediate result -lf = ( - pl.scan_parquet("data.parquet") - .filter(...) - .cache() # Forces materialization here - .group_by(...) - .agg(...) -) -``` - -## Rust Lazy API - -```rust -use polars::prelude::*; - -fn main() -> PolarsResult<()> { - let lf = LazyCsvReader::new("trades.csv") - .finish()? - .filter(col("volume").gt(lit(1000000))) - .select([col("symbol"), col("price")]) - .group_by([col("symbol")]) - .agg([col("price").mean()]); - - // View plan - println!("{}", lf.explain(true)?); - - // Execute - let df = lf.collect()?; - - Ok(()) -} -``` - -## Best Practices - -1. **Start lazy, end eager**: Use `scan_*` functions, collect only when needed -2. **Check plans**: Use `explain()` to verify optimizations work -3. **Use `collect_all`**: For diverging queries from same source -4. **Sink for ETL**: Use `sink_*` for write-heavy pipelines -5. **Profile in production**: Use `.profile()` to find bottlenecks -6. **Streaming for big data**: Set `engine="streaming"` for larger-than-memory -7. **Don't reuse LazyFrames**: They recompute each time - use `collect_all` or materialize diff --git a/.cursor/skills/polars-expertise/references/migration_pandas.md b/.cursor/skills/polars-expertise/references/migration_pandas.md deleted file mode 100644 index 947ba453..00000000 --- a/.cursor/skills/polars-expertise/references/migration_pandas.md +++ /dev/null @@ -1,418 +0,0 @@ -# Pandas to Polars Migration Guide - -This guide helps you migrate from pandas to Polars with comprehensive operation mappings and key differences. - -## Core Conceptual Differences - -### 1. No Index System - -**Pandas:** Uses row-based indexing system -```python -df.loc[0, "column"] -df.iloc[0:5] -df.set_index("id") -``` - -**Polars:** Uses integer positions only -```python -df[0, "column"] # Row position, column name -df[0:5] # Row slice -# No set_index equivalent - use group_by instead -``` - -### 2. Memory Format - -**Pandas:** Row-oriented NumPy arrays -**Polars:** Columnar Apache Arrow format - -**Implications:** -- Polars is faster for column operations -- Polars uses less memory -- Polars has better data sharing capabilities - -### 3. Parallelization - -**Pandas:** Primarily single-threaded (requires Dask for parallelism) -**Polars:** Parallel by default using Rust's concurrency - -### 4. Lazy Evaluation - -**Pandas:** Only eager evaluation -**Polars:** Both eager (DataFrame) and lazy (LazyFrame) with query optimization - -### 5. Type Strictness - -**Pandas:** Allows silent type conversions -**Polars:** Strict typing, explicit casts required - -**Example:** -```python -# Pandas: Silently converts to float -pd_df["int_col"] = [1, 2, None, 4] # dtype: float64 - -# Polars: Keeps as integer with null -pl_df = pl.DataFrame({"int_col": [1, 2, None, 4]}) # dtype: Int64 -``` - -## Operation Mappings - -### Data Selection - -| Operation | Pandas | Polars | -|-----------|--------|--------| -| Select column | `df["col"]` or `df.col` | `df.select("col")` or `df["col"]` | -| Select multiple | `df[["a", "b"]]` | `df.select("a", "b")` | -| Select by position | `df.iloc[:, 0:3]` | `df.select(pl.col(df.columns[0:3]))` | -| Select by condition | `df[df["age"] > 25]` | `df.filter(pl.col("age") > 25)` | - -### Data Filtering - -| Operation | Pandas | Polars | -|-----------|--------|--------| -| Single condition | `df[df["age"] > 25]` | `df.filter(pl.col("age") > 25)` | -| Multiple conditions | `df[(df["age"] > 25) & (df["city"] == "NY")]` | `df.filter(pl.col("age") > 25, pl.col("city") == "NY")` | -| Query method | `df.query("age > 25")` | `df.filter(pl.col("age") > 25)` | -| isin | `df[df["city"].isin(["NY", "LA"])]` | `df.filter(pl.col("city").is_in(["NY", "LA"]))` | -| isna | `df[df["value"].isna()]` | `df.filter(pl.col("value").is_null())` | -| notna | `df[df["value"].notna()]` | `df.filter(pl.col("value").is_not_null())` | - -### Adding/Modifying Columns - -| Operation | Pandas | Polars | -|-----------|--------|--------| -| Add column | `df["new"] = df["old"] * 2` | `df.with_columns((pl.col("old") * 2).alias("new"))` | -| Multiple columns | `df.assign(a=..., b=...)` | `df.with_columns(a=..., b=...)` | -| Conditional column | `np.where(condition, a, b)` | `pl.when(condition).then(a).otherwise(b)` | - -**Important difference - Parallel execution:** - -```python -# Pandas: Sequential (lambda sees previous results) -df.assign( - a=lambda df_: df_.value * 10, - b=lambda df_: df_.value * 100 -) - -# Polars: Parallel (all computed together) -df.with_columns( - (pl.col("value") * 10).alias("a"), - (pl.col("value") * 100).alias("b") -) -``` - -### Grouping and Aggregation - -| Operation | Pandas | Polars | -|-----------|--------|--------| -| Group by | `df.groupby("col")` | `df.group_by("col")` | -| Agg single | `df.groupby("col")["val"].mean()` | `df.group_by("col").agg(pl.col("val").mean())` | -| Agg multiple | `df.groupby("col").agg({"val": ["mean", "sum"]})` | `df.group_by("col").agg(pl.col("val").mean(), pl.col("val").sum())` | -| Size | `df.groupby("col").size()` | `df.group_by("col").agg(pl.len())` | -| Count | `df.groupby("col").count()` | `df.group_by("col").agg(pl.col("*").count())` | - -### Window Functions - -| Operation | Pandas | Polars | -|-----------|--------|--------| -| Transform | `df.groupby("col").transform("mean")` | `df.with_columns(pl.col("val").mean().over("col"))` | -| Rank | `df.groupby("col")["val"].rank()` | `df.with_columns(pl.col("val").rank().over("col"))` | -| Shift | `df.groupby("col")["val"].shift(1)` | `df.with_columns(pl.col("val").shift(1).over("col"))` | -| Cumsum | `df.groupby("col")["val"].cumsum()` | `df.with_columns(pl.col("val").cum_sum().over("col"))` | - -### Joins - -| Operation | Pandas | Polars | -|-----------|--------|--------| -| Inner join | `df1.merge(df2, on="id")` | `df1.join(df2, on="id", how="inner")` | -| Left join | `df1.merge(df2, on="id", how="left")` | `df1.join(df2, on="id", how="left")` | -| Different keys | `df1.merge(df2, left_on="a", right_on="b")` | `df1.join(df2, left_on="a", right_on="b")` | - -### Concatenation - -| Operation | Pandas | Polars | -|-----------|--------|--------| -| Vertical | `pd.concat([df1, df2], axis=0)` | `pl.concat([df1, df2], how="vertical")` | -| Horizontal | `pd.concat([df1, df2], axis=1)` | `pl.concat([df1, df2], how="horizontal")` | - -### Sorting - -| Operation | Pandas | Polars | -|-----------|--------|--------| -| Sort by column | `df.sort_values("col")` | `df.sort("col")` | -| Descending | `df.sort_values("col", ascending=False)` | `df.sort("col", descending=True)` | -| Multiple columns | `df.sort_values(["a", "b"])` | `df.sort("a", "b")` | - -### Reshaping - -| Operation | Pandas | Polars | -|-----------|--------|--------| -| Pivot | `df.pivot(index="a", columns="b", values="c")` | `df.pivot(values="c", index="a", columns="b")` | -| Melt | `df.melt(id_vars="id")` | `df.unpivot(index="id")` | - -### I/O Operations - -| Operation | Pandas | Polars | -|-----------|--------|--------| -| Read CSV | `pd.read_csv("file.csv")` | `pl.read_csv("file.csv")` or `pl.scan_csv()` | -| Write CSV | `df.to_csv("file.csv")` | `df.write_csv("file.csv")` | -| Read Parquet | `pd.read_parquet("file.parquet")` | `pl.read_parquet("file.parquet")` | -| Write Parquet | `df.to_parquet("file.parquet")` | `df.write_parquet("file.parquet")` | -| Read Excel | `pd.read_excel("file.xlsx")` | `pl.read_excel("file.xlsx")` | - -### String Operations - -| Operation | Pandas | Polars | -|-----------|--------|--------| -| Upper | `df["col"].str.upper()` | `df.select(pl.col("col").str.to_uppercase())` | -| Lower | `df["col"].str.lower()` | `df.select(pl.col("col").str.to_lowercase())` | -| Contains | `df["col"].str.contains("pattern")` | `df.filter(pl.col("col").str.contains("pattern"))` | -| Replace | `df["col"].str.replace("old", "new")` | `df.select(pl.col("col").str.replace("old", "new"))` | -| Split | `df["col"].str.split(" ")` | `df.select(pl.col("col").str.split(" "))` | - -### Datetime Operations - -| Operation | Pandas | Polars | -|-----------|--------|--------| -| Parse dates | `pd.to_datetime(df["col"])` | `df.select(pl.col("col").str.strptime(pl.Date, "%Y-%m-%d"))` | -| Year | `df["date"].dt.year` | `df.select(pl.col("date").dt.year())` | -| Month | `df["date"].dt.month` | `df.select(pl.col("date").dt.month())` | -| Day | `df["date"].dt.day` | `df.select(pl.col("date").dt.day())` | - -### Missing Data - -| Operation | Pandas | Polars | -|-----------|--------|--------| -| Drop nulls | `df.dropna()` | `df.drop_nulls()` | -| Fill nulls | `df.fillna(0)` | `df.fill_null(0)` | -| Check null | `df["col"].isna()` | `df.select(pl.col("col").is_null())` | -| Forward fill | `df.fillna(method="ffill")` | `df.select(pl.col("col").fill_null(strategy="forward"))` | - -### Other Operations - -| Operation | Pandas | Polars | -|-----------|--------|--------| -| Unique values | `df["col"].unique()` | `df["col"].unique()` | -| Value counts | `df["col"].value_counts()` | `df["col"].value_counts()` | -| Describe | `df.describe()` | `df.describe()` | -| Sample | `df.sample(n=100)` | `df.sample(n=100)` | -| Head | `df.head()` | `df.head()` | -| Tail | `df.tail()` | `df.tail()` | - -## Common Migration Patterns - -### Pattern 1: Chained Operations - -**Pandas:** -```python -result = (df - .assign(new_col=lambda x: x["old_col"] * 2) - .query("new_col > 10") - .groupby("category") - .agg({"value": "sum"}) - .reset_index() -) -``` - -**Polars:** -```python -result = (df - .with_columns((pl.col("old_col") * 2).alias("new_col")) - .filter(pl.col("new_col") > 10) - .group_by("category") - .agg(pl.col("value").sum()) -) -# No reset_index needed - Polars doesn't have index -``` - -### Pattern 2: Apply Functions - -**Pandas:** -```python -# Avoid in Polars - breaks parallelization -df["result"] = df["value"].apply(lambda x: x * 2) -``` - -**Polars:** -```python -# Use expressions instead -df = df.with_columns((pl.col("value") * 2).alias("result")) - -# If custom function needed -df = df.with_columns( - pl.col("value").map_elements(lambda x: x * 2, return_dtype=pl.Float64).alias("result") -) -``` - -### Pattern 3: Conditional Column Creation - -**Pandas:** -```python -df["category"] = np.where( - df["value"] > 100, - "high", - np.where(df["value"] > 50, "medium", "low") -) -``` - -**Polars:** -```python -df = df.with_columns( - pl.when(pl.col("value") > 100) - .then("high") - .when(pl.col("value") > 50) - .then("medium") - .otherwise("low") - .alias("category") -) -``` - -### Pattern 4: Group Transform - -**Pandas:** -```python -df["group_mean"] = df.groupby("category")["value"].transform("mean") -``` - -**Polars:** -```python -df = df.with_columns( - pl.col("value").mean().over("category").alias("group_mean") -) -``` - -### Pattern 5: Multiple Aggregations - -**Pandas:** -```python -result = df.groupby("category").agg({ - "value": ["mean", "sum", "count"], - "price": ["min", "max"] -}) -``` - -**Polars:** -```python -result = df.group_by("category").agg( - pl.col("value").mean().alias("value_mean"), - pl.col("value").sum().alias("value_sum"), - pl.col("value").count().alias("value_count"), - pl.col("price").min().alias("price_min"), - pl.col("price").max().alias("price_max") -) -``` - -## Performance Anti-Patterns to Avoid - -### Anti-Pattern 1: Sequential Pipe Operations - -**Bad (disables parallelization):** -```python -df = df.pipe(function1).pipe(function2).pipe(function3) -``` - -**Good (enables parallelization):** -```python -df = df.with_columns( - function1_result(), - function2_result(), - function3_result() -) -``` - -### Anti-Pattern 2: Python Functions in Hot Paths - -**Bad:** -```python -df = df.with_columns( - pl.col("value").map_elements(lambda x: x * 2).alias("result") -) -``` - -**Good:** -```python -df = df.with_columns((pl.col("value") * 2).alias("result")) -``` - -### Anti-Pattern 3: Using Eager Reading for Large Files - -**Bad:** -```python -df = pl.read_csv("large_file.csv") -result = df.filter(pl.col("age") > 25).select("name", "age") -``` - -**Good:** -```python -lf = pl.scan_csv("large_file.csv") -result = lf.filter(pl.col("age") > 25).select("name", "age").collect() -``` - -### Anti-Pattern 4: Row Iteration - -**Bad:** -```python -for row in df.iter_rows(): - # Process row - pass -``` - -**Good:** -```python -# Use vectorized operations -df = df.with_columns( - # Vectorized computation -) -``` - -## Migration Checklist - -When migrating from pandas to Polars: - -1. **Remove index operations** - Use integer positions or group_by -2. **Replace apply/map with expressions** - Use Polars native operations -3. **Update column assignment** - Use `with_columns()` instead of direct assignment -4. **Change groupby.transform to .over()** - Window functions work differently -5. **Update string operations** - Use `.str.to_uppercase()` instead of `.str.upper()` -6. **Add explicit type casts** - Polars won't silently convert types -7. **Consider lazy evaluation** - Use `scan_*` instead of `read_*` for large data -8. **Update aggregation syntax** - More explicit in Polars -9. **Remove reset_index calls** - Not needed in Polars -10. **Update conditional logic** - Use `when().then().otherwise()` pattern - -## Compatibility Layer - -For gradual migration, you can use both libraries: - -```python -import pandas as pd -import polars as pl - -# Convert pandas to Polars -pl_df = pl.from_pandas(pd_df) - -# Convert Polars to pandas -pd_df = pl_df.to_pandas() - -# Use Arrow for zero-copy (when possible) -pl_df = pl.from_arrow(pd_df) -pd_df = pl_df.to_arrow().to_pandas() -``` - -## When to Stick with Pandas - -Consider staying with pandas when: -- Working with time series requiring complex index operations -- Need extensive ecosystem support (some libraries only support pandas) -- Team lacks Rust/Polars expertise -- Data is small and performance isn't critical -- Using advanced pandas features without Polars equivalents - -## When to Switch to Polars - -Switch to Polars when: -- Performance is critical -- Working with large datasets (>1GB) -- Need lazy evaluation and query optimization -- Want better type safety -- Need parallel execution by default -- Starting a new project diff --git a/.cursor/skills/polars-expertise/references/migration_qkdb.md b/.cursor/skills/polars-expertise/references/migration_qkdb.md deleted file mode 100644 index 62045e3a..00000000 --- a/.cursor/skills/polars-expertise/references/migration_qkdb.md +++ /dev/null @@ -1,411 +0,0 @@ -# q/kdb+ to Polars Migration Guide - -This guide helps you migrate from q/kdb+ to Polars, covering vector operations, table manipulations, and financial data patterns common in trading systems. - -## Philosophical Differences - -### q/kdb+ Philosophy -- Right-to-left evaluation -- Implicit iteration over lists -- Tables are lists of dictionaries -- Column-oriented with native time-series support -- Interpreted, terse syntax - -### Polars Philosophy -- Left-to-right chained operations -- Explicit expression API -- DataFrame as collection of Series (columns) -- Column-oriented with Apache Arrow -- Compiled Rust backend, Python/Rust frontend - -## Core Type Mappings - -| q Type | Polars Type | Notes | -|--------|-------------|-------| -| `boolean` / `b` | `pl.Boolean` | | -| `byte` / `x` | `pl.UInt8` | | -| `short` / `h` | `pl.Int16` | | -| `int` / `i` | `pl.Int32` | | -| `long` / `j` | `pl.Int64` | Default integer | -| `real` / `e` | `pl.Float32` | | -| `float` / `f` | `pl.Float64` | Default float | -| `char` / `c` | `pl.String` | Single chars become strings | -| `symbol` / `s` | `pl.Categorical` or `pl.String` | Use Categorical for low-cardinality | -| `timestamp` / `p` | `pl.Datetime("ns")` | Nanosecond precision | -| `date` / `d` | `pl.Date` | | -| `time` / `t` | `pl.Time` | | -| `timespan` / `n` | `pl.Duration` | | - -## Vector Operation Mappings - -### Basic Arithmetic - -| q | Polars | Notes | -|---|--------|-------| -| `x + y` | `pl.col("x") + pl.col("y")` | | -| `x * y` | `pl.col("x") * pl.col("y")` | | -| `sum x` | `pl.col("x").sum()` | | -| `avg x` | `pl.col("x").mean()` | | -| `max x` | `pl.col("x").max()` | | -| `min x` | `pl.col("x").min()` | | -| `med x` | `pl.col("x").median()` | | -| `dev x` | `pl.col("x").std()` | Standard deviation | -| `var x` | `pl.col("x").var()` | Variance | - -### List/Array Operations - -| q | Polars | Notes | -|---|--------|-------| -| `count x` | `pl.col("x").len()` | Length | -| `first x` | `pl.col("x").first()` | | -| `last x` | `pl.col("x").last()` | | -| `reverse x` | `pl.col("x").reverse()` | | -| `asc x` | `pl.col("x").sort()` | Ascending sort | -| `desc x` | `pl.col("x").sort(descending=True)` | | -| `distinct x` | `pl.col("x").unique()` | | -| `x?y` | `pl.col("x").search_sorted(y)` | Find index (binary search) | -| `x in y` | `pl.col("x").is_in(y)` | Membership | - -### Running/Cumulative Operations - -| q | Polars | Notes | -|---|--------|-------| -| `sums x` | `pl.col("x").cum_sum()` | Running sum | -| `prds x` | `pl.col("x").cum_prod()` | Running product | -| `maxs x` | `pl.col("x").cum_max()` | Running max | -| `mins x` | `pl.col("x").cum_min()` | Running min | - -### Sliding Windows - -| q | Polars | Notes | -|---|--------|-------| -| `mavg[n;x]` | `pl.col("x").rolling_mean(n)` | Moving average | -| `msum[n;x]` | `pl.col("x").rolling_sum(n)` | Moving sum | -| `mmax[n;x]` | `pl.col("x").rolling_max(n)` | Moving max | -| `mmin[n;x]` | `pl.col("x").rolling_min(n)` | Moving min | -| `mdev[n;x]` | `pl.col("x").rolling_std(n)` | Moving std dev | - -### Deltas and Differences - -| q | Polars | Notes | -|---|--------|-------| -| `deltas x` | `pl.col("x").diff()` | First differences | -| `ratios x` | `pl.col("x") / pl.col("x").shift(1)` | Ratio to previous | -| `1 _ x` | `pl.col("x").slice(1)` | Drop first | -| `-1 _ x` | `pl.col("x").head(-1)` | Drop last | -| `n # x` | `pl.col("x").head(n)` | Take first n | -| `-n # x` | `pl.col("x").tail(n)` | Take last n | - -### Prev/Next - -| q | Polars | Notes | -|---|--------|-------| -| `prev x` | `pl.col("x").shift(1)` | Previous value | -| `next x` | `pl.col("x").shift(-1)` | Next value | -| `xprev[n;x]` | `pl.col("x").shift(n)` | N periods back | - -## Table Operations - -### Creating Tables - -**q:** -```q -t:([] sym:`AAPL`MSFT`AAPL; price:150.0 280.0 152.0; size:100 200 150) -``` - -**Polars:** -```python -t = pl.DataFrame({ - "sym": ["AAPL", "MSFT", "AAPL"], - "price": [150.0, 280.0, 152.0], - "size": [100, 200, 150] -}) -``` - -### Selection (qSQL vs Polars) - -**q:** -```q -select sym, price from t where size > 100 -``` - -**Polars:** -```python -t.filter(pl.col("size") > 100).select("sym", "price") -``` - -### Aggregation - -**q:** -```q -select avg price, sum size by sym from t -``` - -**Polars:** -```python -t.group_by("sym").agg( - pl.col("price").mean(), - pl.col("size").sum() -) -``` - -### Update (Adding Columns) - -**q:** -```q -update vwap: size wavg price by sym from t -``` - -**Polars:** -```python -t.with_columns( - vwap=(pl.col("price") * pl.col("size")).sum().over("sym") / - pl.col("size").sum().over("sym") -) -``` - -### Delete (Dropping Rows) - -**q:** -```q -delete from t where size < 100 -``` - -**Polars:** -```python -t.filter(pl.col("size") >= 100) -``` - -### Sorting - -**q:** -```q -`sym`price xasc t -``` - -**Polars:** -```python -t.sort("sym", "price") -``` - -### Joins - -| q | Polars | Notes | -|---|--------|-------| -| `t1 lj t2` | `t1.join(t2, on="key", how="left")` | Left join | -| `t1 ij t2` | `t1.join(t2, on="key", how="inner")` | Inner join | -| `t1 uj t2` | `pl.concat([t1, t2])` | Union (vertical) | -| `t1 aj \`time\`sym t2` | `t1.join_asof(t2, on="time", by="sym")` | As-of join | -| `t1 wj ...` | `t1.join_asof(...).with_columns(...)` | Window join (manual) | - -## Financial Data Patterns - -### OHLCV Bars from Ticks - -**q:** -```q -select o:first price, h:max price, l:min price, c:last price, v:sum size -by sym, time.minute from trades -``` - -**Polars:** -```python -trades.group_by("sym", pl.col("time").dt.truncate("1m")).agg( - pl.col("price").first().alias("o"), - pl.col("price").max().alias("h"), - pl.col("price").min().alias("l"), - pl.col("price").last().alias("c"), - pl.col("size").sum().alias("v") -) -``` - -### VWAP - -**q:** -```q -select vwap: size wavg price by sym from trades -``` - -**Polars:** -```python -trades.group_by("sym").agg( - vwap=(pl.col("price") * pl.col("size")).sum() / pl.col("size").sum() -) -``` - -### Intraday Returns - -**q:** -```q -update ret: 1 - price % prev price by sym from trades -``` - -**Polars:** -```python -trades.with_columns( - ret=(pl.col("price") / pl.col("price").shift(1) - 1).over("sym") -) -``` - -### Rolling Volatility - -**q:** -```q -update vol: mdev[20; log price - log prev price] by sym from trades -``` - -**Polars:** -```python -trades.with_columns( - pl.col("price").log().diff().rolling_std(20).over("sym").alias("vol") -) -``` - -### As-of Joins (Quote/Trade Matching) - -**q:** -```q -aj[`sym`time; trades; quotes] -``` - -**Polars:** -```python -trades.sort("time").join_asof( - quotes.sort("time"), - on="time", - by="sym", - strategy="backward" # Last quote before trade -) -``` - -### Time-Weighted Average - -**q:** -```q -update twap: (deltas time) wavg price by sym from quotes -``` - -**Polars:** -```python -quotes.with_columns( - pl.col("time").diff().over("sym").alias("dt") -).with_columns( - twap=(pl.col("price") * pl.col("dt")).sum().over("sym") / - pl.col("dt").sum().over("sym") -) -``` - -## Performance Considerations - -### What q/kdb+ Does Better -- Native nanosecond timestamps with timezone -- Built-in IPC and pub/sub -- Extremely fast as-of joins on sorted data -- Integrated timeseries database (kdb+) -- Lower memory footprint for timeseries - -### What Polars Does Better -- Complex query optimization (predicate pushdown) -- Multi-threaded by default -- Better for ad-hoc analysis -- Larger ecosystem (Python) -- No licensing costs -- Better for batch processing - -### Memory Comparison - -q/kdb+ uses compact in-memory representation optimized for timeseries. Polars uses Arrow format optimized for analytics. For dense timeseries with many symbols, kdb+ may use less memory. For wide tables with mixed types, they're comparable. - -### Speed Expectations - -| Operation | Relative Performance | -|-----------|---------------------| -| Simple aggregations | Polars often faster (multi-threaded) | -| As-of joins (sorted) | q/kdb+ faster | -| Complex multi-step queries | Polars faster (optimization) | -| Tick data ingestion | kdb+ faster (built-in) | -| Ad-hoc analysis | Polars faster (better optimization) | - -## Migration Strategy - -### Phase 1: Batch Analytics -Move batch analytics (EOD processing, reporting) to Polars first. Keep kdb+ for real-time. - -### Phase 2: Historical Analysis -Use Polars for historical backtesting and research. Export kdb+ data to Parquet. - -### Phase 3: Evaluate Real-time -Consider Polars streaming for lower-frequency real-time needs. Keep kdb+ for HFT. - -## Code Translation Examples - -### Example 1: Daily Stats - -**q:** -```q -select - open: first price, - high: max price, - low: min price, - close: last price, - volume: sum size, - trades: count i -by sym, date from trades -``` - -**Polars:** -```python -( - trades - .group_by("sym", pl.col("timestamp").dt.date().alias("date")) - .agg( - pl.col("price").first().alias("open"), - pl.col("price").max().alias("high"), - pl.col("price").min().alias("low"), - pl.col("price").last().alias("close"), - pl.col("size").sum().alias("volume"), - pl.len().alias("trades") - ) -) -``` - -### Example 2: Moving Spread - -**q:** -```q -update spread: mavg[100; ask - bid] by sym from quotes -``` - -**Polars:** -```python -quotes.with_columns( - spread=(pl.col("ask") - pl.col("bid")).rolling_mean(100).over("sym") -) -``` - -### Example 3: Fill Forward - -**q:** -```q -update price: fills price by sym from quotes -``` - -**Polars:** -```python -quotes.with_columns( - pl.col("price").forward_fill().over("sym").alias("price") -) -``` - -## Key Syntax Differences Summary - -| q Pattern | Polars Pattern | -|-----------|----------------| -| Right-to-left: `avg x` | Left-to-right: `pl.col("x").mean()` | -| Implicit iteration | Explicit `.over()` for groups | -| `select ... by` | `.group_by().agg()` | -| `update col: expr` | `.with_columns(col=expr)` | -| `x?y` (find) | `.search_sorted()` or `.filter()` | -| `x wavg y` | `(x * y).sum() / y.sum()` | -| `fills x` | `.forward_fill()` | -| `aj` | `.join_asof()` | diff --git a/.cursor/skills/polars-expertise/references/migration_spark.md b/.cursor/skills/polars-expertise/references/migration_spark.md deleted file mode 100644 index 635d1d00..00000000 --- a/.cursor/skills/polars-expertise/references/migration_spark.md +++ /dev/null @@ -1,295 +0,0 @@ -# PySpark to Polars Migration Guide - -This guide helps you migrate from PySpark to Polars, covering fundamental differences and common operation mappings. - -## Core Architectural Differences - -### Row-Based vs Column-Based - -**Spark**: DataFrame is a collection of rows - operations preserve row relationships -**Polars**: DataFrame is a collection of columns - columns can be computed independently - -```python -# Polars: Independent column operations -df.select( - pl.col("foo").sort().head(2), # Sorts foo, takes first 2 - pl.col("bar").filter(pl.col("x") > 0).sum() # Filters bar, sums -) -# Result: 2 rows with foo values paired with single bar sum (broadcast) - -# Spark: Operations must maintain row alignment -# Requires separate computations and explicit joins -``` - -### Execution Model - -| Aspect | Spark | Polars | -|--------|-------|--------| -| Execution | Distributed across cluster | Single-machine, parallel threads | -| Lazy by default | Yes (transformations) | No (use LazyFrame explicitly) | -| Memory | Spills to disk | In-memory (streaming for large data) | -| Partitioning | Explicit partition management | Automatic parallelism | -| Fault tolerance | Checkpoint/recompute | None (single machine) | - -### When to Use Which - -**Use Polars when:** -- Data fits on a single machine (even with streaming) -- Need lowest latency -- Don't need cluster infrastructure -- Interactive analysis - -**Use Spark when:** -- Data is truly distributed (multiple TB) -- Need fault tolerance -- Existing Spark infrastructure -- Complex distributed operations - -## Operation Mappings - -### Reading Data - -| Operation | PySpark | Polars | -|-----------|---------|--------| -| Read CSV | `spark.read.csv("file.csv")` | `pl.read_csv("file.csv")` | -| Read CSV lazy | `spark.read.csv("file.csv")` | `pl.scan_csv("file.csv")` | -| Read Parquet | `spark.read.parquet("file.parquet")` | `pl.read_parquet("file.parquet")` | -| Read Parquet lazy | `spark.read.parquet("file.parquet")` | `pl.scan_parquet("file.parquet")` | -| Read multiple files | `spark.read.parquet("path/*.parquet")` | `pl.scan_parquet("path/*.parquet")` | - -### Column Selection - -| Operation | PySpark | Polars | -|-----------|---------|--------| -| Select columns | `df.select("a", "b")` | `df.select("a", "b")` | -| Select with expr | `df.select(col("a"), col("b") * 2)` | `df.select(pl.col("a"), pl.col("b") * 2)` | -| Rename | `df.withColumnRenamed("old", "new")` | `df.rename({"old": "new"})` | -| Drop columns | `df.drop("a", "b")` | `df.drop("a", "b")` | - -### Filtering - -| Operation | PySpark | Polars | -|-----------|---------|--------| -| Filter | `df.filter(col("a") > 5)` | `df.filter(pl.col("a") > 5)` | -| Where (alias) | `df.where(col("a") > 5)` | `df.filter(pl.col("a") > 5)` | -| Multiple conditions | `df.filter((col("a") > 5) & (col("b") < 10))` | `df.filter((pl.col("a") > 5) & (pl.col("b") < 10))` | -| Is in | `df.filter(col("a").isin([1,2,3]))` | `df.filter(pl.col("a").is_in([1,2,3]))` | -| Is null | `df.filter(col("a").isNull())` | `df.filter(pl.col("a").is_null())` | - -### Adding/Modifying Columns - -| Operation | PySpark | Polars | -|-----------|---------|--------| -| Add column | `df.withColumn("new", col("a") * 2)` | `df.with_columns((pl.col("a") * 2).alias("new"))` | -| Multiple columns | `df.withColumn("a", ...).withColumn("b", ...)` | `df.with_columns(a=..., b=...)` | -| Conditional | `df.withColumn("x", when(cond, a).otherwise(b))` | `df.with_columns(pl.when(cond).then(a).otherwise(b).alias("x"))` | - -### Aggregation - -| Operation | PySpark | Polars | -|-----------|---------|--------| -| Group by | `df.groupBy("a")` | `df.group_by("a")` | -| Agg single | `df.groupBy("a").agg({"b": "sum"})` | `df.group_by("a").agg(pl.col("b").sum())` | -| Agg multiple | `df.groupBy("a").agg(sum("b"), avg("c"))` | `df.group_by("a").agg(pl.col("b").sum(), pl.col("c").mean())` | -| Count | `df.groupBy("a").count()` | `df.group_by("a").len()` | - -### Window Functions - -**Spark** requires explicit window specification: - -```python -from pyspark.sql import Window -from pyspark.sql.functions import row_number, lag, mean - -window = Window.partitionBy("symbol").orderBy("date") -rolling_window = window.rowsBetween(-6, 0) - -df = ( - df - .withColumn("rank", row_number().over(window)) - .withColumn("prev_price", lag("price", 1).over(window)) - .withColumn("rolling_mean", mean("price").over(rolling_window)) -) -``` - -**Polars** uses `.over()` for partitioning: - -```python -df = df.with_columns( - pl.col("price").rank().over("symbol").alias("rank"), - pl.col("price").shift(1).over("symbol").alias("prev_price"), - pl.col("price").rolling_mean(7).over("symbol").alias("rolling_mean") -) -``` - -### Composing Window Expressions - -**Spark limitation**: Cannot compose window functions - -```python -# Spark: NOT ALLOWED - lag is a window function -F.mean(F.lag("price", 1)).over(window) # Error - -# Spark workaround: Multiple windows -df = ( - df - .withColumn("lagged_price", F.lag("price", 7).over(window)) - .withColumn("feature", F.mean("lagged_price").over(rolling_window)) -) -``` - -**Polars**: Window expressions can be composed freely - -```python -# Polars: This works - compose shift and rolling_mean -df = df.with_columns( - pl.col("price").shift(7).rolling_mean(7).over("symbol", order_by="date").alias("feature") -) -``` - -### Joins - -| Operation | PySpark | Polars | -|-----------|---------|--------| -| Inner join | `df1.join(df2, on="id", how="inner")` | `df1.join(df2, on="id", how="inner")` | -| Left join | `df1.join(df2, on="id", how="left")` | `df1.join(df2, on="id", how="left")` | -| Different keys | `df1.join(df2, df1.a == df2.b)` | `df1.join(df2, left_on="a", right_on="b")` | - -### Sorting - -| Operation | PySpark | Polars | -|-----------|---------|--------| -| Sort ascending | `df.orderBy("a")` | `df.sort("a")` | -| Sort descending | `df.orderBy(col("a").desc())` | `df.sort("a", descending=True)` | -| Multiple columns | `df.orderBy("a", col("b").desc())` | `df.sort("a", pl.col("b").sort(descending=True))` | - -### SQL Interface - -**Spark:** -```python -df.createOrReplaceTempView("trades") -result = spark.sql("SELECT symbol, AVG(price) FROM trades GROUP BY symbol") -``` - -**Polars:** -```python -ctx = pl.SQLContext(trades=df) -result = ctx.execute("SELECT symbol, AVG(price) FROM trades GROUP BY symbol").collect() -``` - -## Common Migration Patterns - -### Pattern 1: Feature Engineering Pipeline - -**Spark:** -```python -from pyspark.sql.functions import col, lag, avg -from pyspark.sql import Window - -window = Window.partitionBy("symbol").orderBy("date") -rolling = window.rowsBetween(-19, 0) - -result = ( - df - .withColumn("returns", (col("close") - lag("close", 1).over(window)) / lag("close", 1).over(window)) - .withColumn("ma_20", avg("close").over(rolling)) - .withColumn("signal", when(col("close") > col("ma_20"), 1).otherwise(-1)) -) -``` - -**Polars:** -```python -result = df.with_columns( - pl.col("close").pct_change().over("symbol").alias("returns"), - pl.col("close").rolling_mean(20).over("symbol").alias("ma_20"), -).with_columns( - pl.when(pl.col("close") > pl.col("ma_20")).then(1).otherwise(-1).alias("signal") -) -``` - -### Pattern 2: Large File Processing - -**Spark:** -```python -# Spark partitions automatically, uses cluster -df = spark.read.parquet("s3://bucket/data/") -result = df.filter(col("date") >= "2024-01-01").groupBy("symbol").agg(...) -``` - -**Polars:** -```python -# Polars uses lazy eval + streaming for single-machine large data -lf = pl.scan_parquet("s3://bucket/data/**/*.parquet") -result = ( - lf - .filter(pl.col("date") >= "2024-01-01") - .group_by("symbol") - .agg(...) - .collect(engine="streaming") # Streaming for large data -) -``` - -### Pattern 3: UDFs - -**Spark:** -```python -from pyspark.sql.functions import udf -from pyspark.sql.types import DoubleType - -@udf(returnType=DoubleType()) -def custom_calc(x): - return x * 1.1 - -df = df.withColumn("result", custom_calc(col("value"))) -``` - -**Polars** - prefer expressions, avoid map_elements: -```python -# Good: Use native expressions when possible -df = df.with_columns((pl.col("value") * 1.1).alias("result")) - -# If UDF truly needed (performance penalty): -df = df.with_columns( - pl.col("value").map_elements(lambda x: x * 1.1, return_dtype=pl.Float64).alias("result") -) -``` - -## Performance Comparison - -| Scenario | Spark | Polars | -|----------|-------|--------| -| Small-medium data (<100GB) | Overhead from distribution | Faster - no distribution overhead | -| Large data (100GB-1TB) | Scales with cluster | Fast with streaming | -| Very large data (>1TB) | Native territory | May need data partitioning strategy | -| Complex joins | Shuffle-heavy | Very fast on single machine | -| Simple aggregations | Good | Often 10-100x faster | - -## Migration Checklist - -1. **Replace SparkSession** with Polars imports -2. **Change `col()` import** to `pl.col()` -3. **Replace `withColumn`** with `with_columns` -4. **Replace `orderBy`** with `sort` -5. **Replace `groupBy`** with `group_by` -6. **Simplify window functions** - use `.over()` directly -7. **Replace UDFs** with native expressions where possible -8. **Add `.collect()`** after lazy operations -9. **Use `scan_*`** for large files instead of `read_*` -10. **Remove partition management** - Polars handles parallelism automatically - -## What You Lose from Spark - -- Distributed execution across cluster -- Fault tolerance and checkpointing -- Spark ecosystem (MLlib, Spark Streaming, GraphX) -- Delta Lake / Iceberg native integration -- Cluster resource management - -## What You Gain with Polars - -- No cluster setup/maintenance -- Lower latency (no network overhead) -- Simpler code (no explicit windows for most operations) -- Better single-machine performance -- Composable expressions -- Smaller memory footprint diff --git a/.cursor/skills/polars-expertise/references/python/best_practices.md b/.cursor/skills/polars-expertise/references/python/best_practices.md deleted file mode 100644 index 4a76a73d..00000000 --- a/.cursor/skills/polars-expertise/references/python/best_practices.md +++ /dev/null @@ -1,428 +0,0 @@ -# Polars Best Practices (Python) - -## Table of Contents -- [Performance Anti-Patterns](#performance-anti-patterns) -- [Time Series Patterns](#time-series-patterns) -- [Financial Data Patterns](#financial-data-patterns) -- [Large File Processing](#large-file-processing) -- [Memory Optimization](#memory-optimization) -- [Expression Patterns](#expression-patterns) - -## Performance Anti-Patterns - -### NEVER Use These - -```python -# ANTI-PATTERN 1: map_elements with Python functions -# This kills parallelization and is 10-100x slower -df.with_columns( - pl.col("price").map_elements(lambda x: x * 2) # TERRIBLE -) -# CORRECT: -df.with_columns(pl.col("price") * 2) # Fast, parallel - -# ANTI-PATTERN 2: Row iteration -for row in df.iter_rows(): # TERRIBLE - process(row) -# CORRECT: -df.with_columns(process_expression) - -# ANTI-PATTERN 3: Late projection -df = pl.read_parquet("data.parquet") -result = df.filter(...).select("a", "b") # Reads ALL columns first -# CORRECT: -lf = pl.scan_parquet("data.parquet") -result = lf.select("a", "b").filter(...).collect() # Reads only a, b - -# ANTI-PATTERN 4: Eager for large data -df = pl.read_csv("huge.csv") # Loads everything into RAM -# CORRECT: -lf = pl.scan_csv("huge.csv") -result = lf.filter(...).collect(engine="streaming") - -# ANTI-PATTERN 5: Converting to pandas unnecessarily -pandas_df = df.to_pandas() -result = pandas_df.groupby("x").mean() # Why? -# CORRECT: -result = df.group_by("x").agg(pl.all().mean()) - -# ANTI-PATTERN 6: Creating many intermediate DataFrames -df1 = df.filter(...) -df2 = df1.select(...) -df3 = df2.with_columns(...) # Each creates a copy -# CORRECT: -result = df.filter(...).select(...).with_columns(...) # Chained - -# ANTI-PATTERN 7: Wrong dtype selection -df = pl.read_csv("data.csv") # Infers Int64 for small integers -# CORRECT: -df = pl.read_csv("data.csv", dtypes={"small_int": pl.Int16}) -``` - -## Time Series Patterns - -### OHLCV Resampling - -```python -# Tick data to OHLCV bars -def resample_ohlcv(lf: pl.LazyFrame, interval: str) -> pl.LazyFrame: - return lf.group_by_dynamic( - "timestamp", - every=interval, - group_by="symbol" - ).agg( - pl.col("price").first().alias("open"), - pl.col("price").max().alias("high"), - pl.col("price").min().alias("low"), - pl.col("price").last().alias("close"), - pl.col("volume").sum().alias("volume"), - pl.col("price").count().alias("trades") - ) - -# Usage -bars_1m = resample_ohlcv(trades_lf, "1m").collect() -bars_5m = resample_ohlcv(trades_lf, "5m").collect() -``` - -### Rolling Statistics - -```python -# Efficient rolling calculations -df.with_columns( - # Simple moving averages - pl.col("close").rolling_mean(window_size=20).alias("sma_20"), - pl.col("close").rolling_mean(window_size=50).alias("sma_50"), - - # Volatility (rolling std) - pl.col("close").pct_change().rolling_std(window_size=20).alias("volatility"), - - # Rolling correlation - pl.rolling_corr("close", "volume", window_size=20).alias("corr_20"), - - # Exponential moving average - pl.col("close").ewm_mean(span=20).alias("ema_20"), - - # Bollinger Bands - (pl.col("close").rolling_mean(20) + 2 * pl.col("close").rolling_std(20)).alias("bb_upper"), - (pl.col("close").rolling_mean(20) - 2 * pl.col("close").rolling_std(20)).alias("bb_lower") -) -``` - -### Time-Based Windows - -```python -# Rolling by time duration (not row count) -df.with_columns( - # 5-minute rolling average - pl.col("price").rolling_mean( - window_size="5m", - by="timestamp" - ).alias("rolling_5m"), - - # Daily high/low - pl.col("price").rolling_max( - window_size="1d", - by="timestamp" - ).alias("daily_high") -) -``` - -### Lag/Lead for Returns - -```python -df.with_columns( - # Returns - pl.col("close").pct_change().alias("ret_1"), - pl.col("close").pct_change(5).alias("ret_5"), - - # Log returns - pl.col("close").log().diff().alias("log_ret"), - - # Forward returns (for labels) - pl.col("close").pct_change().shift(-1).alias("fwd_ret_1"), - pl.col("close").pct_change(5).shift(-5).alias("fwd_ret_5"), - - # Lagged features - pl.col("close").shift(1).alias("close_lag_1"), - pl.col("volume").shift(1).alias("volume_lag_1") -).over("symbol") # Per-symbol calculations -``` - -## Financial Data Patterns - -### VWAP Calculation - -```python -# Volume-Weighted Average Price -def calculate_vwap(df: pl.DataFrame) -> pl.DataFrame: - return df.with_columns( - vwap=( - (pl.col("price") * pl.col("volume")).cum_sum() / - pl.col("volume").cum_sum() - ).over("symbol") - ) -``` - -### As-Of Joins for Market Data - -```python -# Align trades with quotes -trades.join_asof( - quotes, - on="timestamp", - by="symbol", - strategy="backward" # Most recent quote before trade -) - -# Join with tolerance -trades.join_asof( - quotes, - on="timestamp", - by="symbol", - strategy="backward", - tolerance="100ms" # Only if quote within 100ms -) -``` - -### Bid-Ask Spread - -```python -quotes.with_columns( - (pl.col("ask") - pl.col("bid")).alias("spread"), - (((pl.col("ask") - pl.col("bid")) / pl.col("mid")) * 10000).alias("spread_bps"), - ((pl.col("bid") + pl.col("ask")) / 2).alias("mid") -) -``` - -### Position/PnL Tracking - -```python -# Calculate position and PnL from trades -trades.with_columns( - # Cumulative position - pl.col("signed_qty").cum_sum().over("symbol").alias("position"), - - # Mark-to-market PnL - ( - pl.col("signed_qty") * pl.col("price") * -1 - ).cum_sum().over("symbol").alias("realized_pnl") -) -``` - -## Large File Processing - -### Partitioned Reads - -```python -# Read partitioned parquet efficiently -lf = pl.scan_parquet( - "data/year=*/month=*/*.parquet", - hive_partitioning=True -) - -# Filter on partitions (predicate pushdown) -result = lf.filter( - (pl.col("year") == 2024) & (pl.col("month") >= 6) -).collect() -``` - -### Streaming for Massive Files - -```python -# Process larger-than-RAM data -lf = pl.scan_csv("massive.csv") - -# Streaming aggregation -result = ( - lf.filter(pl.col("value") > threshold) - .group_by("category") - .agg(pl.col("value").sum()) - .collect(engine="streaming") -) - -# Streaming write -lf.filter(pl.col("value") > threshold).sink_parquet("output.parquet") -``` - -### Chunked Processing - -```python -# Process in chunks when streaming not possible -def process_in_chunks(path: str, chunk_size: int = 1_000_000): - results = [] - reader = pl.read_csv_batched(path, batch_size=chunk_size) - - while True: - batch = reader.next_batches(1) - if batch is None: - break - # Process each batch - result = process_batch(batch[0]) - results.append(result) - - return pl.concat(results) -``` - -### Multi-File Processing - -```python -# Parallel file reading -lf = pl.scan_parquet("data/*.parquet") # Reads files in parallel - -# Or explicit list -files = ["data1.parquet", "data2.parquet", "data3.parquet"] -lf = pl.scan_parquet(files) - -# With schema enforcement -lf = pl.scan_parquet( - "data/*.parquet", - schema=expected_schema -) -``` - -## Memory Optimization - -### Dtype Selection - -```python -# Read with optimal types -df = pl.read_csv( - "data.csv", - dtypes={ - "trade_id": pl.UInt32, # Not Int64 if fits - "symbol": pl.Categorical, # Not String for symbols - "side": pl.Categorical, # "buy"/"sell" -> Categorical - "price": pl.Float32, # Float32 if precision OK - "quantity": pl.UInt32, # Unsigned if always positive - } -) -``` - -### Downcast After Load - -```python -def optimize_dtypes(df: pl.DataFrame) -> pl.DataFrame: - """Downcast numeric columns to smallest fitting type.""" - for col in df.columns: - dtype = df[col].dtype - if dtype in [pl.Int64, pl.Int32]: - min_val, max_val = df[col].min(), df[col].max() - if min_val >= 0: - if max_val <= 255: - df = df.with_columns(pl.col(col).cast(pl.UInt8)) - elif max_val <= 65535: - df = df.with_columns(pl.col(col).cast(pl.UInt16)) - elif max_val <= 4294967295: - df = df.with_columns(pl.col(col).cast(pl.UInt32)) - return df -``` - -### Categorical for Low Cardinality - -```python -# Identify candidates -for col in df.columns: - if df[col].dtype == pl.Utf8: - n_unique = df[col].n_unique() - total = len(df) - if n_unique / total < 0.5: # Less than 50% unique - print(f"{col}: {n_unique} unique / {total} total - use Categorical") -``` - -## Expression Patterns - -### Reusable Expression Library - -```python -# Define expression library for your domain -class FinanceExpr: - @staticmethod - def returns(col: str = "close") -> pl.Expr: - return pl.col(col).pct_change() - - @staticmethod - def log_returns(col: str = "close") -> pl.Expr: - return pl.col(col).log().diff() - - @staticmethod - def volatility(col: str = "close", window: int = 20) -> pl.Expr: - return pl.col(col).pct_change().rolling_std(window) - - @staticmethod - def sharpe(returns_col: str, rf: float = 0.0, window: int = 252) -> pl.Expr: - excess = pl.col(returns_col) - rf / 252 - return ( - excess.rolling_mean(window) / - excess.rolling_std(window) * - (252 ** 0.5) - ) - -# Usage -df.with_columns( - ret=FinanceExpr.returns(), - vol=FinanceExpr.volatility(window=20), - sharpe=FinanceExpr.sharpe("ret", window=60) -) -``` - -### Conditional Aggregations - -```python -# Complex conditional aggregations -df.group_by("symbol").agg( - # Count by condition - up_days=(pl.col("close") > pl.col("open")).sum(), - down_days=(pl.col("close") < pl.col("open")).sum(), - - # Conditional averages - pl.col("volume").filter( - pl.col("close") > pl.col("open") - ).mean().alias("avg_up_volume"), - - # Weighted conditionals - vwap_up=( - pl.when(pl.col("close") > pl.col("open")) - .then(pl.col("price") * pl.col("volume")) - .otherwise(0) - .sum() / - pl.when(pl.col("close") > pl.col("open")) - .then(pl.col("volume")) - .otherwise(0) - .sum() - ) -) -``` - -### Pipeline Functions - -```python -def clean_market_data(lf: pl.LazyFrame) -> pl.LazyFrame: - """Standard market data cleaning pipeline.""" - return ( - lf - .filter(pl.col("price") > 0) - .filter(pl.col("volume") > 0) - .with_columns( - pl.col("timestamp").cast(pl.Datetime), - pl.col("symbol").cast(pl.Categorical) - ) - .sort("symbol", "timestamp") - .unique(subset=["symbol", "timestamp"], keep="last") - ) - -def add_technical_features(lf: pl.LazyFrame) -> pl.LazyFrame: - """Add standard technical indicators.""" - return lf.with_columns( - pl.col("close").pct_change().over("symbol").alias("returns"), - pl.col("close").rolling_mean(20).over("symbol").alias("sma_20"), - pl.col("close").rolling_std(20).over("symbol").alias("volatility_20") - ) - -# Compose pipeline -result = ( - pl.scan_parquet("trades/*.parquet") - .pipe(clean_market_data) - .pipe(add_technical_features) - .collect() -) -``` diff --git a/.cursor/skills/polars-expertise/references/python/core_concepts.md b/.cursor/skills/polars-expertise/references/python/core_concepts.md deleted file mode 100644 index 1b7867cb..00000000 --- a/.cursor/skills/polars-expertise/references/python/core_concepts.md +++ /dev/null @@ -1,271 +0,0 @@ -# Polars Core Concepts (Python) - -## Table of Contents -- [Expressions](#expressions) -- [Data Types](#data-types) -- [Lazy vs Eager Evaluation](#lazy-vs-eager-evaluation) -- [Streaming Mode](#streaming-mode) -- [Parallelization](#parallelization) - -## Expressions - -Expressions are the foundation of Polars. They describe transformations without executing immediately. - -### Expression Contexts - -Expressions only execute within specific contexts: - -```python -import polars as pl - -df = pl.DataFrame({ - "symbol": ["AAPL", "GOOG", "AAPL", "GOOG"], - "price": [150.0, 140.0, 151.0, 141.0], - "volume": [1000, 2000, 1500, 2500] -}) - -# select() - choose and transform columns -df.select("symbol", pl.col("price") * pl.col("volume")) - -# with_columns() - add/modify while preserving existing -df.with_columns( - (pl.col("price") * pl.col("volume")).alias("notional"), - pl.col("price").pct_change().over("symbol").alias("price_pct") -) - -# filter() - row selection -df.filter(pl.col("volume") > 1500) - -# group_by().agg() - aggregation -df.group_by("symbol").agg( - pl.col("price").mean().alias("avg_price"), - pl.col("volume").sum().alias("total_volume") -) -``` - -### Expression Composition - -Expressions can be stored and reused: - -```python -# Define reusable expressions for financial calculations -vwap = (pl.col("price") * pl.col("volume")).sum() / pl.col("volume").sum() -volatility = pl.col("price").std() / pl.col("price").mean() -returns = pl.col("price").pct_change() - -# Use in multiple contexts -df.group_by("symbol").agg( - vwap.alias("vwap"), - volatility.alias("volatility") -) -``` - -### Expression Expansion - -Apply operations to multiple columns: - -```python -# All numeric columns -df.select(pl.col(pl.NUMERIC_DTYPES) * 100) - -# Pattern matching -df.select(pl.col("^bid_.*$") - pl.col("^ask_.*$")) # Spread calculation - -# Exclude patterns -df.select(pl.all().exclude("timestamp", "id")) -``` - -## Data Types - -### Core Types - -| Type | Python | Use Case | -|------|--------|----------| -| Int64/Int32 | `pl.Int64` | Trade IDs, counts | -| Float64 | `pl.Float64` | Prices, returns | -| Utf8/String | `pl.Utf8` | Symbols, names | -| Datetime | `pl.Datetime` | Timestamps | -| Date | `pl.Date` | Calendar dates | -| Duration | `pl.Duration` | Time differences | -| Categorical | `pl.Categorical` | Low-cardinality strings | -| List | `pl.List` | Variable-length arrays | - -### Type Casting - -```python -df.with_columns( - # Downcast for memory efficiency - pl.col("trade_id").cast(pl.UInt32), - - # Parse timestamps - pl.col("timestamp_str").str.strptime(pl.Datetime, "%Y-%m-%d %H:%M:%S%.f"), - - # Categorical for symbols (faster grouping) - pl.col("symbol").cast(pl.Categorical) -) -``` - -### Null Handling - -```python -# Check nulls -df.filter(pl.col("price").is_not_null()) - -# Fill strategies -df.with_columns( - # Forward fill (common for market data) - pl.col("price").fill_null(strategy="forward"), - - # Fill with group mean - pl.col("price").fill_null(pl.col("price").mean().over("symbol")), - - # Interpolate - pl.col("price").interpolate() -) -``` - -## Lazy vs Eager Evaluation - -### When to Use Each - -| Lazy (`scan_*`, `.lazy()`) | Eager (`read_*`) | -|---------------------------|------------------| -| Large files (>1GB) | Small exploration | -| Complex pipelines | Interactive work | -| Production jobs | Quick analysis | -| Memory constrained | Simple one-offs | - -### Lazy Example - -```python -# Lazy: builds query plan, optimizes, then executes -lf = ( - pl.scan_parquet("trades/*.parquet") - .filter(pl.col("symbol") == "AAPL") - .filter(pl.col("timestamp") >= "2024-01-01") - .group_by_dynamic("timestamp", every="1m") - .agg( - pl.col("price").first().alias("open"), - pl.col("price").max().alias("high"), - pl.col("price").min().alias("low"), - pl.col("price").last().alias("close"), - pl.col("volume").sum() - ) -) - -# View optimized plan -print(lf.explain()) - -# Execute -df = lf.collect() -``` - -### Query Optimization - -Polars automatically applies: - -| Optimization | Effect | -|-------------|--------| -| Predicate pushdown | Filters at scan level | -| Projection pushdown | Reads only needed columns | -| Slice pushdown | Limits rows early | -| Common subplan elimination | Caches repeated subqueries | -| Join ordering | Optimizes join sequence | - -```python -# Example: only reads symbol="AAPL" rows and price/volume columns -lf = ( - pl.scan_parquet("trades.parquet") - .filter(pl.col("symbol") == "AAPL") # Pushed to scan - .select("price", "volume") # Only these columns read -) -``` - -## Streaming Mode - -Process data larger than RAM: - -```python -# Streaming execution -lf = pl.scan_csv("massive_file.csv") -result = lf.filter(pl.col("value") > 100).collect(engine="streaming") - -# Streaming write (sink) -lf.filter(pl.col("value") > 100).sink_parquet("output.parquet") - -# Check streaming compatibility -print(lf.explain(streaming=True)) -``` - -### Streaming Limitations - -Operations that may not stream: -- Sorts on large data -- Some join types -- Certain aggregations - -Polars falls back to in-memory automatically when needed. - -## Parallelization - -### Automatic Parallelization - -Polars parallelizes: -- Aggregations within groups -- Window functions -- Expression evaluations -- Multi-file reads - -### What Kills Parallelization - -```python -# BAD: Python UDF - sequential, slow -df.with_columns( - pl.col("price").map_elements(lambda x: custom_func(x)) # AVOID -) - -# GOOD: Native expressions - parallel, fast -df.with_columns( - pl.col("price") * 1.1 # Parallelized -) - -# BAD: Row iteration -for row in df.iter_rows(): # AVOID - process(row) - -# GOOD: Columnar operations -df.with_columns(processed=process_expr) -``` - -### Thread Pool Configuration - -```python -import os - -# Set before importing polars -os.environ["POLARS_MAX_THREADS"] = "8" - -# Or check current setting -import polars as pl -print(pl.thread_pool_size()) -``` - -## Memory Format - -Polars uses Apache Arrow columnar format: - -- Zero-copy sharing with other Arrow libraries -- Efficient SIMD vectorization -- Reduced memory overhead -- Fast serialization - -```python -# Check DataFrame memory usage -print(f"Size: {df.estimated_size('mb'):.2f} MB") - -# Convert to Arrow -arrow_table = df.to_arrow() - -# From Arrow (zero-copy when possible) -df = pl.from_arrow(arrow_table) -``` diff --git a/.cursor/skills/polars-expertise/references/python/io_guide.md b/.cursor/skills/polars-expertise/references/python/io_guide.md deleted file mode 100644 index 07eb8c8d..00000000 --- a/.cursor/skills/polars-expertise/references/python/io_guide.md +++ /dev/null @@ -1,559 +0,0 @@ -# Polars Data I/O Guide - -Comprehensive guide to reading and writing data in various formats with Polars. - -**Contents:** CSV | Parquet | JSON | Excel | Database | Cloud Storage | BigQuery | Arrow | In-Memory | Streaming | Best Practices | Error Handling | Schema - -## CSV Files - -### Reading CSV - -**Eager mode (loads into memory):** -```python -import polars as pl - -# Basic read -df = pl.read_csv("data.csv") - -# With options -df = pl.read_csv( - "data.csv", - separator=",", - has_header=True, - columns=["col1", "col2"], # Select specific columns - n_rows=1000, # Read only first 1000 rows - skip_rows=10, # Skip first 10 rows - dtypes={"col1": pl.Int64, "col2": pl.Utf8}, # Specify types - null_values=["NA", "null", ""], # Define null values - encoding="utf-8", - ignore_errors=False -) -``` - -**Lazy mode (scans without loading - recommended for large files):** -```python -# Scan CSV (builds query plan) -lf = pl.scan_csv("data.csv") - -# Apply operations -result = lf.filter(pl.col("age") > 25).select("name", "age") - -# Execute and load -df = result.collect() -``` - -### Writing CSV - -```python -# Basic write -df.write_csv("output.csv") - -# With options -df.write_csv( - "output.csv", - separator=",", - include_header=True, - null_value="", # How to represent nulls - quote_char='"', - line_terminator="\n" -) -``` - -### Multiple CSV Files - -**Read multiple files:** -```python -# Read all CSVs in directory -lf = pl.scan_csv("data/*.csv") - -# Read specific files -lf = pl.scan_csv(["file1.csv", "file2.csv", "file3.csv"]) -``` - -## Parquet Files - -Parquet is the recommended format for performance and compression. - -### Reading Parquet - -**Eager:** -```python -df = pl.read_parquet("data.parquet") - -# With options -df = pl.read_parquet( - "data.parquet", - columns=["col1", "col2"], # Select specific columns - n_rows=1000, # Read first N rows - parallel="auto" # Control parallelization -) -``` - -**Lazy (recommended):** -```python -lf = pl.scan_parquet("data.parquet") - -# Automatic predicate and projection pushdown -result = lf.filter(pl.col("age") > 25).select("name", "age").collect() -``` - -### Writing Parquet - -```python -# Basic write -df.write_parquet("output.parquet") - -# With compression -df.write_parquet( - "output.parquet", - compression="snappy", # Options: "snappy", "gzip", "brotli", "lz4", "zstd" - statistics=True, # Write statistics (enables predicate pushdown) - use_pyarrow=False # Use Rust writer (faster) -) -``` - -### Partitioned Parquet (Hive-style) - -**Write partitioned:** -```python -# Write with partitioning -df.write_parquet( - "output_dir", - partition_by=["year", "month"] # Creates directory structure -) -# Creates: output_dir/year=2023/month=01/data.parquet -``` - -**Read partitioned:** -```python -lf = pl.scan_parquet("output_dir/**/*.parquet") - -# Hive partitioning columns are automatically added -result = lf.filter(pl.col("year") == 2023).collect() -``` - -## JSON Files - -### Reading JSON - -**NDJSON (newline-delimited JSON) - recommended:** -```python -df = pl.read_ndjson("data.ndjson") - -# Lazy -lf = pl.scan_ndjson("data.ndjson") -``` - -**Standard JSON:** -```python -df = pl.read_json("data.json") - -# From JSON string -df = pl.read_json('{"col1": [1, 2], "col2": ["a", "b"]}') -``` - -### Writing JSON - -```python -# Write NDJSON -df.write_ndjson("output.ndjson") - -# Write standard JSON -df.write_json("output.json") - -# Pretty printed -df.write_json("output.json", pretty=True, row_oriented=False) -``` - -## Excel Files - -### Reading Excel - -```python -# Read first sheet -df = pl.read_excel("data.xlsx") - -# Specific sheet -df = pl.read_excel("data.xlsx", sheet_name="Sheet1") -# Or by index -df = pl.read_excel("data.xlsx", sheet_id=0) - -# With options -df = pl.read_excel( - "data.xlsx", - sheet_name="Sheet1", - columns=["A", "B", "C"], # Excel columns - n_rows=100, - skip_rows=5, - has_header=True -) -``` - -### Writing Excel - -```python -# Write to Excel -df.write_excel("output.xlsx") - -# Multiple sheets -with pl.ExcelWriter("output.xlsx") as writer: - df1.write_excel(writer, worksheet="Sheet1") - df2.write_excel(writer, worksheet="Sheet2") -``` - -## Database Connectivity - -### Read from Database - -```python -import polars as pl - -# Read entire table -df = pl.read_database("SELECT * FROM users", connection_uri="postgresql://...") - -# Using connectorx for better performance -df = pl.read_database_uri( - "SELECT * FROM users WHERE age > 25", - uri="postgresql://user:pass@localhost/db" -) -``` - -### Write to Database - -```python -# Using SQLAlchemy -from sqlalchemy import create_engine - -engine = create_engine("postgresql://user:pass@localhost/db") -df.write_database("table_name", connection=engine) - -# With options -df.write_database( - "table_name", - connection=engine, - if_exists="replace", # or "append", "fail" -) -``` - -### Common Database Connectors - -**PostgreSQL:** -```python -uri = "postgresql://username:password@localhost:5432/database" -df = pl.read_database_uri("SELECT * FROM table", uri=uri) -``` - -**MySQL:** -```python -uri = "mysql://username:password@localhost:3306/database" -df = pl.read_database_uri("SELECT * FROM table", uri=uri) -``` - -**SQLite:** -```python -uri = "sqlite:///path/to/database.db" -df = pl.read_database_uri("SELECT * FROM table", uri=uri) -``` - -## Cloud Storage - -### AWS S3 - -```python -# Read from S3 -df = pl.read_parquet("s3://bucket/path/to/file.parquet") -lf = pl.scan_parquet("s3://bucket/path/*.parquet") - -# Write to S3 -df.write_parquet("s3://bucket/path/output.parquet") - -# With credentials -import os -os.environ["AWS_ACCESS_KEY_ID"] = "your_key" -os.environ["AWS_SECRET_ACCESS_KEY"] = "your_secret" -os.environ["AWS_REGION"] = "us-west-2" - -df = pl.read_parquet("s3://bucket/file.parquet") -``` - -### Azure Blob Storage - -```python -# Read from Azure -df = pl.read_parquet("az://container/path/file.parquet") - -# Write to Azure -df.write_parquet("az://container/path/output.parquet") - -# With credentials -os.environ["AZURE_STORAGE_ACCOUNT_NAME"] = "account" -os.environ["AZURE_STORAGE_ACCOUNT_KEY"] = "key" -``` - -### Google Cloud Storage (GCS) - -```python -# Read from GCS -df = pl.read_parquet("gs://bucket/path/file.parquet") - -# Write to GCS -df.write_parquet("gs://bucket/path/output.parquet") - -# With credentials -os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/path/to/credentials.json" -``` - -## Google BigQuery - -```python -# Read from BigQuery -df = pl.read_database( - "SELECT * FROM project.dataset.table", - connection_uri="bigquery://project" -) - -# Or using Google Cloud SDK -from google.cloud import bigquery -client = bigquery.Client() - -query = "SELECT * FROM project.dataset.table WHERE date > '2023-01-01'" -df = pl.from_pandas(client.query(query).to_dataframe()) -``` - -## Apache Arrow - -### IPC/Feather Format - -**Read:** -```python -df = pl.read_ipc("data.arrow") -lf = pl.scan_ipc("data.arrow") -``` - -**Write:** -```python -df.write_ipc("output.arrow") - -# Compressed -df.write_ipc("output.arrow", compression="zstd") -``` - -### Arrow Streaming - -```python -# Write streaming format -df.write_ipc("output.arrows", compression="zstd") - -# Read streaming -df = pl.read_ipc("output.arrows") -``` - -### From/To Arrow - -```python -import pyarrow as pa - -# From Arrow Table -arrow_table = pa.table({"col": [1, 2, 3]}) -df = pl.from_arrow(arrow_table) - -# To Arrow Table -arrow_table = df.to_arrow() -``` - -## In-Memory Formats - -### Python Dictionaries - -```python -# From dict -df = pl.DataFrame({ - "col1": [1, 2, 3], - "col2": ["a", "b", "c"] -}) - -# To dict -data_dict = df.to_dict() # Column-oriented -data_dict = df.to_dict(as_series=False) # Lists instead of Series -``` - -### NumPy Arrays - -```python -import numpy as np - -# From NumPy -arr = np.array([[1, 2], [3, 4], [5, 6]]) -df = pl.DataFrame(arr, schema=["col1", "col2"]) - -# To NumPy -arr = df.to_numpy() -``` - -### Pandas DataFrames - -```python -import pandas as pd - -# From Pandas -pd_df = pd.DataFrame({"col": [1, 2, 3]}) -pl_df = pl.from_pandas(pd_df) - -# To Pandas -pd_df = pl_df.to_pandas() - -# Zero-copy when possible -pl_df = pl.from_arrow(pd_df) -``` - -### Lists of Rows - -```python -# From list of dicts -data = [ - {"name": "Alice", "age": 25}, - {"name": "Bob", "age": 30} -] -df = pl.DataFrame(data) - -# To list of dicts -rows = df.to_dicts() - -# From list of tuples -data = [("Alice", 25), ("Bob", 30)] -df = pl.DataFrame(data, schema=["name", "age"]) -``` - -## Streaming Large Files - -For datasets larger than memory, use lazy mode with streaming: - -```python -# Streaming mode -lf = pl.scan_csv("very_large.csv") -result = lf.filter(pl.col("value") > 100).collect(streaming=True) - -# Streaming with multiple files -lf = pl.scan_parquet("data/*.parquet") -result = lf.group_by("category").agg(pl.col("value").sum()).collect(streaming=True) -``` - -## Best Practices - -### Format Selection - -**Use Parquet when:** -- Need compression (up to 10x smaller than CSV) -- Want fast reads/writes -- Need to preserve data types -- Working with large datasets -- Need predicate pushdown - -**Use CSV when:** -- Need human-readable format -- Interfacing with legacy systems -- Data is small -- Need universal compatibility - -**Use JSON when:** -- Working with nested/hierarchical data -- Need web API compatibility -- Data has flexible schema - -**Use Arrow IPC when:** -- Need zero-copy data sharing -- Fastest serialization required -- Working between Arrow-compatible systems - -### Reading Large Files - -```python -# 1. Always use lazy mode -lf = pl.scan_csv("large.csv") # NOT read_csv - -# 2. Filter and select early (pushdown optimization) -result = ( - lf - .select("col1", "col2", "col3") # Only needed columns - .filter(pl.col("date") > "2023-01-01") # Filter early - .collect() -) - -# 3. Use streaming for very large data -result = lf.filter(...).select(...).collect(streaming=True) - -# 4. Read only needed rows during development -df = pl.read_csv("large.csv", n_rows=10000) # Sample for testing -``` - -### Writing Large Files - -```python -# 1. Use Parquet with compression -df.write_parquet("output.parquet", compression="zstd") - -# 2. Use partitioning for very large datasets -df.write_parquet("output", partition_by=["year", "month"]) - -# 3. Write streaming -lf = pl.scan_csv("input.csv") -lf.sink_parquet("output.parquet") # Streaming write -``` - -### Performance Tips - -```python -# 1. Specify dtypes when reading CSV -df = pl.read_csv( - "data.csv", - dtypes={"id": pl.Int64, "name": pl.Utf8} # Avoids inference -) - -# 2. Use appropriate compression -df.write_parquet("output.parquet", compression="snappy") # Fast -df.write_parquet("output.parquet", compression="zstd") # Better compression - -# 3. Parallel reading -df = pl.read_csv("data.csv", parallel="auto") - -# 4. Read multiple files in parallel -lf = pl.scan_parquet("data/*.parquet") # Automatic parallel read -``` - -## Error Handling - -```python -try: - df = pl.read_csv("data.csv") -except pl.exceptions.ComputeError as e: - print(f"Error reading CSV: {e}") - -# Ignore errors during parsing -df = pl.read_csv("messy.csv", ignore_errors=True) - -# Handle missing files -from pathlib import Path -if Path("data.csv").exists(): - df = pl.read_csv("data.csv") -else: - print("File not found") -``` - -## Schema Management - -```python -# Infer schema from sample -schema = pl.read_csv("data.csv", n_rows=1000).schema - -# Use inferred schema for full read -df = pl.read_csv("data.csv", dtypes=schema) - -# Define schema explicitly -schema = { - "id": pl.Int64, - "name": pl.Utf8, - "date": pl.Date, - "value": pl.Float64 -} -df = pl.read_csv("data.csv", dtypes=schema) -``` diff --git a/.cursor/skills/polars-expertise/references/python/operations.md b/.cursor/skills/polars-expertise/references/python/operations.md deleted file mode 100644 index d44e6354..00000000 --- a/.cursor/skills/polars-expertise/references/python/operations.md +++ /dev/null @@ -1,607 +0,0 @@ -# Polars Operations Reference - -This reference covers all common Polars operations with comprehensive examples. - -**Contents:** Selection | Filtering | Grouping/Aggregation | Window Functions | Sorting | Conditional | String | Date/Time | List | Struct | Unique/Duplicate | Sampling | Renaming - -## Selection Operations - -### Select Columns - -**Basic selection:** -```python -# Select specific columns -df.select("name", "age", "city") - -# Using expressions -df.select(pl.col("name"), pl.col("age")) -``` - -**Pattern-based selection:** -```python -# All columns starting with "sales_" -df.select(pl.col("^sales_.*$")) - -# All numeric columns -df.select(pl.col(pl.NUMERIC_DTYPES)) - -# All columns except specific ones -df.select(pl.all().exclude("id", "timestamp")) -``` - -**Computed columns:** -```python -df.select( - "name", - (pl.col("age") * 12).alias("age_in_months"), - (pl.col("salary") * 1.1).alias("salary_after_raise") -) -``` - -### With Columns (Add/Modify) - -Add new columns or modify existing ones while preserving all other columns: - -```python -# Add new columns -df.with_columns( - (pl.col("age") * 2).alias("age_doubled"), - (pl.col("first_name") + " " + pl.col("last_name")).alias("full_name") -) - -# Modify existing columns -df.with_columns( - pl.col("name").str.to_uppercase().alias("name"), - pl.col("salary").cast(pl.Float64).alias("salary") -) - -# Multiple operations in parallel -df.with_columns( - pl.col("value") * 10, - pl.col("value") * 100, - pl.col("value") * 1000, -) -``` - -## Filtering Operations - -### Basic Filtering - -```python -# Single condition -df.filter(pl.col("age") > 25) - -# Multiple conditions (AND) -df.filter( - pl.col("age") > 25, - pl.col("city") == "NY" -) - -# OR conditions -df.filter( - (pl.col("age") > 30) | (pl.col("salary") > 100000) -) - -# NOT condition -df.filter(~pl.col("active")) -df.filter(pl.col("city") != "NY") -``` - -### Advanced Filtering - -**String operations:** -```python -# Contains substring -df.filter(pl.col("name").str.contains("John")) - -# Starts with -df.filter(pl.col("email").str.starts_with("admin")) - -# Regex match -df.filter(pl.col("phone").str.contains(r"^\d{3}-\d{3}-\d{4}$")) -``` - -**Membership checks:** -```python -# In list -df.filter(pl.col("city").is_in(["NY", "LA", "SF"])) - -# Not in list -df.filter(~pl.col("status").is_in(["inactive", "deleted"])) -``` - -**Range filters:** -```python -# Between values -df.filter(pl.col("age").is_between(25, 35)) - -# Date range -df.filter( - pl.col("date") >= pl.date(2023, 1, 1), - pl.col("date") <= pl.date(2023, 12, 31) -) -``` - -**Null filtering:** -```python -# Filter out nulls -df.filter(pl.col("value").is_not_null()) - -# Keep only nulls -df.filter(pl.col("value").is_null()) -``` - -## Grouping and Aggregation - -### Basic Group By - -```python -# Group by single column -df.group_by("department").agg( - pl.col("salary").mean().alias("avg_salary"), - pl.len().alias("employee_count") -) - -# Group by multiple columns -df.group_by("department", "location").agg( - pl.col("salary").sum() -) - -# Maintain order -df.group_by("category", maintain_order=True).agg( - pl.col("value").sum() -) -``` - -### Aggregation Functions - -**Count and length:** -```python -df.group_by("category").agg( - pl.len().alias("count"), - pl.col("id").count().alias("non_null_count"), - pl.col("id").n_unique().alias("unique_count") -) -``` - -**Statistical aggregations:** -```python -df.group_by("group").agg( - pl.col("value").sum().alias("total"), - pl.col("value").mean().alias("average"), - pl.col("value").median().alias("median"), - pl.col("value").std().alias("std_dev"), - pl.col("value").var().alias("variance"), - pl.col("value").min().alias("minimum"), - pl.col("value").max().alias("maximum"), - pl.col("value").quantile(0.95).alias("p95") -) -``` - -**First and last:** -```python -df.group_by("user_id").agg( - pl.col("timestamp").first().alias("first_seen"), - pl.col("timestamp").last().alias("last_seen"), - pl.col("event").first().alias("first_event") -) -``` - -**List aggregation:** -```python -# Collect values into lists -df.group_by("category").agg( - pl.col("item").alias("all_items") # Creates list column -) -``` - -### Conditional Aggregations - -Filter within aggregations: - -```python -df.group_by("department").agg( - # Count high earners - (pl.col("salary") > 100000).sum().alias("high_earners"), - - # Average of filtered values - pl.col("salary").filter(pl.col("bonus") > 0).mean().alias("avg_with_bonus"), - - # Conditional sum - pl.when(pl.col("active")) - .then(pl.col("sales")) - .otherwise(0) - .sum() - .alias("active_sales") -) -``` - -### Multiple Aggregations - -Combine multiple aggregations efficiently: - -```python -df.group_by("store_id").agg( - pl.col("transaction_id").count().alias("num_transactions"), - pl.col("amount").sum().alias("total_sales"), - pl.col("amount").mean().alias("avg_transaction"), - pl.col("customer_id").n_unique().alias("unique_customers"), - pl.col("amount").max().alias("largest_transaction"), - pl.col("timestamp").min().alias("first_transaction_date"), - pl.col("timestamp").max().alias("last_transaction_date") -) -``` - -## Window Functions - -Window functions apply aggregations while preserving the original row count. - -### Basic Window Operations - -**Group statistics:** -```python -# Add group mean to each row -df.with_columns( - pl.col("age").mean().over("department").alias("avg_age_by_dept") -) - -# Multiple group columns -df.with_columns( - pl.col("value").mean().over("category", "region").alias("group_avg") -) -``` - -**Ranking:** -```python -df.with_columns( - # Rank within groups - pl.col("score").rank().over("team").alias("rank"), - - # Dense rank (no gaps) - pl.col("score").rank(method="dense").over("team").alias("dense_rank"), - - # Row number - pl.col("timestamp").sort().rank(method="ordinal").over("user_id").alias("row_num") -) -``` - -### Window Mapping Strategies - -**group_to_rows (default):** -Preserves original row order: -```python -df.with_columns( - pl.col("value").mean().over("category", mapping_strategy="group_to_rows").alias("group_mean") -) -``` - -**explode:** -Faster, groups rows together: -```python -df.with_columns( - pl.col("value").mean().over("category", mapping_strategy="explode").alias("group_mean") -) -``` - -**join:** -Creates list columns: -```python -df.with_columns( - pl.col("value").over("category", mapping_strategy="join").alias("group_values") -) -``` - -### Rolling Windows - -**Time-based rolling:** -```python -df.with_columns( - pl.col("value").rolling_mean( - window_size="7d", - by="date" - ).alias("rolling_avg") -) -``` - -**Row-based rolling:** -```python -df.with_columns( - pl.col("value").rolling_sum(window_size=3).alias("rolling_sum"), - pl.col("value").rolling_max(window_size=5).alias("rolling_max") -) -``` - -### Cumulative Operations - -```python -df.with_columns( - pl.col("value").cum_sum().over("group").alias("cumsum"), - pl.col("value").cum_max().over("group").alias("cummax"), - pl.col("value").cum_min().over("group").alias("cummin"), - pl.col("value").cum_prod().over("group").alias("cumprod") -) -``` - -### Shift and Lag/Lead - -```python -df.with_columns( - # Previous value (lag) - pl.col("value").shift(1).over("user_id").alias("prev_value"), - - # Next value (lead) - pl.col("value").shift(-1).over("user_id").alias("next_value"), - - # Calculate difference from previous - (pl.col("value") - pl.col("value").shift(1).over("user_id")).alias("diff") -) -``` - -## Sorting - -### Basic Sorting - -```python -# Sort by single column -df.sort("age") - -# Sort descending -df.sort("age", descending=True) - -# Sort by multiple columns -df.sort("department", "age") - -# Mixed sorting order -df.sort(["department", "salary"], descending=[False, True]) -``` - -### Advanced Sorting - -**Null handling:** -```python -# Nulls first -df.sort("value", nulls_last=False) - -# Nulls last -df.sort("value", nulls_last=True) -``` - -**Sort by expression:** -```python -# Sort by computed value -df.sort(pl.col("first_name").str.len()) - -# Sort by multiple expressions -df.sort( - pl.col("last_name").str.to_lowercase(), - pl.col("age").abs() -) -``` - -## Conditional Operations - -### When/Then/Otherwise - -```python -# Basic conditional -df.with_columns( - pl.when(pl.col("age") >= 18) - .then("adult") - .otherwise("minor") - .alias("status") -) - -# Multiple conditions -df.with_columns( - pl.when(pl.col("score") >= 90) - .then("A") - .when(pl.col("score") >= 80) - .then("B") - .when(pl.col("score") >= 70) - .then("C") - .otherwise("F") - .alias("category") -) - -# Conditional computation -df.with_columns( - pl.when(pl.col("is_member")) - .then(pl.col("price") * 0.9) - .otherwise(pl.col("price")) - .alias("adjusted_price") -) -``` - -## String Operations - -### Common String Methods - -```python -df.with_columns( - # Case conversion - pl.col("name").str.to_uppercase().alias("upper"), - pl.col("name").str.to_lowercase().alias("lower"), - pl.col("name").str.to_titlecase().alias("title"), - - # Trimming - pl.col("text").str.strip_chars().alias("trimmed"), - - # Substring - pl.col("name").str.slice(0, 3).alias("first_3"), - - # Replace - pl.col("text").str.replace("old", "new").alias("cleaned"), - pl.col("text").str.replace_all("old", "new").alias("cleaned_all"), - - # Split - pl.col("full_name").str.split(" ").alias("parts"), - - # Length - pl.col("name").str.len_chars().alias("name_length") -) -``` - -### String Filtering - -```python -# Contains -df.filter(pl.col("email").str.contains("@gmail.com")) - -# Starts/ends with -df.filter(pl.col("name").str.starts_with("A")) -df.filter(pl.col("file").str.ends_with(".csv")) - -# Regex matching -df.filter(pl.col("phone").str.contains(r"^\d{3}-\d{4}$")) -``` - -## Date and Time Operations - -### Date Parsing - -```python -# Parse strings to dates -df.with_columns( - pl.col("date_str").str.strptime(pl.Date, "%Y-%m-%d").alias("date"), - pl.col("dt_str").str.strptime(pl.Datetime, "%Y-%m-%d %H:%M:%S").alias("datetime") -) -``` - -### Date Components - -```python -df.with_columns( - pl.col("date").dt.year().alias("year"), - pl.col("date").dt.month().alias("month"), - pl.col("date").dt.day().alias("day"), - pl.col("date").dt.weekday().alias("weekday"), - pl.col("datetime").dt.hour().alias("hour"), - pl.col("datetime").dt.minute().alias("minute") -) -``` - -### Date Arithmetic - -```python -# Add duration -df.with_columns( - (pl.col("date") + pl.duration(weeks=1)).alias("next_week"), - (pl.col("date") + pl.duration(months=1)).alias("next_month") -) - -# Difference between dates -df.with_columns( - days_diff=(pl.col("end_date") - pl.col("start_date")).dt.total_days() -) -``` - -### Date Filtering - -```python -# Filter by date range -df.filter( - pl.col("date").is_between(pl.date(2023, 1, 1), pl.date(2023, 12, 31)) -) - -# Filter by year -df.filter(pl.col("date").dt.year() == 2023) - -# Filter by month -df.filter(pl.col("date").dt.month().is_in([6, 7, 8])) # Summer months -``` - -## List Operations - -### Working with List Columns - -```python -# Create list column -df.with_columns( - pl.col("item1", "item2", "item3").to_list().alias("items_list") -) - -# List operations -df.with_columns( - pl.col("items").list.len().alias("list_len"), - pl.col("items").list.first().alias("first_item"), - pl.col("items").list.last().alias("last_item"), - pl.col("items").list.unique().alias("unique_items"), - pl.col("items").list.sort().alias("sorted_items") -) - -# Explode lists to rows -df.explode("items") - -# Filter list elements -df.with_columns( - pl.col("items").list.eval(pl.element() > 10).alias("filtered") -) -``` - -## Struct Operations - -### Working with Nested Structures - -```python -# Create struct column -df.with_columns( - address=pl.struct(["street", "city", "zip"]) -) - -# Access struct fields -df.with_columns( - pl.col("address").struct.field("city").alias("city") -) - -# Unnest struct to columns -df.unnest("address") -``` - -## Unique and Duplicate Operations - -```python -# Get unique rows -df.unique() - -# Unique on specific columns -df.unique(subset=["name", "email"]) - -# Keep first/last duplicate -df.unique(subset=["id"], keep="first") -df.unique(subset=["id"], keep="last") - -# Identify duplicates -df.with_columns( - pl.col("id").is_duplicated().alias("is_duplicate") -) - -# Count duplicates -df.group_by("email").agg( - pl.len().alias("count") -).filter(pl.col("count") > 1) -``` - -## Sampling - -```python -# Random sample -df.sample(n=100) - -# Sample fraction -df.sample(fraction=0.1) - -# Sample with seed for reproducibility -df.sample(n=100, seed=42) -``` - -## Column Renaming - -```python -# Rename specific columns -df.rename({"old_name": "new_name", "age": "years"}) - -# Rename with expression -df.select(pl.col("*").name.suffix("_renamed")) -df.select(pl.col("*").name.prefix("data_")) -df.select(pl.col("*").name.to_uppercase()) -``` diff --git a/.cursor/skills/polars-expertise/references/python/transformations.md b/.cursor/skills/polars-expertise/references/python/transformations.md deleted file mode 100644 index 5392457e..00000000 --- a/.cursor/skills/polars-expertise/references/python/transformations.md +++ /dev/null @@ -1,551 +0,0 @@ -# Polars Data Transformations - -Comprehensive guide to joins, concatenation, and reshaping operations in Polars. - -**Contents:** Joins | Concatenation | Pivoting | Unpivoting/Melting | Exploding | Transposing | Reshaping Patterns | Advanced Transformations | Performance | Use Cases - -## Joins - -Joins combine data from multiple DataFrames based on common columns. - -### Basic Join Types - -**Inner Join (intersection):** -```python -# Keep only matching rows from both DataFrames -result = df1.join(df2, on="id", how="inner") -``` - -**Left Join (all left + matches from right):** -```python -# Keep all rows from left, add matching rows from right -result = df1.join(df2, on="id", how="left") -``` - -**Outer Join (union):** -```python -# Keep all rows from both DataFrames -result = df1.join(df2, on="id", how="outer") -``` - -**Cross Join (Cartesian product):** -```python -# Every row from left with every row from right -result = df1.join(df2, how="cross") -``` - -**Semi Join (filtered left):** -```python -# Keep only left rows that have a match in right -result = df1.join(df2, on="id", how="semi") -``` - -**Anti Join (non-matching left):** -```python -# Keep only left rows that DON'T have a match in right -result = df1.join(df2, on="id", how="anti") -``` - -### Join Syntax Variations - -**Single column join:** -```python -df1.join(df2, on="id") -``` - -**Multiple columns join:** -```python -df1.join(df2, on=["id", "date"]) -``` - -**Different column names:** -```python -df1.join(df2, left_on="user_id", right_on="id") -``` - -**Multiple different columns:** -```python -df1.join( - df2, - left_on=["user_id", "date"], - right_on=["id", "timestamp"] -) -``` - -### Suffix Handling - -When both DataFrames have columns with the same name (other than join keys): - -```python -# Add suffixes to distinguish columns -result = df1.join(df2, on="id", suffix="_right") - -# Results in: value, value_right (if both had "value" column) -``` - -### Join Examples - -**Example 1: Customer Orders** -```python -customers = pl.DataFrame({ - "customer_id": [1, 2, 3, 4], - "name": ["Alice", "Bob", "Charlie", "David"] -}) - -orders = pl.DataFrame({ - "order_id": [101, 102, 103], - "customer_id": [1, 2, 1], - "amount": [100, 200, 150] -}) - -# Inner join - only customers with orders -result = customers.join(orders, on="customer_id", how="inner") - -# Left join - all customers, even without orders -result = customers.join(orders, on="customer_id", how="left") -``` - -**Example 2: Time-series data** -```python -prices = pl.DataFrame({ - "date": ["2023-01-01", "2023-01-02", "2023-01-03"], - "stock": ["AAPL", "AAPL", "AAPL"], - "price": [150, 152, 151] -}) - -volumes = pl.DataFrame({ - "date": ["2023-01-01", "2023-01-02"], - "stock": ["AAPL", "AAPL"], - "volume": [1000000, 1100000] -}) - -result = prices.join( - volumes, - on=["date", "stock"], - how="left" -) -``` - -### Asof Joins (Nearest Match) - -For time-series data, join to nearest timestamp: - -```python -# Join to nearest earlier timestamp -quotes = pl.DataFrame({ - "timestamp": [1, 2, 3, 4, 5], - "stock": ["A", "A", "A", "A", "A"], - "quote": [100, 101, 102, 103, 104] -}) - -trades = pl.DataFrame({ - "timestamp": [1.5, 3.5, 4.2], - "stock": ["A", "A", "A"], - "trade": [50, 75, 100] -}) - -result = trades.join_asof( - quotes, - on="timestamp", - by="stock", - strategy="backward" # or "forward", "nearest" -) -``` - -## Concatenation - -Concatenation stacks DataFrames together. - -### Vertical Concatenation (Stack Rows) - -```python -df1 = pl.DataFrame({"a": [1, 2], "b": [3, 4]}) -df2 = pl.DataFrame({"a": [5, 6], "b": [7, 8]}) - -# Stack rows -result = pl.concat([df1, df2], how="vertical") -# Result: 4 rows, same columns -``` - -**Handling mismatched schemas:** -```python -df1 = pl.DataFrame({"a": [1, 2], "b": [3, 4]}) -df2 = pl.DataFrame({"a": [5, 6], "c": [7, 8]}) - -# Diagonal concat - fills missing columns with nulls -result = pl.concat([df1, df2], how="diagonal") -# Result: columns a, b, c (with nulls where not present) -``` - -### Horizontal Concatenation (Stack Columns) - -```python -df1 = pl.DataFrame({"a": [1, 2, 3]}) -df2 = pl.DataFrame({"b": [4, 5, 6]}) - -# Stack columns -result = pl.concat([df1, df2], how="horizontal") -# Result: 3 rows, columns a and b -``` - -**Note:** Horizontal concat requires same number of rows. - -### Concatenation Options - -```python -# Rechunk after concatenation (better performance for subsequent operations) -result = pl.concat([df1, df2], rechunk=True) - -# Parallel execution -result = pl.concat([df1, df2], parallel=True) -``` - -### Use Cases - -**Combining data from multiple sources:** -```python -# Read multiple files and concatenate -files = ["data_2023.csv", "data_2024.csv", "data_2025.csv"] -dfs = [pl.read_csv(f) for f in files] -combined = pl.concat(dfs, how="vertical") -``` - -**Adding computed columns:** -```python -base = pl.DataFrame({"value": [1, 2, 3]}) -computed = pl.DataFrame({"doubled": [2, 4, 6]}) -result = pl.concat([base, computed], how="horizontal") -``` - -## Pivoting (Wide Format) - -Convert unique values from one column into multiple columns. - -### Basic Pivot - -```python -df = pl.DataFrame({ - "date": ["2023-01", "2023-01", "2023-02", "2023-02"], - "product": ["A", "B", "A", "B"], - "sales": [100, 150, 120, 160] -}) - -# Pivot: products become columns -pivoted = df.pivot( - values="sales", - index="date", - columns="product" -) -# Result: -# date | A | B -# 2023-01 | 100 | 150 -# 2023-02 | 120 | 160 -``` - -### Pivot with Aggregation - -When there are duplicate combinations, aggregate: - -```python -df = pl.DataFrame({ - "date": ["2023-01", "2023-01", "2023-01"], - "product": ["A", "A", "B"], - "sales": [100, 110, 150] -}) - -# Aggregate duplicates -pivoted = df.pivot( - values="sales", - index="date", - columns="product", - aggregate_function="sum" # or "mean", "max", "min", etc. -) -``` - -### Multiple Index Columns - -```python -df = pl.DataFrame({ - "region": ["North", "North", "South", "South"], - "date": ["2023-01", "2023-01", "2023-01", "2023-01"], - "product": ["A", "B", "A", "B"], - "sales": [100, 150, 120, 160] -}) - -pivoted = df.pivot( - values="sales", - index=["region", "date"], - columns="product" -) -``` - -## Unpivoting/Melting (Long Format) - -Convert multiple columns into rows (opposite of pivot). - -### Basic Unpivot - -```python -df = pl.DataFrame({ - "date": ["2023-01", "2023-02"], - "product_A": [100, 120], - "product_B": [150, 160] -}) - -# Unpivot: convert columns to rows -unpivoted = df.unpivot( - index="date", - on=["product_A", "product_B"] -) -# Result: -# date | variable | value -# 2023-01 | product_A | 100 -# 2023-01 | product_B | 150 -# 2023-02 | product_A | 120 -# 2023-02 | product_B | 160 -``` - -### Custom Column Names - -```python -unpivoted = df.unpivot( - index="date", - on=["product_A", "product_B"], - variable_name="product", - value_name="sales" -) -``` - -### Unpivot by Pattern - -```python -# Unpivot all columns matching pattern -df = pl.DataFrame({ - "id": [1, 2], - "sales_Q1": [100, 200], - "sales_Q2": [150, 250], - "sales_Q3": [120, 220], - "revenue_Q1": [1000, 2000] -}) - -# Unpivot all sales columns -unpivoted = df.unpivot( - index="id", - on=pl.col("^sales.*$") -) -``` - -## Exploding (Unnesting Lists) - -Convert list columns into multiple rows. - -### Basic Explode - -```python -df = pl.DataFrame({ - "id": [1, 2], - "values": [[1, 2, 3], [4, 5]] -}) - -# Explode list into rows -exploded = df.explode("values") -# Result: -# id | values -# 1 | 1 -# 1 | 2 -# 1 | 3 -# 2 | 4 -# 2 | 5 -``` - -### Multiple Column Explode - -```python -df = pl.DataFrame({ - "id": [1, 2], - "letters": [["a", "b"], ["c", "d"]], - "numbers": [[1, 2], [3, 4]] -}) - -# Explode multiple columns (must be same length) -exploded = df.explode("letters", "numbers") -``` - -## Transposing - -Swap rows and columns: - -```python -df = pl.DataFrame({ - "metric": ["sales", "costs", "profit"], - "Q1": [100, 60, 40], - "Q2": [150, 80, 70] -}) - -# Transpose -transposed = df.transpose( - include_header=True, - header_name="quarter", - column_names="metric" -) -# Result: quarters as rows, metrics as columns -``` - -## Reshaping Patterns - -### Pattern 1: Wide to Long to Wide - -```python -# Start wide -wide = pl.DataFrame({ - "id": [1, 2], - "A": [10, 20], - "B": [30, 40] -}) - -# To long -long = wide.unpivot(index="id", on=["A", "B"]) - -# Back to wide (maybe with transformations) -wide_again = long.pivot(values="value", index="id", columns="variable") -``` - -### Pattern 2: Nested to Flat - -```python -# Nested data -df = pl.DataFrame({ - "user": [1, 2], - "purchases": [ - [{"item": "A", "qty": 2}, {"item": "B", "qty": 1}], - [{"item": "C", "qty": 3}] - ] -}) - -# Explode and unnest -flat = ( - df.explode("purchases") - .unnest("purchases") -) -``` - -### Pattern 3: Aggregation to Pivot - -```python -# Raw data -sales = pl.DataFrame({ - "date": ["2023-01", "2023-01", "2023-02"], - "product": ["A", "B", "A"], - "sales": [100, 150, 120] -}) - -# Aggregate then pivot -result = ( - sales - .group_by("date", "product") - .agg(pl.col("sales").sum()) - .pivot(values="sales", index="date", columns="product") -) -``` - -## Advanced Transformations - -### Conditional Reshaping - -```python -# Pivot only certain values -df.filter(pl.col("year") >= 2020).pivot(...) - -# Unpivot with filtering -df.unpivot(index="id", on=pl.col("^sales.*$")) -``` - -### Multi-level Transformations - -```python -# Complex reshaping pipeline -result = ( - df - .unpivot(index="id", on=pl.col("^Q[0-9]_.*$")) - .with_columns( - pl.col("variable").str.extract(r"Q([0-9])", 1).alias("quarter"), - pl.col("variable").str.extract(r"Q[0-9]_(.*)", 1).alias("metric") - ) - .drop("variable") - .pivot(values="value", index=["id", "quarter"], columns="metric") -) -``` - -## Performance Considerations - -### Join Performance - -```python -# 1. Join on indexed/sorted columns when possible -df1_sorted = df1.sort("id") -df2_sorted = df2.sort("id") -result = df1_sorted.join(df2_sorted, on="id") - -# 2. Use appropriate join type -# semi/anti are faster than inner+filter -matches = df1.join(df2, on="id", how="semi") # Better than filtering after inner join - -# 3. Filter before joining -df1_filtered = df1.filter(pl.col("active")) -result = df1_filtered.join(df2, on="id") # Smaller join -``` - -### Concatenation Performance - -```python -# 1. Rechunk after concatenation -result = pl.concat(dfs, rechunk=True) - -# 2. Use lazy mode for large concatenations -lf1 = pl.scan_parquet("file1.parquet") -lf2 = pl.scan_parquet("file2.parquet") -result = pl.concat([lf1, lf2]).collect() -``` - -### Pivot Performance - -```python -# 1. Filter before pivoting -pivoted = df.filter(pl.col("year") == 2023).pivot(...) - -# 2. Specify aggregate function explicitly -pivoted = df.pivot(..., aggregate_function="first") # Faster than "sum" if only one value -``` - -## Common Use Cases - -### Time Series Alignment - -```python -# Align two time series with different timestamps -ts1.join_asof(ts2, on="timestamp", strategy="backward") -``` - -### Feature Engineering - -```python -# Create lag features -df.with_columns( - pl.col("value").shift(1).over("user_id").alias("prev_value"), - pl.col("value").shift(2).over("user_id").alias("prev_prev_value") -) -``` - -### Data Denormalization - -```python -# Combine normalized tables -orders.join(customers, on="customer_id").join(products, on="product_id") -``` - -### Report Generation - -```python -# Pivot for reporting -sales.pivot(values="amount", index="month", columns="product") -``` diff --git a/.cursor/skills/polars-expertise/references/rust/arrow_interop.md b/.cursor/skills/polars-expertise/references/rust/arrow_interop.md deleted file mode 100644 index e4bc676e..00000000 --- a/.cursor/skills/polars-expertise/references/rust/arrow_interop.md +++ /dev/null @@ -1,323 +0,0 @@ -# Arrow Interoperability (Rust) - -## Table of Contents -- [Polars and Arrow](#polars-and-arrow) -- [Zero-Copy Sharing](#zero-copy-sharing) -- [FFI Boundaries](#ffi-boundaries) -- [Integration with Other Libraries](#integration-with-other-libraries) -- [Memory Layout](#memory-layout) - -## Polars and Arrow - -Polars is built on Apache Arrow's columnar memory format, enabling efficient interoperability with the Arrow ecosystem. - -### Internal Structure - -``` -Polars ChunkedArray - └── Vec> (Arrow arrays) - └── ArrayData - ├── Buffers (actual data) - ├── Null bitmap - └── Child data (for nested types) -``` - -## Zero-Copy Sharing - -### To Arrow - -```rust -use polars::prelude::*; -use arrow::record_batch::RecordBatch; -use arrow::array::ArrayRef; - -fn to_arrow(df: &DataFrame) -> Vec { - // Convert DataFrame to Arrow RecordBatches - df.iter_chunks(false) - .map(|chunk| { - // Each chunk becomes a RecordBatch - let arrays: Vec = chunk - .into_iter() - .map(|arr| arr.into()) - .collect(); - - RecordBatch::try_new( - df.schema().to_arrow(CompatLevel::newest()), - arrays, - ).unwrap() - }) - .collect() -} - -// Direct conversion -fn df_to_arrow_table(df: DataFrame) -> arrow::array::RecordBatch { - let schema = df.schema().to_arrow(CompatLevel::newest()); - let chunks = df.iter_chunks(false).next().unwrap(); - let arrays: Vec = chunks.into_iter().map(|a| a.into()).collect(); - RecordBatch::try_new(schema, arrays).unwrap() -} -``` - -### From Arrow - -```rust -use polars::prelude::*; -use arrow::record_batch::RecordBatch; - -fn from_arrow(batch: RecordBatch) -> PolarsResult { - let schema = Schema::from_arrow_schema(batch.schema().as_ref()); - - let columns: Vec = batch - .columns() - .iter() - .zip(schema.iter_names()) - .map(|(arr, name)| { - let series = Series::from_arrow(name, arr.clone()).unwrap(); - series.into_column() - }) - .collect(); - - DataFrame::new_infer_height(columns) -} -``` - -### Series Conversion - -```rust -use polars::prelude::*; - -fn series_arrow_conversion() -> PolarsResult<()> { - // Create Series - let s = Series::new("values".into(), &[1i64, 2, 3, 4, 5]); - - // Get underlying Arrow arrays - let chunks = s.chunks(); - for chunk in chunks { - // chunk is a Box - println!("Array length: {}", chunk.len()); - } - - // From Arrow array - let arr = arrow::array::Int64Array::from(vec![1, 2, 3, 4, 5]); - let s = Series::from_arrow("from_arrow".into(), Box::new(arr))?; - - Ok(()) -} -``` - -## FFI Boundaries - -### C Data Interface - -Arrow's C Data Interface allows zero-copy sharing across language boundaries. - -```rust -use polars::prelude::*; -use arrow::ffi::{FFI_ArrowArray, FFI_ArrowSchema}; - -// Export to C -fn export_to_c(s: &Series) -> (FFI_ArrowSchema, FFI_ArrowArray) { - let chunks = s.chunks(); - let array = chunks.first().unwrap().clone(); - - let schema = FFI_ArrowSchema::try_from(s.dtype().to_arrow(CompatLevel::newest())).unwrap(); - let arr = FFI_ArrowArray::new(&*array); - - (schema, arr) -} - -// Import from C -fn import_from_c(schema: FFI_ArrowSchema, array: FFI_ArrowArray) -> PolarsResult { - let field = arrow::ffi::import_field_from_c(&schema)?; - let arr = arrow::ffi::import_array_from_c(array, field.data_type().clone())?; - - Series::from_arrow(field.name(), arr) -} -``` - -### PyArrow Integration - -When using pyo3: - -```rust -use pyo3::prelude::*; -use polars::prelude::*; - -#[pyfunction] -fn process_arrow_table(py: Python, table: &PyAny) -> PyResult<()> { - // Convert PyArrow table to Polars DataFrame - // This is handled by polars-python bindings - Ok(()) -} -``` - -## Integration with Other Libraries - -### DataFusion - -```rust -use polars::prelude::*; -use datafusion::prelude::*; -use datafusion::arrow::record_batch::RecordBatch; - -async fn with_datafusion(df: DataFrame) -> PolarsResult { - // Convert to RecordBatches - let batches: Vec = df - .iter_chunks(false) - .map(|chunk| { - let schema = df.schema().to_arrow(CompatLevel::newest()); - let arrays: Vec<_> = chunk.into_iter().map(|a| a.into()).collect(); - RecordBatch::try_new(schema, arrays).unwrap() - }) - .collect(); - - // Use with DataFusion - let ctx = SessionContext::new(); - let table = MemTable::try_new(batches[0].schema(), vec![batches])?; - ctx.register_table("df", Arc::new(table))?; - - let df_result = ctx - .sql("SELECT * FROM df WHERE x > 10") - .await? - .collect() - .await?; - - // Convert back to Polars - // ... conversion code - Ok(df) -} -``` - -### DuckDB - -```rust -use polars::prelude::*; -use duckdb::Connection; - -fn with_duckdb(df: &DataFrame) -> PolarsResult { - let conn = Connection::open_in_memory()?; - - // Register Arrow data with DuckDB - conn.register_arrow("my_table", df.iter_chunks(false).collect())?; - - // Query using DuckDB - let mut stmt = conn.prepare("SELECT * FROM my_table WHERE x > 10")?; - let arrow_result = stmt.query_arrow([])?; - - // Convert back to Polars - let batches: Vec<_> = arrow_result.collect(); - // ... conversion - Ok(df.clone()) -} -``` - -## Memory Layout - -### Buffer Structure - -``` -Arrow Primitive Array (e.g., Int64): -┌─────────────────────────────────────┐ -│ Validity Bitmap (bit-packed nulls) │ -├─────────────────────────────────────┤ -│ Data Buffer (contiguous i64 values) │ -└─────────────────────────────────────┘ - -Arrow String Array: -┌─────────────────────────────────────┐ -│ Validity Bitmap │ -├─────────────────────────────────────┤ -│ Offsets Buffer (i32/i64 offsets) │ -├─────────────────────────────────────┤ -│ Data Buffer (UTF-8 bytes) │ -└─────────────────────────────────────┘ -``` - -### Memory Efficiency - -```rust -use polars::prelude::*; - -fn memory_info(df: &DataFrame) { - // Estimated memory usage - println!("Estimated size: {} bytes", df.estimated_size()); - - // Per-column info - for col in df.get_columns() { - println!( - "Column '{}': {} bytes, {} chunks", - col.name(), - col.estimated_size(), - col.n_chunks() - ); - } -} -``` - -### Rechunking - -Multiple chunks can occur after operations like concatenation. Rechunking consolidates: - -```rust -use polars::prelude::*; - -fn optimize_memory(mut df: DataFrame) -> DataFrame { - // Check chunk count - let n_chunks: usize = df.get_columns() - .iter() - .map(|c| c.n_chunks()) - .max() - .unwrap_or(1); - - if n_chunks > 1 { - // Rechunk to single contiguous buffer - df = df.rechunk(); - } - - df -} -``` - -## Best Practices - -### Zero-Copy Guidelines - -1. **Avoid unnecessary copies** - - Use references when possible - - Share Arrow buffers directly - -2. **Watch for implicit copies** - - Type casting creates new buffers - - String operations often copy - -3. **Align with Arrow expectations** - - Use Arrow-native types - - Respect null semantics - -### Cross-Language Sharing - -```rust -// Good: Direct Arrow export -let arrow_table = df.to_arrow(); // Zero-copy when possible - -// Good: Stream large data -for batch in df.iter_chunks(false) { - // Process/send each batch -} - -// Avoid: Serialization when sharing -let json = df.write_json(); // Expensive for interop -``` - -### Memory Alignment - -Arrow requires 64-byte alignment for SIMD operations: - -```rust -use polars::prelude::*; - -// Polars handles alignment automatically -// But be aware when interfacing with raw buffers -let s = Series::new("a".into(), &[1i64, 2, 3]); -// Internal buffers are 64-byte aligned -``` diff --git a/.cursor/skills/polars-expertise/references/rust/core_concepts.md b/.cursor/skills/polars-expertise/references/rust/core_concepts.md deleted file mode 100644 index 96c29476..00000000 --- a/.cursor/skills/polars-expertise/references/rust/core_concepts.md +++ /dev/null @@ -1,438 +0,0 @@ -# Polars Core Concepts (Rust) - -## Table of Contents -- [Data Structures](#data-structures) -- [Creating DataFrames and Series](#creating-dataframes-and-series) -- [Expressions](#expressions) -- [Lazy vs Eager](#lazy-vs-eager) -- [Error Handling](#error-handling) - -## Data Structures - -### Hierarchy - -``` -DataFrame - └── Column (Vec) - └── Series - └── ChunkedArray - └── Arrow Arrays (Vec) -``` - -### ChunkedArray - -The fundamental data structure - a typed wrapper around Arrow arrays: - -```rust -use polars::prelude::*; - -// Create from slice -let ca = UInt32Chunked::new("foo".into(), &[1, 2, 3]); - -// From iterator -let ca: UInt32Chunked = (0..10).map(Some).collect(); - -// With builder (more control) -let mut builder = PrimitiveChunkedBuilder::::new("foo".into(), 10); -for value in 0..10 { - builder.append_value(value); -} -let ca = builder.finish(); -``` - -### Series - -Type-agnostic columnar data: - -```rust -use polars::prelude::*; - -// From slice -let s = Series::new("foo".into(), &[1, 2, 3]); - -// From iterator -let s: Series = (0..10).map(Some).collect(); - -// From ChunkedArray -let ca = UInt32Chunked::new("foo".into(), &[Some(1), None, Some(3)]); -let s = ca.into_series(); - -// Convert to Column -let col = s.into_column(); -``` - -### DataFrame - -Two-dimensional data backed by columns: - -```rust -use polars::prelude::*; -use polars::df; - -// Using df! macro -let df = df![ - "symbol" => ["AAPL", "GOOG"], - "price" => [150.0, 140.0], - "volume" => [Some(1000), None] -]?; - -// From Vec -let c1 = Column::new("symbol".into(), &["AAPL", "GOOG"]); -let c2 = Column::new("price".into(), &[150.0, 140.0]); -let df = DataFrame::new_infer_height(vec![c1, c2])?; -``` - -## Creating DataFrames and Series - -### Typed ChunkedArrays - -```rust -use polars::prelude::*; - -// Numeric types -let int32: Int32Chunked = Int32Chunked::new("a".into(), &[1, 2, 3]); -let float64: Float64Chunked = Float64Chunked::new("b".into(), &[1.0, 2.0, 3.0]); - -// String type -let utf8: StringChunked = StringChunked::new("c".into(), &["foo", "bar"]); - -// Boolean -let bool_ca: BooleanChunked = BooleanChunked::new("d".into(), &[true, false, true]); - -// With nulls -let with_nulls = Int64Chunked::new("e".into(), &[Some(1), None, Some(3)]); -``` - -### Downcasting Series - -Access the underlying ChunkedArray: - -```rust -use polars::prelude::*; - -fn process_series(s: &Series) -> PolarsResult<()> { - // Downcast to specific type - let ca: &Int32Chunked = s.i32()?; - let ca: &Float64Chunked = s.f64()?; - let ca: &StringChunked = s.str()?; - let ca: &BooleanChunked = s.bool()?; - - // Access values - for opt_val in ca.into_iter() { - match opt_val { - Some(val) => println!("{}", val), - None => println!("null"), - } - } - Ok(()) -} -``` - -## Expressions - -### Expression Contexts - -Expressions execute within specific contexts: - -```rust -use polars::prelude::*; - -fn expression_contexts(df: DataFrame) -> PolarsResult<()> { - let lf = df.lazy(); - - // select() - choose and transform columns - let selected = lf.clone() - .select([col("symbol"), col("price") * lit(100)]) - .collect()?; - - // with_columns() - add/modify while preserving existing - let with_new = lf.clone() - .with_columns([ - (col("price") * col("volume")).alias("notional"), - col("price").pct_change(lit(1)).alias("return") - ]) - .collect()?; - - // filter() - row selection - let filtered = lf.clone() - .filter(col("volume").gt(lit(1000))) - .collect()?; - - // group_by().agg() - aggregation - let grouped = lf.clone() - .group_by([col("symbol")]) - .agg([ - col("price").mean().alias("avg_price"), - col("volume").sum().alias("total_volume") - ]) - .collect()?; - - Ok(()) -} -``` - -### Expression Composition - -```rust -use polars::prelude::*; - -// Define reusable expressions -fn vwap() -> Expr { - (col("price") * col("volume")).sum() / col("volume").sum() -} - -fn volatility(window: usize) -> Expr { - col("price").pct_change(lit(1)).rolling_std(RollingOptionsFixedWindow { - window_size: window, - min_periods: window, - ..Default::default() - }) -} - -// Use in contexts -fn apply_expressions(lf: LazyFrame) -> PolarsResult { - lf.group_by([col("symbol")]) - .agg([ - vwap().alias("vwap"), - col("price").std(1).alias("std_dev") - ]) - .collect() -} -``` - -### Conditional Expressions - -```rust -use polars::prelude::*; - -fn conditionals(lf: LazyFrame) -> PolarsResult { - lf.with_columns([ - // when/then/otherwise - when(col("price").gt(lit(100))) - .then(lit("high")) - .when(col("price").gt(lit(50))) - .then(lit("medium")) - .otherwise(lit("low")) - .alias("price_tier") - ]) - .collect() -} -``` - -## Lazy vs Eager - -### Eager Mode - -Operations execute immediately: - -```rust -use polars::prelude::*; -use std::fs::File; - -fn eager_operations() -> PolarsResult<()> { - // Read file immediately - let file = File::open("data.csv")?; - let df = CsvReader::new(file).finish()?; - - // Each operation executes right away - let mask = df.column("price")?.gt(100)?; - let filtered = df.filter(&mask)?; - - Ok(()) -} -``` - -### Lazy Mode (Preferred) - -Operations build a query plan, optimized before execution: - -```rust -use polars::prelude::*; - -fn lazy_operations() -> PolarsResult { - // Build query plan (no execution yet) - let lf = LazyFrame::scan_parquet("trades.parquet", Default::default())? - .filter(col("symbol").eq(lit("AAPL"))) - .filter(col("timestamp").gt(lit("2024-01-01"))) - .group_by([col("symbol")]) - .agg([ - col("price").first().alias("open"), - col("price").max().alias("high"), - col("price").min().alias("low"), - col("price").last().alias("close"), - col("volume").sum() - ]); - - // View the optimized plan - println!("{}", lf.explain(true)?); - - // Execute - lf.collect() -} -``` - -### Converting Between Modes - -```rust -use polars::prelude::*; - -fn convert_modes(df: DataFrame) -> PolarsResult { - // Eager to Lazy - let lf: LazyFrame = df.lazy(); - - // Lazy to Eager - let df: DataFrame = lf.collect()?; - - Ok(df) -} -``` - -### Query Optimizations - -Polars automatically applies: - -| Optimization | Effect | -|-------------|--------| -| Predicate pushdown | Filters at scan level | -| Projection pushdown | Reads only needed columns | -| Slice pushdown | Limits rows early | -| Common subplan elimination | Caches repeated subqueries | -| Join ordering | Optimizes join sequence | - -```rust -use polars::prelude::*; - -// Example: only reads symbol="AAPL" rows and price/volume columns -fn optimized_read() -> PolarsResult { - LazyFrame::scan_parquet("trades.parquet", Default::default())? - .filter(col("symbol").eq(lit("AAPL"))) // Pushed to scan - .select([col("price"), col("volume")]) // Only these columns read - .collect() -} -``` - -## Error Handling - -### PolarsResult - -```rust -use polars::prelude::*; - -fn handle_errors() -> PolarsResult { - // Use ? for propagation - let df = df![ - "a" => [1, 2, 3] - ]?; - - // Explicit error handling - let result = df.column("nonexistent"); - match result { - Ok(col) => println!("Found column"), - Err(e) => println!("Error: {}", e), - } - - Ok(df) -} -``` - -### Common Error Patterns - -```rust -use polars::prelude::*; - -fn safe_operations(df: &DataFrame) -> PolarsResult<()> { - // Column access - returns Result - let col = df.column("price")?; - - // Downcasting - returns Result - let ca = col.f64()?; - - // Type casting - returns Result - let casted = col.cast(&DataType::Float32)?; - - // Arithmetic - may fail on type mismatch - let s1 = Series::new("a".into(), &[1, 2, 3]); - let s2 = Series::new("b".into(), &[1.0, 2.0, 3.0]); - let sum = &s1 + &s2; // Coerces types automatically - - Ok(()) -} -``` - -## Type System - -### Core Data Types - -```rust -use polars::prelude::*; - -// Numeric -DataType::Int8, DataType::Int16, DataType::Int32, DataType::Int64 -DataType::UInt8, DataType::UInt16, DataType::UInt32, DataType::UInt64 -DataType::Float32, DataType::Float64 - -// Text -DataType::String // UTF-8 strings -DataType::Categorical(None, CategoricalOrdering::Physical) - -// Temporal (requires dtype-* features) -DataType::Date -DataType::Datetime(TimeUnit::Microseconds, None) -DataType::Duration(TimeUnit::Microseconds) -DataType::Time - -// Complex -DataType::List(Box::new(DataType::Int64)) -DataType::Struct(vec![Field::new("a".into(), DataType::Int64)]) - -// Other -DataType::Boolean -DataType::Binary -DataType::Null -``` - -### Type Casting - -```rust -use polars::prelude::*; - -fn type_operations(s: &Series) -> PolarsResult { - // Cast to different type - let as_f64 = s.cast(&DataType::Float64)?; - - // Check type - if s.dtype() == &DataType::Int64 { - println!("It's an Int64"); - } - - // Strict casting (fails on overflow) - let strict = s.strict_cast(&DataType::Int32)?; - - Ok(as_f64) -} -``` - -### Null Handling - -```rust -use polars::prelude::*; - -fn null_operations(df: DataFrame) -> PolarsResult { - df.lazy() - .with_columns([ - // Check nulls - col("value").is_null().alias("is_missing"), - - // Fill nulls with constant - col("value").fill_null(lit(0)).alias("filled_const"), - - // Fill with strategy - col("value").forward_fill(None).alias("filled_forward"), - - // Drop nulls (use filter) - // Filter rows where value is not null - ]) - .filter(col("value").is_not_null()) - .collect() -} -``` diff --git a/.cursor/skills/polars-expertise/references/rust/features.md b/.cursor/skills/polars-expertise/references/rust/features.md deleted file mode 100644 index 042e4144..00000000 --- a/.cursor/skills/polars-expertise/references/rust/features.md +++ /dev/null @@ -1,352 +0,0 @@ -# Polars Feature Flags (Rust) - -## Table of Contents -- [Feature System Overview](#feature-system-overview) -- [Core Features](#core-features) -- [I/O Features](#io-features) -- [Data Type Features](#data-type-features) -- [Operation Features](#operation-features) -- [Performance Features](#performance-features) -- [Common Configurations](#common-configurations) - -## Feature System Overview - -Polars uses Rust's feature flags to reduce compile times and binary size. Only enable what you need. - -### Minimal Setup - -```toml -[dependencies] -polars = "0.46" # Uses default features -``` - -Default features include: `docs`, `zip_with`, `csv`, `parquet`, `temporal`, `fmt`, `dtype-slim` - -### Custom Feature Selection - -```toml -[dependencies] -polars = { version = "0.46", default-features = false, features = [ - "lazy", - "parquet", - "temporal", - "dtype-datetime" -] } -``` - -## Core Features - -### Lazy API - -```toml -features = ["lazy"] # Required for LazyFrame -``` - -Enables: -- `LazyFrame` and query optimization -- Expression DSL (`col()`, `lit()`, etc.) -- Streaming execution - -```rust -use polars::prelude::*; - -// Requires "lazy" feature -let lf = LazyFrame::scan_parquet("data.parquet", Default::default())?; -let result = lf.filter(col("x").gt(lit(10))).collect()?; -``` - -### SQL Support - -```toml -features = ["sql"] -``` - -```rust -use polars::prelude::*; -use polars::sql::SQLContext; - -let mut ctx = SQLContext::new(); -ctx.register("df", df.lazy()); -let result = ctx.execute("SELECT * FROM df WHERE x > 10")?.collect()?; -``` - -### Regex Support - -```toml -features = ["lazy", "regex"] -``` - -```rust -// Column selection by regex -df.lazy().select([col("^sales_.*$")]) -``` - -## I/O Features - -| Feature | Description | -|---------|-------------| -| `csv` | CSV reading/writing | -| `parquet` | Parquet format (recommended) | -| `ipc` | Arrow IPC format | -| `ipc_streaming` | Streaming IPC | -| `json` | JSON reading/writing | -| `avro` | Apache Avro format | -| `decompress` | Auto-decompress gzip/zlib/zstd | - -### Cloud Storage - -```toml -features = ["cloud"] # Base cloud support -features = ["aws"] # AWS S3 -features = ["azure"] # Azure Blob -features = ["gcp"] # Google Cloud Storage -features = ["http"] # HTTP sources -``` - -```rust -// Requires "aws" feature -let lf = LazyFrame::scan_parquet("s3://bucket/data.parquet", Default::default())?; -``` - -## Data Type Features - -### Minimal Set (dtype-slim) - -```toml -features = ["dtype-slim"] # Included in default -``` - -Includes: `Date`, `Datetime`, `Duration` - -### Full Set (dtype-full) - -```toml -features = ["dtype-full"] -``` - -Includes all optional types: -- `dtype-date`, `dtype-datetime`, `dtype-duration`, `dtype-time` -- `dtype-array` (fixed-size arrays) -- `dtype-i8`, `dtype-i16`, `dtype-i128` -- `dtype-u8`, `dtype-u16`, `dtype-u128` -- `dtype-f16` -- `dtype-decimal` -- `dtype-categorical` -- `dtype-struct` - -### Individual Type Features - -```toml -# Enable only what you need -features = [ - "dtype-datetime", - "dtype-categorical", - "dtype-struct" -] -``` - -**Note:** If you get compile errors about missing types, you likely need to enable the corresponding dtype feature. - -## Operation Features - -### DataFrame Operations - -| Feature | Description | -|---------|-------------| -| `rows` | Row-based operations, pivot, transpose | -| `pivot` | Pivot tables (requires `rows`, `dtype-struct`) | -| `asof_join` | As-of joins for time series | -| `cross_join` | Cartesian product joins | -| `semi_anti_join` | Semi and anti joins | -| `dynamic_group_by` | Time-based group by | -| `partition_by` | Partition DataFrame by groups | -| `diagonal_concat` | Concat with different schemas | -| `dataframe_arithmetic` | DataFrame arithmetic ops | - -### Series/Expression Operations - -| Feature | Description | -|---------|-------------| -| `abs` | Absolute values | -| `cum_agg` | Cumulative sum/min/max | -| `diff` | Difference between elements | -| `pct_change` | Percentage change | -| `rolling_window` | Rolling aggregations | -| `rolling_window_by` | Time-based rolling | -| `rank` | Ranking algorithms | -| `is_in` | Membership checks | -| `is_between` | Range checks | -| `mode` | Most frequent values | -| `ewma` | Exponential moving average | -| `interpolate` | Interpolate nulls | -| `strings` | String utilities | -| `trigonometry` | Trig functions | -| `log` | Logarithms | - -### List Operations - -```toml -features = [ - "list_eval", # Apply expressions over lists - "list_gather", # Take sublist by indices - "list_to_struct", # Convert list to struct - "list_sets", # Set operations on lists -] -``` - -## Performance Features - -### Nightly Optimizations - -```toml -features = ["nightly"] # Requires nightly Rust -``` - -Enables: -- SIMD acceleration -- Specialization -- Additional optimizations - -```bash -# Build with nightly -rustup override set nightly -cargo build --release --features nightly -``` - -### SIMD - -```toml -features = ["simd"] # Included with "nightly" -``` - -### AVX-512 - -```toml -features = ["avx512"] # For CPUs with AVX-512 -``` - -### Performant Mode - -```toml -features = ["performant"] -``` - -Enables additional fast paths at the cost of slower compilation: -- `chunked_ids` -- `dtype-u8`, `dtype-u16`, `dtype-f16`, `dtype-struct` -- `cse` (common subexpression elimination) -- `fused` (fused operations) - -### Big Index - -```toml -features = ["bigidx"] # For >2^32 rows -``` - -Uses 64-bit indices instead of 32-bit. Slightly slower but supports massive datasets. - -## Common Configurations - -### Financial Data Processing - -```toml -[dependencies] -polars = { version = "0.46", default-features = false, features = [ - "lazy", - "parquet", - "csv", - "temporal", - "dtype-datetime", - "dtype-duration", - "dtype-f64", - "asof_join", - "dynamic_group_by", - "rolling_window", - "rolling_window_by", - "cum_agg", - "pct_change", - "diff", - "rank", - "fmt" -] } -``` - -### Data Pipeline / ETL - -```toml -[dependencies] -polars = { version = "0.46", default-features = false, features = [ - "lazy", - "parquet", - "csv", - "json", - "ipc", - "decompress", - "cloud", - "aws", - "dtype-full", - "streaming", - "diagonal_concat", - "partition_by", - "fmt" -] } -``` - -### ML Feature Engineering - -```toml -[dependencies] -polars = { version = "0.46", default-features = false, features = [ - "lazy", - "parquet", - "dtype-full", - "ndarray", - "strings", - "is_in", - "is_between", - "rank", - "mode", - "interpolate", - "pivot", - "to_dummies", - "fmt" -] } -``` - -### Maximum Performance - -```toml -[dependencies] -polars = { version = "0.46", features = [ - "nightly", - "performant" -] } -``` - -Build command: -```bash -RUSTFLAGS='-C target-cpu=native' cargo build --release -``` - -### Minimal Binary Size - -```toml -[dependencies] -polars = { version = "0.46", default-features = false, features = [ - "lazy", - "parquet" # Or just "csv" if you don't need parquet -] } -``` - -## Compile Time Tips - -1. **Start minimal** - Add features as needed -2. **Use sccache** - Cache compilation artifacts -3. **Incremental builds** - Avoid clean builds -4. **Separate dev/prod** - More features for dev, minimal for prod - -```bash -# Install sccache -cargo install sccache -export RUSTC_WRAPPER=sccache -``` diff --git a/.cursor/skills/polars-expertise/references/rust/io_guide.md b/.cursor/skills/polars-expertise/references/rust/io_guide.md deleted file mode 100644 index a8bdc74b..00000000 --- a/.cursor/skills/polars-expertise/references/rust/io_guide.md +++ /dev/null @@ -1,382 +0,0 @@ -# Polars I/O Guide (Rust) - -## Table of Contents -- [CSV](#csv) -- [Parquet](#parquet) -- [IPC/Arrow](#ipcarrow) -- [JSON](#json) -- [Cloud Storage](#cloud-storage) -- [Streaming](#streaming) - -## CSV - -**Feature:** `csv` - -### Reading CSV - -```rust -use polars::prelude::*; -use std::fs::File; - -// Basic read -fn read_csv() -> PolarsResult { - let file = File::open("data.csv")?; - CsvReader::new(file).finish() -} - -// With options -fn read_csv_options() -> PolarsResult { - let file = File::open("data.csv")?; - CsvReader::new(file) - .with_has_header(true) - .with_separator(b',') - .with_n_rows(Some(1000)) // Limit rows - .with_columns(Some(Arc::new(vec!["col1".into(), "col2".into()]))) - .with_dtypes(Some(Arc::new(Schema::from_iter([ - Field::new("id".into(), DataType::Int64), - Field::new("price".into(), DataType::Float64), - ])))) - .with_null_values(Some(NullValues::AllColumnsSingle("NA".into()))) - .finish() -} - -// Lazy (recommended for large files) -fn scan_csv() -> PolarsResult { - LazyCsvReader::new("data.csv") - .with_has_header(true) - .with_separator(b',') - .finish() -} -``` - -### Writing CSV - -```rust -use polars::prelude::*; -use std::fs::File; - -fn write_csv(df: &mut DataFrame) -> PolarsResult<()> { - let mut file = File::create("output.csv")?; - CsvWriter::new(&mut file) - .include_header(true) - .with_separator(b',') - .with_quote_char(b'"') - .finish(df)?; - Ok(()) -} -``` - -## Parquet - -**Feature:** `parquet` - -### Reading Parquet - -```rust -use polars::prelude::*; -use std::fs::File; - -// Eager -fn read_parquet() -> PolarsResult { - let file = File::open("data.parquet")?; - ParquetReader::new(file).finish() -} - -// With options -fn read_parquet_options() -> PolarsResult { - let file = File::open("data.parquet")?; - ParquetReader::new(file) - .with_columns(Some(vec!["col1".into(), "col2".into()])) - .with_n_rows(Some(1000)) - .with_parallel(ParallelStrategy::Auto) - .finish() -} - -// Lazy (recommended) -fn scan_parquet() -> PolarsResult { - LazyFrame::scan_parquet("data.parquet", Default::default()) -} - -// Multiple files -fn scan_parquet_glob() -> PolarsResult { - LazyFrame::scan_parquet("data/*.parquet", Default::default()) -} - -// With scan options -fn scan_parquet_options() -> PolarsResult { - let args = ScanArgsParquet { - n_rows: Some(10000), - parallel: ParallelStrategy::Auto, - rechunk: false, - ..Default::default() - }; - LazyFrame::scan_parquet("data.parquet", args) -} -``` - -### Writing Parquet - -```rust -use polars::prelude::*; -use std::fs::File; - -fn write_parquet(df: &mut DataFrame) -> PolarsResult { - let file = File::create("output.parquet")?; - ParquetWriter::new(file) - .with_compression(ParquetCompression::Snappy) // Fast - // .with_compression(ParquetCompression::Zstd(Some(ZstdLevel::try_new(3)?))) // Better compression - .with_statistics(StatisticsOptions::full()) // Enables predicate pushdown - .finish(df) -} - -// Partitioned write -fn write_partitioned(lf: LazyFrame) -> PolarsResult<()> { - lf.sink_parquet( - "output/", - ParquetWriteOptions { - compression: ParquetCompression::Snappy, - statistics: StatisticsOptions::full(), - ..Default::default() - }, - ) -} -``` - -## IPC/Arrow - -**Feature:** `ipc` - -### Reading IPC - -```rust -use polars::prelude::*; -use std::fs::File; - -fn read_ipc() -> PolarsResult { - let file = File::open("data.arrow")?; - IpcReader::new(file).finish() -} - -fn scan_ipc() -> PolarsResult { - LazyFrame::scan_ipc("data.arrow", Default::default()) -} -``` - -### Writing IPC - -```rust -use polars::prelude::*; -use std::fs::File; - -fn write_ipc(df: &mut DataFrame) -> PolarsResult<()> { - let file = File::create("output.arrow")?; - IpcWriter::new(file) - .with_compression(Some(IpcCompression::ZSTD)) - .finish(df) -} -``` - -## JSON - -**Feature:** `json` - -### Reading JSON - -```rust -use polars::prelude::*; -use std::fs::File; - -// NDJSON (newline-delimited JSON) - recommended -fn read_ndjson() -> PolarsResult { - let file = File::open("data.ndjson")?; - JsonReader::new(file) - .with_json_format(JsonFormat::JsonLines) - .finish() -} - -// Standard JSON -fn read_json() -> PolarsResult { - let file = File::open("data.json")?; - JsonReader::new(file) - .with_json_format(JsonFormat::Json) - .finish() -} - -// Lazy NDJSON scan -fn scan_ndjson() -> PolarsResult { - LazyFrame::scan_ndjson("data.ndjson", Default::default()) -} -``` - -### Writing JSON - -```rust -use polars::prelude::*; -use std::fs::File; - -fn write_json(df: &mut DataFrame) -> PolarsResult<()> { - let mut file = File::create("output.json")?; - JsonWriter::new(&mut file) - .with_json_format(JsonFormat::Json) - .finish(df) -} - -fn write_ndjson(df: &mut DataFrame) -> PolarsResult<()> { - let mut file = File::create("output.ndjson")?; - JsonWriter::new(&mut file) - .with_json_format(JsonFormat::JsonLines) - .finish(df) -} -``` - -## Cloud Storage - -**Features:** `cloud`, `aws`, `azure`, `gcp` - -### AWS S3 - -```rust -use polars::prelude::*; - -// Set credentials via environment -// AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION - -fn read_s3() -> PolarsResult { - LazyFrame::scan_parquet("s3://bucket/path/data.parquet", Default::default()) -} - -fn read_s3_glob() -> PolarsResult { - LazyFrame::scan_parquet("s3://bucket/path/*.parquet", Default::default()) -} -``` - -### Azure Blob - -```rust -use polars::prelude::*; - -// Set credentials via environment -// AZURE_STORAGE_ACCOUNT_NAME, AZURE_STORAGE_ACCOUNT_KEY - -fn read_azure() -> PolarsResult { - LazyFrame::scan_parquet("az://container/path/data.parquet", Default::default()) -} -``` - -### Google Cloud Storage - -```rust -use polars::prelude::*; - -// Set credentials via environment -// GOOGLE_APPLICATION_CREDENTIALS - -fn read_gcs() -> PolarsResult { - LazyFrame::scan_parquet("gs://bucket/path/data.parquet", Default::default()) -} -``` - -### HTTP - -```rust -use polars::prelude::*; - -fn read_http() -> PolarsResult { - LazyFrame::scan_parquet("https://example.com/data.parquet", Default::default()) -} -``` - -## Streaming - -### Streaming Reads - -```rust -use polars::prelude::*; - -fn streaming_process() -> PolarsResult { - // Streaming is handled automatically with new_streaming feature - LazyFrame::scan_parquet("very_large.parquet", Default::default())? - .filter(col("value").gt(lit(100))) - .group_by([col("category")]) - .agg([col("value").sum()]) - .collect() // Streaming execution when possible -} -``` - -### Streaming Writes (Sink) - -```rust -use polars::prelude::*; - -fn sink_to_parquet() -> PolarsResult<()> { - LazyFrame::scan_csv("large.csv", Default::default())? - .filter(col("value").gt(lit(100))) - .sink_parquet( - "output.parquet", - ParquetWriteOptions::default(), - ) -} -``` - -### Batched CSV Reading - -```rust -use polars::prelude::*; -use std::fs::File; - -fn process_csv_batches() -> PolarsResult> { - let file = File::open("large.csv")?; - let reader = CsvReader::new(file) - .batched(None)?; // Returns batched reader - - let mut results = Vec::new(); - while let Some(batch) = reader.next_batches(1)? { - for df in batch { - // Process each batch - let processed = df.filter(&df.column("x")?.gt(100)?)?; - results.push(processed); - } - } - Ok(results) -} -``` - -## Best Practices - -### Format Selection - -| Format | Use When | -|--------|----------| -| Parquet | Large files, archival, data lakes | -| CSV | Human-readable, legacy systems | -| IPC/Arrow | Fast transfer, zero-copy | -| NDJSON | Streaming JSON, logs | - -### Reading Large Files - -```rust -// 1. Always use lazy mode -let lf = LazyFrame::scan_parquet("large.parquet", Default::default())?; - -// 2. Project columns early -let lf = lf.select([col("a"), col("b")]); - -// 3. Filter early (predicate pushdown) -let lf = lf.filter(col("a").gt(lit(100))); - -// 4. Collect at the end -let df = lf.collect()?; -``` - -### Writing Large Files - -```rust -// 1. Use Parquet with compression -ParquetWriter::new(file) - .with_compression(ParquetCompression::Zstd(Some(ZstdLevel::try_new(3)?))) - .with_statistics(StatisticsOptions::full()) - .finish(&mut df)?; - -// 2. Use sink for streaming writes -lf.sink_parquet("output.parquet", ParquetWriteOptions::default())?; -``` diff --git a/.cursor/skills/polars-expertise/references/rust/operations.md b/.cursor/skills/polars-expertise/references/rust/operations.md deleted file mode 100644 index 8273c975..00000000 --- a/.cursor/skills/polars-expertise/references/rust/operations.md +++ /dev/null @@ -1,479 +0,0 @@ -# Polars Operations (Rust) - -## Table of Contents -- [Selection and Filtering](#selection-and-filtering) -- [Aggregations](#aggregations) -- [Window Functions](#window-functions) -- [Joins](#joins) -- [Sorting](#sorting) -- [Transformations](#transformations) - -## Selection and Filtering - -### Column Selection - -```rust -use polars::prelude::*; - -fn select_columns(df: DataFrame) -> PolarsResult { - // Select specific columns - let selected = df.lazy() - .select([col("a"), col("b"), col("c")]) - .collect()?; - - // Select by pattern (requires "regex" feature) - let by_pattern = df.lazy() - .select([col("^sales_.*$")]) // All columns starting with "sales_" - .collect()?; - - // Select all except some - let excluded = df.lazy() - .select([all().exclude(["id", "timestamp"])]) - .collect()?; - - // Select by dtype - let numeric = df.lazy() - .select([dtype_cols([DataType::Float64, DataType::Int64])]) - .collect()?; - - Ok(selected) -} -``` - -### Row Filtering - -```rust -use polars::prelude::*; - -fn filter_rows(df: DataFrame) -> PolarsResult { - let lf = df.lazy(); - - // Single condition - let filtered = lf.clone() - .filter(col("age").gt(lit(25))) - .collect()?; - - // Multiple conditions (AND) - let multi = lf.clone() - .filter( - col("age").gt(lit(25)) - .and(col("city").eq(lit("NY"))) - ) - .collect()?; - - // OR condition - let or_cond = lf.clone() - .filter( - col("age").gt(lit(30)) - .or(col("salary").gt(lit(100000))) - ) - .collect()?; - - // NOT condition - let not_cond = lf.clone() - .filter(col("active").not()) - .collect()?; - - // Membership - let in_list = lf.clone() - .filter(col("city").is_in(lit(Series::new("".into(), ["NY", "LA", "SF"])))) - .collect()?; - - // Range - let in_range = lf.clone() - .filter(col("age").is_between(lit(25), lit(35), ClosedInterval::Both)) - .collect()?; - - // Null checks - let not_null = lf.clone() - .filter(col("value").is_not_null()) - .collect()?; - - Ok(filtered) -} -``` - -### Add/Modify Columns - -```rust -use polars::prelude::*; - -fn add_columns(df: DataFrame) -> PolarsResult { - df.lazy() - .with_columns([ - // Arithmetic - (col("price") * col("quantity")).alias("total"), - - // Conditional - when(col("age").gt(lit(18))) - .then(lit("adult")) - .otherwise(lit("minor")) - .alias("status"), - - // String operations - col("name").str().to_uppercase().alias("name_upper"), - - // Cast types - col("id").cast(DataType::String).alias("id_str"), - ]) - .collect() -} -``` - -## Aggregations - -### Group By - -```rust -use polars::prelude::*; - -fn group_by_operations(df: DataFrame) -> PolarsResult { - // Basic group by - let grouped = df.clone().lazy() - .group_by([col("category")]) - .agg([ - col("value").sum().alias("total"), - col("value").mean().alias("average"), - col("value").count().alias("count"), - ]) - .collect()?; - - // Multiple group columns - let multi_group = df.clone().lazy() - .group_by([col("category"), col("region")]) - .agg([col("value").sum()]) - .collect()?; - - // Maintain order - let ordered = df.clone().lazy() - .group_by([col("category")]) - .agg([col("value").sum()]) - .sort([col("category")], Default::default()) - .collect()?; - - Ok(grouped) -} -``` - -### Aggregation Functions - -```rust -use polars::prelude::*; - -fn aggregation_functions(df: DataFrame) -> PolarsResult { - df.lazy() - .group_by([col("group")]) - .agg([ - // Count - len().alias("count"), - col("id").n_unique().alias("unique_count"), - - // Statistics - col("value").sum().alias("total"), - col("value").mean().alias("average"), - col("value").median().alias("median"), - col("value").std(1).alias("std_dev"), - col("value").var(1).alias("variance"), - col("value").min().alias("minimum"), - col("value").max().alias("maximum"), - col("value").quantile(lit(0.95), QuantileMethod::Linear).alias("p95"), - - // First/Last - col("timestamp").first().alias("first_ts"), - col("timestamp").last().alias("last_ts"), - - // List aggregation - col("item").alias("all_items"), // Collects into list - ]) - .collect() -} -``` - -### Conditional Aggregations - -```rust -use polars::prelude::*; - -fn conditional_aggregations(df: DataFrame) -> PolarsResult { - df.lazy() - .group_by([col("category")]) - .agg([ - // Count by condition - col("value").filter(col("value").gt(lit(100))).count().alias("high_count"), - - // Sum by condition - col("value").filter(col("active")).sum().alias("active_total"), - - // Conditional expression - when(col("type").eq(lit("A"))) - .then(col("value")) - .otherwise(lit(0)) - .sum() - .alias("type_a_total"), - ]) - .collect() -} -``` - -## Window Functions - -### Basic Window Operations - -```rust -use polars::prelude::*; - -fn window_functions(df: DataFrame) -> PolarsResult { - df.lazy() - .with_columns([ - // Group statistics (preserves row count) - col("value").mean().over([col("category")]).alias("group_mean"), - col("value").sum().over([col("category")]).alias("group_total"), - - // Ranking - col("value").rank(RankOptions::default(), None).over([col("category")]).alias("rank"), - - // Cumulative - col("value").cum_sum(false).over([col("category")]).alias("cumsum"), - - // Lag/Lead - col("value").shift(lit(1)).over([col("category")]).alias("prev_value"), - col("value").shift(lit(-1)).over([col("category")]).alias("next_value"), - - // Difference from previous - (col("value") - col("value").shift(lit(1))).over([col("category")]).alias("diff"), - ]) - .collect() -} -``` - -### Rolling Windows - -```rust -use polars::prelude::*; - -fn rolling_windows(df: DataFrame) -> PolarsResult { - df.lazy() - .with_columns([ - // Row-based rolling - col("value").rolling_mean(RollingOptionsFixedWindow { - window_size: 20, - min_periods: 20, - ..Default::default() - }).alias("rolling_mean"), - - col("value").rolling_std(RollingOptionsFixedWindow { - window_size: 20, - min_periods: 20, - ..Default::default() - }).alias("rolling_std"), - - col("value").rolling_sum(RollingOptionsFixedWindow { - window_size: 10, - min_periods: 1, // Allow partial windows - ..Default::default() - }).alias("rolling_sum"), - ]) - .collect() -} -``` - -## Joins - -### Basic Joins - -```rust -use polars::prelude::*; - -fn join_operations(df1: DataFrame, df2: DataFrame) -> PolarsResult<()> { - let lf1 = df1.clone().lazy(); - let lf2 = df2.clone().lazy(); - - // Inner join - let inner = lf1.clone() - .inner_join(lf2.clone(), col("id"), col("id")) - .collect()?; - - // Left join - let left = lf1.clone() - .left_join(lf2.clone(), col("id"), col("id")) - .collect()?; - - // Full outer join - let outer = lf1.clone() - .full_join(lf2.clone(), col("id"), col("id")) - .collect()?; - - // Multiple keys - let multi_key = lf1.clone() - .join( - lf2.clone(), - [col("id"), col("date")], - [col("id"), col("date")], - JoinArgs::new(JoinType::Inner), - ) - .collect()?; - - // Different column names - let diff_names = lf1.clone() - .join( - lf2.clone(), - [col("user_id")], - [col("id")], - JoinArgs::new(JoinType::Left), - ) - .collect()?; - - Ok(()) -} -``` - -### Semi and Anti Joins - -```rust -use polars::prelude::*; - -fn semi_anti_joins(df1: DataFrame, df2: DataFrame) -> PolarsResult<()> { - let lf1 = df1.lazy(); - let lf2 = df2.lazy(); - - // Semi join: keep left rows that have a match in right - let semi = lf1.clone() - .join(lf2.clone(), [col("id")], [col("id")], JoinArgs::new(JoinType::Semi)) - .collect()?; - - // Anti join: keep left rows that don't have a match in right - let anti = lf1.clone() - .join(lf2.clone(), [col("id")], [col("id")], JoinArgs::new(JoinType::Anti)) - .collect()?; - - Ok(()) -} -``` - -### As-Of Joins - -**Feature:** `asof_join` - -```rust -use polars::prelude::*; - -fn asof_join(trades: DataFrame, quotes: DataFrame) -> PolarsResult { - // Join to nearest earlier timestamp - trades.lazy() - .join_asof_by( - quotes.lazy(), - col("timestamp"), - col("timestamp"), - [col("symbol")], // Match by symbol first - [col("symbol")], - AsofStrategy::Backward, // Nearest earlier - None, // No tolerance - ) - .collect() -} -``` - -## Sorting - -```rust -use polars::prelude::*; - -fn sort_operations(df: DataFrame) -> PolarsResult { - // Single column - let sorted = df.clone().lazy() - .sort([col("value")], Default::default()) - .collect()?; - - // Descending - let desc = df.clone().lazy() - .sort( - [col("value")], - SortMultipleOptions::default().with_order_descending(true), - ) - .collect()?; - - // Multiple columns - let multi = df.clone().lazy() - .sort_by_exprs( - vec![col("category"), col("value")], - SortMultipleOptions::default() - .with_order_descending_multi([false, true]), // category asc, value desc - ) - .collect()?; - - // Nulls last - let nulls_last = df.clone().lazy() - .sort( - [col("value")], - SortMultipleOptions::default().with_nulls_last(true), - ) - .collect()?; - - Ok(sorted) -} -``` - -## Transformations - -### Concatenation - -```rust -use polars::prelude::*; - -fn concat_operations(df1: DataFrame, df2: DataFrame) -> PolarsResult { - // Vertical (stack rows) - let stacked = concat( - [df1.clone().lazy(), df2.clone().lazy()], - UnionArgs::default(), - )?.collect()?; - - // Horizontal (stack columns) - let horizontal = concat( - [df1.lazy(), df2.lazy()], - UnionArgs { - how: JoinType::Cross, // Cross join for horizontal - ..Default::default() - }, - )?.collect()?; - - Ok(stacked) -} -``` - -### Pivot and Unpivot - -```rust -use polars::prelude::*; - -fn pivot_operations(df: DataFrame) -> PolarsResult<()> { - // Pivot (wide format) - requires "pivot" feature - let pivoted = pivot::pivot( - &df, - [PlSmallStr::from_static("date")], // index - Some([PlSmallStr::from_static("product")]), // columns - Some([PlSmallStr::from_static("sales")]), // values - false, - Some(first()), // Aggregation function - None, - )?; - - // Unpivot (long format) - let unpivoted = df.unpivot( - [PlSmallStr::from_static("date")], // id columns - [PlSmallStr::from_static("A"), PlSmallStr::from_static("B")], // value columns - )?; - - Ok(()) -} -``` - -### Explode - -```rust -use polars::prelude::*; - -fn explode_operation(df: DataFrame) -> PolarsResult { - // Explode list column into rows - df.explode([PlSmallStr::from("items")]) -} -``` diff --git a/.cursor/skills/polars-expertise/references/rust/performance.md b/.cursor/skills/polars-expertise/references/rust/performance.md deleted file mode 100644 index 61426834..00000000 --- a/.cursor/skills/polars-expertise/references/rust/performance.md +++ /dev/null @@ -1,322 +0,0 @@ -# Polars Performance Optimization (Rust) - -## Table of Contents -- [Custom Allocators](#custom-allocators) -- [Compiler Optimizations](#compiler-optimizations) -- [Feature Flags for Performance](#feature-flags-for-performance) -- [Environment Variables](#environment-variables) -- [Code Patterns](#code-patterns) -- [Benchmarking](#benchmarking) - -## Custom Allocators - -Using a custom allocator can improve performance by 10-25%. - -### Jemalloc (Recommended for Linux/macOS) - -```toml -# Cargo.toml -[dependencies] -tikv-jemallocator = "0.6" -``` - -```rust -// main.rs or lib.rs - at the top -use tikv_jemallocator::Jemalloc; - -#[global_allocator] -static GLOBAL: Jemalloc = Jemalloc; -``` - -### Mimalloc (Cross-platform) - -```toml -# Cargo.toml -[dependencies] -mimalloc = { version = "0.1", default-features = false } -``` - -```rust -use mimalloc::MiMalloc; - -#[global_allocator] -static GLOBAL: MiMalloc = MiMalloc; -``` - -### Allocator Comparison - -| Allocator | Best For | Notes | -|-----------|----------|-------| -| Jemalloc | Linux, macOS | Best overall for OLAP | -| Mimalloc | Windows, cross-platform | Good general purpose | -| System | Default | Baseline, no extra deps | - -## Compiler Optimizations - -### Release Profile - -```toml -# Cargo.toml -[profile.release] -lto = "thin" # Link-time optimization -codegen-units = 1 # Better optimization, slower compile -opt-level = 3 # Maximum optimization -``` - -### Native CPU Features - -```bash -# Build for current CPU architecture -RUSTFLAGS='-C target-cpu=native' cargo build --release - -# Or specify architecture -RUSTFLAGS='-C target-cpu=skylake' cargo build --release -``` - -### Profile-Guided Optimization (PGO) - -```bash -# Step 1: Build with instrumentation -RUSTFLAGS='-Cprofile-generate=/tmp/pgo-data' cargo build --release - -# Step 2: Run representative workload -./target/release/my_app --process-data - -# Step 3: Merge profile data -llvm-profdata merge -o /tmp/pgo-data/merged.profdata /tmp/pgo-data - -# Step 4: Build with profile data -RUSTFLAGS='-Cprofile-use=/tmp/pgo-data/merged.profdata' cargo build --release -``` - -## Feature Flags for Performance - -### Nightly Features - -```toml -features = ["nightly"] -``` - -Requires nightly Rust: -```bash -rustup override set nightly -``` - -Enables: -- SIMD vectorization -- Specialization -- Additional optimizations - -### SIMD - -```toml -features = ["simd"] # Enabled with "nightly" -``` - -### Performant Mode - -```toml -features = ["performant"] -``` - -Enables fast paths at cost of compile time: -- Common subexpression elimination -- Fused operations -- Optimal dtype handling - -### AVX-512 - -```toml -features = ["avx512"] -``` - -For modern Intel CPUs with AVX-512 support. - -## Environment Variables - -### Thread Pool - -```bash -# Set number of threads (default: num_cpus) -export POLARS_MAX_THREADS=8 - -# Or in Rust -std::env::set_var("POLARS_MAX_THREADS", "8"); -``` - -### Partitioning - -```bash -# Disable partitioned group_by (for debugging) -export POLARS_NO_PARTITION=1 - -# Force partitioned group_by -export POLARS_FORCE_PARTITION=1 - -# Partition threshold (default 1000) -export POLARS_PARTITION_UNIQUE_COUNT=500 -``` - -### Debugging - -```bash -# Verbose output -export POLARS_VERBOSE=1 - -# Panic on error (for debugging) -export POLARS_PANIC_ON_ERR=1 - -# Include backtrace in errors -export POLARS_BACKTRACE_IN_ERR=1 -``` - -### Parquet - -```bash -# Ignore parquet statistics (for debugging) -export POLARS_NO_PARQUET_STATISTICS=1 -``` - -## Code Patterns - -### Use Lazy Mode - -```rust -// GOOD: Lazy mode enables optimizations -let result = LazyFrame::scan_parquet("data.parquet", Default::default())? - .filter(col("x").gt(lit(10))) - .select([col("x"), col("y")]) - .collect()?; - -// BAD: Eager reads all data first -let df = ParquetReader::new(File::open("data.parquet")?).finish()?; -let filtered = df.filter(&df.column("x")?.gt(10)?)?; -``` - -### Project Early - -```rust -// GOOD: Project columns early -lf.select([col("a"), col("b")]) - .filter(col("a").gt(lit(10))) - .collect()? - -// BAD: Read all columns, filter, then project -lf.filter(col("a").gt(lit(10))) - .collect()? - .select(["a", "b"])? -``` - -### Avoid Python-style UDFs - -```rust -// GOOD: Native expressions -lf.with_columns([col("x") * lit(2)]) - -// BAD: map function (sequential, not parallelized) -lf.with_columns([col("x").map(|s| { - Ok(Some(s.multiply(&Series::new("".into(), [2]))?)) -}, GetOutput::same_type())]) -``` - -### Use Categorical for Low Cardinality - -```rust -// GOOD: Categorical for repeated strings -let df = df.lazy() - .with_columns([col("symbol").cast(DataType::Categorical(None, Default::default()))]) - .collect()?; - -// Faster groupby and joins with categorical -``` - -### Rechunk After Operations - -```rust -// After multiple concatenations, rechunk for better cache locality -let combined = pl::concat(dfs, UnionArgs::default())?; -let combined = combined.rechunk(); -``` - -### Streaming for Large Data - -```rust -// Use streaming for out-of-memory data -let result = lf - .filter(col("x").gt(lit(10))) - .group_by([col("category")]) - .agg([col("value").sum()]) - .collect()?; // Streaming is automatic in new_streaming -``` - -## Benchmarking - -### Setup - -```toml -# Cargo.toml -[dev-dependencies] -criterion = { version = "0.5", features = ["html_reports"] } - -[[bench]] -name = "my_benchmark" -harness = false -``` - -### Basic Benchmark - -```rust -// benches/my_benchmark.rs -use criterion::{criterion_group, criterion_main, Criterion, black_box}; -use polars::prelude::*; - -fn benchmark_filter(c: &mut Criterion) { - let df = df![ - "x" => (0..1_000_000).collect::>(), - "y" => (0..1_000_000).map(|i| i as f64).collect::>() - ].unwrap(); - - c.bench_function("filter_large", |b| { - b.iter(|| { - let mask = black_box(&df).column("x").unwrap().gt(500_000).unwrap(); - black_box(&df).filter(&mask).unwrap() - }) - }); -} - -criterion_group!(benches, benchmark_filter); -criterion_main!(benches); -``` - -### Run Benchmarks - -```bash -cargo bench - -# With specific feature flags -cargo bench --features nightly,performant -``` - -### Profiling - -```bash -# Using perf (Linux) -perf record --call-graph dwarf ./target/release/my_app -perf report - -# Using flamegraph -cargo install flamegraph -cargo flamegraph --bin my_app -``` - -## Performance Checklist - -- [ ] Using custom allocator (jemalloc/mimalloc)? -- [ ] Building with `--release`? -- [ ] Using `RUSTFLAGS='-C target-cpu=native'`? -- [ ] Using lazy mode for complex pipelines? -- [ ] Projecting columns early? -- [ ] Using categorical for low-cardinality strings? -- [ ] Avoiding map functions where expressions work? -- [ ] Appropriate thread pool size for workload? -- [ ] Features enabled: `performant`, `nightly` (if applicable)? diff --git a/.cursor/skills/polars-expertise/references/sql_interface.md b/.cursor/skills/polars-expertise/references/sql_interface.md deleted file mode 100644 index 9f633bd5..00000000 --- a/.cursor/skills/polars-expertise/references/sql_interface.md +++ /dev/null @@ -1,313 +0,0 @@ -# SQL Interface - -Polars provides SQL support through `SQLContext`, translating SQL queries into expressions for native execution with full query optimization. - -## SQLContext Setup - -### Basic Initialization - -```python -import polars as pl - -# Create context with DataFrame registration -df = pl.DataFrame({ - "symbol": ["AAPL", "AAPL", "MSFT", "MSFT"], - "price": [150.0, 152.0, 280.0, 285.0], - "volume": [1000000, 1200000, 800000, 900000] -}) - -ctx = pl.SQLContext(frames={"trades": df}) -``` - -### Registration Methods - -```python -# Register single DataFrame -ctx = pl.SQLContext() -ctx.register("trades", df) - -# Register multiple DataFrames -ctx.register_many({ - "trades": trades_df, - "quotes": quotes_df, - "orders": orders_df -}) - -# Register LazyFrames (recommended for large data) -lf = pl.scan_parquet("trades.parquet") -ctx.register("trades", lf) - -# Register all DataFrames/LazyFrames in global namespace -ctx.register_globals() -``` - -### Eager vs Lazy Execution - -```python -# Lazy execution (default) - returns LazyFrame -ctx = pl.SQLContext(trades=df) -result = ctx.execute("SELECT * FROM trades WHERE price > 150") -# result is LazyFrame - call .collect() to materialize - -# Eager execution - returns DataFrame directly -ctx = pl.SQLContext(trades=df, eager_execution=True) -result = ctx.execute("SELECT * FROM trades WHERE price > 150") -# result is DataFrame - -# Per-query eager execution -result = ctx.execute("SELECT * FROM trades", eager=True) -``` - -## Query Patterns - -### Basic SELECT with Aggregations - -```python -ctx.execute(""" - SELECT - symbol, - AVG(price) as avg_price, - SUM(volume) as total_volume, - COUNT(*) as trade_count - FROM trades - GROUP BY symbol - ORDER BY avg_price DESC -""").collect() -``` - -### Financial Data Patterns - -```python -# VWAP calculation -ctx.execute(""" - SELECT - symbol, - SUM(price * volume) / SUM(volume) as vwap - FROM trades - GROUP BY symbol -""").collect() - -# Price change analysis -ctx.execute(""" - SELECT - symbol, - price, - LAG(price, 1) OVER (PARTITION BY symbol ORDER BY timestamp) as prev_price, - price - LAG(price, 1) OVER (PARTITION BY symbol ORDER BY timestamp) as price_change - FROM trades -""").collect() -``` - -### JOINs - -```python -quotes = pl.DataFrame({ - "symbol": ["AAPL", "MSFT"], - "bid": [149.5, 279.5], - "ask": [150.5, 280.5] -}) - -ctx.register("quotes", quotes) - -# Inner join -ctx.execute(""" - SELECT - t.symbol, - t.price, - q.bid, - q.ask, - t.price - (q.bid + q.ask) / 2 as mid_deviation - FROM trades t - INNER JOIN quotes q ON t.symbol = q.symbol -""").collect() -``` - -### Common Table Expressions (CTEs) - -```python -ctx.execute(""" - WITH daily_stats AS ( - SELECT - symbol, - DATE_TRUNC('day', timestamp) as date, - MAX(price) as high, - MIN(price) as low, - FIRST_VALUE(price) OVER (PARTITION BY symbol, DATE_TRUNC('day', timestamp) ORDER BY timestamp) as open, - LAST_VALUE(price) OVER (PARTITION BY symbol, DATE_TRUNC('day', timestamp) ORDER BY timestamp) as close - FROM trades - GROUP BY symbol, DATE_TRUNC('day', timestamp) - ), - with_returns AS ( - SELECT - *, - (close - open) / open * 100 as daily_return - FROM daily_stats - ) - SELECT * FROM with_returns - WHERE daily_return > 1.0 - ORDER BY daily_return DESC -""").collect() -``` - -### Table Functions - Direct File Reading - -```python -# Read directly from files in SQL -ctx.execute(""" - SELECT * - FROM read_parquet('trades/*.parquet') - WHERE date >= '2024-01-01' -""").collect() - -# Join files -ctx.execute(""" - SELECT t.*, r.reference_price - FROM read_csv('trades.csv') t - JOIN read_parquet('reference.parquet') r - ON t.symbol = r.symbol -""").collect() -``` - -### CREATE TABLE - -```python -# Create derived tables -ctx.execute(""" - CREATE TABLE high_volume_trades AS - SELECT * - FROM trades - WHERE volume > 1000000 -""") - -# Query created table -ctx.execute("SELECT * FROM high_volume_trades").collect() - -# Show registered tables -ctx.execute("SHOW TABLES").collect() -``` - -## Supported SQL Features - -### Statements - -| Statement | Support | -|-----------|---------| -| SELECT | Full support | -| CREATE TABLE AS | Full support | -| DROP TABLE | Full support | -| TRUNCATE TABLE | Full support | -| EXPLAIN | Full support | -| SHOW TABLES | Full support | -| INSERT/UPDATE/DELETE | Not supported | - -### Clauses - -| Clause | Support | -|--------|---------| -| WHERE | Full support | -| GROUP BY | Full support | -| HAVING | Full support | -| ORDER BY | Full support | -| LIMIT/OFFSET | Full support | -| JOIN (INNER, LEFT, RIGHT, FULL, CROSS) | Full support | -| UNION/UNION ALL | Full support | -| WITH (CTE) | Full support | - -### Functions - -| Category | Examples | -|----------|----------| -| Aggregation | SUM, AVG, MIN, MAX, COUNT, STDDEV, VARIANCE, FIRST, LAST | -| Window | ROW_NUMBER, RANK, DENSE_RANK, LAG, LEAD, FIRST_VALUE, LAST_VALUE | -| Math | ABS, CEIL, FLOOR, ROUND, EXP, LN, LOG, POWER, SQRT | -| String | UPPER, LOWER, LENGTH, TRIM, SUBSTR, CONCAT, REPLACE, STARTS_WITH, ENDS_WITH | -| Date/Time | DATE_TRUNC, DATE_PART, EXTRACT, NOW, CURRENT_DATE | -| Array | EXPLODE, UNNEST, ARRAY_LENGTH, ARRAY_SUM | -| Conditional | CASE WHEN, COALESCE, NULLIF, IIF | - -## Best Practices - -### 1. Use LazyFrames for Large Data - -```python -# Good - lazy reading with SQL optimization -lf = pl.scan_parquet("large_dataset.parquet") -ctx = pl.SQLContext(data=lf) -result = ctx.execute(""" - SELECT symbol, AVG(price) - FROM data - WHERE date >= '2024-01-01' - GROUP BY symbol -""").collect() -# Predicate and projection pushdown applied - -# Bad - eager reading wastes memory -df = pl.read_parquet("large_dataset.parquet") # Loads everything -ctx = pl.SQLContext(data=df) -``` - -### 2. Combine SQL with Expression API - -```python -# SQL for initial query -lf = ctx.execute(""" - SELECT symbol, date, price, volume - FROM trades - WHERE volume > 100000 -""") - -# Expression API for complex transformations -result = ( - lf.with_columns( - pl.col("price").rolling_mean(20).over("symbol").alias("ma_20"), - pl.col("volume").rank("ordinal").over("date").alias("volume_rank") - ) - .collect() -) -``` - -### 3. EXPLAIN for Query Debugging - -```python -# View query plan -print(ctx.execute("EXPLAIN SELECT * FROM trades WHERE price > 150")) -``` - -## Rust SQL Support - -Enable the `sql` feature in Cargo.toml: - -```toml -[dependencies] -polars = { version = "0.46", features = ["sql"] } -``` - -```rust -use polars::prelude::*; -use polars::sql::SQLContext; - -fn main() -> PolarsResult<()> { - let df = df!( - "symbol" => ["AAPL", "MSFT"], - "price" => [150.0, 280.0] - )?; - - let mut ctx = SQLContext::new(); - ctx.register("trades", df.lazy()); - - let result = ctx.execute("SELECT * FROM trades WHERE price > 200")? - .collect()?; - - println!("{}", result); - Ok(()) -} -``` - -## Limitations - -1. **No DML**: INSERT, UPDATE, DELETE not supported - use expression API -2. **No DDL schema**: No column type definitions in CREATE TABLE -3. **PostgreSQL dialect**: Some vendor-specific syntax may not work -4. **Expression priority**: New features land in expression API first - -For complex transformations not supported in SQL, use the expression API directly. diff --git a/.cursor/skills/python-pro/SKILL.md b/.cursor/skills/python-pro/SKILL.md deleted file mode 100644 index 45856f4b..00000000 --- a/.cursor/skills/python-pro/SKILL.md +++ /dev/null @@ -1,92 +0,0 @@ ---- -name: python-pro -description: Use when building Python 3.11+ applications requiring type safety, async programming, or production-grade patterns. Invoke for type hints, pytest, async/await, dataclasses, mypy configuration. -triggers: - - Python development - - type hints - - async Python - - pytest - - mypy - - dataclasses - - Python best practices - - Pythonic code -role: specialist -scope: implementation -output-format: code ---- - -# Python Pro - -Senior Python developer with 10+ years experience specializing in type-safe, async-first, production-ready Python 3.11+ code. - -## Role Definition - -You are a senior Python engineer mastering modern Python 3.11+ and its ecosystem. You write idiomatic, type-safe, performant code across web development, data science, automation, and system programming with focus on production best practices. - -## When to Use This Skill - -- Writing type-safe Python with complete type coverage -- Implementing async/await patterns for I/O operations -- Setting up pytest test suites with fixtures and mocking -- Creating Pythonic code with comprehensions, generators, context managers -- Building packages with uv and proper project structure -- Performance optimization and profiling - -## Core Workflow - -1. **Analyze codebase** - Review structure, dependencies, type coverage, test suite -2. **Design interfaces** - Define protocols, dataclasses, type aliases -3. **Implement** - Write Pythonic code with full type hints and error handling -4. **Test** - Create comprehensive pytest suite with >90% coverage -5. **Validate** - Run mypy, black, ruff; ensure quality standards met - -## Reference Guide - -Load detailed guidance based on context: - -| Topic | Reference | Load When | -|-------|-----------|-----------| -| Type System | `references/type-system.md` | Type hints, mypy, generics, Protocol | -| Async Patterns | `references/async-patterns.md` | async/await, asyncio, task groups | -| Standard Library | `references/standard-library.md` | pathlib, dataclasses, functools, itertools | -| Testing | `references/testing.md` | pytest, fixtures, mocking, parametrize | -| Packaging | `references/packaging.md` | uv, pip, pyproject.toml, distribution | - -## Constraints - -### MUST DO -- Type hints for all function signatures and class attributes -- PEP 8 compliance with black formatting -- Comprehensive docstrings (Google style) -- Test coverage exceeding 90% with pytest -- Use `X | None` instead of `Optional[X]` (Python 3.10+) -- Async/await for I/O-bound operations -- Dataclasses over manual __init__ methods -- Context managers for resource handling - -### MUST NOT DO -- Skip type annotations on public APIs -- Use mutable default arguments -- Mix sync and async code improperly -- Ignore mypy errors in strict mode -- Use bare except clauses -- Hardcode secrets or configuration -- Use deprecated stdlib modules (use pathlib not os.path) - -## Output Templates - -When implementing Python features, provide: -1. Module file with complete type hints -2. Test file with pytest fixtures -3. Type checking confirmation (mypy --strict passes) -4. Brief explanation of Pythonic patterns used - -## Knowledge Reference - -Python 3.11+, typing module, mypy, pytest, black, ruff, dataclasses, async/await, asyncio, pathlib, functools, itertools, uv, Pydantic, contextlib, collections.abc, Protocol - -## Related Skills - -- **FastAPI Expert** - Async Python APIs -- **Data Science Pro** - NumPy, Pandas, ML -- **DevOps Engineer** - Python automation and tooling diff --git a/.cursor/skills/python-pro/references/async-patterns.md b/.cursor/skills/python-pro/references/async-patterns.md deleted file mode 100644 index 6160247c..00000000 --- a/.cursor/skills/python-pro/references/async-patterns.md +++ /dev/null @@ -1,359 +0,0 @@ -# Async Programming Patterns - -> Reference for: Python Pro -> Load when: async/await, asyncio, concurrent operations, task groups - -## Basic Async/Await - -```python -import asyncio -from collections.abc import Coroutine - -# Basic async function -async def fetch_data(url: str) -> dict[str, str]: - await asyncio.sleep(1) # Simulate I/O - return {"url": url, "status": "ok"} - -# Running async code -async def main() -> None: - result = await fetch_data("https://api.example.com") - print(result) - -if __name__ == "__main__": - asyncio.run(main()) - -# Multiple concurrent operations -async def fetch_all(urls: list[str]) -> list[dict[str, str]]: - tasks = [fetch_data(url) for url in urls] - return await asyncio.gather(*tasks) - -# Error handling with gather -async def safe_fetch_all(urls: list[str]) -> list[dict[str, str] | None]: - tasks = [fetch_data(url) for url in urls] - results = await asyncio.gather(*tasks, return_exceptions=True) - return [r if not isinstance(r, Exception) else None for r in results] -``` - -## Task Groups (Python 3.11+) - -```python -from asyncio import TaskGroup - -# Task groups for structured concurrency -async def process_batch(items: list[int]) -> list[int]: - results: list[int] = [] - - async with TaskGroup() as tg: - tasks = [tg.create_task(process_item(item)) for item in items] - - # All tasks complete before this line - return [task.result() for task in tasks] - -# Error handling with TaskGroup -async def robust_processing(items: list[str]) -> tuple[list[str], list[Exception]]: - results: list[str] = [] - errors: list[Exception] = [] - - try: - async with TaskGroup() as tg: - for item in items: - tg.create_task(process_item_safe(item)) - except ExceptionGroup as eg: - for exc in eg.exceptions: - errors.append(exc) - - return results, errors -``` - -## Async Context Managers - -```python -from typing import Self -from collections.abc import AsyncIterator - -class AsyncDatabaseConnection: - def __init__(self, url: str) -> None: - self.url = url - self._conn: Connection | None = None - - async def __aenter__(self) -> Self: - self._conn = await connect(self.url) - return self - - async def __aexit__( - self, - exc_type: type[BaseException] | None, - exc_val: BaseException | None, - exc_tb: Any, - ) -> None: - if self._conn: - await self._conn.close() - - async def query(self, sql: str) -> list[dict[str, Any]]: - if not self._conn: - raise RuntimeError("Not connected") - return await self._conn.execute(sql) - -# Usage -async def get_users() -> list[dict[str, Any]]: - async with AsyncDatabaseConnection("postgresql://...") as db: - return await db.query("SELECT * FROM users") - -# Async context manager with contextlib -from contextlib import asynccontextmanager - -@asynccontextmanager -async def get_db_session() -> AsyncIterator[Session]: - session = await create_session() - try: - yield session - await session.commit() - except Exception: - await session.rollback() - raise - finally: - await session.close() -``` - -## Async Generators - -```python -from collections.abc import AsyncIterator - -# Async generator for streaming data -async def read_lines(filepath: str) -> AsyncIterator[str]: - async with aiofiles.open(filepath) as f: - async for line in f: - yield line.strip() - -# Process stream -async def process_file(filepath: str) -> int: - count = 0 - async for line in read_lines(filepath): - await process_line(line) - count += 1 - return count - -# Async generator with cleanup -async def fetch_paginated(url: str) -> AsyncIterator[dict[str, Any]]: - page = 1 - session = await create_session() - try: - while True: - data = await session.get(f"{url}?page={page}") - if not data: - break - yield data - page += 1 - finally: - await session.close() -``` - -## Async Comprehensions - -```python -# Async list comprehension -async def fetch_all_users(user_ids: list[int]) -> list[User]: - return [user async for user in fetch_users(user_ids)] - -# Async dict comprehension -async def build_user_map(user_ids: list[int]) -> dict[int, User]: - return { - user.id: user - async for user in fetch_users(user_ids) - } - -# Conditional async comprehension -async def get_active_users(user_ids: list[int]) -> list[User]: - return [ - user - async for user in fetch_users(user_ids) - if user.is_active - ] -``` - -## Synchronization Primitives - -```python -import asyncio - -# Lock for critical sections -class SharedResource: - def __init__(self) -> None: - self._lock = asyncio.Lock() - self._data: dict[str, Any] = {} - - async def update(self, key: str, value: Any) -> None: - async with self._lock: - # Critical section - current = self._data.get(key, 0) - await asyncio.sleep(0.1) # Simulate processing - self._data[key] = current + value - -# Semaphore for rate limiting -class RateLimiter: - def __init__(self, max_concurrent: int) -> None: - self._semaphore = asyncio.Semaphore(max_concurrent) - - async def process(self, item: str) -> str: - async with self._semaphore: - return await expensive_operation(item) - -# Event for coordination -class AsyncWorker: - def __init__(self) -> None: - self._ready = asyncio.Event() - self._shutdown = asyncio.Event() - - async def start(self) -> None: - # Initialization - await self._initialize() - self._ready.set() - - # Wait for shutdown - await self._shutdown.wait() - - async def wait_ready(self) -> None: - await self._ready.wait() - - def stop(self) -> None: - self._shutdown.set() -``` - -## Async Queue Patterns - -```python -from asyncio import Queue - -# Producer-consumer pattern -async def producer(queue: Queue[int], n: int) -> None: - for i in range(n): - await queue.put(i) - await asyncio.sleep(0.1) - -async def consumer(queue: Queue[int], name: str) -> None: - while True: - item = await queue.get() - try: - await process_item(item) - finally: - queue.task_done() - -async def run_pipeline(num_items: int, num_workers: int) -> None: - queue: Queue[int] = Queue(maxsize=10) - - # Start producer and consumers - async with TaskGroup() as tg: - tg.create_task(producer(queue, num_items)) - for i in range(num_workers): - tg.create_task(consumer(queue, f"worker-{i}")) - - # Wait for all items to be processed - await queue.join() -``` - -## Async Timeouts - -```python -# Timeout for single operation -async def fetch_with_timeout(url: str, timeout: float) -> dict[str, Any]: - try: - async with asyncio.timeout(timeout): - return await fetch_data(url) - except TimeoutError: - return {"error": "timeout"} - -# Timeout for multiple operations -async def fetch_all_with_timeout( - urls: list[str], - timeout: float -) -> list[dict[str, Any] | None]: - try: - async with asyncio.timeout(timeout): - return await fetch_all(urls) - except TimeoutError: - return [None] * len(urls) -``` - -## Background Tasks - -```python -from asyncio import create_task, Task - -class BackgroundTaskManager: - def __init__(self) -> None: - self._tasks: set[Task[None]] = set() - - def create_task(self, coro: Coroutine[None, None, None]) -> Task[None]: - task = create_task(coro) - self._tasks.add(task) - task.add_done_callback(self._tasks.discard) - return task - - async def shutdown(self) -> None: - # Cancel all background tasks - for task in self._tasks: - task.cancel() - # Wait for cancellation - await asyncio.gather(*self._tasks, return_exceptions=True) - -# Usage -manager = BackgroundTaskManager() -manager.create_task(background_job()) -``` - -## Async Iteration Protocol - -```python -class AsyncRange: - def __init__(self, start: int, end: int) -> None: - self.start = start - self.end = end - self.current = start - - def __aiter__(self) -> Self: - return self - - async def __anext__(self) -> int: - if self.current >= self.end: - raise StopAsyncIteration - await asyncio.sleep(0.1) # Simulate async work - value = self.current - self.current += 1 - return value - -# Usage -async for i in AsyncRange(0, 5): - print(i) -``` - -## Mixing Sync and Async - -```python -from concurrent.futures import ThreadPoolExecutor -import functools - -# Run sync code in executor -async def run_in_executor(func: Callable[..., T], *args: Any) -> T: - loop = asyncio.get_running_loop() - return await loop.run_in_executor(None, func, *args) - -# Run async code from sync context -def sync_wrapper(coro: Coroutine[None, None, T]) -> T: - loop = asyncio.new_event_loop() - try: - return loop.run_until_complete(coro) - finally: - loop.close() - -# Async wrapper for sync function -def to_async(func: Callable[..., T]) -> Callable[..., Coroutine[None, None, T]]: - @functools.wraps(func) - async def wrapper(*args: Any, **kwargs: Any) -> T: - loop = asyncio.get_running_loop() - return await loop.run_in_executor( - None, - functools.partial(func, *args, **kwargs) - ) - return wrapper -``` diff --git a/.cursor/skills/python-pro/references/packaging.md b/.cursor/skills/python-pro/references/packaging.md deleted file mode 100644 index 1621efb2..00000000 --- a/.cursor/skills/python-pro/references/packaging.md +++ /dev/null @@ -1,561 +0,0 @@ -# Python Packaging and Project Setup - -> Reference for: Python Pro -> Load when: uv, pip, pyproject.toml, package distribution, virtual environments - -## Project Structure - -``` -myproject/ -├── pyproject.toml # Project metadata and dependencies -├── README.md # Project description -├── .gitignore # Git ignore patterns -├── .python-version # Python version for pyenv -├── src/ -│ └── myproject/ -│ ├── __init__.py # Package initialization -│ ├── py.typed # PEP 561 type marker -│ ├── core.py # Core functionality -│ └── utils.py # Utilities -├── tests/ -│ ├── __init__.py -│ ├── conftest.py # Pytest configuration -│ └── test_core.py # Tests -└── docs/ - └── index.md # Documentation -``` - -## Pyproject.toml Configuration - -```toml -[build-system] -requires = ["hatchling"] -build-backend = "hatchling.build" - -[project] -name = "myproject" -version = "0.1.0" -description = "A Python project" -readme = "README.md" -requires-python = ">=3.11" -license = {text = "MIT"} -authors = [ - {name = "Your Name", email = "you@example.com"} -] -keywords = ["python", "package"] -classifiers = [ - "Development Status :: 4 - Beta", - "Intended Audience :: Developers", - "License :: OSI Approved :: MIT License", - "Programming Language :: Python :: 3.11", - "Programming Language :: Python :: 3.12", - "Typing :: Typed", -] - -dependencies = [ - "requests>=2.31.0", - "pydantic>=2.5.0", -] - -[project.optional-dependencies] -dev = [ - "pytest>=7.4.0", - "pytest-cov>=4.1.0", - "mypy>=1.7.0", - "black>=23.11.0", - "ruff>=0.1.6", -] -docs = [ - "mkdocs>=1.5.0", - "mkdocs-material>=9.4.0", -] - -[project.scripts] -myproject = "myproject.cli:main" - -[project.urls] -Homepage = "https://github.com/username/myproject" -Documentation = "https://myproject.readthedocs.io" -Repository = "https://github.com/username/myproject" -Changelog = "https://github.com/username/myproject/blob/main/CHANGELOG.md" - -# Tool configurations -[tool.black] -line-length = 100 -target-version = ["py311"] -include = '\.pyi?$' - -[tool.ruff] -line-length = 100 -target-version = "py311" -select = [ - "E", # pycodestyle errors - "W", # pycodestyle warnings - "F", # pyflakes - "I", # isort - "B", # flake8-bugbear - "C4", # flake8-comprehensions - "UP", # pyupgrade -] -ignore = [] - -[tool.ruff.per-file-ignores] -"__init__.py" = ["F401"] # Ignore unused imports in __init__.py - -[tool.mypy] -python_version = "3.11" -strict = true -warn_return_any = true -warn_unused_configs = true -disallow_untyped_defs = true - -[[tool.mypy.overrides]] -module = "third_party.*" -ignore_missing_imports = true - -[tool.pytest.ini_options] -minversion = "7.0" -addopts = [ - "-ra", - "--strict-markers", - "--strict-config", - "--cov=myproject", - "--cov-report=term-missing", - "--cov-report=html", -] -testpaths = ["tests"] -pythonpath = ["src"] - -[tool.coverage.run] -source = ["src"] -branch = true - -[tool.coverage.report] -exclude_lines = [ - "pragma: no cover", - "def __repr__", - "raise AssertionError", - "raise NotImplementedError", - "if __name__ == .__main__.:", - "if TYPE_CHECKING:", -] -``` - -## UV Project Management - -```bash -# UV commands -uv init myproject # Initialize new project -cd myproject -uv venv --python=3.13 # Create environement with python version 3.13 -source .venv/bin/activate # Activate environemnt on posix -.venv/Scripts/activate # Activate environment on windows -uv add requests # Add dependency -uv add --dev pytest # Add dev dependency -uv add --optional docs mkdocs # Add optional dependency group -uv sync # Install dependencies from pyproject.toml -uv lock # Update lock file -uv pip install -e . # Install in editable mode -uv run pytest # Run command in venv -uv build # Build package -uv publish # Publish to PyPI -uv pip compile pyproject.toml # Generate requirements.txt -uv pip list # List installed packages -uv pip freeze # Export exact versions -uv remove requests # Remove dependency -uv update # Update all dependencies -uv update requests # Update specific dependency -``` - -```toml -# pyproject.toml for uv (standard PEP 621 format) -[build-system] -requires = ["hatchling"] -build-backend = "hatchling.build" - -[project] -name = "myproject" -version = "0.1.0" -description = "A Python project" -readme = "README.md" -requires-python = ">=3.11" -license = {text = "MIT"} -authors = [ - {name = "Your Name", email = "you@example.com"} -] -keywords = ["python", "package"] -classifiers = [ - "Development Status :: 4 - Beta", - "Intended Audience :: Developers", - "License :: OSI Approved :: MIT License", - "Programming Language :: Python :: 3.11", - "Programming Language :: Python :: 3.12", - "Typing :: Typed", -] - -dependencies = [ - "requests>=2.31.0", - "pydantic>=2.5.0", -] - -[project.optional-dependencies] -dev = [ - "pytest>=7.4.0", - "pytest-cov>=4.1.0", - "mypy>=1.7.0", - "black>=23.11.0", - "ruff>=0.1.6", -] -docs = [ - "mkdocs>=1.5.0", - "mkdocs-material>=9.4.0", -] - -[project.scripts] -myproject = "myproject.cli:main" - -[project.urls] -Homepage = "https://github.com/username/myproject" -Documentation = "https://myproject.readthedocs.io" -Repository = "https://github.com/username/myproject" -Changelog = "https://github.com/username/myproject/blob/main/CHANGELOG.md" -``` - -```bash -# UV virtual environment management -uv venv # Create virtual environment in .venv -uv venv --python 3.13 # Create venv with specific Python version -source .venv/bin/activate # Activate venv (Linux/Mac) -.venv\Scripts\activate # Activate venv (Windows) -uv pip install package # Install in active venv -uv pip install -e ".[dev]" # Install with optional dependencies -uv pip uninstall package # Uninstall package -uv pip sync requirements.txt # Sync from requirements file -``` - -## UV Best Practices - -```bash -# Project initialization workflow -uv init myproject # Creates project with pyproject.toml -cd myproject -uv add package # Adds to dependencies -uv add --dev pytest # Adds to dev dependencies -uv sync # Installs all dependencies - -# Working with existing projects -uv sync # Install from pyproject.toml and uv.lock -uv sync --dev # Include dev dependencies -uv sync --extra docs # Include optional dependency groups -uv lock # Update lock file without installing - -# Running commands -uv run pytest # Run in project environment -uv run python script.py # Run script in project environment -uv run --with pytest pytest # Temporarily add dependency - -# Package management -uv add "package>=1.0.0" # Add with version constraint -uv add "package@git+https://..." # Add from git -uv remove package # Remove dependency -uv update # Update all dependencies -uv update package # Update specific package - -# Building and publishing -uv build # Build wheel and sdist -uv publish # Publish to PyPI -uv publish --publish-url https://... # Publish to custom index - -# Dependency resolution -uv pip compile pyproject.toml # Generate requirements.txt -uv pip compile --extra dev pyproject.toml -o requirements-dev.txt -uv tree # Show dependency tree -uv pip list # List installed packages -``` - -## Virtual Environments - -```bash -# Using uv (recommended) -uv venv # Create .venv in current directory -uv venv --python=3.13 # Create with specific Python version -source .venv/bin/activate # Activate (Linux/Mac) -.venv\Scripts\activate # Activate (Windows) -uv sync # Install dependencies from pyproject.toml -uv pip install -e . # Install in editable mode -uv pip install -e ".[dev]" # With optional dependencies - -# Using venv (built-in, alternative) -python -m venv .venv -source .venv/bin/activate # Linux/Mac -.venv\Scripts\activate # Windows -pip install -e . -pip install -e ".[dev]" - -# Using virtualenv (legacy) -pip install virtualenv -virtualenv venv -source venv/bin/activate - -# Using pyenv for Python version management -pyenv install 3.11.6 -pyenv local 3.11.6 # Set for current directory -echo "3.11.6" > .python-version -``` - -## Package __init__.py - -```python -# src/myproject/__init__.py -"""MyProject - A Python package.""" - -from myproject.core import main_function, CoreClass -from myproject.utils import helper_function - -__version__ = "0.1.0" -__all__ = ["main_function", "CoreClass", "helper_function"] - -# Package-level configuration -import logging - -logger = logging.getLogger(__name__) -logger.addHandler(logging.NullHandler()) -``` - -## Type Stub Files (py.typed) - -```python -# src/myproject/py.typed -# Empty file indicates package includes type hints - -# src/myproject/__init__.pyi (optional stub file) -from typing import Any - -__version__: str - -def main_function(arg: str) -> dict[str, Any]: ... - -class CoreClass: - def __init__(self, name: str) -> None: ... - def process(self) -> str: ... -``` - -## CLI Entry Points - -```python -# src/myproject/cli.py -import sys -from typing import NoReturn - -def main() -> NoReturn: - """Main CLI entry point.""" - print("MyProject CLI") - sys.exit(0) - -if __name__ == "__main__": - main() -``` - -## Requirements Files - -```bash -# requirements.txt - Production dependencies -requests>=2.31.0,<3.0.0 -pydantic>=2.5.0,<3.0.0 - -# requirements-dev.txt - Development dependencies --r requirements.txt -pytest>=7.4.0 -pytest-cov>=4.1.0 -mypy>=1.7.0 -black>=23.11.0 -ruff>=0.1.6 - -# Generate from uv -uv pip compile pyproject.toml -o requirements.txt -uv pip compile pyproject.toml --extra dev -o requirements-dev.txt -uv pip freeze > requirements-lock.txt -``` - -## Building and Distribution - -```bash -# Build package -python -m build - -# Check package -twine check dist/* - -# Upload to PyPI -twine upload dist/* - -# Upload to Test PyPI -twine upload --repository testpypi dist/* - -# Install from Test PyPI -pip install --index-url https://test.pypi.org/simple/ myproject -``` - -## Setuptools Configuration (Legacy) - -```python -# setup.py (if not using pyproject.toml) -from setuptools import setup, find_packages - -setup( - name="myproject", - version="0.1.0", - packages=find_packages(where="src"), - package_dir={"": "src"}, - python_requires=">=3.11", - install_requires=[ - "requests>=2.31.0", - "pydantic>=2.5.0", - ], - extras_require={ - "dev": [ - "pytest>=7.4.0", - "mypy>=1.7.0", - ], - }, - entry_points={ - "console_scripts": [ - "myproject=myproject.cli:main", - ], - }, -) -``` - -## Manifest for Package Data - -``` -# MANIFEST.in -include README.md -include LICENSE -include pyproject.toml -recursive-include src/myproject *.py -recursive-include src/myproject py.typed -recursive-include tests *.py -prune docs/_build -``` - -## Version Management - -```python -# src/myproject/__version__.py -__version__ = "0.1.0" - -# src/myproject/__init__.py -from myproject.__version__ import __version__ - -# Read version in pyproject.toml -import tomli -from pathlib import Path - -def get_version() -> str: - pyproject = Path(__file__).parent.parent / "pyproject.toml" - with open(pyproject, "rb") as f: - data = tomli.load(f) - return data["project"]["version"] -``` - -## Dependency Management Best Practices - -```python -# Pin dependencies for applications -requests==2.31.0 -pydantic==2.5.2 - -# Use ranges for libraries -requests>=2.31.0,<3.0.0 -pydantic>=2.5.0,<3.0.0 - -# Lock files -# uv: uv.lock (automatically generated) -# pip: requirements.txt with exact versions -uv pip freeze > requirements-lock.txt - -# Update dependencies -uv lock --upgrade -uv update -uv pip install --upgrade -r requirements.txt -``` - -## CI/CD Integration - -```yaml -# .github/workflows/test.yml -name: Tests - -on: [push, pull_request] - -jobs: - test: - runs-on: ubuntu-latest - strategy: - matrix: - python-version: ["3.11", "3.12"] - - steps: - - uses: actions/checkout@v4 - - name: Set up Python - uses: actions/setup-python@v4 - with: - python-version: ${{ matrix.python-version }} - - - name: Install uv - uses: astral-sh/setup-uv@v3 - with: - version: "latest" - - - name: Install dependencies - run: | - uv sync --dev - - - name: Run tests - run: | - pytest --cov --cov-report=xml - - - name: Type check - run: mypy src - - - name: Lint - run: | - black --check src tests - ruff check src tests - - - name: Upload coverage - uses: codecov/codecov-action@v3 -``` - -## Pre-commit Hooks - -```yaml -# .pre-commit-config.yaml -repos: - - repo: https://github.com/psf/black - rev: 23.11.0 - hooks: - - id: black - - - repo: https://github.com/astral-sh/ruff-pre-commit - rev: v0.1.6 - hooks: - - id: ruff - args: [--fix, --exit-non-zero-on-fix] - - - repo: https://github.com/pre-commit/mirrors-mypy - rev: v1.7.1 - hooks: - - id: mypy - additional_dependencies: [types-requests] -``` - -```bash -# Install pre-commit -pip install pre-commit -pre-commit install - -# Run manually -pre-commit run --all-files -``` diff --git a/.cursor/skills/python-pro/references/standard-library.md b/.cursor/skills/python-pro/references/standard-library.md deleted file mode 100644 index 5667fe2c..00000000 --- a/.cursor/skills/python-pro/references/standard-library.md +++ /dev/null @@ -1,381 +0,0 @@ -# Standard Library Mastery - -> Reference for: Python Pro -> Load when: pathlib, dataclasses, functools, itertools, collections - -## Pathlib for File Operations - -```python -from pathlib import Path - -# Path creation and manipulation -project_root = Path(__file__).parent.parent -config_file = project_root / "config" / "settings.toml" -data_dir = Path.home() / "data" - -# File operations -def read_config(config_path: Path) -> dict[str, str]: - if not config_path.exists(): - raise FileNotFoundError(f"Config not found: {config_path}") - - # Read text - content = config_path.read_text(encoding="utf-8") - - # Read bytes - binary = config_path.read_bytes() - - return parse_config(content) - -# Path traversal -def find_python_files(directory: Path) -> list[Path]: - # Recursive glob - return list(directory.rglob("*.py")) - -def get_file_info(path: Path) -> dict[str, Any]: - stat = path.stat() - return { - "size": stat.st_size, - "modified": stat.st_mtime, - "is_file": path.is_file(), - "is_dir": path.is_dir(), - "suffix": path.suffix, - "stem": path.stem, - } - -# Creating directories -def ensure_dir(path: Path) -> None: - path.mkdir(parents=True, exist_ok=True) - -# Temporary files -from tempfile import TemporaryDirectory -from pathlib import Path - -def process_with_temp() -> None: - with TemporaryDirectory() as tmpdir: - temp_path = Path(tmpdir) / "output.txt" - temp_path.write_text("data") -``` - -## Dataclasses for Data Structures - -```python -from dataclasses import dataclass, field, asdict, replace -from typing import ClassVar - -# Basic dataclass -@dataclass -class User: - id: int - name: str - email: str - active: bool = True - -# Post-init processing -@dataclass -class Product: - name: str - price: float - discount: float = 0.0 - - def __post_init__(self) -> None: - if self.discount > 1.0: - raise ValueError("Discount must be <= 1.0") - - @property - def final_price(self) -> float: - return self.price * (1 - self.discount) - -# Field with factory -@dataclass -class ShoppingCart: - user_id: int - items: list[str] = field(default_factory=list) - metadata: dict[str, Any] = field(default_factory=dict) - -# Frozen dataclass (immutable) -@dataclass(frozen=True) -class Point: - x: float - y: float - - def distance(self, other: "Point") -> float: - return ((self.x - other.x)**2 + (self.y - other.y)**2)**0.5 - -# Class variables -@dataclass -class Config: - API_VERSION: ClassVar[str] = "v1" - BASE_URL: ClassVar[str] = "https://api.example.com" - - timeout: int = 30 - retries: int = 3 - -# Ordered dataclass for comparison -@dataclass(order=True) -class Priority: - level: int - name: str = field(compare=False) - -# Convert to/from dict -user = User(1, "Alice", "alice@example.com") -user_dict = asdict(user) -updated = replace(user, name="Alice Smith") -``` - -## Functools for Function Tools - -```python -from functools import ( - cache, lru_cache, cached_property, - partial, wraps, reduce, singledispatch -) - -# Caching -@cache # Unlimited cache (Python 3.9+) -def fibonacci(n: int) -> int: - if n < 2: - return n - return fibonacci(n - 1) + fibonacci(n - 2) - -@lru_cache(maxsize=128) # LRU cache with size limit -def fetch_user(user_id: int) -> dict[str, Any]: - # Expensive database call - return {"id": user_id, "name": "User"} - -# Cached property -class DataProcessor: - def __init__(self, data: list[int]) -> None: - self._data = data - - @cached_property - def mean(self) -> float: - """Computed once, then cached.""" - return sum(self._data) / len(self._data) - -# Partial application -from operator import mul - -double = partial(mul, 2) -triple = partial(mul, 3) -print(double(5)) # 10 - -# Decorator preservation -def timing_decorator(func: Callable[P, R]) -> Callable[P, R]: - @wraps(func) - def wrapper(*args: P.args, **kwargs: P.kwargs) -> R: - start = time.time() - result = func(*args, **kwargs) - print(f"{func.__name__} took {time.time() - start:.2f}s") - return result - return wrapper - -# Reduce for aggregation -from operator import add - -total = reduce(add, [1, 2, 3, 4, 5]) # 15 -product = reduce(mul, [1, 2, 3, 4], 1) # 24 - -# Single dispatch for polymorphism -@singledispatch -def process(arg: Any) -> str: - return f"Unknown type: {type(arg)}" - -@process.register -def _(arg: int) -> str: - return f"Integer: {arg * 2}" - -@process.register -def _(arg: str) -> str: - return f"String: {arg.upper()}" - -@process.register(list) -def _(arg: list[Any]) -> str: - return f"List with {len(arg)} items" -``` - -## Itertools for Iteration - -```python -from itertools import ( - chain, islice, cycle, repeat, - groupby, accumulate, combinations, permutations, - product, zip_longest, tee, filterfalse -) - -# Chain multiple iterables -combined = list(chain([1, 2], [3, 4], [5, 6])) # [1,2,3,4,5,6] - -# Slice iterator (memory efficient) -first_10 = list(islice(range(1000), 10)) - -# Infinite iterators -from itertools import count -counter = count(start=1, step=2) # 1, 3, 5, 7, ... - -# Groupby for grouping -data = [("A", 1), ("A", 2), ("B", 1), ("B", 2)] -grouped = {k: list(v) for k, v in groupby(data, key=lambda x: x[0])} - -# Accumulate for running totals -cumsum = list(accumulate([1, 2, 3, 4, 5])) # [1, 3, 6, 10, 15] - -# Combinations and permutations -combos = list(combinations([1, 2, 3], 2)) # [(1,2), (1,3), (2,3)] -perms = list(permutations([1, 2, 3], 2)) # [(1,2), (1,3), (2,1), ...] - -# Cartesian product -pairs = list(product([1, 2], ['a', 'b'])) # [(1,'a'), (1,'b'), (2,'a'), (2,'b')] - -# Zip with different lengths -from itertools import zip_longest -paired = list(zip_longest([1, 2], ['a', 'b', 'c'], fillvalue=0)) - -# Tee for multiple iterators -it1, it2 = tee(range(5), 2) - -# Filter false -odds = list(filterfalse(lambda x: x % 2 == 0, range(10))) -``` - -## Collections for Data Structures - -```python -from collections import ( - defaultdict, Counter, deque, namedtuple, - ChainMap, OrderedDict -) - -# defaultdict for automatic defaults -word_index: defaultdict[str, list[int]] = defaultdict(list) -for i, word in enumerate(["hello", "world", "hello"]): - word_index[word].append(i) - -# Counter for counting -from collections import Counter - -word_counts = Counter(["apple", "banana", "apple", "cherry", "banana", "apple"]) -print(word_counts.most_common(2)) # [('apple', 3), ('banana', 2)] - -# Counter operations -c1 = Counter(a=3, b=1) -c2 = Counter(a=1, b=2) -print(c1 + c2) # Counter({'a': 4, 'b': 3}) - -# deque for efficient queue operations -from collections import deque - -queue: deque[str] = deque() -queue.append("first") -queue.append("second") -queue.appendleft("priority") -item = queue.popleft() # "priority" - -# Ring buffer with maxlen -recent: deque[int] = deque(maxlen=3) -for i in range(5): - recent.append(i) # Only keeps last 3 - -# namedtuple for lightweight classes -from collections import namedtuple - -Point = namedtuple('Point', ['x', 'y']) -p = Point(1, 2) -print(p.x, p.y) - -# ChainMap for layered configs -from collections import ChainMap - -defaults = {'color': 'red', 'user': 'guest'} -environment = {'user': 'admin'} -combined = ChainMap(environment, defaults) -print(combined['user']) # 'admin' (from environment) -``` - -## Context Managers - -```python -from contextlib import contextmanager, suppress, ExitStack - -# Custom context manager -@contextmanager -def managed_resource(resource_id: str) -> Iterator[Resource]: - resource = acquire_resource(resource_id) - try: - yield resource - finally: - release_resource(resource) - -# Suppress exceptions -with suppress(FileNotFoundError): - Path("nonexistent.txt").unlink() - -# ExitStack for dynamic context managers -def process_files(filenames: list[str]) -> None: - with ExitStack() as stack: - files = [stack.enter_context(open(fn)) for fn in filenames] - # All files auto-closed on exit - for f in files: - process(f.read()) -``` - -## Enum for Constants - -```python -from enum import Enum, auto, IntEnum, Flag - -# Basic enum -class Status(Enum): - PENDING = "pending" - APPROVED = "approved" - REJECTED = "rejected" - -# Auto values -class Color(Enum): - RED = auto() - GREEN = auto() - BLUE = auto() - -# IntEnum for numeric values -class Priority(IntEnum): - LOW = 1 - MEDIUM = 2 - HIGH = 3 - -# Flag for bit flags -class Permission(Flag): - READ = auto() - WRITE = auto() - EXECUTE = auto() - -user_perms = Permission.READ | Permission.WRITE -if Permission.READ in user_perms: - print("Can read") -``` - -## Logging - -```python -import logging -from pathlib import Path - -# Configure logging -logging.basicConfig( - level=logging.INFO, - format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', - handlers=[ - logging.FileHandler('app.log'), - logging.StreamHandler() - ] -) - -logger = logging.getLogger(__name__) - -# Structured logging -def process_user(user_id: int) -> None: - logger.info("Processing user", extra={"user_id": user_id}) - try: - # Process... - logger.debug("User data loaded", extra={"user_id": user_id}) - except Exception as e: - logger.exception("Failed to process user", extra={"user_id": user_id}) -``` diff --git a/.cursor/skills/python-pro/references/testing.md b/.cursor/skills/python-pro/references/testing.md deleted file mode 100644 index 4362f74a..00000000 --- a/.cursor/skills/python-pro/references/testing.md +++ /dev/null @@ -1,407 +0,0 @@ -# Testing with Pytest - -> Reference for: Python Pro -> Load when: pytest, fixtures, mocking, test coverage, parametrize - -## Basic Pytest Structure - -```python -# test_user.py -import pytest -from myapp.user import User, UserService - -# Simple test function -def test_user_creation() -> None: - user = User(id=1, name="Alice", email="alice@example.com") - assert user.name == "Alice" - assert user.is_active is True - -# Test with multiple assertions -def test_user_validation() -> None: - with pytest.raises(ValueError, match="Invalid email"): - User(id=1, name="Alice", email="invalid") - -# Test class for grouping -class TestUserService: - def test_find_user(self) -> None: - service = UserService() - user = service.find(1) - assert user is not None - - def test_create_user(self) -> None: - service = UserService() - user = service.create(name="Bob", email="bob@example.com") - assert user.id > 0 -``` - -## Fixtures for Setup/Teardown - -```python -# conftest.py - shared fixtures -import pytest -from typing import Iterator -from myapp.database import Database, Session - -@pytest.fixture -def db() -> Iterator[Database]: - """Provide database instance with cleanup.""" - database = Database("test.db") - database.create_tables() - yield database - database.drop_tables() - database.close() - -@pytest.fixture -def db_session(db: Database) -> Iterator[Session]: - """Provide database session with rollback.""" - session = db.create_session() - yield session - session.rollback() - session.close() - -@pytest.fixture -def sample_user() -> User: - """Provide test user.""" - return User(id=1, name="Test User", email="test@example.com") - -# Using fixtures in tests -def test_user_creation(db_session: Session, sample_user: User) -> None: - db_session.add(sample_user) - db_session.commit() - - retrieved = db_session.query(User).filter_by(id=1).first() - assert retrieved.name == "Test User" - -# Fixture with parameters -@pytest.fixture(params=["sqlite", "postgresql", "mysql"]) -def db_engine(request: pytest.FixtureRequest) -> str: - return request.param - -def test_connection(db_engine: str) -> None: - # Test runs 3 times with different engines - assert create_connection(db_engine) - -# Autouse fixture (runs automatically) -@pytest.fixture(autouse=True) -def reset_state() -> Iterator[None]: - """Reset global state before each test.""" - clear_caches() - yield - cleanup_temp_files() -``` - -## Parametrize for Multiple Cases - -```python -import pytest - -# Parametrize test function -@pytest.mark.parametrize( - "input,expected", - [ - (2, 4), - (3, 9), - (4, 16), - (-2, 4), - ] -) -def test_square(input: int, expected: int) -> None: - assert square(input) == expected - -# Multiple parameters -@pytest.mark.parametrize("base", [2, 10]) -@pytest.mark.parametrize("exponent", [0, 1, 2]) -def test_power(base: int, exponent: int) -> None: - result = base ** exponent - assert result >= 0 - -# Parametrize with IDs -@pytest.mark.parametrize( - "email,valid", - [ - ("user@example.com", True), - ("invalid", False), - ("@example.com", False), - ("user@", False), - ], - ids=["valid", "no_at", "no_user", "no_domain"] -) -def test_email_validation(email: str, valid: bool) -> None: - assert is_valid_email(email) == valid - -# Parametrize with fixtures -@pytest.fixture -def user_factory(): - def _make_user(name: str, active: bool = True) -> User: - return User(name=name, active=active) - return _make_user - -@pytest.mark.parametrize("name", ["Alice", "Bob", "Charlie"]) -def test_user_names(user_factory, name: str) -> None: - user = user_factory(name) - assert user.name == name -``` - -## Mocking and Patching - -```python -from unittest.mock import Mock, MagicMock, patch, AsyncMock, call -import pytest - -# Mock object -def test_api_call_with_mock() -> None: - mock_client = Mock() - mock_client.get.return_value = {"status": "ok"} - - service = ApiService(mock_client) - result = service.fetch_data() - - mock_client.get.assert_called_once_with("/api/data") - assert result["status"] == "ok" - -# Patch function/method -def test_database_call() -> None: - with patch("myapp.database.connect") as mock_connect: - mock_connect.return_value = Mock() - - db = Database() - db.connect() - - mock_connect.assert_called_once() - -# Patch as decorator -@patch("myapp.user.send_email") -def test_user_registration(mock_send_email: Mock) -> None: - service = UserService() - service.register("user@example.com") - - mock_send_email.assert_called_with( - to="user@example.com", - subject="Welcome" - ) - -# Multiple patches -@patch("myapp.api.requests.get") -@patch("myapp.api.cache.get") -def test_cached_api(mock_cache: Mock, mock_requests: Mock) -> None: - mock_cache.return_value = None - mock_requests.return_value.json.return_value = {"data": "value"} - - result = fetch_with_cache("key") - - mock_cache.assert_called_once_with("key") - mock_requests.assert_called_once() - -# Mock side effects -def test_retry_logic() -> None: - mock_api = Mock() - mock_api.call.side_effect = [ - ConnectionError("Failed"), - ConnectionError("Failed"), - {"status": "ok"} - ] - - result = retry_api_call(mock_api) - assert result["status"] == "ok" - assert mock_api.call.call_count == 3 - -# Async mock -@pytest.mark.asyncio -async def test_async_function() -> None: - mock_db = AsyncMock() - mock_db.fetch_user.return_value = User(id=1, name="Alice") - - service = AsyncUserService(mock_db) - user = await service.get_user(1) - - mock_db.fetch_user.assert_awaited_once_with(1) - assert user.name == "Alice" -``` - -## Async Testing - -```python -import pytest -import asyncio - -# Mark async test -@pytest.mark.asyncio -async def test_async_fetch() -> None: - result = await fetch_data("https://api.example.com") - assert result["status"] == "ok" - -# Async fixture -@pytest.fixture -async def async_db() -> AsyncIterator[AsyncDatabase]: - db = AsyncDatabase() - await db.connect() - yield db - await db.disconnect() - -@pytest.mark.asyncio -async def test_async_query(async_db: AsyncDatabase) -> None: - result = await async_db.query("SELECT * FROM users") - assert len(result) > 0 - -# Test concurrent operations -@pytest.mark.asyncio -async def test_concurrent_requests() -> None: - urls = ["http://example.com/1", "http://example.com/2"] - results = await asyncio.gather(*[fetch(url) for url in urls]) - assert len(results) == 2 -``` - -## Pytest Markers - -```python -import pytest - -# Skip test -@pytest.mark.skip(reason="Not implemented yet") -def test_future_feature() -> None: - pass - -# Conditional skip -@pytest.mark.skipif(sys.version_info < (3, 11), reason="Requires Python 3.11+") -def test_new_feature() -> None: - pass - -# Expected failure -@pytest.mark.xfail(reason="Known bug #123") -def test_known_bug() -> None: - assert buggy_function() == expected_value - -# Custom markers -@pytest.mark.slow -def test_slow_operation() -> None: - time.sleep(5) - assert True - -@pytest.mark.integration -def test_integration() -> None: - assert external_service.ping() - -# Run with: pytest -m "not slow" -``` - -## Test Coverage - -```python -# Run with coverage -# pytest --cov=myapp --cov-report=html --cov-report=term - -# conftest.py - coverage configuration -def pytest_configure(config): - config.addinivalue_line( - "markers", "unit: mark test as unit test" - ) - -# pytest.ini or pyproject.toml -""" -[tool.pytest.ini_options] -minversion = "7.0" -addopts = [ - "--cov=myapp", - "--cov-report=term-missing", - "--cov-fail-under=90", - "-ra", - "--strict-markers", -] -testpaths = ["tests"] -""" -``` - -## Property-Based Testing - -```python -from hypothesis import given, strategies as st - -# Property-based test -@given(st.integers(), st.integers()) -def test_addition_commutative(a: int, b: int) -> None: - assert a + b == b + a - -@given(st.lists(st.integers())) -def test_sorted_is_ordered(lst: list[int]) -> None: - sorted_lst = sorted(lst) - for i in range(len(sorted_lst) - 1): - assert sorted_lst[i] <= sorted_lst[i + 1] - -# Custom strategies -@given(st.emails()) -def test_email_validation(email: str) -> None: - assert "@" in email - assert validate_email(email) - -# Composite strategies -from hypothesis import strategies as st -from hypothesis.strategies import composite - -@composite -def users(draw) -> User: - return User( - id=draw(st.integers(min_value=1)), - name=draw(st.text(min_size=1, max_size=50)), - email=draw(st.emails()), - age=draw(st.integers(min_value=18, max_value=120)) - ) - -@given(users()) -def test_user_creation(user: User) -> None: - assert user.age >= 18 - assert len(user.name) > 0 -``` - -## Test Organization - -```python -# tests/ -# conftest.py - Shared fixtures -# test_user.py - User tests -# test_api.py - API tests -# integration/ -# test_workflow.py - Integration tests -# unit/ -# test_models.py - Unit tests - -# Fixture factory pattern -@pytest.fixture -def user_factory(db_session: Session): - created_users: list[User] = [] - - def _create_user( - name: str = "Test User", - email: str | None = None, - **kwargs - ) -> User: - if email is None: - email = f"{name.lower().replace(' ', '.')}@example.com" - - user = User(name=name, email=email, **kwargs) - db_session.add(user) - db_session.commit() - created_users.append(user) - return user - - yield _create_user - - # Cleanup - for user in created_users: - db_session.delete(user) - db_session.commit() -``` - -## Snapshot Testing - -```python -import pytest -from syrupy.assertion import SnapshotAssertion - -def test_api_response(snapshot: SnapshotAssertion) -> None: - response = api.get_user(1) - assert response == snapshot - -def test_rendered_template(snapshot: SnapshotAssertion) -> None: - html = render_template("user.html", user=get_user(1)) - assert html == snapshot -``` diff --git a/.cursor/skills/python-pro/references/type-system.md b/.cursor/skills/python-pro/references/type-system.md deleted file mode 100644 index 958be2bd..00000000 --- a/.cursor/skills/python-pro/references/type-system.md +++ /dev/null @@ -1,293 +0,0 @@ -# Type System Mastery - -> Reference for: Python Pro -> Load when: Type hints, mypy configuration, generics, Protocol definitions - -## Basic Type Annotations - -```python -from typing import Any -from collections.abc import Sequence, Mapping - -# Function signatures -def process_user(name: str, age: int, active: bool = True) -> dict[str, Any]: - return {"name": name, "age": age, "active": active} - -# Use | for unions (Python 3.10+) -def find_user(user_id: int | str) -> dict[str, Any] | None: - if isinstance(user_id, int): - return {"id": user_id} - return None - -# Collections - prefer collections.abc -def process_items(items: Sequence[str]) -> list[str]: - """Accepts list, tuple, or any sequence.""" - return [item.upper() for item in items] - -def merge_configs(base: Mapping[str, int], override: dict[str, int]) -> dict[str, int]: - """Mapping for read-only, dict for mutable.""" - return {**base, **override} -``` - -## Generic Types - -```python -from typing import TypeVar, Generic, Protocol -from collections.abc import Callable - -T = TypeVar('T') -K = TypeVar('K') -V = TypeVar('V') - -# Generic function -def first_element(items: Sequence[T]) -> T | None: - return items[0] if items else None - -# Generic class -class Cache(Generic[K, V]): - def __init__(self) -> None: - self._data: dict[K, V] = {} - - def get(self, key: K) -> V | None: - return self._data.get(key) - - def set(self, key: K, value: V) -> None: - self._data[key] = value - -# Usage -user_cache: Cache[int, str] = Cache() -user_cache.set(1, "Alice") - -# Constrained TypeVar -from numbers import Number -NumT = TypeVar('NumT', bound=Number) - -def add_numbers(a: NumT, b: NumT) -> NumT: - return a + b # type: ignore[return-value] -``` - -## Protocol for Structural Typing - -```python -from typing import Protocol, runtime_checkable - -# Define interface without inheritance -class Drawable(Protocol): - def draw(self) -> str: - ... - - @property - def color(self) -> str: - ... - -class Circle: - def __init__(self, radius: float, color: str) -> None: - self.radius = radius - self._color = color - - def draw(self) -> str: - return f"Drawing {self._color} circle" - - @property - def color(self) -> str: - return self._color - -# Circle implements Drawable without inheriting -def render(shape: Drawable) -> str: - return shape.draw() - -# Runtime checkable protocol -@runtime_checkable -class Closeable(Protocol): - def close(self) -> None: - ... - -def cleanup(resource: Closeable) -> None: - if isinstance(resource, Closeable): - resource.close() -``` - -## Advanced Type Features - -```python -from typing import Literal, TypeAlias, TypedDict, NotRequired, Self, overload - -# Literal types for constants -Mode = Literal["read", "write", "append"] - -def open_file(path: str, mode: Mode) -> None: - ... - -# Type aliases for complex types -JsonDict: TypeAlias = dict[str, Any] -UserId: TypeAlias = int | str - -# TypedDict for structured dictionaries -class UserDict(TypedDict): - id: int - name: str - email: str - age: NotRequired[int] # Optional field - -def create_user(data: UserDict) -> None: - print(data["name"]) # Type-safe access - -# Self type for method chaining -class Builder: - def __init__(self) -> None: - self._value = 0 - - def add(self, n: int) -> Self: - self._value += n - return self - - def multiply(self, n: int) -> Self: - self._value *= n - return self - -# Overload for different signatures -@overload -def process(data: str) -> str: ... - -@overload -def process(data: int) -> int: ... - -def process(data: str | int) -> str | int: - if isinstance(data, str): - return data.upper() - return data * 2 -``` - -## Callable Types - -```python -from collections.abc import Callable -from typing import ParamSpec, Concatenate - -# Basic callable -def apply(func: Callable[[int, int], int], a: int, b: int) -> int: - return func(a, b) - -# ParamSpec for preserving signatures -P = ParamSpec('P') -R = TypeVar('R') - -def logging_decorator(func: Callable[P, R]) -> Callable[P, R]: - def wrapper(*args: P.args, **kwargs: P.kwargs) -> R: - print(f"Calling {func.__name__}") - return func(*args, **kwargs) - return wrapper - -# Concatenate for dependency injection -def with_connection( - func: Callable[Concatenate[Connection, P], R] -) -> Callable[P, R]: - def wrapper(*args: P.args, **kwargs: P.kwargs) -> R: - conn = get_connection() - return func(conn, *args, **kwargs) - return wrapper - -# Usage -@with_connection -def query_user(conn: Connection, user_id: int) -> User: - return conn.execute(f"SELECT * FROM users WHERE id = {user_id}") -``` - -## Mypy Configuration - -```toml -# pyproject.toml -[tool.mypy] -python_version = "3.11" -strict = true -warn_return_any = true -warn_unused_configs = true -disallow_untyped_defs = true -disallow_any_generics = true -disallow_subclassing_any = true -disallow_untyped_calls = true -disallow_incomplete_defs = true -check_untyped_defs = true -no_implicit_optional = true -warn_redundant_casts = true -warn_unused_ignores = true -warn_no_return = true -warn_unreachable = true -strict_equality = true - -[[tool.mypy.overrides]] -module = "third_party.*" -ignore_missing_imports = true -``` - -## Common Type Patterns - -```python -# Result type pattern -from dataclasses import dataclass - -@dataclass -class Success(Generic[T]): - value: T - -@dataclass -class Error: - message: str - -Result = Success[T] | Error - -def divide(a: int, b: int) -> Result[float]: - if b == 0: - return Error("Division by zero") - return Success(a / b) - -# Option/Maybe type -def safe_get(items: Sequence[T], index: int) -> T | None: - try: - return items[index] - except IndexError: - return None - -# Sentinel value with typing -from typing import Final - -MISSING: Final = object() - -def get_value(key: str, default: T | type[MISSING] = MISSING) -> T: - if default is MISSING: - raise KeyError(key) - return default # type: ignore[return-value] -``` - -## Type Narrowing - -```python -from typing import assert_type, assert_never - -def process_value(value: int | str | None) -> str: - # Type guards - if value is None: - return "null" - - if isinstance(value, int): - # Type narrowed to int - return str(value * 2) - - # Type narrowed to str - return value.upper() - -# Exhaustiveness checking -def handle_mode(mode: Literal["read", "write"]) -> str: - if mode == "read": - return "Reading" - elif mode == "write": - return "Writing" - else: - # Mypy will error if mode can be anything else - assert_never(mode) - -# Custom type guard -def is_string_list(val: list[Any]) -> bool: - """Runtime check for list of strings.""" - return all(isinstance(x, str) for x in val) -``` diff --git a/.cursor/skills/rust-engineer/SKILL.md b/.cursor/skills/rust-engineer/SKILL.md deleted file mode 100644 index fe5b1c05..00000000 --- a/.cursor/skills/rust-engineer/SKILL.md +++ /dev/null @@ -1,96 +0,0 @@ ---- -name: rust-engineer -description: Use when building Rust applications requiring memory safety, systems programming, or zero-cost abstractions. Invoke for ownership patterns, lifetimes, traits, async/await with tokio. -triggers: - - Rust - - Cargo - - ownership - - borrowing - - lifetimes - - async Rust - - tokio - - zero-cost abstractions - - memory safety - - systems programming -role: specialist -scope: implementation -output-format: code ---- - -# Rust Engineer - -Senior Rust engineer with deep expertise in Rust 2021 edition, systems programming, memory safety, and zero-cost abstractions. Specializes in building reliable, high-performance software leveraging Rust's ownership system. - -## Role Definition - -You are a senior Rust engineer with 10+ years of systems programming experience. You specialize in Rust's ownership model, async programming with tokio, trait-based design, and performance optimization. You build memory-safe, concurrent systems with zero-cost abstractions. - -## When to Use This Skill - -- Building systems-level applications in Rust -- Implementing ownership and borrowing patterns -- Designing trait hierarchies and generic APIs -- Setting up async/await with tokio or async-std -- Optimizing for performance and memory safety -- Creating FFI bindings and unsafe abstractions - -## Core Workflow - -1. **Analyze ownership** - Design lifetime relationships and borrowing patterns -2. **Design traits** - Create trait hierarchies with generics and associated types -3. **Implement safely** - Write idiomatic Rust with minimal unsafe code -4. **Handle errors** - Use Result/Option with ? operator and custom error types -5. **Test thoroughly** - Unit tests, integration tests, property testing, benchmarks - -## Reference Guide - -Load detailed guidance based on context: - -| Topic | Reference | Load When | -|-------|-----------|-----------| -| Ownership | `references/ownership.md` | Lifetimes, borrowing, smart pointers, Pin | -| Traits | `references/traits.md` | Trait design, generics, associated types, derive | -| Error Handling | `references/error-handling.md` | Result, Option, ?, custom errors, thiserror | -| Async | `references/async.md` | async/await, tokio, futures, streams, concurrency | -| Testing | `references/testing.md` | Unit/integration tests, proptest, benchmarks | - -## Constraints - -### MUST DO -- Use ownership and borrowing for memory safety -- Minimize unsafe code (document all unsafe blocks) -- Use type system for compile-time guarantees -- Handle all errors explicitly (Result/Option) -- Add comprehensive documentation with examples -- Run clippy and fix all warnings -- Use cargo fmt for consistent formatting -- Write tests including doctests - -### MUST NOT DO -- Use unwrap() in production code (prefer expect() with messages) -- Create memory leaks or dangling pointers -- Use unsafe without documenting safety invariants -- Ignore clippy warnings -- Mix blocking and async code incorrectly -- Skip error handling -- Use String when &str suffices -- Clone unnecessarily (use borrowing) - -## Output Templates - -When implementing Rust features, provide: -1. Type definitions (structs, enums, traits) -2. Implementation with proper ownership -3. Error handling with custom error types -4. Tests (unit, integration, doctests) -5. Brief explanation of design decisions - -## Knowledge Reference - -Rust 2021, Cargo, ownership/borrowing, lifetimes, traits, generics, async/await, tokio, Result/Option, thiserror/anyhow, serde, clippy, rustfmt, cargo-test, criterion benchmarks, MIRI, unsafe Rust - -## Related Skills - -- **Systems Architect** - Low-level system design -- **Performance Engineer** - Optimization and profiling -- **Test Master** - Comprehensive testing strategies diff --git a/.cursor/skills/rust-engineer/references/async.md b/.cursor/skills/rust-engineer/references/async.md deleted file mode 100644 index 94190a82..00000000 --- a/.cursor/skills/rust-engineer/references/async.md +++ /dev/null @@ -1,461 +0,0 @@ -# Async Programming in Rust - -> Reference for: Rust Engineer -> Load when: async/await, tokio, futures, streams, concurrency patterns - -## Basic Async/Await - -```rust -use tokio; - -// Async function returns a Future -async fn fetch_data(url: &str) -> Result { - let response = reqwest::get(url).await?; - let body = response.text().await?; - Ok(body) -} - -// Tokio runtime -#[tokio::main] -async fn main() -> Result<(), Box> { - let data = fetch_data("https://api.example.com").await?; - println!("Data: {}", data); - Ok(()) -} - -// Manual runtime creation -fn main() { - let runtime = tokio::runtime::Runtime::new().unwrap(); - runtime.block_on(async { - println!("Hello from async context"); - }); -} -``` - -## Concurrent Execution - -```rust -use tokio; - -// Sequential execution -async fn sequential() { - let result1 = async_operation1().await; - let result2 = async_operation2().await; // Waits for operation1 -} - -// Concurrent execution with join! -async fn concurrent() { - let (result1, result2) = tokio::join!( - async_operation1(), - async_operation2() - ); -} - -// Concurrent with try_join! (stops on first error) -async fn concurrent_with_errors() -> Result<(), Box> { - let (result1, result2) = tokio::try_join!( - fallible_operation1(), - fallible_operation2() - )?; - Ok(()) -} - -// Spawning tasks -async fn spawn_tasks() { - let handle1 = tokio::spawn(async { - // This runs on a separate task - expensive_computation().await - }); - - let handle2 = tokio::spawn(async { - another_computation().await - }); - - // Wait for both to complete - let result1 = handle1.await.unwrap(); - let result2 = handle2.await.unwrap(); -} -``` - -## Select and Race Conditions - -```rust -use tokio::time::{sleep, Duration}; - -// select! - wait for first to complete -async fn first_to_complete() { - tokio::select! { - result = async_operation1() => { - println!("Operation 1 completed first: {:?}", result); - } - result = async_operation2() => { - println!("Operation 2 completed first: {:?}", result); - } - } -} - -// Timeout pattern -async fn with_timeout() -> Result { - tokio::select! { - result = fetch_data("https://api.example.com") => { - result.map_err(|_| "Fetch failed") - } - _ = sleep(Duration::from_secs(5)) => { - Err("Timeout") - } - } -} - -// Cancellation with select! -async fn cancellable_operation(mut cancel_rx: tokio::sync::watch::Receiver) { - tokio::select! { - result = long_running_task() => { - println!("Task completed: {:?}", result); - } - _ = cancel_rx.changed() => { - println!("Task cancelled"); - } - } -} -``` - -## Streams - -```rust -use tokio_stream::{self as stream, StreamExt}; - -// Creating streams -async fn stream_example() { - let mut stream = stream::iter(vec![1, 2, 3, 4, 5]); - - while let Some(value) = stream.next().await { - println!("Value: {}", value); - } -} - -// Stream combinators -async fn stream_combinators() { - let stream = stream::iter(vec![1, 2, 3, 4, 5]) - .filter(|x| *x % 2 == 0) - .map(|x| x * 2); - - let results: Vec<_> = stream.collect().await; - println!("Results: {:?}", results); -} - -// Async stream processing -use futures::stream::{self, StreamExt}; - -async fn process_stream() { - let stream = stream::iter(vec![1, 2, 3, 4, 5]) - .then(|x| async move { - tokio::time::sleep(Duration::from_millis(100)).await; - x * 2 - }); - - stream.for_each(|x| async move { - println!("Processed: {}", x); - }).await; -} -``` - -## Channels for Communication - -```rust -use tokio::sync::{mpsc, oneshot, broadcast, watch}; - -// mpsc: multiple producer, single consumer -async fn mpsc_example() { - let (tx, mut rx) = mpsc::channel(32); - - tokio::spawn(async move { - tx.send("Hello").await.unwrap(); - tx.send("World").await.unwrap(); - }); - - while let Some(msg) = rx.recv().await { - println!("Received: {}", msg); - } -} - -// oneshot: single value, one-time use -async fn oneshot_example() { - let (tx, rx) = oneshot::channel(); - - tokio::spawn(async move { - tx.send("Result").unwrap(); - }); - - let result = rx.await.unwrap(); - println!("Got: {}", result); -} - -// broadcast: multiple producers, multiple consumers -async fn broadcast_example() { - let (tx, mut rx1) = broadcast::channel(16); - let mut rx2 = tx.subscribe(); - - tokio::spawn(async move { - tx.send("Message").unwrap(); - }); - - println!("rx1: {}", rx1.recv().await.unwrap()); - println!("rx2: {}", rx2.recv().await.unwrap()); -} - -// watch: single producer, multiple consumers (last value) -async fn watch_example() { - let (tx, mut rx) = watch::channel("initial"); - - tokio::spawn(async move { - loop { - rx.changed().await.unwrap(); - println!("Value changed to: {}", *rx.borrow()); - } - }); - - tx.send("updated").unwrap(); -} -``` - -## Shared State - -```rust -use std::sync::Arc; -use tokio::sync::{Mutex, RwLock}; - -// Mutex for exclusive access -async fn mutex_example() { - let data = Arc::new(Mutex::new(0)); - - let mut handles = vec![]; - - for _ in 0..10 { - let data = Arc::clone(&data); - let handle = tokio::spawn(async move { - let mut lock = data.lock().await; - *lock += 1; - }); - handles.push(handle); - } - - for handle in handles { - handle.await.unwrap(); - } - - println!("Final value: {}", *data.lock().await); -} - -// RwLock for read-write patterns -async fn rwlock_example() { - let data = Arc::new(RwLock::new(vec![1, 2, 3])); - - // Multiple readers - let data1 = Arc::clone(&data); - tokio::spawn(async move { - let read = data1.read().await; - println!("Read: {:?}", *read); - }); - - let data2 = Arc::clone(&data); - tokio::spawn(async move { - let read = data2.read().await; - println!("Read: {:?}", *read); - }); - - // Single writer - tokio::time::sleep(Duration::from_millis(100)).await; - let mut write = data.write().await; - write.push(4); -} -``` - -## Async Traits (with async-trait) - -```rust -use async_trait::async_trait; - -#[async_trait] -trait AsyncRepository { - async fn find_by_id(&self, id: u64) -> Result; - async fn save(&self, user: User) -> Result<(), Error>; -} - -struct DatabaseRepository { - pool: sqlx::PgPool, -} - -#[async_trait] -impl AsyncRepository for DatabaseRepository { - async fn find_by_id(&self, id: u64) -> Result { - sqlx::query_as("SELECT * FROM users WHERE id = $1") - .bind(id) - .fetch_one(&self.pool) - .await - .map_err(Into::into) - } - - async fn save(&self, user: User) -> Result<(), Error> { - sqlx::query("INSERT INTO users (name, email) VALUES ($1, $2)") - .bind(&user.name) - .bind(&user.email) - .execute(&self.pool) - .await?; - Ok(()) - } -} -``` - -## Pin and Futures - -```rust -use std::pin::Pin; -use std::future::Future; -use std::task::{Context, Poll}; - -// Manual Future implementation -struct DelayedValue { - value: i32, - delay: tokio::time::Sleep, -} - -impl Future for DelayedValue { - type Output = i32; - - fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { - match Pin::new(&mut self.delay).poll(cx) { - Poll::Ready(_) => Poll::Ready(self.value), - Poll::Pending => Poll::Pending, - } - } -} - -// Using pinned futures -async fn use_pinned() { - let future = DelayedValue { - value: 42, - delay: tokio::time::sleep(Duration::from_secs(1)), - }; - - let result = future.await; - println!("Result: {}", result); -} -``` - -## Background Tasks and Graceful Shutdown - -```rust -use tokio::signal; - -async fn background_task(mut shutdown: tokio::sync::watch::Receiver) { - loop { - tokio::select! { - _ = tokio::time::sleep(Duration::from_secs(1)) => { - println!("Background task running..."); - } - _ = shutdown.changed() => { - println!("Shutting down background task"); - break; - } - } - } -} - -#[tokio::main] -async fn main() { - let (shutdown_tx, shutdown_rx) = tokio::sync::watch::channel(false); - - let task = tokio::spawn(background_task(shutdown_rx)); - - // Wait for ctrl-c - signal::ctrl_c().await.unwrap(); - println!("Received shutdown signal"); - - // Signal shutdown - shutdown_tx.send(true).unwrap(); - - // Wait for task to complete - task.await.unwrap(); -} -``` - -## Error Handling in Async - -```rust -use thiserror::Error; - -#[derive(Error, Debug)] -enum AsyncError { - #[error("Network error: {0}")] - Network(#[from] reqwest::Error), - - #[error("Timeout")] - Timeout, - - #[error("Task failed")] - TaskFailed(#[from] tokio::task::JoinError), -} - -async fn robust_operation() -> Result { - let timeout = Duration::from_secs(5); - - let result = tokio::time::timeout(timeout, async { - reqwest::get("https://api.example.com") - .await? - .text() - .await - }) - .await - .map_err(|_| AsyncError::Timeout)??; - - Ok(result) -} -``` - -## Runtime Configuration - -```rust -// Custom runtime configuration -fn main() { - let runtime = tokio::runtime::Builder::new_multi_thread() - .worker_threads(4) - .thread_name("my-worker") - .thread_stack_size(3 * 1024 * 1024) - .enable_all() - .build() - .unwrap(); - - runtime.block_on(async { - println!("Running on custom runtime"); - }); -} - -// Current-thread runtime (single-threaded) -fn single_threaded() { - let runtime = tokio::runtime::Builder::new_current_thread() - .enable_all() - .build() - .unwrap(); - - runtime.block_on(async { - println!("Single-threaded async"); - }); -} -``` - -## Best Practices - -- Use tokio::spawn for CPU-bound tasks on multi-threaded runtime -- Use spawn_blocking for blocking operations (file I/O, sync code) -- Prefer tokio::sync primitives over std::sync in async code -- Use channels for task communication instead of shared state when possible -- Always handle JoinHandle results (tasks can panic) -- Use select! for cancellation patterns -- Avoid holding locks across .await points -- Use timeout for all external I/O operations -- Implement graceful shutdown with channels -- Use async-trait for trait-based async code -- Prefer try_join! over manual error handling -- Use Arc> sparingly (channels often better) -- Test async code with tokio::test macro -- Monitor task spawning to prevent unbounded growth diff --git a/.cursor/skills/rust-engineer/references/error-handling.md b/.cursor/skills/rust-engineer/references/error-handling.md deleted file mode 100644 index f09cfc66..00000000 --- a/.cursor/skills/rust-engineer/references/error-handling.md +++ /dev/null @@ -1,337 +0,0 @@ -# Error Handling in Rust - -> Reference for: Rust Engineer -> Load when: Handling errors, Result, Option, custom error types, thiserror - -## Result and Option Basics - -```rust -// Result: operation that can fail -fn divide(a: f64, b: f64) -> Result { - if b == 0.0 { - Err("Division by zero".to_string()) - } else { - Ok(a / b) - } -} - -// Option: value that might be absent -fn find_user(id: u64) -> Option { - if id == 1 { - Some(User { id, name: "Alice".to_string() }) - } else { - None - } -} - -// Using ? operator for propagation -fn calculate(a: f64, b: f64, c: f64) -> Result { - let x = divide(a, b)?; // Returns Err early if division fails - let y = divide(x, c)?; - Ok(y) -} -``` - -## Custom Error Types - -```rust -use std::fmt; - -// Manual error type -#[derive(Debug)] -enum AppError { - NotFound(String), - InvalidInput(String), - DatabaseError(String), -} - -impl fmt::Display for AppError { - fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - match self { - AppError::NotFound(msg) => write!(f, "Not found: {}", msg), - AppError::InvalidInput(msg) => write!(f, "Invalid input: {}", msg), - AppError::DatabaseError(msg) => write!(f, "Database error: {}", msg), - } - } -} - -impl std::error::Error for AppError {} - -// Usage -fn get_user(id: u64) -> Result { - if id == 0 { - return Err(AppError::InvalidInput("ID cannot be zero".to_string())); - } - // ... fetch user - Err(AppError::NotFound(format!("User {} not found", id))) -} -``` - -## Using thiserror - -```rust -use thiserror::Error; - -#[derive(Error, Debug)] -enum DataError { - #[error("Data not found: {0}")] - NotFound(String), - - #[error("Invalid ID: {id}, reason: {reason}")] - InvalidId { id: u64, reason: String }, - - #[error("IO error")] - Io(#[from] std::io::Error), - - #[error("Parse error")] - Parse(#[from] std::num::ParseIntError), - - #[error("Database error: {0}")] - Database(#[from] sqlx::Error), -} - -// Usage with automatic conversions -fn read_config(path: &str) -> Result { - let content = std::fs::read_to_string(path)?; // Auto-converts io::Error - let port: u16 = content.parse()?; // Auto-converts ParseIntError - Ok(Config { port }) -} -``` - -## Using anyhow for Applications - -```rust -use anyhow::{Result, Context, bail, ensure}; - -// Simple error handling for applications -fn process_file(path: &str) -> Result<()> { - let content = std::fs::read_to_string(path) - .context(format!("Failed to read file: {}", path))?; - - ensure!(!content.is_empty(), "File is empty"); - - if content.len() > 1000 { - bail!("File too large"); - } - - // Process content... - Ok(()) -} - -// Adding context to errors -fn main() -> Result<()> { - process_file("config.txt") - .context("Failed to process configuration")?; - Ok(()) -} -``` - -## Option Combinators - -```rust -// map: transform Option to Option -let num: Option = Some(5); -let doubled = num.map(|n| n * 2); // Some(10) - -// and_then: chain operations -let result = Some(5) - .and_then(|n| if n > 0 { Some(n * 2) } else { None }) - .and_then(|n| Some(n + 1)); // Some(11) - -// or: provide alternative -let value = None.or(Some(42)); // Some(42) - -// unwrap_or: provide default -let value = None.unwrap_or(42); // 42 - -// unwrap_or_else: compute default lazily -let value = None.unwrap_or_else(|| expensive_computation()); - -// filter: conditional None -let num = Some(5).filter(|&n| n > 10); // None - -// Pattern matching -match find_user(1) { - Some(user) => println!("Found: {}", user.name), - None => println!("User not found"), -} - -// if let for simple cases -if let Some(user) = find_user(1) { - println!("Found: {}", user.name); -} -``` - -## Result Combinators - -```rust -// map: transform Ok value -let result: Result = Ok(5); -let doubled = result.map(|n| n * 2); // Ok(10) - -// map_err: transform error -let result: Result = Err("error"); -let mapped = result.map_err(|e| e.to_uppercase()); // Err("ERROR") - -// and_then: chain fallible operations -fn parse_then_double(s: &str) -> Result { - s.parse::() - .and_then(|n| Ok(n * 2)) -} - -// or_else: provide alternative computation -let result = Err("error").or_else(|_| Ok(42)); // Ok(42) - -// unwrap_or: provide default -let value = Err("error").unwrap_or(42); // 42 - -// expect: unwrap with custom panic message -let value = result.expect("Failed to parse number"); - -// Pattern matching -match divide(10.0, 2.0) { - Ok(result) => println!("Result: {}", result), - Err(e) => eprintln!("Error: {}", e), -} -``` - -## Error Conversion and From Trait - -```rust -use std::io; -use std::num::ParseIntError; - -#[derive(Debug)] -enum MyError { - Io(io::Error), - Parse(ParseIntError), -} - -impl From for MyError { - fn from(err: io::Error) -> Self { - MyError::Io(err) - } -} - -impl From for MyError { - fn from(err: ParseIntError) -> Self { - MyError::Parse(err) - } -} - -// Now ? operator works with automatic conversion -fn read_and_parse(path: &str) -> Result { - let content = std::fs::read_to_string(path)?; // io::Error -> MyError - let number = content.trim().parse()?; // ParseIntError -> MyError - Ok(number) -} -``` - -## Advanced Error Patterns - -```rust -// Multiple error sources with Box -use std::error::Error; - -fn complex_operation() -> Result> { - let file = std::fs::read_to_string("data.txt")?; - let number: i32 = file.trim().parse()?; - Ok(format!("Number: {}", number)) -} - -// Error with backtrace (nightly) -#[derive(Debug)] -struct DetailedError { - message: String, - backtrace: std::backtrace::Backtrace, -} - -impl DetailedError { - fn new(message: impl Into) -> Self { - Self { - message: message.into(), - backtrace: std::backtrace::Backtrace::capture(), - } - } -} - -// Recoverable vs unrecoverable errors -fn might_fail(value: i32) -> Result { - if value < 0 { - Err("Negative value".to_string()) // Recoverable - } else if value > 1000 { - panic!("Value too large!"); // Unrecoverable - } else { - Ok(value * 2) - } -} -``` - -## Try Blocks (Nightly) - -```rust -#![feature(try_blocks)] - -// Try block for localized error handling -let result: Result> = try { - let file = std::fs::read_to_string("config.txt")?; - let num: i32 = file.trim().parse()?; - num * 2 -}; -``` - -## Error Context Pattern - -```rust -use thiserror::Error; - -#[derive(Error, Debug)] -#[error("{message}")] -struct ContextError { - message: String, - #[source] - source: Option>, -} - -impl ContextError { - fn new(message: impl Into) -> Self { - Self { - message: message.into(), - source: None, - } - } - - fn with_source(mut self, source: impl Error + Send + Sync + 'static) -> Self { - self.source = Some(Box::new(source)); - self - } -} - -// Extension trait for adding context -trait Context { - fn context(self, message: impl Into) -> Result; -} - -impl Context for Result { - fn context(self, message: impl Into) -> Result { - self.map_err(|e| ContextError::new(message).with_source(e)) - } -} -``` - -## Best Practices - -- Use Result for recoverable errors, panic! for unrecoverable bugs -- Prefer ? operator over unwrap() in production code -- Use expect() with descriptive messages instead of unwrap() -- Use thiserror for libraries (structured errors) -- Use anyhow for applications (simple error handling) -- Implement std::error::Error trait for custom error types -- Add context to errors as they propagate up the stack -- Use #[from] in thiserror for automatic conversions -- Document error conditions in function documentation -- Use Option::ok_or() to convert Option to Result -- Use Result::ok() to convert Result to Option (discarding error) -- Avoid String as error type (use custom types instead) -- Use ensure! and bail! from anyhow for cleaner checks -- Log errors at boundaries, return them in library code diff --git a/.cursor/skills/rust-engineer/references/ownership.md b/.cursor/skills/rust-engineer/references/ownership.md deleted file mode 100644 index 6c972d3d..00000000 --- a/.cursor/skills/rust-engineer/references/ownership.md +++ /dev/null @@ -1,281 +0,0 @@ -# Ownership, Borrowing, and Lifetimes - -> Reference for: Rust Engineer -> Load when: Working with ownership, lifetimes, smart pointers, borrowing - -## Ownership Patterns - -```rust -// Move semantics (ownership transfer) -fn take_ownership(s: String) { - println!("{}", s); -} // s dropped here - -// Borrowing (immutable reference) -fn borrow(s: &String) { - println!("{}", s); -} // s NOT dropped, caller still owns - -// Mutable borrowing -fn borrow_mut(s: &mut String) { - s.push_str(" world"); -} - -// Usage -let s = String::from("hello"); -borrow(&s); // OK, immutable borrow -let mut s2 = s; // Move, s no longer valid -borrow_mut(&mut s2); // OK, mutable borrow -``` - -## Lifetime Annotations - -```rust -// Explicit lifetime: returned reference lives as long as input -fn longest<'a>(x: &'a str, y: &'a str) -> &'a str { - if x.len() > y.len() { x } else { y } -} - -// Multiple lifetimes -fn first_word<'a, 'b>(s: &'a str, _other: &'b str) -> &'a str { - s.split_whitespace().next().unwrap_or("") -} - -// Lifetime in structs -struct Excerpt<'a> { - part: &'a str, -} - -impl<'a> Excerpt<'a> { - fn announce_and_return(&self, announcement: &str) -> &'a str { - println!("Attention: {}", announcement); - self.part - } -} - -// Static lifetime (lives for entire program) -const GREETING: &'static str = "Hello, world!"; -``` - -## Smart Pointers - -```rust -use std::rc::Rc; -use std::cell::RefCell; -use std::sync::{Arc, Mutex}; - -// Box: heap allocation, single owner -let b = Box::new(5); - -// Rc: reference counting (single-threaded) -let rc1 = Rc::new(vec![1, 2, 3]); -let rc2 = Rc::clone(&rc1); // Increment count -println!("Count: {}", Rc::strong_count(&rc1)); // 2 - -// Arc: atomic reference counting (thread-safe) -let arc1 = Arc::new(vec![1, 2, 3]); -let arc2 = Arc::clone(&arc1); -std::thread::spawn(move || { - println!("{:?}", arc2); -}); - -// RefCell: interior mutability (runtime borrow checking) -let data = RefCell::new(5); -*data.borrow_mut() += 1; // Mutable borrow at runtime - -// Combining Rc + RefCell for shared mutable state -let shared = Rc::new(RefCell::new(vec![1, 2, 3])); -shared.borrow_mut().push(4); - -// Combining Arc + Mutex for thread-safe shared state -let counter = Arc::new(Mutex::new(0)); -let counter_clone = Arc::clone(&counter); -std::thread::spawn(move || { - let mut num = counter_clone.lock().unwrap(); - *num += 1; -}); -``` - -## Interior Mutability - -```rust -use std::cell::{Cell, RefCell}; - -// Cell: Copy types only -let c = Cell::new(5); -c.set(10); -let val = c.get(); - -// RefCell: runtime borrow checking -let data = RefCell::new(vec![1, 2, 3]); -data.borrow_mut().push(4); - -// Pattern: mock objects with interior mutability -struct MockLogger { - messages: RefCell>, -} - -impl MockLogger { - fn new() -> Self { - Self { messages: RefCell::new(Vec::new()) } - } - - fn log(&self, msg: &str) { - self.messages.borrow_mut().push(msg.to_string()); - } - - fn get_messages(&self) -> Vec { - self.messages.borrow().clone() - } -} -``` - -## Pin and Self-Referential Types - -```rust -use std::pin::Pin; -use std::marker::PhantomPinned; - -// Self-referential struct (requires Pin) -struct SelfReferential { - data: String, - pointer: *const String, - _pin: PhantomPinned, -} - -impl SelfReferential { - fn new(data: String) -> Pin> { - let mut boxed = Box::pin(Self { - data, - pointer: std::ptr::null(), - _pin: PhantomPinned, - }); - - // Safe: we're not moving the data after this - let ptr = &boxed.data as *const String; - unsafe { - let mut_ref = Pin::as_mut(&mut boxed); - Pin::get_unchecked_mut(mut_ref).pointer = ptr; - } - - boxed - } -} - -// Pin in async contexts -async fn pinned_future() { - // Futures are often self-referential, hence Pin - let fut = async { 42 }; - let pinned = Box::pin(fut); - pinned.await; -} -``` - -## Cow (Clone on Write) - -```rust -use std::borrow::Cow; - -fn process_text(input: &str) -> Cow { - if input.contains("bad") { - // Need to modify: allocate new String - Cow::Owned(input.replace("bad", "good")) - } else { - // No modification needed: just borrow - Cow::Borrowed(input) - } -} - -// Usage -let text1 = "hello world"; -let result1 = process_text(text1); // Borrowed (no allocation) - -let text2 = "bad word"; -let result2 = process_text(text2); // Owned (allocated) -``` - -## Drop Trait and RAII - -```rust -struct FileGuard { - name: String, -} - -impl FileGuard { - fn new(name: String) -> Self { - println!("Opening {}", name); - Self { name } - } -} - -impl Drop for FileGuard { - fn drop(&mut self) { - println!("Closing {}", self.name); - } -} - -// Usage: automatic cleanup -{ - let _file = FileGuard::new("data.txt".to_string()); - // Use file... -} // Drop called automatically here -``` - -## Common Patterns - -```rust -// Builder pattern with ownership -struct Config { - host: String, - port: u16, -} - -impl Config { - fn builder() -> ConfigBuilder { - ConfigBuilder::default() - } -} - -struct ConfigBuilder { - host: Option, - port: Option, -} - -impl ConfigBuilder { - fn host(mut self, host: impl Into) -> Self { - self.host = Some(host.into()); - self - } - - fn port(mut self, port: u16) -> Self { - self.port = Some(port); - self - } - - fn build(self) -> Result { - Ok(Config { - host: self.host.ok_or("host required")?, - port: self.port.unwrap_or(8080), - }) - } -} - -// Usage -let config = Config::builder() - .host("localhost") - .port(3000) - .build()?; -``` - -## Best Practices - -- Prefer borrowing (&T) over ownership transfer when possible -- Use &str over String for function parameters -- Use &[T] over Vec for function parameters -- Clone only when necessary (profile first) -- Use Cow<'a, T> for conditional cloning -- Document lifetime relationships in complex cases -- Use Arc> for shared mutable state across threads -- Use Rc> for shared mutable state in single thread -- Implement Drop for RAII patterns -- Use PhantomData to constrain variance when needed diff --git a/.cursor/skills/rust-engineer/references/testing.md b/.cursor/skills/rust-engineer/references/testing.md deleted file mode 100644 index 982f311a..00000000 --- a/.cursor/skills/rust-engineer/references/testing.md +++ /dev/null @@ -1,473 +0,0 @@ -# Testing in Rust - -> Reference for: Rust Engineer -> Load when: Unit tests, integration tests, property testing, benchmarks - -## Unit Tests - -```rust -// Tests in same file -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn test_addition() { - assert_eq!(2 + 2, 4); - } - - #[test] - fn test_subtraction() { - assert!(10 - 5 == 5); - } - - #[test] - #[should_panic(expected = "division by zero")] - fn test_panic() { - divide(10, 0); - } - - #[test] - fn test_result() -> Result<(), String> { - let result = divide(10, 2)?; - assert_eq!(result, 5); - Ok(()) - } - - #[test] - #[ignore] - fn expensive_test() { - // Run with: cargo test -- --ignored - } -} - -// Assertions -fn assert_examples() { - assert!(true); - assert_eq!(2 + 2, 4); - assert_ne!(2 + 2, 5); - - // Custom messages - assert!(value > 0, "Value must be positive, got {}", value); - assert_eq!(result, expected, "Calculation failed"); -} -``` - -## Doctests - -```rust -/// Adds two numbers together. -/// -/// # Examples -/// -/// ``` -/// use mylib::add; -/// -/// let result = add(2, 3); -/// assert_eq!(result, 5); -/// ``` -/// -/// ```should_panic -/// use mylib::divide; -/// -/// divide(10, 0); // This will panic -/// ``` -/// -/// ```ignore -/// // This code won't compile but won't fail the test -/// let x = undefined_function(); -/// ``` -pub fn add(a: i32, b: i32) -> i32 { - a + b -} -``` - -## Integration Tests - -```rust -// tests/integration_test.rs -use mylib; - -#[test] -fn test_full_workflow() { - let config = mylib::Config::new("test.conf"); - let result = mylib::process(&config); - assert!(result.is_ok()); -} - -// tests/common/mod.rs - shared test utilities -pub fn setup() -> TestContext { - TestContext { - db: create_test_db(), - } -} - -// tests/another_test.rs -mod common; - -#[test] -fn test_with_common() { - let ctx = common::setup(); - // Use ctx... -} -``` - -## Test Organization - -```rust -// Nested test modules -#[cfg(test)] -mod tests { - use super::*; - - mod addition { - use super::*; - - #[test] - fn positive_numbers() { - assert_eq!(add(2, 3), 5); - } - - #[test] - fn negative_numbers() { - assert_eq!(add(-2, -3), -5); - } - } - - mod subtraction { - use super::*; - - #[test] - fn test_subtract() { - assert_eq!(subtract(10, 5), 5); - } - } -} -``` - -## Test Fixtures and Setup - -```rust -struct TestContext { - temp_dir: std::path::PathBuf, - db: Database, -} - -impl TestContext { - fn setup() -> Self { - let temp_dir = std::env::temp_dir().join("test"); - std::fs::create_dir_all(&temp_dir).unwrap(); - - Self { - temp_dir, - db: Database::connect_test(), - } - } -} - -impl Drop for TestContext { - fn drop(&mut self) { - // Cleanup - std::fs::remove_dir_all(&self.temp_dir).ok(); - self.db.disconnect(); - } -} - -#[test] -fn test_with_fixture() { - let ctx = TestContext::setup(); - // Test uses ctx... - // Automatic cleanup via Drop -} -``` - -## Async Tests - -```rust -use tokio; - -#[tokio::test] -async fn test_async_function() { - let result = async_operation().await; - assert_eq!(result, 42); -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 2)] -async fn test_with_custom_runtime() { - let result = concurrent_operation().await; - assert!(result.is_ok()); -} - -// Testing async with timeout -#[tokio::test] -async fn test_with_timeout() { - let timeout = std::time::Duration::from_secs(5); - let result = tokio::time::timeout(timeout, slow_operation()).await; - assert!(result.is_ok()); -} -``` - -## Property-Based Testing (proptest) - -```rust -use proptest::prelude::*; - -// Simple property test -proptest! { - #[test] - fn test_reversing_twice_is_identity(ref s in ".*") { - let reversed: String = s.chars().rev().collect(); - let double_reversed: String = reversed.chars().rev().collect(); - assert_eq!(s, &double_reversed); - } -} - -// Custom strategies -proptest! { - #[test] - fn test_addition_commutative(a in 0..1000i32, b in 0..1000i32) { - assert_eq!(a + b, b + a); - } - - #[test] - fn test_vector_push_pop( - ref v in prop::collection::vec(0..100i32, 0..100), - item in 0..100i32 - ) { - let mut v = v.clone(); - v.push(item); - assert_eq!(v.pop(), Some(item)); - } -} - -// Complex custom strategies -fn user_strategy() -> impl Strategy { - (1..1000u64, "[a-z]{3,10}", "[a-z0-9.]+@[a-z]+\\.[a-z]+") - .prop_map(|(id, name, email)| User { id, name, email }) -} - -proptest! { - #[test] - fn test_user_serialization(user in user_strategy()) { - let json = serde_json::to_string(&user).unwrap(); - let deserialized: User = serde_json::from_str(&json).unwrap(); - assert_eq!(user, deserialized); - } -} -``` - -## Mocking - -```rust -// Using mockall -use mockall::*; -use mockall::predicate::*; - -#[automock] -trait Database { - fn get_user(&self, id: u64) -> Option; - fn save_user(&mut self, user: User) -> Result<(), Error>; -} - -#[test] -fn test_with_mock() { - let mut mock = MockDatabase::new(); - - mock.expect_get_user() - .with(eq(1)) - .times(1) - .returning(|_| Some(User { id: 1, name: "Alice".to_string() })); - - mock.expect_save_user() - .times(1) - .returning(|_| Ok(())); - - // Use mock in test - let user = mock.get_user(1); - assert!(user.is_some()); -} -``` - -## Benchmarks (Criterion) - -```rust -// benches/my_benchmark.rs -use criterion::{black_box, criterion_group, criterion_main, Criterion}; - -fn fibonacci(n: u64) -> u64 { - match n { - 0 => 1, - 1 => 1, - n => fibonacci(n - 1) + fibonacci(n - 2), - } -} - -fn criterion_benchmark(c: &mut Criterion) { - c.bench_function("fib 20", |b| b.iter(|| fibonacci(black_box(20)))); -} - -criterion_group!(benches, criterion_benchmark); -criterion_main!(benches); - -// Cargo.toml: -// [dev-dependencies] -// criterion = "0.5" -// -// [[bench]] -// name = "my_benchmark" -// harness = false -``` - -## Advanced Benchmarking - -```rust -use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main}; - -fn bench_multiple_sizes(c: &mut Criterion) { - let mut group = c.benchmark_group("sorting"); - - for size in [10, 100, 1000, 10000].iter() { - group.bench_with_input(BenchmarkId::from_parameter(size), size, |b, &size| { - b.iter_batched( - || generate_random_vec(size), - |mut v| v.sort(), - criterion::BatchSize::SmallInput, - ); - }); - } - - group.finish(); -} - -// Comparing implementations -fn bench_comparison(c: &mut Criterion) { - let mut group = c.benchmark_group("string_search"); - - group.bench_function("naive", |b| { - b.iter(|| naive_search(black_box("haystack"), black_box("needle"))) - }); - - group.bench_function("optimized", |b| { - b.iter(|| optimized_search(black_box("haystack"), black_box("needle"))) - }); - - group.finish(); -} - -criterion_group!(benches, bench_multiple_sizes, bench_comparison); -criterion_main!(benches); -``` - -## Testing with External Resources - -```rust -// Testing file I/O -#[test] -fn test_file_operations() { - use std::io::Write; - - let temp_dir = std::env::temp_dir(); - let file_path = temp_dir.join("test_file.txt"); - - // Write - let mut file = std::fs::File::create(&file_path).unwrap(); - file.write_all(b"test content").unwrap(); - - // Read - let content = std::fs::read_to_string(&file_path).unwrap(); - assert_eq!(content, "test content"); - - // Cleanup - std::fs::remove_file(&file_path).unwrap(); -} - -// Testing with databases (using sqlx) -#[sqlx::test] -async fn test_database_operations(pool: sqlx::PgPool) -> sqlx::Result<()> { - sqlx::query("INSERT INTO users (name) VALUES ($1)") - .bind("Alice") - .execute(&pool) - .await?; - - let count: (i64,) = sqlx::query_as("SELECT COUNT(*) FROM users") - .fetch_one(&pool) - .await?; - - assert_eq!(count.0, 1); - Ok(()) -} -``` - -## Snapshot Testing - -```rust -// Using insta crate -use insta::assert_snapshot; - -#[test] -fn test_output_format() { - let data = generate_complex_output(); - assert_snapshot!(data); -} - -#[test] -fn test_json_output() { - let json = serde_json::to_string_pretty(&get_data()).unwrap(); - assert_snapshot!(json); -} - -// Run with: cargo insta test -// Review snapshots: cargo insta review -``` - -## Code Coverage - -```rust -// Using tarpaulin -// cargo install cargo-tarpaulin -// cargo tarpaulin --out Html --output-dir coverage - -// Using llvm-cov -// cargo install cargo-llvm-cov -// cargo llvm-cov --html -``` - -## Fuzzing - -```rust -// Using cargo-fuzz -// cargo install cargo-fuzz -// cargo fuzz init - -// fuzz/fuzz_targets/fuzz_target_1.rs -#![no_main] -use libfuzzer_sys::fuzz_target; - -fuzz_target!(|data: &[u8]| { - if let Ok(s) = std::str::from_utf8(data) { - let _ = mylib::parse_input(s); - } -}); - -// Run with: cargo fuzz run fuzz_target_1 -``` - -## Best Practices - -- Write tests alongside production code in #[cfg(test)] modules -- Use integration tests in tests/ directory for end-to-end testing -- Include doctests in documentation for examples that must work -- Use descriptive test names that explain what is being tested -- Test edge cases (empty inputs, max values, etc.) -- Use property-based testing for algorithmic code -- Benchmark performance-critical code with criterion -- Run tests in CI with cargo test --all-features -- Use cargo test -- --nocapture to see println! output -- Test error conditions with #[should_panic] or Result -- Mock external dependencies for unit tests -- Use test fixtures for complex setup/teardown -- Run clippy on test code too -- Measure code coverage and aim for high coverage -- Use fuzzing for security-critical parsers -- Test async code with tokio::test -- Use snapshot testing for complex output validation diff --git a/.cursor/skills/rust-engineer/references/traits.md b/.cursor/skills/rust-engineer/references/traits.md deleted file mode 100644 index 19122a91..00000000 --- a/.cursor/skills/rust-engineer/references/traits.md +++ /dev/null @@ -1,416 +0,0 @@ -# Traits, Generics, and Type System - -> Reference for: Rust Engineer -> Load when: Designing traits, generics, associated types, derive macros - -## Basic Trait Definition - -```rust -// Simple trait -trait Drawable { - fn draw(&self); -} - -// Trait with default implementation -trait Describable { - fn describe(&self) -> String { - String::from("No description available") - } -} - -// Implementing traits -struct Circle { - radius: f64, -} - -impl Drawable for Circle { - fn draw(&self) { - println!("Drawing circle with radius {}", self.radius); - } -} - -impl Describable for Circle { - fn describe(&self) -> String { - format!("A circle with radius {}", self.radius) - } -} -``` - -## Associated Types - -```rust -// Associated types vs generic parameters -trait Container { - type Item; - - fn add(&mut self, item: Self::Item); - fn get(&self, index: usize) -> Option<&Self::Item>; -} - -impl Container for Vec { - type Item = i32; - - fn add(&mut self, item: i32) { - self.push(item); - } - - fn get(&self, index: usize) -> Option<&i32> { - self.get(index) - } -} - -// Iterator trait (standard library example) -trait MyIterator { - type Item; - - fn next(&mut self) -> Option; -} -``` - -## Generic Traits and Bounds - -```rust -// Generic trait with multiple bounds -fn print_info(item: &T) -where - T: std::fmt::Display + std::fmt::Debug, -{ - println!("Display: {}", item); - println!("Debug: {:?}", item); -} - -// Generic struct with trait bounds -struct Pair { - first: T, - second: T, -} - -impl Pair { - fn new(first: T, second: T) -> Self { - Self { first, second } - } - - fn larger(&self) -> &T { - if self.first > self.second { - &self.first - } else { - &self.second - } - } -} - -// Blanket implementation -trait MyTrait { - fn do_something(&self); -} - -impl MyTrait for T { - fn do_something(&self) { - println!("Value: {}", self); - } -} -``` - -## Trait Objects (Dynamic Dispatch) - -```rust -// Static dispatch (monomorphization) -fn static_dispatch(item: &T) { - item.draw(); -} - -// Dynamic dispatch (trait objects) -fn dynamic_dispatch(item: &dyn Drawable) { - item.draw(); -} - -// Storing trait objects -struct Canvas { - shapes: Vec>, -} - -impl Canvas { - fn new() -> Self { - Self { shapes: Vec::new() } - } - - fn add_shape(&mut self, shape: Box) { - self.shapes.push(shape); - } - - fn draw_all(&self) { - for shape in &self.shapes { - shape.draw(); - } - } -} - -// Object safety: traits must meet criteria -trait ObjectSafe { - fn method(&self); // OK: takes &self -} - -trait NotObjectSafe { - fn generic(&self); // NOT OK: generic method - fn by_value(self); // NOT OK: takes self by value -} -``` - -## Derive Macros - -```rust -// Standard derive macros -#[derive(Debug, Clone, PartialEq, Eq, Hash)] -struct User { - id: u64, - name: String, -} - -// Deriving more traits -#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] -struct Point { - x: i32, - y: i32, -} - -// Custom derive with serde -use serde::{Deserialize, Serialize}; - -#[derive(Debug, Serialize, Deserialize)] -struct Config { - host: String, - port: u16, -} -``` - -## Advanced Trait Patterns - -```rust -// Extension trait pattern -trait StringExt { - fn truncate_to(&self, max_len: usize) -> String; -} - -impl StringExt for str { - fn truncate_to(&self, max_len: usize) -> String { - if self.len() <= max_len { - self.to_string() - } else { - format!("{}...", &self[..max_len]) - } - } -} - -// Sealed trait pattern (prevent external implementation) -mod sealed { - pub trait Sealed {} -} - -pub trait MySealed: sealed::Sealed { - fn method(&self); -} - -struct MyType; -impl sealed::Sealed for MyType {} -impl MySealed for MyType { - fn method(&self) { - println!("Implemented"); - } -} - -// Supertraits -trait Printable { - fn print(&self); -} - -trait Loggable: Printable { // Supertrait: must also impl Printable - fn log(&self) { - self.print(); // Can call supertrait methods - } -} -``` - -## Associated Constants - -```rust -trait Config { - const MAX_SIZE: usize; - const DEFAULT_TIMEOUT: u64; -} - -struct ServerConfig; - -impl Config for ServerConfig { - const MAX_SIZE: usize = 1024; - const DEFAULT_TIMEOUT: u64 = 30; -} - -fn use_config() { - println!("Max size: {}", T::MAX_SIZE); -} -``` - -## Generic Associated Types (GATs) - -```rust -// GATs allow generics in associated types -trait LendingIterator { - type Item<'a> where Self: 'a; - - fn next<'a>(&'a mut self) -> Option>; -} - -struct WindowsMut<'data, T> { - data: &'data mut [T], - index: usize, -} - -impl<'data, T> LendingIterator for WindowsMut<'data, T> { - type Item<'a> = &'a mut [T] where Self: 'a; - - fn next<'a>(&'a mut self) -> Option> { - if self.index >= self.data.len() { - return None; - } - - let start = self.index; - self.index += 2; - - Some(&mut self.data[start..start.min(self.data.len())]) - } -} -``` - -## Marker Traits - -```rust -use std::marker::{PhantomData, Send, Sync}; - -// Send: type can be transferred across thread boundaries -// Sync: type can be shared between threads (&T is Send) - -// Custom marker trait -trait Trusted {} - -struct TrustedData { - data: T, - _marker: PhantomData, -} - -impl TrustedData { - fn new(data: T) -> Self { - Self { - data, - _marker: PhantomData, - } - } -} -``` - -## Operator Overloading - -```rust -use std::ops::{Add, Mul}; - -#[derive(Debug, Clone, Copy)] -struct Vector2D { - x: f64, - y: f64, -} - -impl Add for Vector2D { - type Output = Self; - - fn add(self, other: Self) -> Self { - Self { - x: self.x + other.x, - y: self.y + other.y, - } - } -} - -impl Mul for Vector2D { - type Output = Self; - - fn mul(self, scalar: f64) -> Self { - Self { - x: self.x * scalar, - y: self.y * scalar, - } - } -} - -// Usage -let v1 = Vector2D { x: 1.0, y: 2.0 }; -let v2 = Vector2D { x: 3.0, y: 4.0 }; -let v3 = v1 + v2; -let v4 = v1 * 2.5; -``` - -## From/Into Conversion Traits - -```rust -struct UserId(u64); - -impl From for UserId { - fn from(id: u64) -> Self { - UserId(id) - } -} - -// Into is automatically implemented -fn accept_user_id(id: impl Into) { - let user_id = id.into(); - println!("User ID: {}", user_id.0); -} - -// TryFrom for fallible conversions -use std::convert::TryFrom; - -impl TryFrom for UserId { - type Error = &'static str; - - fn try_from(value: i64) -> Result { - if value < 0 { - Err("User ID cannot be negative") - } else { - Ok(UserId(value as u64)) - } - } -} -``` - -## Const Traits (Nightly) - -```rust -// Const trait implementations (requires nightly) -#![feature(const_trait_impl)] - -#[const_trait] -trait ConstAdd { - fn add(self, other: Self) -> Self; -} - -impl const ConstAdd for i32 { - fn add(self, other: Self) -> Self { - self + other - } -} - -const fn compute() -> i32 { - 5.add(10) // Can use in const context -} -``` - -## Best Practices - -- Prefer associated types when there's one clear type per implementation -- Use generic parameters when multiple types might be used simultaneously -- Keep traits small and focused (single responsibility) -- Use extension traits to add functionality to existing types -- Document trait requirements and invariants -- Use marker traits for compile-time guarantees -- Prefer static dispatch for performance, dynamic dispatch for flexibility -- Use #[derive] when possible instead of manual implementations -- Implement standard traits (Debug, Clone, etc.) for better ecosystem integration -- Use sealed traits to prevent external implementations when needed diff --git a/.cursor/skills/rust-pyo3/SKILL.md b/.cursor/skills/rust-pyo3/SKILL.md new file mode 100644 index 00000000..a77ead28 --- /dev/null +++ b/.cursor/skills/rust-pyo3/SKILL.md @@ -0,0 +1,12 @@ +--- +author: dotagents +name: rust-pyo3 +description: Applies dotagent conventions for Rust PyO3 and Maturin extension crates on top of the base Rust stack. +--- + +When this skill applies: + +- Keep the Python surface small and stable; align naming between `pyproject.toml`, `Cargo.toml`, and the published module. +- Use Maturin as the build entry point unless the repository already standardizes on a different documented flow. +- Be explicit about GIL, `Send`, and exception mapping across the FFI boundary. +- Prefer thin Python modules that re-export a narrow Rust API over exposing many low-level handles. diff --git a/.cursor/skills/rust-tui/SKILL.md b/.cursor/skills/rust-tui/SKILL.md new file mode 100644 index 00000000..ce5b1e15 --- /dev/null +++ b/.cursor/skills/rust-tui/SKILL.md @@ -0,0 +1,12 @@ +--- +author: dotagents +name: rust-tui +description: Applies dotagent conventions for Rust terminal user interfaces on top of the base Rust stack. +--- + +When this skill applies: + +- Follow the TUI stack already present in the repository; avoid introducing a second framework without a migration plan. +- Separate rendering from state: keep model updates independent of widget internals where the codebase already does. +- Treat resize, focus changes, and partial redraws as normal; avoid assuming a fixed terminal size. +- Prefer keyboard-first flows; document mouse or alternative input when the project exposes them. diff --git a/.cursor/skills/rust/SKILL.md b/.cursor/skills/rust/SKILL.md new file mode 100644 index 00000000..27f5cecf --- /dev/null +++ b/.cursor/skills/rust/SKILL.md @@ -0,0 +1,12 @@ +--- +author: dotagents +name: rust +description: Applies the dotagent Rust stack defaults when editing Rust crates and workspaces. +--- + +When this skill applies: + +- Use cargo for builds, tests, and dependency changes; add crates with `cargo add` when introducing new dependencies. +- Respect the repository edition, MSRV, and any clippy or rustfmt configuration already checked in. +- Prefer explicit `Result` handling at API boundaries; keep `unwrap` out of library code unless the project documents an exception. +- Align module layout and visibility with existing crates in the workspace before introducing new patterns. diff --git a/.cursor/skills/xray-pro/ALS_11012_manual_Jun2021.pdf b/.cursor/skills/xray-pro/ALS_11012_manual_Jun2021.pdf deleted file mode 100644 index 12590e35..00000000 Binary files a/.cursor/skills/xray-pro/ALS_11012_manual_Jun2021.pdf and /dev/null differ diff --git a/.cursor/skills/xray-pro/SKILL.md b/.cursor/skills/xray-pro/SKILL.md deleted file mode 100644 index ff3468a4..00000000 --- a/.cursor/skills/xray-pro/SKILL.md +++ /dev/null @@ -1,133 +0,0 @@ ---- -name: xray-pro -description: Use when planning, implementing and running API connections and scripts associated with X-ray absorption fine structure spectroscopy (NEXAFS/XAS). Invoke for computational, numerical, and experimental planning, beamline control, sample alignment, and data acquisition. -triggers: - - nexafs - - xas - - spectroscopy - - beamline - - alignment - - motor - - scan - - plan - - run - - reflectivity - - rsoxs - - als - - bcs - - api -role: expert -scope: implementation -output-format: code ---- - -# Xray Pro - -Expert experimentalist specializing in the efficient collection and analysis of NEXAFS, Reflectivity, Scattering and Diffraction experimental data at synchrotron beamlines, with deep expertise in beamline control systems, sample alignment, and experimental planning. - -## Role Definition - -You are a senior experimental scientist with deep expertise in the ALS 11.0.1.2 RSoXS beamline control system, motor operations, coordinate systems, and the running, scheduling, and planning of X-ray spectroscopy, and scattering experiments. You understand beamline geometry, detector positioning, sample alignment algorithms, and best practices for NEXAFS and reflectivity measurements. - -## When to Use This Skill - -- Writing experimental scans and data acquisition routines -- Planning system architecture and design for running NEXAFS experiments -- Building pre and post processing scripts for spectroscopy data -- Implementing sample alignment and beamline setup procedures -- Interacting with BCS API for motor control and data acquisition -- Understanding beamline coordinate systems and motor naming conventions -- Planning reflectivity and scattering experiments - -## Core Workflow - -1. **Instrument Alignment**: Align detectors (photodiode and beamstop) to the direct beam using CCD Theta optimization -2. **Sample Alignment**: Iteratively align sample Z and Sample Theta positions to the incident beam using half-maximum finding and peak centroid calculations -3. **Scan Planning**: Design energy scans, angle scans, or multi-dimensional scans based on experimental requirements -4. **Data Acquisition**: Execute scans with proper normalization, exposure times, and data collection -5. **Data Processing**: Normalize spectra, handle edge jumps, and analyze NEXAFS features - -## Reference Guide - -Load detailed guidance based on context: - -| Topic | Reference | Load When | -|-------|-----------|-----------| -| Beamline Controls & Motors | `references/beamline-controls.md` | Figuring out motor names, AI channels, DIO channels, coordinate systems, or motor operations | -| Sample Alignment Algorithms | `references/lineup.md` | At the start of beamtime experiments, when planning alignment code, or implementing sample positioning routines | -| BCS API Documentation | `references/bcsz.md` | Interacting with beamline control systems via Python, using asyncio/ZMQ, or requiring hardware status and command definitions | -| Coordinate System Diagram | `references/coords.png` | Visual reference for sample-detector geometry and coordinate system relationships | - -### Reference Details - -#### `references/beamline-controls.md` -Complete reference for all beamline control components: -- **84 Motors**: Complete motor reference organized by category (Sample, Detector, Energy, EPU, Mirrors, Slits, etc.) -- **21 AI Channels**: All analog input channels (0-20) with descriptions and typical units -- **13 DIO Channels**: Digital input/output channels for shutter control, triggers, and status signals -- **Coordinate Systems**: Detailed explanation of Sample (X, Y, Z, Theta) and Detector (CCD Theta, X, Y) coordinate systems -- **Beamline Layout**: Upstream components, optical path, and in-chamber geometry -- **API Commands**: Complete BCS API command reference organized by subsystem - -#### `references/lineup.md` -Step-by-step alignment algorithms with pseudo code and Python implementations: -- **Instrument Alignment**: Finding optimal CCD Theta positions for photodiode and beamstop detectors -- **Sample Alignment**: Iterative algorithm for Sample Z (half-maximum finding) and Sample Theta (peak centroid) alignment -- **Fine-Grained Alignment**: High-precision alignment using slitted beamstop photodiode -- **15 Algorithm Components**: Broken down into reusable functions with descriptions, pseudo code, and Python implementations -- **Checkpointing**: All functions include checkpointing for important statistics and debugging - -#### `references/bcsz.md` -BCS API client library documentation: -- **Connection Setup**: Async/await patterns, ZMQ integration, event loop compatibility -- **API Methods**: Complete method reference for motors, AI channels, DIO, instruments, scans -- **Status Enums**: Motor status, command types, error handling -- **Best Practices**: Race condition prevention, proper async usage, error recovery - -#### `references/coords.png` -Visual diagram showing: -- Sample-detector geometry -- Coordinate system relationships -- Sample Theta and CCD Theta definitions -- Beam path and detector positioning - -## Constraints - -### MUST DO -- Always perform instrument alignment before sample alignment -- Use checkpointing to save important statistics at each alignment step -- Verify motor names against the complete motor reference before use -- Check convergence criteria in iterative alignment procedures -- Use proper 2θ geometry when calculating detector positions for reflection measurements -- Normalize intensity measurements using photodiode or Izero readings -- Account for beam drift and motor drift throughout experiments -- Use appropriate AI channels (Photodiode vs AI 6 BeamStop) based on precision requirements - -### MUST NOT DO -- Skip instrument alignment before sample alignment -- Use motor names without verifying they exist in the motor reference -- Ignore convergence criteria in iterative algorithms -- Mix coordinate systems without proper transformation -- Use area detector (CCD) when photodiode should be used for alignment -- Assume motor positions without checking current state -- Hardcode motor positions that may drift over time - -## Output Templates - -When implementing beamline control solutions, provide: -1. Code with proper async/await patterns for BCS API calls -2. Checkpointing at critical steps for debugging and analysis -3. Error handling for motor movements and data acquisition -4. Convergence checking in iterative algorithms -5. Comments explaining coordinate system transformations and geometry -6. Validation of motor names and AI/DIO channel names before use - -## Knowledge Reference - -ALS 11.0.1.2 RSoXS beamline, BCS API, asyncio, ZMQ, motor control, sample alignment algorithms, NEXAFS spectroscopy, X-ray reflectivity, coordinate systems, detector positioning, beamline geometry, EPU polarization control, monochromator operation, scatter slits, higher order suppressor, photodiode measurements, centroid calculations, half-maximum finding, convergence algorithms, checkpointing strategies - -## Related Skills - -- **Python Pro** - Async/await patterns, type hints, error handling -- **Pandas Pro** - Data processing and analysis of experimental data -- **Data Scientist** - Statistical analysis of spectroscopy results diff --git a/.cursor/skills/xray-pro/references/bcsz.md b/.cursor/skills/xray-pro/references/bcsz.md deleted file mode 100644 index c33b6e52..00000000 --- a/.cursor/skills/xray-pro/references/bcsz.md +++ /dev/null @@ -1,1881 +0,0 @@ - -# BCSz Core API - -> Reference for: Xray Pro -> Load when: Interacting with beamline control systems via Python, using asyncio, zmq, or requiring hardware status and command definitions - ---- - - -## Overview - -Reference for: BCS API client usage, communication protocols, and control/status interfaces for XRAY-PRO hardware - -Documentation covers connection setup, event loop compatibility, ZMQ/asyncio integration, and motor/status enums. - ---- - -```python - -""" -A python client interface to the BCS API, using zmq and asyncio. - -All API calls ultimately call bcs_request with specialized JSON. Calling bcs_request from client -application code is possible, but discouraged, and not supported, as the call signature may change in future versions. - -Contact bcs@lbl.gov with questions / requests. -""" -import sys -import asyncio - -if sys.platform[:3] == 'win': - # zmq.asyncio does not support the default (proactor) event loop on windows. - # so set the event loop to one zmq supports - asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) - -import zmq -import zmq.asyncio -import zmq.utils.z85 - -import json -import time - -# helper classes -from enum import Flag # for MotorStatus - - -class MotorStatus(Flag): - HOME = 1 - FORWARD_LIMIT = 2 - REVERSE_LIMIT = 4 - MOTOR_DIRECTION = 8 - MOTOR_OFF = 16 - MOVE_COMPLETE = 32 - FOLLOWING_ERROR = 64 - NOT_IN_DEAD_BAND = 128 - FORWARD_SW_LIMIT = 256 - REVERSE_SW_LIMIT = 512 - MOTOR_DISABLED = 1024 - RAW_MOTOR_DIRECTION = 2048 - RAW_FORWARD_LIMIT = 4096 - RAW_REVERSE_LIMIT = 8192 - RAW_FORWARD_SW_LIMIT = 16384 - RAW_REVERSE_SW_LIMIT = 32768 - RAW_MOVE_COMPLETE = 65536 - MOVE_LT_THRESHOLD = 131072 - - def is_set(self, flag): - return bool(self._value_ & flag._value_) - - -# helper functions -def bytes_from_blob(blob): - """Deserializes binary data blobs. This implementation uses zmq's Base85 encoding, but future ones may not.""" - blob_len = blob['length'] - blob_str = blob['blob'] - return zmq.utils.z85.decode(blob_str)[:blob_len] - - -_zmq_context = None - - -class BCSServer: - """Represents a remote BCS endstation or beamline system, running the BCS zmq server. - - Each endpoint returns a dictionary of results. In addition to endpoint-specific keys, the following - global keys are included. - - * **success** (*bool*) - API endpoint completion status. Note that this does *not* indicate endpoint errors. - * **error_description** (*str*) - If the server failed to execute the endpoint, this should indicate why. There may be warnings or information here even if the endpoint completed successfully. - * **log** (*bool*) - True if the server logged this request. - - """ - _zmq_socket = None - - @staticmethod - async def _get_server_public_key(addr, port): - clear_socket = _zmq_context.socket(zmq.REQ) - clear_socket.connect(f'tcp://{addr}:{port}') - await clear_socket.send('public'.encode()) - server_public = await clear_socket.recv() - clear_socket.close() - return server_public - - # def __init__(self, addr='127.0.0.1', port=5577): - async def connect(self, addr='127.0.0.1', port=5577): - - """(formerly the Constructor) Supply the zmq address string, addr, to reach this endstation.""" - - global _zmq_context - - # the first server object will create the global zmq context - if not _zmq_context: - _zmq_context = zmq.asyncio.Context() - - self._zmq_socket = _zmq_context.socket(zmq.REQ) - - (client_public_key, client_secret_key) = zmq.curve_keypair() - - # server_public_key = asyncio.get_running_loop().run_until_complete(self._get_server_public_key(addr, port)) - - server_public_key = await self._get_server_public_key(addr, port) - - print(f'Server Public Key {server_public_key}') - - self._zmq_socket.setsockopt(zmq.CURVE_SERVERKEY, server_public_key) - self._zmq_socket.setsockopt(zmq.CURVE_PUBLICKEY, client_public_key) - self._zmq_socket.setsockopt(zmq.CURVE_SECRETKEY, client_secret_key) - - self._zmq_socket.connect(f'tcp://{addr}:{port + 1}') - - async def bcs_request(self, command_name, param_dict, debugging=False): - """ - The method responsible for direct communication to the BCS server - - :param command_name: Name of the API endpoint - :type command_name: str - :param param_dict: Parameter dictionary - :type param_dict: dict - """ - if debugging: - print(f"API command {command_name} BEGIN.") - - api_call_start = time.time() - param_dict['command'] = command_name - param_dict['_unused'] = '_unused' - if 'self' in param_dict: - del param_dict['self'] - await self._zmq_socket.send(json.dumps(param_dict).encode()) - response_dict = json.loads(await self._zmq_socket.recv()) - response_dict['API_delta_t'] = time.time() - api_call_start - - if debugging: - print(f"API command {command_name} END {response_dict['API_delta_t']} s.") - - return response_dict - -# end BCSz_header.py - async def acquire_data(self, chans=[], time=0, counts=0) -> dict: - """ - Acquires for ``time`` seconds, or ``counts`` counts. whichever is non-zero. **Waits for the acquision to complete** and returns data for channels specified in ``chans``. - If both ``counts`` *and* ``time`` are non-zero, which parameter takes precedence is not defined. - - :param chans: AI channel names to acquire. An empty array will return data for all AI channels on the server. - :type chans: list - :param time: If non-zero, the amount of time to acquire. - :type time: float - :param counts: If non-zero, the number of counts to acquire. - :type counts: int - - :return: Dictionary of results, with the following key(s). - - * **chans** (*list*) - The channel names that have data in the **data** array, in the same order. - - * **not_found** (*list*) - The requested channels in ``chans`` that were not found on the host system, if any. - - * **data** (*list*) - The acquired data, in the same order as the channel names in **chans**. - - - """ - return await self.bcs_request('AcquireData', dict(locals())) - - async def at_preset(self, name="_none") -> dict: - """ - Checks if the associated motor is at the preset position. The associated motor and preset position are defined on the server. - - :param name: Name of the preset (not the motor name). - :type name: str - - :return: Dictionary of results, with the following key(s). - - * **at_preset** (*bool*) - True if the target motor is at the preset position. - - * **position** (*float*) - The preset position, in units of the target motor. - - - """ - return await self.bcs_request('AtPreset', dict(locals())) - - async def at_trajectory(self, name="success") -> dict: - """ - Checks if all requirement been satisfied to be "At trajectory" (usually just means that each motor is at its trajectory goal). - - :param name: Trajectory name (defined on the server). - :type name: str - - :return: Dictionary of results, with the following key(s). - - * **at_trajectory** (*bool*) - Are the motors "At trajectory?" - - * **running** (*bool*) - Is the trajectory still running (are the motors still moving)? - - - """ - return await self.bcs_request('AtTrajectory', dict(locals())) - - async def command_motor(self, commands=[], motors=[], goals=[]) -> dict: - """ - Command one or more motors. A large number of commands are available. - - :param commands: Array of motor commands, one for each motor in ``motors``. Current valid commands are {None, Normal Move, Backlash Move, Velocity Move, Move to Home, Stop Motor, Set Position, Enable Motor, Disable Motor, Move to Index, Run Home Routine, Set Velocity, Set Acceleration, Set Deceleration, Enable and Move, Disable SW Limits, Enable SW Limits, Start Time Delay, Check Time Delay, Set Output Pulses, Backlash Jog, Normal Jog, Run Coord Program, Halt Coord Program, Gearing ON, Gearing OFF, Set Forward SW Limit, Set Reverse SW Limit, Revert Forward SW Limit, Revert Reverse SW Limit}. - :type commands: list - :param motors: Array of motor names - :type motors: list - :param goals: Goal (or associated value for the command) for each motor provided in ``motors`` - :type goals: list - - :return: Dictionary of results, with the following key(s). - - * **timed_out** (*list*) - List of motors that timed-out, if any. - - * **not_found** (*list*) - List of motors that were not found, if any. - - - """ - return await self.bcs_request('CommandMotor', dict(locals())) - - async def current_scan_running(self) -> dict: - """ - Returns the currently running 'integrated scan' run, or an empty string if none is running. - - - - :return: Dictionary of results, with the following key(s). - - * **running_scan** (*str*) - The name of the currently running scan, or an empty string if none. - - - """ - return await self.bcs_request('CurrentScanRunning', dict(locals())) - - async def disable_breakpoints(self, name="") -> dict: - """ - Disables the breakpoint output (output-on-position) of motor controller for the named motor. Or, 'takes the motor out of flying mode'. - - :param name: Motor name - :type name: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('DisableBreakpoints', dict(locals())) - - async def disable_motor(self, name="success") -> dict: - """ - Disables the named motor. - - :param name: Motor name. - :type name: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('DisableMotor', dict(locals())) - - async def enable_motor(self, name="success") -> dict: - """ - Enables the named motor. - - :param name: Motor name. - :type name: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('EnableMotor', dict(locals())) - - async def get_acquired(self, chans=[]) -> dict: - """ - Retrieve the average value from the most recent *single-shot* acquisition (see StartAcquire). - - :param chans: AI channel name(s). An empty array returns data for all AI channels on the server. - :type chans: list - - :return: Dictionary of results, with the following key(s). - - * **chans** (*list*) - AI channel names that have data in the **data** array. - - * **not_found** (*list*) - The requested channels in ``chans`` that were not found on the host system, if any. - - * **data** (*list*) - The acquired data, in the same order as the channel names in **chans**. - - - """ - return await self.bcs_request('GetAcquired', dict(locals())) - - async def get_acquired_array(self, chans=[]) -> dict: - """ - Retrieve the array acquired from the most recent *single-shot* acquisition (see StartAcquire). - - :param chans: AI channel name(s). An empty array returns data for all AI channels on the server. - :type chans: list - - :return: Dictionary of results, with the following key(s). - - * **not_found** (*list*) - The requested channels in ``chans`` that were not found on the server, if any. - - * **chans** (*list*) - Array of clusters/dictionaries. Each array element contains the channel (**chan**), **period**, **time**, and **data**. Data is an array of sampled values. - - - """ - return await self.bcs_request('GetAcquiredArray', dict(locals())) - - async def get_acquire_status(self) -> dict: - """ - Read the current acquisition state of the AI subsystem - - - - :return: Dictionary of results, with the following key(s). - - * **acquiring** (*bool*) - True if an acquisition is in progress - - * **started** (*bool*) - True if an acquisition has been started - - * **reading** (*bool*) - True if acquisition data is currently being retrieved from an acquisition device - - - """ - return await self.bcs_request('GetAcquireStatus', dict(locals())) - - async def get_di(self, chans=[]) -> dict: - """ - Get digital input (DI) channel value - - :param chans: DI channel name(s). An empty array returns data for all channels on the server. - :type chans: list - - :return: Dictionary of results, with the following key(s). - - * **chans** (*list*) - DI channel names that have data in the **data** array. - - * **not_found** (*list*) - The requested channels in ``chans`` that were not found on the host system, if any. - - * **enabled** (*list*) - Boolean list, in the same order as **chans**, indicating if the channel is used (enabled). - - * **data** (*list*) - Boolean value of the DI's, in the same order as the channel names in **chans**. - - - """ - return await self.bcs_request('GetDI', dict(locals())) - - async def get_flying_positions(self, name="") -> dict: - """ - Retrieve the locations of motor **name** that will trigger acquisitions in a flying scan. - - :param name: Motor name - :type name: str - - :return: Dictionary of results, with the following key(s). - - * **pulse_positions** (*list*) - Trigger (pulse) positions. - - - """ - return await self.bcs_request('GetFlyingPositions', dict(locals())) - - async def get_folder_listing(self, path=".", pattern="*.txt", recurse=False) -> dict: - """ - Get lists of all files and folders in a location descended from "C:\\\\Beamline Controls\\\\BCS Setup Data" - - :param path: Path to text file, relative to C:\\\\Beamline Controls\\\\BCS Setup Data. Use backslashes (\\\\) not forward slashes (/). - :type path: str - :param pattern: Pattern for files for which you want to search. - :type pattern: str - :param recurse: If True, list all descendent files and folders that match pattern. - :type recurse: bool - - :return: Dictionary of results, with the following key(s). - - * **files** (*list*) - Array of paths, relative to C:\\\\Beamline Controls\\\\BCS Setup Data. - - * **folders** (*list*) - Array of paths, relative to C:\\\\Beamline Controls\\\\BCS Setup Data. - - - """ - return await self.bcs_request('GetFolderListing', dict(locals())) - - async def get_freerun(self, chans=[]) -> dict: - """ - Get freerun AI data for one or more channels - - :param chans: Array of channel names to get. An empty array retrieves all channels' data. - :type chans: list - - :return: Dictionary of results, with the following key(s). - - * **chans** (*list*) - The retrieved channel names, in the same order as **data**. - - * **not_found** (*list*) - The channels requested in ``chans`` that were not found on the server. - - * **data** (*list*) - The retrieved data, corresponding to the channels in **chans**. - - - """ - return await self.bcs_request('GetFreerun', dict(locals())) - - async def get_freerun_array(self, chans=[]) -> dict: - """ - Retrieve most recent AI freerun data. - - :param chans: AI channel name(s). An empty array returns data for all AI channels on the server. - :type chans: list - - :return: Dictionary of results, with the following key(s). - - * **not_found** (*list*) - The requested channels in ``chans`` that were not found on the server, if any. - - * **chans** (*list*) - Array of clusters/dictionaries. Each array element contains the channel (**chan**), **data**, and **x_values**. - - - """ - return await self.bcs_request('GetFreerunArray', dict(locals())) - - async def get_instrument_acquired1d(self, name="") -> dict: - """ - Retrieve data (1D array) from the most recent acquisition. - - :param name: Instrument name - :type name: str - - :return: Dictionary of results, with the following key(s). - - * **x** (*list*) - 1D array of abscissae - - * **y** (*list*) - 1D array of data - - - """ - return await self.bcs_request('GetInstrumentAcquired1D', dict(locals())) - - async def get_instrument_acquired2d(self, name="") -> dict: - """ - Retrieve data (2D array) from the most recent acquisition. - - :param name: Instrument name - :type name: str - - :return: Dictionary of results, with the following key(s). - - * **data** (*list*) - 2D array of u32. - - - """ - return await self.bcs_request('GetInstrumentAcquired2D', dict(locals())) - - async def get_instrument_acquired3d(self, name="") -> dict: - """ - Retrieve data (3D array) from the most recent acquisition. - - :param name: Instrument name - :type name: str - - :return: Dictionary of results, with the following key(s). - - * **data** (*list*) - 3D array of u32. - - - """ - return await self.bcs_request('GetInstrumentAcquired3D', dict(locals())) - - async def get_instrument_acquisition_info(self, name="") -> dict: - """ - Retrieve miscellanaeous info about the instrument and acquisition: file_name, sensor temperature (if applicable), live time, and dead time. - - :param name: Instrument name - :type name: str - - :return: Dictionary of results, with the following key(s). - - * **file_name** (*str*) - Fully qualified path name to latest data file. - - * **temperature** (*float*) - The temperature of an internal element, such as a CCD, if the instrument supports it. - - * **live_time** (*list*) - Live time for the acquistion. - - * **dead_time** (*list*) - Dead time for the acquisition. - - - """ - return await self.bcs_request('GetInstrumentAcquisitionInfo', dict(locals())) - - async def get_instrument_acquisition_status(self, name="") -> dict: - """ - Retrieve all available status bits from the instrument subsystem for the named instrument. - - :param name: Instrument name - :type name: str - - :return: Dictionary of results, with the following key(s). - - * **scan_params_set_up** (*bool*) - Are the scan parameters set up? - - * **acquiring** (*bool*) - Is the instrument acquiring data? - - * **single_shot** (*bool*) - Unclear at the moment. TBD - - * **aborted** (*bool*) - Is the acquisition aborted? - - * **data_available** (*bool*) - Is data available? - - * **ins_single_shot** (*bool*) - 'Instrument Single Shot': Also unclear. TBD. - - - """ - return await self.bcs_request('GetInstrumentAcquisitionStatus', dict(locals())) - - async def get_instrument_count_rates(self, name="") -> dict: - """ - Retreive instrument count rates - - :param name: Instrument name - :type name: str - - :return: Dictionary of results, with the following key(s). - - * **input_cr** (*list*) - Input count rate - - * **output_cr** (*list*) - Output count rate - - - """ - return await self.bcs_request('GetInstrumentCountRates', dict(locals())) - - async def get_instrument_driver_status(self, name="") -> dict: - """ - Returns the status of the BCS Instrument Driver for the named instrument. - - :param name: Instrument name - :type name: str - - :return: Dictionary of results, with the following key(s). - - * **running** (*bool*) - True if the driver is running. - - - """ - return await self.bcs_request('GetInstrumentDriverStatus', dict(locals())) - - async def get_motor(self, motors=[]) -> dict: - """ - Get information and status for the motors in ``motors`` - - :param motors: Array of motor names to retrieve. - :type motors: list - - :return: Dictionary of results, with the following key(s). - - * **not_found** (*list*) - The requested motors in ``motors`` that were not found on the host system, if any. - - * **data** (*list*) - Array of clusters/dictionaries. Each array element contains the motor name (**motor**), **position**, **position_raw**, **goal**, **goal_raw**, **status**, and **time**. - - **status** is a bit-field. TODO: helper function to test status - - - """ - return await self.bcs_request('GetMotor', dict(locals())) - - async def get_motor_full(self, motors=[]) -> dict: - """ - Returns the complete state of the requested motors. - - :param motors: Names of motors to query. - :type motors: list - - :return: Dictionary of results, with the following key(s). - - * **not_found** (*list*) - Names in **motors** that were not found, if any. - - * **data** (*list*) - dictionary of dictionaries all relateing to state. Too many parameters to list atm. - - - """ - return await self.bcs_request('GetMotorFull', dict(locals())) - - async def get_panel_image(self, name="", quality=0) -> dict: - """ - Send a current image of the LabVIEW panel, in jpg format. - - :param name: Full path to panel - :type name: str - :param quality: JPEG quality 0-100 - :type quality: int - - :return: Dictionary of results, with the following key(s). - - * **image_blob** (*dict*) - Ideally opaque blob for holding binary data. Helper functions should be available to clients to extract blob contents. Contains **length** and **blob**. - - - """ - return await self.bcs_request('GetPanelImage', dict(locals())) - - async def get_state_variable(self, name="") -> dict: - """ - Get the value of the named BCS State Variable. - - :param name: Name of the state variable to retrieve - :type name: str - - :return: Dictionary of results, with the following key(s). - - * **value** (*void*) - Value of the state variable - - * **found** (*bool*) - True if requested name was found on the server - - * **type** (*enum u16*) - 'String', 'Boolean', 'Integer', or 'Double' - - * **name** (*str*) - Same as ``name`` - - - """ - return await self.bcs_request('GetStateVariable', dict(locals())) - - async def get_subsystem_status(self) -> dict: - """ - Returns the current status of all BCS subsystems on the server - - - - :return: Dictionary of results, with the following key(s). - - * **status** (*list*) - Array of dictionaryies of keys: **name** - Subsystem Name, and **status** - one of {Initializing, Running, Stopping, Stopped, Force Stop, Disabled, Bad Path, VI Broken Not in List} - - - """ - return await self.bcs_request('GetSubsystemStatus', dict(locals())) - - async def get_text_file(self, path="You must have at least one input control in the Parameter cluster. Delete this (_unused) if you don't need it.") -> dict: - """ - Get contents of a text file (usually acquired data) from any file in a location descended from "C:\\\\Beamline Controls\\\\BCS Setup Data". - - :param path: Path to text file, relative to C:\\\\Beamline Controls\\\\BCS Setup Data. Use backslashes (\\\\) not forward slashes (/). - :type path: str - - :return: Dictionary of results, with the following key(s). - - * **text** (*str*) - Text of the file. Platform-dependent end-of-line characters have been converted to line feed characters - - - """ - return await self.bcs_request('GetTextFile', dict(locals())) - - async def get_video_image(self, name="", quality=0, type="default") -> dict: - """ - Return the most recent image from the named camera, in jpg format. - - :param name: Video (camera) name - :type name: str - :param quality: JPEG quality 0-100 - :type quality: int - :param type: One of [default, roi, or threshold] - :type type: enum u16 - - :return: Dictionary of results, with the following key(s). - - * **image_blob** (*dict*) - Ideally opaque blob for holding binary data. Helper functions should be available to clients to extract blob contents. Contains **length** and **blob**. - - - """ - return await self.bcs_request('GetVideoImage', dict(locals())) - - async def home_motor(self, motors=[]) -> dict: - """ - Home the motors in the input array, ``motors``. - - :param motors: Names of the motors to home. - :type motors: list - - :return: Dictionary of results, with the following key(s). - - * **timed_out** (*list*) - True if the home operation timed out. - - * **not_found** (*list*) - The motors in ``motors`` that are not found on the server, if any. - - - """ - return await self.bcs_request('HomeMotor', dict(locals())) - - async def last_scan_run(self) -> dict: - """ - Returns the last 'integrated scan' run, or an empty string if none have run. - - - - :return: Dictionary of results, with the following key(s). - - * **last_scan** (*str*) - The name of the last run scan, or an empty string if none. - - - """ - return await self.bcs_request('LastScanRun', dict(locals())) - - async def list_ais(self) -> dict: - """ - Return the list of all AI channels that are defined on the server. - - - - :return: Dictionary of results, with the following key(s). - - * **names** (*list*) - Llist of AI channels that are defined on the server. The list includes disabled and hidden channels. - - * **displayed** (*list*) - The list of AI channel that are displayed (not set to 'hidden' in BCS). - - * **disabled** (*list*) - Iindexes in **displayed** that are disabled. - - - """ - return await self.bcs_request('ListAIs', dict(locals())) - - async def list_dios(self) -> dict: - """ - Retrieves the complete list of digital input channel names defined on the server. - - - - :return: Dictionary of results, with the following key(s). - - * **names** (*list*) - Liist of all digital input channel names on the server. The list includes disabled and hidden channels. - - * **displayed** (*list*) - The list of channels that are displayed (not set to 'hidden' in BCS). - - * **disabled** (*list*) - Iindexes in **displayed** that are disabled. - - - """ - return await self.bcs_request('ListDIOs', dict(locals())) - - async def list_instruments(self) -> dict: - """ - Return the list of instruments that are defined on the server. - - - - :return: Dictionary of results, with the following key(s). - - * **names** (*list*) - The list of all instruments that are defined on the server, including disabled and hidden instruments. - - * **displayed** (*list*) - The list of instruments that are displayed (not set to 'hidden' in BCS). - - * **disabled** (*list*) - Iindexes in **displayed** that are disabled. - - - """ - return await self.bcs_request('ListInstruments', dict(locals())) - - async def list_motors(self) -> dict: - """ - Return the list of motors that are defined on the server. - - - - :return: Dictionary of results, with the following key(s). - - * **names** (*list*) - The list of motors that are defined on the server. The list includes disabled and hidden motors. - - * **displayed** (*list*) - The list of motors that are displayed (not set to 'hidden' in BCS). - - * **disabled** (*list*) - Iindexes in **displayed** that are disabled. - - - """ - return await self.bcs_request('ListMotors', dict(locals())) - - async def list_presets(self) -> dict: - """ - Return array of motor preset positions. Each array entry is a dictionary describing one preset. The dictionary keys are **Preset Name**, **Motor Name**, **Preset Position**, **Tolerance (+/-)**. - - - - :return: Dictionary of results, with the following key(s). - - * **presets** (*list*) - Array of dictionaries, one per preset, each with keys: **Preset Name**, **Motor Name**, **Preset Position**, **Tolerance (+/-)**. - - - """ - return await self.bcs_request('ListPresets', dict(locals())) - - async def list_state_variables(self) -> dict: - """ - Get the complete list of state variable, and their types (Boolean, Integer, String, Double). - - - - :return: Dictionary of results, with the following key(s). - - * **names** (*list*) - Current list of state variable on the server. - - * **types** (*list*) - Variable types (Boolean, Integer, String, Double), in the same order as **names**. - - - """ - return await self.bcs_request('ListStateVariables', dict(locals())) - - async def list_trajectories(self) -> dict: - """ - List all trajectories. - - - - :return: Dictionary of results, with the following key(s). - - * **trajectories** (*list*) - Array of dictionaries, each element describing a trajectory. - - - """ - return await self.bcs_request('ListTrajectories', dict(locals())) - - async def move_motor(self, motors=[], goals=[]) -> dict: - """ - Command one or more motors to begin moves to supplied goals. - - :param motors: Array of motor names - :type motors: list - :param goals: Goal for each motor listed in ``motors`` - :type goals: list - - :return: Dictionary of results, with the following key(s). - - * **timed_out** (*list*) - List of motors that timed-out, if any. - - * **not_found** (*list*) - List of motors that were not found, if any. - - - """ - return await self.bcs_request('MoveMotor', dict(locals())) - - async def move_to_preset(self, names=[]) -> dict: - """ - Move to preset positions. Takes a list of preset names and executes them (sends their motors to their respective positions). - - :param names: Array of preset names to execute. - :type names: list - - :return: Dictionary of results, with the following key(s). - - * **not_found** (*list*) - List of names in ``names`` that were not found on the server, if any. - - - """ - return await self.bcs_request('MoveToPreset', dict(locals())) - - async def move_to_trajectory(self, names=[]) -> dict: - """ - Move to trajectory positions. Takes a list of trajectory names and executes them (sends their motors to their respective positions). - - :param names: Array of trajectory names to execute. - :type names: list - - :return: Dictionary of results, with the following key(s). - - * **not_found** (*list*) - List of names in ``names`` that were not found on the server, if any. - - - """ - return await self.bcs_request('MoveToTrajectory', dict(locals())) - - async def scan_status(self) -> dict: - """ - Returns information about the 'integrated scan' system, namely the last scan (last_scan) run, the currently running scan, and the scanner status. An empty string indicates no scan. - - - - :return: Dictionary of results, with the following key(s). - - * **last_scan** (*str*) - The name of the last run scan, or an empty string if none. - - * **running_scan** (*str*) - The name of the currently running scan, or an empty string if none. - - * **scanner_status** (*str*) - The current status of the scanner. One of a large set of finite-state-machine states used by the scanner. The set also varies by scan. Empty string if the scanner has stopped running. - - * **log_directory** (*str*) - Location of data file, maybe. - - * **last_filename** (*str*) - Data file name - - * **user_path** (*str*) - Or maybe this is the location of the data file - - - """ - return await self.bcs_request('ScanStatus', dict(locals())) - - async def set_breakpoints(self, name="", x0=0, dx=0, n=0, breakpoints=[], counts_or_units=False) -> dict: - """ - Set Breakpoints for the named motor. Breakpoints are the positions at which the controller will generate acquisition triggers during a flying scan. Specify either {**x0**, **dx**, **n**} to generate a regular grid, or send an arbitrary list in **breakpoints**. - - :param name: Motor - :type name: str - :param x0: Starting position. - :type x0: float - :param dx: Interval spacing. - :type dx: float - :param n: Number of breakpoints (triggers) to generate. - :type n: int - :param breakpoints: An array of arbitrary locations to on which to trigger data acquisition. - :type breakpoints: list - :param counts_or_units: Are the locations in motor units or motor counts (False == units)? - :type counts_or_units: bool - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('SetBreakpoints', dict(locals())) - - async def set_do(self, chan="success", value=False) -> dict: - """ - Sets the digital output (DO) channel ``chan`` to ``value`` - - :param chan: Name of the channel to set. - :type chan: str - :param value: New value - :type value: bool - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('SetDO', dict(locals())) - - async def set_motor_velocity(self, motors=[], velocities=[]) -> dict: - """ - Set motor speeds. This endpoint duplicates the functionality of CommandMotor, and may be removed from future releases. - - :param motors: List of motors - :type motors: list - :param velocities: Speeds to set, in the same order as **motors**. - :type velocities: list - - :return: Dictionary of results, with the following key(s). - - * **timed_out** (*list*) - - - * **not_found** (*list*) - - - - """ - return await self.bcs_request('SetMotorVelocity', dict(locals())) - - async def set_state_variable(self, name="variable name", value=0) -> dict: - """ - Set the value of the named BCS State Variable. - - :param name: Name of the state variable to set - :type name: str - :param value: New value of the state variable. Accepts string, numeric, and boolean values. - :type value: int - - :return: Dictionary of results, with the following key(s). - - * **found** (*bool*) - True if requested name was found on the server - - - """ - return await self.bcs_request('SetStateVariable', dict(locals())) - - async def start_acquire(self, time=0, counts=0) -> dict: - """ - Start Acquisition for either ``time`` or ``counts``, whichever is non-zero. The acquisition is started, but the endpoint does not wait for the acquisition to complete. - - :param time: Acquisition time. - :type time: float - :param counts: Acquisition counts. - :type counts: int - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('StartAcquire', dict(locals())) - - async def start_flying_scan(self, name="", xi=0, xf=0, dx=0, speed=0) -> dict: - """ - Start a flying scan with the named motor. The scan is started. The enpoint does not wait for the scan to finish. - - :param name: Motor name - :type name: str - :param xi: Starting point - :type xi: float - :param xf: Stopping point - :type xf: float - :param dx: Interval between triggers (pulses) - :type dx: float - :param speed: Motor speed - :type speed: float - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('StartFlyingScan', dict(locals())) - - async def start_instrument_acquire(self, name="", run_type="Exposure", acq_time_s=0, acq_counts=0) -> dict: - """ - Starts an instrument acquisition and waits for it to complete. Acquires for either **acq_time** seconds, or for **acq_counts** counts (triggers), whichever is non-zero. If both are non-zero, the result is undefined. - - :param name: Instrument name - :type name: str - :param run_type: One of {'Exposure', 'Total Counts'}. Determines whether to acquire for a set time, or a set number of counts/triggers. - :type run_type: enum u32 - :param acq_time_s: Length of time for the acquisition. - :type acq_time_s: float - :param acq_counts: Number of counts or triggers to acquire for. - :type acq_counts: int - - :return: Dictionary of results, with the following key(s). - - * **elapsed_s** (*float*) - Total duration of the acquisition in seconds. - - - """ - return await self.bcs_request('StartInstrumentAcquire', dict(locals())) - - async def start_instrument_driver(self, name="") -> dict: - """ - Starts the named instrument driver (does nothing if the driver is already running). - - :param name: Instrument name - :type name: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('StartInstrumentDriver', dict(locals())) - - async def stop_acquire(self, timeout_ms=0) -> dict: - """ - Stop acquisition in progress. Waits for the acquisition to terminate within ``timeout_ms`` milleseconds. If the acquisition does not stop in that time, returns False in ``success`` and "timed out" in ``error_description``. - - :param timeout_ms: Time to wait for the acquisition to terminate, in milliseconds. - :type timeout_ms: int - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('StopAcquire', dict(locals())) - - async def stop_instrument_acquire(self, name="") -> dict: - """ - Stop acquisition on the named instrument (does nothing if the instument is not acquiring). Waits up to half a second to verify that acquisition has terminated. The **success** field is False if the abort times out without terminating the acquisition. - - :param name: Instrument name - :type name: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('StopInstrumentAcquire', dict(locals())) - - async def stop_instrument_driver(self, name="") -> dict: - """ - Stops the named instrument driver (does nothing if the driver is not running). - - :param name: Instrument name - :type name: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('StopInstrumentDriver', dict(locals())) - - async def stop_motor(self, motors=[]) -> dict: - """ - Immediately issue stop command for the motors provided in ``motors``. - - :param motors: List of motors to stop. The stop command is immediately sent to the motor controller for the listed motors. - :type motors: list - - :return: Dictionary of results, with the following key(s). - - * **timed_out** (*list*) - Motor subsystem error indicating that the command queue is unavailable. Seek help from the beamline scientist or beamline controls. - - * **not_found** (*list*) - Those motors requested in ``motors`` that were not found on the server, if any. - - - """ - return await self.bcs_request('StopMotor', dict(locals())) - - async def stop_scan(self) -> dict: - """ - Immediately issue stop command forthe currently running 'integrated scan'. - - - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('StopScan', dict(locals())) - - async def test_connection(self, chan="success", value=False) -> dict: - """ - - - :param chan: - :type chan: str - :param value: - :type value: bool - - :return: Dictionary of results, with the following key(s). - - * **response_string** (*str*) - - - - """ - return await self.bcs_request('TestConnection', dict(locals())) - - async def sc_alsu_mirror_vibration(self, time_sec=0, start_delay_sec=0, x_motor="", start_position=0, stop_position=0, duration_sec=0, delay_sec=0, final_position=0, left_sensor="", right_sensor="", seperation_mm=1, scale_factor_pm=0, description="", file_pattern="") -> dict: - """ - Setup the ALSU Mirror Vibration Scan - This scan moves one motor in a specific pattern and records the analog data. Additional calculations are performed on the Interferometer Data to provide more relevant calculations into the file. - The setup that goes with this scan involves a mirror on a table. The mirror has cooling water flowed through it using a Mass Flow Controller. The "motor" in this scan is actually the flow of water through the mirror. The scan then is to flow water thorough the mirror and monitor the interferometer to look at the vibrations of the mirror. - - :param time_sec: Time that the motor moves for. - :type time_sec: float - :param start_delay_sec: Delay initial data acquisition. This will give a chance for the Flow controlled by the X motor to stabilize. - :type start_delay_sec: float - :param x_motor: Choose the motor that will move. - :type x_motor: str - :param start_position: Where the motor starts. This is generally flow, so the flow at which the experiement starts. - :type start_position: float - :param stop_position: Where the motor stops. This if generally flow, so the flow at which the experiment stops. - :type stop_position: float - :param duration_sec: Time that the data is acquired for. - :type duration_sec: float - :param delay_sec: Wait this long after acquisition starts before the motor move. - :type delay_sec: float - :param final_position: Position to set the X motor when finished with scan. This should be zero. That will stop the flow of liquid. - :type final_position: float - :param left_sensor: Left interferomter count channel. - :type left_sensor: str - :param right_sensor: Right Interferometer count channel. - :type right_sensor: str - :param seperation_mm: Distance seperating the two interferometers. - :type seperation_mm: float - :param scale_factor_pm: Scales the readings into pico meteres. - :type scale_factor_pm: float - :param description: - :type description: str - :param file_pattern: - :type file_pattern: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('sc_ALSU Mirror Vibration', dict(locals())) - - async def sc_auto_coll_single_axis_scan(self, direction="Forward", index=0, x_motor="Select Motor", start=-3, stop=0, increment=1, delay_after_move_s=0, count_time_s=0.5, number_of_scans=1, bidirect=True, at_end_of_scan="Return", description="", file_pattern="") -> dict: - """ - This is VERY close to a Single Motor Scan. The only difference is how the files are saved. It has been changed so that a TCP command can specify the scans that are run with more control than just the regular Single motor scan. This was created for the Metrology lab. - - :param direction: Name the direction of the scan. This does NOT swap the start and end. It is only used to lable the file created. - :type direction: enum u32 - :param index: Used to save this scan as part of a group of scans. A new folder will be created for every 0 index scan started. Subsequent scans with increasing indices will be added to the same folder. - :type index: int - :param x_motor: Select the name of the motor to move. - :type x_motor: str - :param start: Start the scan with X Motor here. - :type start: float - :param stop: Where to stop the scan. - :type stop: float - :param increment: How far to move between each sample. - :type increment: float - :param delay_after_move_s: How long to pause after each move. - :type delay_after_move_s: float - :param count_time_s: Time to acquire from data source to generate sample - :type count_time_s: float - :param number_of_scans: How many times should the motor make this motion during the scan. - :type number_of_scans: int - :param bidirect: If number of scans is more than 1, will move even scans in the opposite direction. - :type bidirect: bool - :param at_end_of_scan: Choose what the X Motor does at the scan end. - :type at_end_of_scan: enum u32 - :param description: - :type description: str - :param file_pattern: - :type file_pattern: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('sc_AutoColl Single Axis Scan', dict(locals())) - - async def sc_barf_scan(self, instrument="", exposure_time_s=0.001, number_to_average=1, continuous_acquire=True, background_data="Q:\\Commissioning\\OffLineRuby\\background\\background2020-03-12_11-15-40.txt", fit_type="Lorentzian", ambient_peak_2=694.23, ambient_peak_1=692.8, description="", file_pattern="") -> dict: - """ - The Barf scan acquires from a spectrometer and then fits the ruby peaks to estimate the pressure. - - :param instrument: Choose the instrumnet to acquire from. - :type instrument: str - :param exposure_time_s: Choose the exposure time for the instrument. - :type exposure_time_s: float - :param number_to_average: How many exposures to average. - :type number_to_average: int - :param continuous_acquire: Choose if the scan should continuously acquire. Starting a scan with this option will cause the scan to run forever. The user can then stop the scan with the stop button, or by unchecking this box. - :type continuous_acquire: bool - :param background_data: Background data to subtract from acquired data. - :type background_data: str - :param fit_type: Type of fit to do. - :type fit_type: enum u32 - :param ambient_peak_2: Peak position of the 2nd peak when the pressure is zero. - :type ambient_peak_2: float - :param ambient_peak_1: Peak position of the 1st peak when pressure is zero. - :type ambient_peak_1: float - :param description: - :type description: str - :param file_pattern: - :type file_pattern: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('sc_Barf Scan', dict(locals())) - - async def sc_from_file_scan(self, file_path="", delay_after_move_s=0, count_time_s=0.5, at_end="Return", move_motors_sequentially=False, dont_repeat_motor_moves=False, move_at_end_of_scan="", description="", file_pattern="") -> dict: - """ -

This interface hides all the scan detail inside of a 'from file' scan file. Only the four remaining settings are used to affect every step listed in the file.

-

The file contains motor names and positions. At each step, the motors will be set to those positions, then the data will be read. This allows the user to create scans of arbitrary complexity, but they are harder to setup because the file has to be created.

-

Example:

-
- - - - - - - - - - - - - - - - - - - - - -
Motor1Motor2Motor3
10.00.000.00
10.010.00.00
0.0010.010.0
-
-

This is the template of the from file scan. For more complex behavior, use the trajectory scan.

- - :param file_path: Choose the file to run. File should have the motor names across the top and the positions on the rows below. Tab spaces values. - :type file_path: str - :param delay_after_move_s: After the motors move, the signal may need some time to settle. This can be added here to improve data quality. - :type delay_after_move_s: float - :param count_time_s: Specify how long to sample the input data for. All the data sampled during this time will be averaged together, depending on the data source. - :type count_time_s: float - :param at_end: If return to start is used, the motors will receive one last command to come back to the start after the scan has completed. Otherwise the motors will stay where it is left at the end of the scan. - :type at_end: enum u32 - :param move_motors_sequentially: Move the motors one at a time. - :type move_motors_sequentially: bool - :param dont_repeat_motor_moves: Prevent identical positions from moving the motor twice. This could cause a motor to move when the desired action is for the motor to not move. - :type dont_repeat_motor_moves: bool - :param move_at_end_of_scan: Choose a Trajectory to run at the end of the scan. - :type move_at_end_of_scan: str - :param description: - :type description: str - :param file_pattern: - :type file_pattern: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('sc_From File Scan', dict(locals())) - - async def sc_image_from_file_scan(self, instrument="", save_images_as="", file_path="", delay_after_move_s=0, count_time_s=0.5, at_end="Return", move_motors_sequentially=False, dont_repeat_motor_moves=False, move_at_end_of_scan="", description="", file_pattern="") -> dict: - """ - - - :param instrument: - :type instrument: str - :param save_images_as: - :type save_images_as: str - :param file_path: Choose the file to run. File should have the motor names across the top and the positions on the rows below. Tab spaces values. - :type file_path: str - :param delay_after_move_s: After the motors move, the signal may need some time to settle. This can be added here to improve data quality. - :type delay_after_move_s: float - :param count_time_s: Specify how long to sample the input data for. All the data sampled during this time will be averaged together, depending on the data source. - :type count_time_s: float - :param at_end: If return to start is used, the motors will receive one last command to come back to the start after the scan has completed. Otherwise the motors will stay where it is left at the end of the scan. - :type at_end: enum u32 - :param move_motors_sequentially: Move the motors one at a time. - :type move_motors_sequentially: bool - :param dont_repeat_motor_moves: Prevent identical positions from moving the motor twice. This could cause a motor to move when the desired action is for the motor to not move. - :type dont_repeat_motor_moves: bool - :param move_at_end_of_scan: Choose a Trajectory to run at the end of the scan. - :type move_at_end_of_scan: str - :param description: - :type description: str - :param file_pattern: - :type file_pattern: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('sc_Image From File Scan', dict(locals())) - - async def sc_image_one_motor_scan(self, instrument="", save_images_as="", x_motor="Select Motor", start=-3, stop=0, increment=1, delay_after_move_s=0, count_time_s=0.5, number_of_scans=1, bidirect=True, at_end_of_scan="Return", description="", file_pattern="") -> dict: - """ - - - :param instrument: - :type instrument: str - :param save_images_as: - :type save_images_as: str - :param x_motor: Select the name of the motor to move. - :type x_motor: str - :param start: Start the scan with X Motor here. - :type start: float - :param stop: Where to stop the scan. - :type stop: float - :param increment: How far to move between each sample. - :type increment: float - :param delay_after_move_s: How long to pause after each move. - :type delay_after_move_s: float - :param count_time_s: Time to acquire from data source to generate sample - :type count_time_s: float - :param number_of_scans: How many times should the motor make this motion during the scan. - :type number_of_scans: int - :param bidirect: If number of scans is more than 1, will move even scans in the opposite direction. - :type bidirect: bool - :param at_end_of_scan: Choose what the X Motor does at the scan end. - :type at_end_of_scan: enum u32 - :param description: - :type description: str - :param file_pattern: - :type file_pattern: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('sc_Image One Motor Scan', dict(locals())) - - async def sc_image_time_scan(self, instrument="", save_images_as="", total_time_s=0, number_of_samples=0, time_between_points_s=0, count_time_s=0, frequency_hz=0, stop_condition="Samples Taken", description="", file_pattern="") -> dict: - """ - - - :param instrument: - :type instrument: str - :param save_images_as: - :type save_images_as: str - :param total_time_s: Time to run the scan in total. - :type total_time_s: float - :param number_of_samples: Samples to take in the Total Time. - :type number_of_samples: int - :param time_between_points_s: Will combine with Total time to set the number of samples. If Set is selected for Number of Samples, then the total time will change to acomplish the requested time between points. - :type time_between_points_s: float - :param count_time_s: Time that the data is averaged for to create a single data point. - :type count_time_s: float - :param frequency_hz: The user can specify the frequency in samples per second or number of samples. Both work with time and reconfigure the other. - :type frequency_hz: float - :param stop_condition: Should the scan end based on time or number of samples. Sometimes, the time between points cannot be accopmlished and the samples are taken at a longer interval. In cases where this is significant, the user may want to stop the scan based on time and not the number of samples taken. - :type stop_condition: enum u16 - :param description: - :type description: str - :param file_pattern: - :type file_pattern: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('sc_Image Time Scan', dict(locals())) - - async def sc_image_two_motor_scan(self, instrument="", save_images_as="", x_motor="", x_start=-3, x_stop=0, x_increment=1, delay_after_move_s=0, count_time_s=0.5, y_motor="", y_start=-3, y_stop=0, y_increment=1, at_end_of_scan="Return", description="", file_pattern="") -> dict: - """ - - - :param instrument: - :type instrument: str - :param save_images_as: - :type save_images_as: str - :param x_motor: Choose the motor to be moved as the X Motor. - :type x_motor: str - :param x_start: Place where X Motor will start scan. - :type x_start: float - :param x_stop: Furthest position of X Motor travel. - :type x_stop: float - :param x_increment: How far the X Motor moves with each X move. - :type x_increment: float - :param delay_after_move_s: How long to delay after each motor moves. - :type delay_after_move_s: float - :param count_time_s: Time that data is acquired and averaged to create sample. - :type count_time_s: float - :param y_motor: Choose the motor to be moved as the Y Motor. - :type y_motor: str - :param y_start: Place where Y Motor will start scan. - :type y_start: float - :param y_stop: Furthest position of Y Motor travel. - :type y_stop: float - :param y_increment: How far the Y Motor moves with each Y move. - :type y_increment: float - :param at_end_of_scan: Choose what the scan does upon completion. Stay at end will not command an end move. Return to start will. - :type at_end_of_scan: enum u32 - :param description: - :type description: str - :param file_pattern: - :type file_pattern: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('sc_Image Two Motor Scan', dict(locals())) - - async def sc_ltp_single_axis_scan(self, direction="Forward", index=0, instrument="", frames_to_average=0, delay_between_frames_s=0, x_motor="Select Motor", start=-3, stop=0, increment=1, delay_after_move_s=0, count_time_s=0.5, number_of_scans=1, bidirect=True, at_end_of_scan="Return", description="", file_pattern="") -> dict: - """ - - - :param direction: - :type direction: enum u32 - :param index: - :type index: int - :param instrument: - :type instrument: str - :param frames_to_average: - :type frames_to_average: int - :param delay_between_frames_s: - :type delay_between_frames_s: float - :param x_motor: Select the name of the motor to move. - :type x_motor: str - :param start: Start the scan with X Motor here. - :type start: float - :param stop: Where to stop the scan. - :type stop: float - :param increment: How far to move between each sample. - :type increment: float - :param delay_after_move_s: How long to pause after each move. - :type delay_after_move_s: float - :param count_time_s: Time to acquire from data source to generate sample - :type count_time_s: float - :param number_of_scans: How many times should the motor make this motion during the scan. - :type number_of_scans: int - :param bidirect: If number of scans is more than 1, will move even scans in the opposite direction. - :type bidirect: bool - :param at_end_of_scan: Choose what the X Motor does at the scan end. - :type at_end_of_scan: enum u32 - :param description: - :type description: str - :param file_pattern: - :type file_pattern: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('sc_LTP Single Axis Scan', dict(locals())) - - async def sc_move_motor(self, motor_name="", motor_position=0, delay_after_move_sec=0, description="", file_pattern="") -> dict: - """ - Setup this simple scan so that it will move a motor. This is used in the automation scripts to move things around. - - :param motor_name: - :type motor_name: str - :param motor_position: Position to move to. - :type motor_position: float - :param delay_after_move_sec: Scan will wait this long before moving completing. - :type delay_after_move_sec: float - :param description: - :type description: str - :param file_pattern: - :type file_pattern: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('sc_Move Motor', dict(locals())) - - async def sc_move_trajectory(self, trajectory_name="", delay_after_move_sec=0, description="", file_pattern="") -> dict: - """ - Setup this simple scan so that it will activate a trajectory. This not a trajectory scan. Instead the user needs to set up a motor trajectory which is a named configuration of motors moved together or in sequence. This is used in the automation scripts to move things around. - - :param trajectory_name: Choose the name of the desired trajectory. - :type trajectory_name: str - :param delay_after_move_sec: The scan will wait this many seconds after moving. - :type delay_after_move_sec: float - :param description: - :type description: str - :param file_pattern: - :type file_pattern: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('sc_Move Trajectory', dict(locals())) - - async def sc_powder_diffraction(self, exposure_time=0, number_of_samples=0, file_index=0, instrument="", save_images_as="", description="", file_pattern="") -> dict: - """ - - - :param exposure_time: - :type exposure_time: float - :param number_of_samples: - :type number_of_samples: int - :param file_index: - :type file_index: int - :param instrument: - :type instrument: str - :param save_images_as: - :type save_images_as: str - :param description: - :type description: str - :param file_pattern: - :type file_pattern: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('sc_Powder diffraction', dict(locals())) - - async def sc_set_dio(self, description="", file_pattern="") -> dict: - """ - - - :param description: - :type description: str - :param file_pattern: - :type file_pattern: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('sc_Set DIO', dict(locals())) - - async def sc_single_crystal_scan(self, instrument="", save_images_as="", x_motor="", start=-3, stop=0, increment=1, count_time_s=0.5, description="", file_pattern="") -> dict: - """ - Setup the single motor scan that also records an image from a ccd camera instrument, with each data point. - This scan moves a single motor to multiple positions at regular intervals and takes data at each spot. The motor that is moved is recorded and is set as the default x axis when this data is graphed on an xy graph. If you look at this data on an Intensity plot or Image Graph, It will default to the "2D Image" data. - - :param instrument: Choose The Instrument to Capture Images From. - :type instrument: str - :param save_images_as: - :type save_images_as: str - :param x_motor: Selects the motor to scan. - :type x_motor: str - :param start: This is where the scan should start. The motor will move from here to the stop position during the scan. - :type start: float - :param stop: This is where the scan should stop. The motor will move from the start position to here during the scan. - :type stop: float - :param increment: This is specified in the the units of the motor. It determines how far to move the motor for each step. - :type increment: float - :param count_time_s: Specify how long to sample the input data for. All the data sampled during this time will be averaged together, depending on the data source. - :type count_time_s: float - :param description: - :type description: str - :param file_pattern: - :type file_pattern: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('sc_Single Crystal Scan', dict(locals())) - - async def sc_single_motor_flying_scan(self, x_motor="", start=0, stop=10, increment=1, velocity_units=0.5, number_of_scans=1, bidirect=False, shift_ai=False, at_end_of_scan="Return", description="", file_pattern="") -> dict: - """ - Setup a single motor flying scan. - This scan makes one motor move and records data at specific positions during the move. - To accomplish this, the scan sets up special conditions in the motor drive hardware to output pulses when the motor is at a certain position. When those pulses happen, the motor controller latches its current position into an array. The analog aquisition card also latches the analog values with every pulse. - The scan polls the aquisition card and retreives all the data since the last poll. At the end of the scan, the motor positions are retreived and matched up with the analog data. This file is then saved. - Because of how this operates, the data for the flying scan is not displayed until the end of the scan. - - :param x_motor: Choose the name of the motor to scan. This motor must support flying scans. - :type x_motor: str - :param start: Place where the first sample is taken. The scan will start elsewhere and accelerate to the proper velocity before the first sample is taken. - :type start: float - :param stop: Place where the last sample is taken. The scan will begin to decelerate after the last sample is taken. - :type stop: float - :param increment: The spacing between sample points. - :type increment: float - :param velocity_units: Specify how many times the motor moves along the patter for this scan. Cannot be 0. - :type velocity_units: float - :param number_of_scans: Will allow the scan to run back to front on even scans. - :type number_of_scans: int - :param bidirect: Specify the velocity in the underlying motor's units. This is not in counts. - :type bidirect: bool - :param shift_ai: Choose if the motor returns to the start at the end of the scan. - :type shift_ai: bool - :param at_end_of_scan: Choose behaviour at end of the scan. - :type at_end_of_scan: enum u32 - :param description: - :type description: str - :param file_pattern: - :type file_pattern: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('sc_Single Motor Flying Scan', dict(locals())) - - async def sc_single_motor_scan(self, x_motor="Select Motor", start=-3, stop=0, increment=1, delay_after_move_s=0, count_time_s=0.5, number_of_scans=1, bidirect=True, at_end_of_scan="Return", description="", file_pattern="") -> dict: - """ - Setup the single motor scan. - This scan moves a single motor to multiple positions at regular intervals and takes data at each spot. The motor that is moved is recorded and is set as the default x axis when this data is graphed on an xy graph. - - :param x_motor: Select the name of the motor to move. - :type x_motor: str - :param start: Start the scan with X Motor here. - :type start: float - :param stop: Where to stop the scan. - :type stop: float - :param increment: How far to move between each sample. - :type increment: float - :param delay_after_move_s: How long to pause after each move. - :type delay_after_move_s: float - :param count_time_s: Time to acquire from data source to generate sample - :type count_time_s: float - :param number_of_scans: How many times should the motor make this motion during the scan. - :type number_of_scans: int - :param bidirect: If number of scans is more than 1, will move even scans in the opposite direction. - :type bidirect: bool - :param at_end_of_scan: Choose what the X Motor does at the scan end. - :type at_end_of_scan: enum u32 - :param description: - :type description: str - :param file_pattern: - :type file_pattern: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('sc_Single Motor Scan', dict(locals())) - - async def sc_temperature_scan(self, instrument="", exposure_time_msec="", calibration_temperature="", xmin=0, xmax=0, lambda_max=0, lambda_min=0, nd_filter_motor="Select Motor", instrument_2="", exposure_time_msec_2="", calibration_temperature_2="", xmin_2=0, xmax_2=0, lambda_max_2=0, lambda_min_2=0, nd_filter_motor_2="Select Motor", instrument_3="", roi_size=0, temp_background_k=0, instrument_4="", roi_size_2=0, temp_background_k_2=0, acquire_images=False, description="", file_pattern="") -> dict: - """ - The Temperature Scan acquires data from a spectrometer and then with background and calibration data calculates the temperature. It also optionally acquires an image from a 2D instrument and then displays a contour plot of the temperature profile along with recomputing the temperature based on the 2D image. - - :param instrument: - :type instrument: str - :param exposure_time_msec: - :type exposure_time_msec: str - :param calibration_temperature: - :type calibration_temperature: str - :param xmin: - :type xmin: float - :param xmax: - :type xmax: float - :param lambda_max: - :type lambda_max: float - :param lambda_min: - :type lambda_min: float - :param nd_filter_motor: - :type nd_filter_motor: str - :param instrument_2: - :type instrument_2: str - :param exposure_time_msec_2: - :type exposure_time_msec_2: str - :param calibration_temperature_2: - :type calibration_temperature_2: str - :param xmin_2: - :type xmin_2: float - :param xmax_2: - :type xmax_2: float - :param lambda_max_2: - :type lambda_max_2: float - :param lambda_min_2: - :type lambda_min_2: float - :param nd_filter_motor_2: - :type nd_filter_motor_2: str - :param instrument_3: - :type instrument_3: str - :param roi_size: - :type roi_size: float - :param temp_background_k: - :type temp_background_k: float - :param instrument_4: - :type instrument_4: str - :param roi_size_2: - :type roi_size_2: float - :param temp_background_k_2: - :type temp_background_k_2: float - :param acquire_images: Selects whether to also acquire the 2d images - :type acquire_images: bool - :param description: - :type description: str - :param file_pattern: - :type file_pattern: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('sc_Temperature Scan', dict(locals())) - - async def sc_time_scan(self, total_time_s=0, number_of_samples=0, time_between_points_s=0, count_time_s=0, frequency_hz=0, stop_condition="Samples Taken", description="", file_pattern="") -> dict: - """ - Setup the time scan. - This scan takes periodic data aquisitions and stores them. The main parameters that need to be specified are: The number of samples, how often to sample, and how long to aquire data for. All other variables that can be configured relate back to these three. - This scan supports the legacy header. - - :param total_time_s: Time to run the scan in total. - :type total_time_s: float - :param number_of_samples: Samples to take in the Total Time. - :type number_of_samples: int - :param time_between_points_s: Will combine with Total time to set the number of samples. If Set is selected for Number of Samples, then the total time will change to acomplish the requested time between points. - :type time_between_points_s: float - :param count_time_s: Time that the data is averaged for to create a single data point. - :type count_time_s: float - :param frequency_hz: The user can specify the frequency in samples per second or number of samples. Both work with time and reconfigure the other. - :type frequency_hz: float - :param stop_condition: Should the scan end based on time or number of samples. Sometimes, the time between points cannot be accopmlished and the samples are taken at a longer interval. In cases where this is significant, the user may want to stop the scan based on time and not the number of samples taken. - :type stop_condition: enum u16 - :param description: - :type description: str - :param file_pattern: - :type file_pattern: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('sc_Time Scan', dict(locals())) - - async def sc_trajectory_scan(self, file_path="", delay_after_move_s=0, count_time_s=0.5, at_end="Return", move_motors_sequentially=False, dont_repeat_motor_moves=False, shift_flying_data=False, move_at_end_of_scan="", description="", file_pattern="") -> dict: - """ -

This simple interface hides all the scan detail inside of a tracjectory scan file. Only the three remaining settings are used to affect every step listed in the file.

-

The file contains motor names and positions. At each step, the motors will be set to those positions, then the data will be read. This allows the user to create scans of arbitrary complexity, but they are harder to setup because the file has to be created.

-

Example:

-
- - - - - - - - - - - - - - - - - - - - - -
Motor1Motor2Motor3
10.00.000.00
10.010.00.00
0.0010.010.0
-
-

There are many more options for the files.

- - :param file_path: Specify the path to the trajectory scan file. - :type file_path: str - :param delay_after_move_s: After the motors move, the signal may need some time to settle. This can be added here to improve data quality. - :type delay_after_move_s: float - :param count_time_s: Specify how long to sample the input data for. All the data sampled during this time will be averaged together, depending on the data source. - :type count_time_s: float - :param at_end: If return to start is used, the motors will receive one last command to come back to the start after the scan has completed. Otherwise the motors will stay where it is left at the end of the scan. - :type at_end: enum u32 - :param move_motors_sequentially: Move the motors one at a time. - :type move_motors_sequentially: bool - :param dont_repeat_motor_moves: Prevent identical positions from moving the motor twice. This could cause a motor to move when the desired action is for the motor to not move. - :type dont_repeat_motor_moves: bool - :param shift_flying_data: Shift the flying dat ahead one time interval. - :type shift_flying_data: bool - :param move_at_end_of_scan: A trajectory to be called at the end of the scan. Trajectories can be setup in the motor system. The BCS team can help. If the trajectory is left blank, nothing will be done. - :type move_at_end_of_scan: str - :param description: - :type description: str - :param file_pattern: - :type file_pattern: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('sc_Trajectory Scan', dict(locals())) - - async def sc_two_motor_scan(self, x_motor="", x_start=-3, x_stop=0, x_increment=1, delay_after_move_s=0, count_time_s=0.5, y_motor="", y_start=-3, y_stop=0, y_increment=1, at_end_of_scan="Return", description="", file_pattern="") -> dict: - """ - This is the display for the setup when the scan is in the "View Data" mode. It has the same fields as the "Run Scan" mode, but everything is for display only. - - :param x_motor: Choose the motor to be moved as the X Motor. - :type x_motor: str - :param x_start: Place where X Motor will start scan. - :type x_start: float - :param x_stop: Furthest position of X Motor travel. - :type x_stop: float - :param x_increment: How far the X Motor moves with each X move. - :type x_increment: float - :param delay_after_move_s: How long to delay after each motor moves. - :type delay_after_move_s: float - :param count_time_s: Time that data is acquired and averaged to create sample. - :type count_time_s: float - :param y_motor: Choose the motor to be moved as the Y Motor. - :type y_motor: str - :param y_start: Place where Y Motor will start scan. - :type y_start: float - :param y_stop: Furthest position of Y Motor travel. - :type y_stop: float - :param y_increment: How far the Y Motor moves with each Y move. - :type y_increment: float - :param at_end_of_scan: Choose what the scan does upon completion. Stay at end will not command an end move. Return to start will. - :type at_end_of_scan: enum u32 - :param description: - :type description: str - :param file_pattern: - :type file_pattern: str - - :return: Dictionary of results (no endpoint-specific keys). - - - - """ - return await self.bcs_request('sc_Two Motor Scan', dict(locals())) - -``` diff --git a/.cursor/skills/xray-pro/references/beamline-controls.md b/.cursor/skills/xray-pro/references/beamline-controls.md deleted file mode 100644 index 84a25529..00000000 --- a/.cursor/skills/xray-pro/references/beamline-controls.md +++ /dev/null @@ -1,467 +0,0 @@ -# Beamline Controls and Motor Naming - -> Reference for: Xray Pro -> Load when: Figuring out a motor name, AI, etc. - ---- - -## Overview - -This is a detailed documentation describing what each motor is, the naming scheme, and the axes alignments with regard to the instrument and the lab reference frame - ---- - -## Fundamental Naming - -Controls are sent asynchronously through the BCS API. They are separated into core functionalities. - -* **Analog Inputs (AI)**: These are each of the sensors at the beamline, including the Photodiode (photocurrent measured from the synchrotron beam hitting the photodiode sensor), Beamline Energy (from the mono), Ai 3 Izero (upstream beam intensity measured from a gold mesh), and others. Common AI channels are listed in the Analog Input Channels section below. - | Command | Description | - |----------------------|------------------------------------------------------------------------| - | acquire_data | Acquires for `time` seconds, or `counts` counts. | - | get_acquired | Retrieve the average value from the most recent single-shot acquisition (see `start_acquire`). | - | get_acquired_array | Retrieve the array acquired from the most recent single-shot acquisition (see `start_acquire`). | - | get_acquire_status | Read the current acquisition state of the AI subsystem. | - | get_freerun | Get freerun AI data for one or more channels. | - | get_freerun_array | Retrieve most recent AI freerun data. | - | list_ais | Return the list of all AI channels that are defined on the server. | - | start_acquire | Start Acquisition for either `time` or `counts`, whichever is non-zero.| - | stop_acquire | Stop acquisition in progress. | - -* **Digital Input Output (DIO)**: This contains toggles for each of the the toggleable components. This includes the shutter input, the in chamber light and camera trigger, and others. - | Command | Description | - |-------------|--------------------------------------------------------------------------------------------------| - | get_di | Get digital input (DI) channel value | - | list_dios | Retrieves the complete list of digital input channel names defined on the server. | - | set_do | Sets the digital output (DO) channel `chan` to `value` | - -* **Files and Folders**: This gets and sets the disk locations where the data is being saved. In all cases data is saved to the drive, this lets the user control where these files are being written to. - | Command | Description | - |--------------------|---------------------------------------------------------------------------------------------------------------------------------| - | get_folder_listing | Get lists of all files and folders in a location descended from “C:\Beamline Controls\BCS Setup Data” | - | get_text_file | Get contents of a text file (usually acquired data) from any file in a location descended from “C:\Beamline Controls\BCS Setup Data”. | - -* **Global (State) Variables**: This contains state variables for the instrument. In most instances these should not be touched, changed, or set programmatically. - | Command | Description | - |----------------------|-------------------------------------------------------------------------------------------------| - | get_state_variable | Get the value of the named BCS State Variable. | - | list_state_variables | Get the complete list of state variables and their types (Boolean, Integer, String, Double). | - | set_state_variable | Set the value of the named BCS State Variable. | - -* **Instrument Data Acquisition**: This subsystem handles data acquisition from various instruments connected to the beamline, such as detectors and spectrometers. It provides commands to start, stop, and retrieve data from instrument acquisitions. - | Command | Description | - |--------------------------------|-------------------------------------------------------------------------------------------------| - | get_instrument_acquired1d | Retrieve data (1D array) from the most recent acquisition. | - | get_instrument_acquired2d | Retrieve data (2D array) from the most recent acquisition. | - | get_instrument_acquired3d | Retrieve data (3D array) from the most recent acquisition. | - | get_instrument_acquisition_info | Retrieve miscellaneous info about the instrument and acquisition: file_name, sensor temperature (if applicable), live time, and dead time. | - | get_instrument_acquisition_status | Retrieve all available status bits from the instrument subsystem for the named instrument. | - | get_instrument_count_rates | Retrieve instrument count rates. | - | get_instrument_driver_status | Returns the status of the BCS Instrument Driver for the named instrument. | - | list_instruments | Return the list of instruments that are defined on the server. | - | start_instrument_acquire | Starts an instrument acquisition and waits for it to complete. | - | start_instrument_driver | Starts the named instrument driver (does nothing if the driver is already running). | - | stop_instrument_acquire | Stop acquisition on the named instrument (does nothing if the instrument is not acquiring). | - | stop_instrument_driver | Stops the named instrument driver (does nothing if the driver is not running). | - -* **Miscellaneous**: Utility commands for system status, panel images, and video feeds. - | Command | Description | - |--------------------|------------------------------------------------------------------| - | get_panel_image | Send a current image of the LabVIEW panel, in jpg format. | - | get_subsystem_status | Returns the current status of all BCS subsystems on the server. | - | get_video_image | Return the most recent image from the named camera, in jpg format. | - -* **Motors and Motion**: This subsystem controls all motorized components of the beamline. Motors can be moved individually or in coordinated trajectories. Motors can be enabled, disabled, homed, and their positions can be queried. Trajectories allow multiple motors to move in a coordinated fashion to predefined positions. Key motors on this instrument are broken down into Sample (X, Y, Z, Theta) and CCD (X, Y, Theta). These motors define the coordinate system of the beamline are are core to the alignment of the system. - | Command | Description | - |----------------------|-------------------------------------------------------------------------------------------------| - | at_preset | Checks if the associated motor is at the preset position. | - | at_trajectory | Checks if all requirements have been satisfied to be "At trajectory" (usually just means that each motor is at its trajectory goal). | - | command_motor | Command one or more motors. | - | disable_breakpoints | Disables the breakpoint output (output-on-position) of motor controller for the named motor. | - | disable_motor | Disables the named motor. | - | enable_motor | Enables the named motor. | - | get_flying_positions | Retrieve the locations of motor name that will trigger acquisitions in a flying scan. | - | get_motor | Get information and status for the motors in motors. | - | get_motor_full | Returns the complete state of the requested motors. | - | home_motor | Home the motors in the input array, motors. | - | list_motors | Return the list of motors that are defined on the server. | - | list_presets | Return array of motor preset positions. | - | list_trajectories | List all trajectories. | - | move_motor | Command one or more motors to begin moves to supplied goals. | - | move_to_preset | Move to preset positions. | - | move_to_trajectory | Move to trajectory positions. | - | set_breakpoints | Set Breakpoints for the named motor. | - | set_motor_velocity | Set motor speeds. | - | start_flying_scan | Start a flying scan with the named motor. | - | stop_motor | Immediately issue stop command for the motors provided in motors. | - -* **Scans**: This subsystem provides high-level scan routines for common experimental procedures. Scans coordinate motor movements with data acquisition. Various scan types are available for different experimental geometries and measurement types. - | Command | Description | - |----------------------------|-------------------------------------------------------------------------------------------------| - | sc_alsu_mirror_vibration | Setup the ALSU Mirror Vibration Scan. This scan moves one motor in a specific pattern and records the analog data. | - | sc_auto_coll_single_axis_scan | This is VERY close to a Single Motor Scan. | - | sc_barf_scan | The Barf scan acquires from a spectrometer and then fits the ruby peaks to estimate the pressure. | - | sc_from_file_scan | This interface hides all the scan detail inside of a 'from file' scan file. | - | sc_image_from_file_scan | Image scan from file configuration. | - | sc_image_one_motor_scan | Image scan with one motor. | - | sc_image_time_scan | Image scan over time. | - | sc_image_two_motor_scan | Image scan with two motors. | - | sc_ltp_single_axis_scan | LTP single axis scan. | - | sc_move_motor | Setup this simple scan so that it will move a motor. | - | sc_move_trajectory | Setup this simple scan so that it will activate a trajectory. | - | sc_powder_diffraction | Powder diffraction scan. | - | sc_set_dio | Setup scan to set digital output channels. | - | sc_single_crystal_scan | Setup the single motor scan that also records an image from a ccd camera instrument, with each data point. | - | sc_single_motor_flying_scan | Setup a single motor flying scan. | - | sc_single_motor_scan | Single motor scan that moves one motor and acquires data at each position. | - | sc_temperature_scan | The Temperature Scan acquires data from a spectrometer and then with background and calibration data calculates the temperature. | - | sc_time_scan | Time scan that acquires data over time without motor movement. | - | sc_two_motor_scan | Two motor scan that moves two motors and acquires data at each position combination. | - -* **Scan Management**: This subsystem provides commands to manage scan execution, including starting, stopping, and querying scan status. Scans can be executed synchronously or asynchronously. - | Command | Description | - |----------------------|-------------------------------------------------------------------------------------------------| - | current_scan_running | Returns the currently running 'integrated scan' run, or an empty string if none is running. | - | last_scan_run | Returns the last 'integrated scan' run, or an empty string if none have run. | - | scan_status | Returns information about the 'integrated scan' system, namely the last scan (last_scan) run, the currently running scan, and the scanner status. | - | stop_scan | Immediately issue stop command for the currently running 'integrated scan'. | - -## Coordinates and Axes - -The beamline is configured in two regions, the chamber, and the region up stream from the chamber. - -### Upstream Components - -The beamline is set up with an X-ray beam incident into a spherical chamber from an Undulator source. Here is an explanation of the X-ray flight path into the chamber. - -1. **EPU Source**: The beam is generated at an EPU (Elliptically Polarizing Undulator) source. Elliptically polarizing undulator (EPU) – 5 cm period; 165-1,800 eV [A. T. Young et al., J. Synchrotron Rad., 2002, 9, 270-274] - - 4 rows of permanent magnets placed along axis of electron beam - - 2 rows above the plane of the storage ring on either side of the beam; same for the 2 rows below - - Linear polarization from 0° to 90° are available for energies above 160 eV - - Using the fundamental output from the undulator, pure circularly polarized X-rays can be produced from 130-600 eV - - For higher energies, 3rd and 5th harmonics must be used, but this leads to elliptical (P = 0.8 to 0.9) rather than circular polarization - -2. **Horizontal Exit Slit**: After the EPU, the beam is selected using the Horizontal Exit Slit. - -3. **M101 Monochromator**: From there the energy is selected at the M101 MONO (monochromator). The monochromator uses gratings (250 lines/mm for 150-550 eV, 500 lines/mm for energies above 550 eV) to select the desired X-ray energy. - -4. **Mirror Focusing**: The beam is then focused through a series of mirrors, including the M103 Mirror, which provides focusing and beam conditioning. - -5. **Higher Order Suppressor (HOS)**: The beam then hits the Higher Order Suppressor. This is a four-bounce reflective mirror assembly designed to add linear attenuation to the beam and remove higher order light making it past the mono. The incident angle of the mirrors can be changed between 4°-8°, or moved out of the beam path to let unfiltered light into the chamber. - -6. **Upstream JJ Scatter Slits**: The beam then passes through the Upstream JJ scatter slits (approximately 1.5 m upstream of sample chamber). There are four motors for these slits: - - Upstream JJ Vert Trans: vertical translational position - - Upstream JJ Horz Trans: horizontal translational position - - Upstream JJ Vert Aperture: vertical aperture size - - Upstream JJ Horz Aperture: horizontal aperture size - -7. **Shutter System and Beam Monitoring**: The beam then passes through the shutter system and Ai 3 Izero instrument for monitoring beam flux. - - After the 6° deflection mirror, there is a gold mesh assembly that can be lowered into the beam and connected to a picoammeter to measure photoelectric current - - This gold mesh is upstream of the HOS, so the spectrum of X-rays measured here is not suitable when doing scattering experiments at lower energies (e.g. near carbon dip) that need to use the HOS - - Second gold mesh assembly directly downstream of the HOS, but before the first set of slits is used to monitor incident flux (Ai 3 Izero) - -8. **Middle JJ Slits**: The beam then passes through the Middle JJ slits (approximately 0.6 m upstream of sample). These have the same 4 motors, but use the "Middle" prefix. These remove most of the remaining parasitic scattering from the first set of slits. - -9. **In-Chamber JJ Slits**: Lastly, the beam passes through the In-Chamber JJ slits (approximately 0.2 m upstream of sample). Again these have the same 4 motors, but use the "In-Chamber" prefix. These provide final beam definition before the sample. - -### In-Chamber Motors - -The chamber is circular in nature. The beam is incident from upstream and focused onto the sample plate. The detector is then downstream from that. - -#### Sample Coordinate System - -The sample is driven by 4 primary motors: Sample X, Sample Y, Sample Z, and Sample Theta. Plates are rectangular in nature and are designed to be inserted with samples affixed to the top of the plate, and the top of the plate facing upwards towards the top of the chamber in the Lab reference frame. - -**Sample Theta (θ)** dictates the orientation of the plate relative to the incident X-ray beam: -- **Sample θ = 90°**: The X-ray beam is normal to the sample plate surface (perpendicular incidence) -- 0°: Plate facing upwards towards the top of the chamber -- 180°: Plate facing downwards towards the back of the chamber -- Negative angles are valid: -90° = 270° - -The sample coordinate system (Sample X, Y, Z) originates from the center of the sample plate: -- **Sample X**: Translation along the axis of rotation (colinear with Sample Theta axis) - - Sample X + moves the plate inwards towards the motor controlling the motion - - Sample X - moves the plate outwards towards the chamber door -- **Sample Y**: Translation perpendicular to the beam at Sample Theta = 0 - - Sample Y + moves the plate away from the beam - - Sample Y - moves the plate towards the beam -- **Sample Z**: Translation perpendicular to the sample plate - - At Sample Theta = 0°, Sample Z + moves the plate up (in Lab Frame) - - At Sample Theta = 0°, Sample Z - moves the plate down (in Lab Frame) - - **Note**: Changing Sample Z will also change the sample-detector distance (except when Sample θ is at 0°) - -This forms a right-handed coordinate system. When Sample Theta rotates, the Sample Y and Z axes rotate with it, while Sample X remains colinear with the rotation axis. - -#### Detector Coordinate System - -![Geometry](coords.png "Optional title") - -The Sample Theta axis of rotation is colinear with a goniometer arm that controls the CCD motors. **CCD** is the prefix given to all the motors that control the geometry of the instrument positioning arm. This arm rests on a goniometer facilitating rotation through approximately 120°. - -The detector system consists of two components mounted on the same arm: -- **CCD detector**: Large area detector (2048 × 2048 pixels, 13.5 µm pixel size) for collecting scattering patterns -- **Photodiode**: 5 mm × 5 mm GaAs detector (Hamamatsu) for direct beam monitoring - -**CCD Theta (θ)** controls the detector angle: -- At CCD Theta = 0°: The instrument is directly aligned with the incident beam and positioned to the left of the sample from the chamber door perspective -- CCD Theta + rotates the detector along the same direction as the Sample Theta motor -- Typical range: -25° to +160° for scattering measurements - -**CCD Y** controls the sample-detector distance: -- CCD Y + moves the detector further away from the sample (increases sample-detector distance) -- CCD Y - moves the detector closer to the sample (decreases sample-detector distance) -- **Note**: Changing CCD Y will change the sample-detector distance - -**CCD X** moves the detector colinear to the Sample X motor (horizontal position). - -#### Photodiode Positioning - -Both the CCD detector and photodiode are attached to the same arm and controlled by the same motors. To select the Photodiode position: -- Use the "Photodiode Far" trajectory/preset position -- This moves CCD Y = 100 mm, CCD X = 6 mm, CCD Theta close to 0° -- Alternatively, manually position Sample X to move the photodiode into line with the incident beam -- The photodiode is designed to be in the direct beam with CCD out of the way -- There is also a beamstop photodiode (1 mm × 3 mm Si photodiode from Advanced Photonics, accessible via "AI 6 BeamStop") for flux monitoring during scattering measurements - -## Complete Motor Reference - -This section provides a comprehensive list of all motors available on the beamline control system, organized by functional category. - -### Sample Positioning Motors - -These motors control the position and orientation of the sample within the chamber. - -| Motor Name | Description | Units | Notes | -|------------|-------------|-------|-------| -| Sample X | Translation along the axis of rotation (colinear with Sample Theta axis). Positive moves toward motor, negative toward chamber door. | mm | Primary sample positioning axis | -| Sample Y | Translation perpendicular to beam at Sample Theta = 0. Positive moves away from beam, negative toward beam. | mm | Forms right-handed coordinate system with Sample X and Z | -| Sample Z | Translation perpendicular to sample plate. At Sample Theta = 0, positive moves up, negative moves down in lab frame. | mm | Height control relative to beam | -| Sample Theta | Rotation about the sample normal axis. 0° = plate facing up, 90° = normal to incident beam, 180° = facing down. | degrees | Primary sample orientation control | -| Sample Azimuthal Rotation | Additional rotational degree of freedom for sample orientation. | degrees | Secondary rotation axis | -| Sample Y Scaled | Scaled version of Sample Y position, used for specific scan geometries. | mm | Derived motor position | - -### Detector Positioning Motors - -These motors control the position and orientation of the CCD detector and photodiode. - -| Motor Name | Description | Units | Notes | -|------------|-------------|-------|-------| -| CCD Theta | Rotation of detector arm on goniometer. 0° = aligned with incident beam, positioned left of sample from chamber door. Positive rotates in same direction as Sample Theta. Range: ~120°. | degrees | Primary detector angle control | -| CCD X | Translation colinear with Sample X motor. | mm | Detector horizontal position | -| CCD Y | Sample-detector distance. Positive moves detector away from sample, negative moves closer. | mm | Controls scattering geometry | -| Pollux CCD X | Alternative CCD X position for Pollux detector system. | mm | Pollux-specific positioning | -| Pollux CCD Y | Alternative CCD Y position for Pollux detector system. | mm | Pollux-specific positioning | -| T-2T | Theta-2Theta coupling for reflectivity measurements. Automatically couples Sample Theta and CCD Theta. | degrees | Reflectivity scan mode | -| Beam Stop | Position of beam stop to block direct beam from hitting detector. | mm | Protects detector from direct beam | - -### Beamline Energy and Monochromator Controls - -These motors control the X-ray energy selection and monochromator settings. - -| Motor Name | Description | Units | Notes | -|------------|-------------|-------|-------| -| Beamline Energy | Current X-ray energy from monochromator. | eV | Primary energy control | -| Beamline Energy Goal | Target energy for energy scans. | eV | Used in energy scan routines | -| Mono Energy | Monochromator energy setting. | eV | Internal mono control | -| Mono 101 Grating | Grating selection on M101 monochromator. | unitless | Selects grating for energy range | -| Mono 101 Vessel | Monochromator vessel position. | mm | Vessel translation control | -| M101 Feedback | Feedback control for M101 monochromator. | unitless | Energy stabilization | -| M101 Horizontal Deflection | Horizontal beam deflection at M101. | mm | Beam steering | -| M101 Vertical Deflection | Vertical beam deflection at M101. | mm | Beam steering | - -### EPU (Elliptically Polarizing Undulator) Controls - -These motors control the X-ray source polarization and gap. - -| Motor Name | Description | Units | Notes | -|------------|-------------|-------|-------| -| EPU Gap | Undulator gap setting. Controls fundamental energy and flux. | mm | Primary EPU control | -| EPU Z | Undulator Z position. | mm | Longitudinal position | -| EPU Polarization | Polarization setting. EPU=1 (or 0.9) for circular, EPU=100 for S-polarized, EPU=190 for P-polarized. | unitless | Polarization control | - -### Mirror Controls - -These motors control the focusing and steering mirrors. - -| Motor Name | Description | Units | Notes | -|------------|-------------|-------|-------| -| M103 Yaw | M103 mirror yaw angle. | degrees | Mirror alignment | -| M103 Bend Up | M103 mirror upward bending. | mm | Mirror focusing | -| M103 Bend Down | M103 mirror downward bending. | mm | Mirror focusing | -| M121 Translation | M121 mirror translation position. | mm | Mirror positioning | - -### Slit Controls - -These motors control the entrance and exit slits that define the beam size and position. - -| Motor Name | Description | Units | Notes | -|------------|-------------|-------|-------| -| Entrance Slit Width | Width of entrance slit to monochromator. | mm | Beam size control. Note: Also exists as "Entrance Slit width" (lowercase) in some configurations - these refer to the same motor. | -| Exit Slit Top | Top blade position of exit slit. | mm | Vertical beam definition | -| Exit Slit Bottom | Bottom blade position of exit slit. | mm | Vertical beam definition | -| Exit Slit Left | Left blade position of exit slit. | mm | Horizontal beam definition | -| Exit Slit Right | Right blade position of exit slit. | mm | Horizontal beam definition | -| Horizontal Exit Slit Size | Size of horizontal exit slit. | mm | Horizontal beam size | -| Horizontal Exit Slit Position | Position of horizontal exit slit. | mm | Horizontal beam position | -| Vertical Exit Slit Size | Size of vertical exit slit. | mm | Vertical beam size | -| Vertical Exit Slit Position | Position of vertical exit slit. | mm | Vertical beam position | -| Vertical Slit Position | Alternative vertical slit position control. | mm | Additional vertical control | -| Vertical Slit Size | Alternative vertical slit size control. | mm | Additional vertical control | -| Horizontal Slit Position | Alternative horizontal slit position control. | mm | Additional horizontal control | -| Horizontal Slit Size | Alternative horizontal slit size control. | mm | Additional horizontal control | - -### Scatter Slit Controls (JJ Slits) - -These motors control the Jaws-Jaws (JJ) scatter slits at three positions along the beamline. Each set has four motors: vertical and horizontal translation, and vertical and horizontal aperture. - -#### Upstream JJ Slits -Located approximately 1.5 m upstream of sample chamber. First set of slits to define beam. - -| Motor Name | Description | Units | Notes | -|------------|-------------|-------|-------| -| Upstream JJ Vert Trans | Vertical translational position of upstream slits. | mm | Vertical beam position | -| Upstream JJ Horz Trans | Horizontal translational position of upstream slits. | mm | Horizontal beam position | -| Upstream JJ Vert Aperture | Vertical aperture size of upstream slits. | mm | Vertical beam size | -| Upstream JJ Horz Aperture | Horizontal aperture size of upstream slits. | mm | Horizontal beam size | - -#### Middle JJ Slits -Located approximately 0.6 m upstream of sample. Removes remaining parasitic scattering. - -| Motor Name | Description | Units | Notes | -|------------|-------------|-------|-------| -| Middle JJ Vert Trans | Vertical translational position of middle slits. | mm | Vertical beam position | -| Middle JJ Horz Trans | Horizontal translational position of middle slits. | mm | Horizontal beam position | -| Middle JJ Vert Aperture | Vertical aperture size of middle slits. | mm | Vertical beam size | -| Middle JJ Horz Aperture | Horizontal aperture size of middle slits. | mm | Horizontal beam size | - -#### In-Chamber JJ Slits -Located approximately 0.2 m upstream of sample. Final beam definition before sample. - -| Motor Name | Description | Units | Notes | -|------------|-------------|-------|-------| -| In-Chamber JJ Vert Trans | Vertical translational position of in-chamber slits. | mm | Vertical beam position | -| In-Chamber JJ Horz Trans | Horizontal translational position of in-chamber slits. | mm | Horizontal beam position | -| In-Chamber JJ Vert Aperture | Vertical aperture size of in-chamber slits. | mm | Vertical beam size | -| In-Chamber JJ Horz Aperture | Horizontal aperture size of in-chamber slits. | mm | Horizontal beam size | - -### Higher Order Suppressor - -| Motor Name | Description | Units | Notes | -|------------|-------------|-------|-------| -| Higher Order Suppressor | Four-bounce mirror assembly position. Removes higher-order harmonics from monochromator. Incident angle adjustable between 4°-8°, or moved out of beam path. | mm | Critical for low-energy experiments | - -### Shutter Controls - -| Motor Name | Description | Units | Notes | -|------------|-------------|-------|-------| -| PiezoShutter Trans | Piezo-actuated shutter translation position. | mm | Fast shutter control | -| PZT Shutter | Piezoelectric shutter control. | unitless | Alternative shutter system | - -### Temperature and Environmental Controls - -| Motor Name | Description | Units | Notes | -|------------|-------------|-------|-------| -| Temperature Controller | Hot stage temperature setpoint. Calibration curve relates setpoint (K) to actual temperature (°C). | K | Sample temperature control | -| Coolstage | Cooled sample stage temperature control. | K | Low-temperature experiments | -| Camera Temp Setpoint | CCD camera temperature setpoint. Typically set to -45°C for operation. | °C | Detector cooling | - -### Camera and Detector Controls - -These are motor-like parameters that control camera settings and readout. - -| Motor Name | Description | Units | Notes | -|------------|-------------|-------|-------| -| CCD Camera Shutter Inhibit | Inhibit signal for CCD camera shutter. | unitless | Shutter control | -| CCD Shutter Control | CCD camera shutter control signal. | unitless | Shutter control | -| Camera ROI X | Region of Interest X position. | pixels | Image cropping | -| Camera ROI Y | Region of Interest Y position. | pixels | Image cropping | -| Camera ROI Width | Region of Interest width. | pixels | Image size control | -| Camera ROI Height | Region of Interest height. | pixels | Image size control | -| Camera ROI X Bin | Binning factor in X direction. | unitless | Pixel binning for faster readout | -| Camera ROI Y Bin | Binning factor in Y direction. | unitless | Pixel binning for faster readout | - -### Sample Rotation Motors - -Additional rotational degrees of freedom for specialized sample holders. - -| Motor Name | Description | Units | Notes | -|------------|-------------|-------|-------| -| SampleRot0 | Sample rotation axis 0. | degrees | Additional rotation | -| SampleRot1 | Sample rotation axis 1. | degrees | Additional rotation | -| SampleRot2 | Sample rotation axis 2. | degrees | Additional rotation | -| SampleRot3 | Sample rotation axis 3. | degrees | Additional rotation | -| SampleRot4 | Sample rotation axis 4. | degrees | Additional rotation | - -### Multi-Channel Scaler (MCS) Axes - -| Motor Name | Description | Units | Notes | -|------------|-------------|-------|-------| -| MCS_axis0 | Multi-channel scaler axis 0. | unitless | MCS control | -| MCS_axis1 | Multi-channel scaler axis 1. | unitless | MCS control | -| MCS_axis2 | Multi-channel scaler axis 2. | unitless | MCS control | -| MCS_axis3 | Multi-channel scaler axis 3. | unitless | MCS control | -| MCS_axis4 | Multi-channel scaler axis 4. | unitless | MCS control | - -### Additional Controls - -| Motor Name | Description | Units | Notes | -|------------|-------------|-------|-------| -| Sample Number | Sample identifier number. | unitless | Sample tracking | -| Piezo Vertical | Piezo vertical position control. | mm | Fine positioning | -| Piezo Horiz | Piezo horizontal position control. | mm | Fine positioning | -| AO 0 | Analog output channel 0. | V | General purpose analog output | -| AO 1 | Analog output channel 1. | V | General purpose analog output | -| OSP Adjustment | Optical sample position adjustment. | mm | Fine sample positioning | -| Diag 106 | Diagnostic element 106 position. | mm | Beam diagnostics | - -## Analog Input (AI) Channels - -Complete list of analog input channels available on the beamline for data acquisition and monitoring. Channels are numbered sequentially from 0-20. - -| CH # | Channel Name | Description | Typical Units | Notes | -|------|--------------|-------------|---------------|-------| -| 0 | EPU Polarization | EPU polarization setting readback. | unitless | Readback of EPU polarization value (0.9 for circular, 100 for S-polarized, 190 for P-polarized). | -| 1 | Coolstage Temp C | Cooled sample stage temperature reading. | °C | Temperature of the coolstage sample holder. | -| 2 | CCD Temperature | CCD detector temperature reading. | °C | Temperature of the CCD detector. Typically operated at -45°C. | -| 3 | Beam Current | Storage ring beam current. | mA | Synchrotron beam current measurement. Negative values may indicate measurement direction or offset. | -| 4 | TEY signal | Total Electron Yield signal from sample. | A or V | Surface-sensitive detection mode for NEXAFS. Measures electron yield from sample surface. | -| 5 | Izero | Upstream beam intensity normalization signal. | A or V | Beam intensity measurement for normalization. Typically from gold mesh upstream of sample. | -| 6 | Photodiode | Main photodiode signal. Photocurrent from 5 mm × 5 mm GaAs photodiode (Hamamatsu) in direct beam path. | A or V | Primary beam intensity monitor. Use when CCD is out of beam path. | -| 7 | AI 0 | General purpose analog input channel 0. | V | Configurable analog input channel. | -| 8 | AI 3 Izero | Upstream beam intensity measured from gold mesh assembly (Ai 3). Located downstream of HOS, before first set of slits. | A or V | Used for flux normalization. Gold mesh can be lowered into beam and connected to picoammeter. | -| 9 | AI 5 | General purpose analog input channel 5. | V | Configurable analog input channel. | -| 10 | AI 6 BeamStop | Beamstop photodiode signal. Signal from 1 mm × 3 mm Si photodiode (Advanced Photonics) in beamstop. | A or V | Flux monitoring during scattering measurements. Can be used for absorption measurements while collecting scattering data. | -| 11 | AI 7 | General purpose analog input channel 7. | V | Configurable analog input channel. | -| 12 | Temperature Controller | Hot stage temperature controller readback. | K or °C | Temperature reading from the hot stage temperature controller. | -| 13 | PZT Shutter | Piezoelectric shutter position/status readback. | unitless | Status or position of the PZT (piezoelectric) shutter. | -| 14 | Pause Trigger | Pause trigger signal status. | unitless | Status of pause trigger for scan operations. | -| 15 | LV Memory | LabVIEW memory usage. | bytes | LabVIEW program memory usage indicator. | -| 16 | Deriv Photodiode | Derivative of photodiode signal. | V/s or A/s | Rate of change of photodiode signal. Useful for detecting beam fluctuations. | -| 17 | Time Stamp Error | Time synchronization error. | s | Error in time synchronization between systems. | -| 18 | Time Stamp Transmit Time | Time stamp transmission time. | s | Time taken to transmit time stamp data. | -| 19 | Time Stamp Server Time | Server time stamp. | s | Server-side time stamp value. | -| 20 | Camera Temp Setpoint | CCD camera temperature setpoint. | °C | Target temperature setting for CCD camera cooling. Typically set to -45°C for operation. | - -Note: The complete list of available AI channels can be obtained using the `list_ais` command. Channel numbers and names are fixed as listed above. Some channels may show negative values due to measurement direction, signal inversion, or offset calibration. - -## Digital Input/Output (DIO) Channels - -Complete list of digital input/output channels available on the beamline for controlling components and reading status signals. - -| Channel Name | Description | Type | Notes | -|--------------|------------|------|-------| -| Shutter Rev | Shutter reverse/reverse direction control. | DO | Controls shutter reverse operation. | -| Lightfiled Frame Loss | Light field frame loss detection signal. | DI | Indicates when light field frame is lost. | -| Nothing | Unused or placeholder DIO channel. | - | Reserved or unused channel. | -| Camera Scan | Camera scan trigger signal. | DO | Triggers camera acquisition during scans. | -| Shutter Output | Main beam shutter output control. Opens/closes X-ray beam. | DO | Critical safety component. Primary shutter control. Always verify shutter state. | -| Air Shutter Output | Air shutter output control. | DO | Controls air-operated shutter mechanism. | -| Light Output | Chamber illumination light control. | DO | Controls in-chamber lighting for sample viewing and alignment. | -| Beam Dumped | Beam dump status indicator. | DI | Indicates when beam is dumped (stopped). Read-only status signal. | -| PZT Shutter Status | Piezoelectric shutter status readback. | DI | Status of the PZT (piezoelectric) shutter position/state. | -| Do Pause Trigger | Pause trigger output control. | DO | Controls pause trigger for scan operations. | -| Trigger Pause Trigger | Trigger signal for pause trigger. | DO | Triggers the pause mechanism during scans. | -| Shutter Inhibit | Shutter inhibit signal. Prevents shutter from opening. | DO | Safety feature to inhibit shutter operation. | -| Trigger + Inhibit | Combined trigger and inhibit signal. | DO | Combined control signal for trigger and inhibit functions. | - -Note: The complete list of available DIO channels can be obtained using the `list_dios` command. The list above includes all visible channels from the DIO Monitor interface. Additional channels may exist that are not shown in the visible portion of the list. Channel types (DI = Digital Input, DO = Digital Output) are inferred from channel names and typical beamline configurations. diff --git a/.cursor/skills/xray-pro/references/lineup.md b/.cursor/skills/xray-pro/references/lineup.md deleted file mode 100644 index a0baa99a..00000000 --- a/.cursor/skills/xray-pro/references/lineup.md +++ /dev/null @@ -1,1299 +0,0 @@ -# Beamline Axis of Rotation Line Up - -> Reference for: Xray Pro -> Load when: At the start of a beamtime experiment. When planning code associated with linning up and sample alignment. - ---- - -## Overview - -This is dedicated for lining up a sample to the incident beam on a given sample. This is a critical component to measuring anything at the beamline, especial NEXAFS spectrsocopy and Reflectivity. These measurements require preciese knowledge of the Sample Z postition, and Sample Theta position with respect to the incident beam. This accounts for beam drift, and motor drift thoughout an experiment. - -First refference `references\beamline-controls.md` to understant the core api, motors, and coordinates. - -## Alignement Algorithm - -The alignement algorithm functions in three stages. First we algin the sensors to the direct beam, by aligning the CCD Theta motor positions to center the beam on the Photodiode, then we align the Sample Z and Sampe Theta positions. This sample alignment step is first done on a large detector, before happening on a more precise instrument. - -### Instrument Alignement and Centering - -1. Move the detector X motor such that we are centered on the photodiode instruemnt and not the Area detector. - -2. Move the Sample Z motor down such that it the beam passes straight though the chamber onto the Photodiode. - -3. Rock the CCD Theta motor arround zero to find the CCD Theta position associated with the maximum intensity of the Photodiode. Save this position for later use - -4. Jog up by about 4 degrees to center the the secondary photodiode placed on the bottom of the detector. This is a slitted photidode to improve the precision of the alignment algorithm. - -5. Rock the CCD Theta arround the 4 degree mark to find the CCD Theta position associated with the maximum intensity on teh AI 6 Beamstop AI. This is the name given to the slitten beamstop. Save this position for later use. - -6. Move back to the large photodiode position saved in step 3. The Instrument is now ligned up. - -### Sample Alginment - -Sample alignement needs to be done for each new sample, and sometimes for each new energy. It may be the case that a quick alignement at the start is allways prefered to the possibility of a mis-aligned measurement. - -What follows is a algorithmic iterative approach for sample alighnment. - -1. Ensure the sample is below the beam allowing the full signal to hit the detector. - -2. Move the sample up (+z) and record the instensity on the large photodiode. - -3. Find the Sample Z position that cut's the beam intesnity in half. Alternatively, fit the signal to a sigmoid/error function, and determine it's position z0. Record this position for note keeping. - -4. Move the sample theta motor to 1 degrees, and jog the sample theta motor by 2 degrees. We will be looking at the reflected intensity on the Photodiode, so ensure that we calculate the new position using the saves position from step 3 of the Instrument alignemtn. - -5. Scan the sample theta position from about 0 - 2 degrees to find the angle that maximizes the reflected intensity as measured by the photodiode. A better measure of this is to find the centroid of the peak that this trace displays. This angle is a better guess of what 1 degrees really is. - -6. Calculate the offset between 1 deg and this "True" angle. Save this offset for note keeping. Move to this offset and treat it as zero., - -7. Move back to step 1. Repeat until a convergence tolerence is reached. - -### Fine grained sample alignement - -This follows the same rules, but uses the slitted photodiode. To use this instrument, be sure that the offset angle for it found in the Instrument Alignment is properly accounted for. - -## Algorithm Components - -The alignment algorithm is broken down into reusable components. Each component includes a description, pseudo code, and Python implementation with checkpointing support. - -### 1. Finding Maximum Value in Data - -**Description:** Finds the position and value of the maximum intensity in a dataset. Used for peak finding in alignment scans. - -**Pseudo Code:** -``` -FUNCTION FIND_MAX(data_points): - // data_points is array of (x, y) pairs - SET max_y = -infinity - SET max_x = undefined - FOR each (x, y) IN data_points: - IF y > max_y: - SET max_y = y - SET max_x = x - END IF - END FOR - RETURN (max_x, max_y) -END FUNCTION -``` - -**Python Implementation:** -```python -def find_max(data_points): - """ - Find the position and value of maximum intensity. - - Parameters: - ----------- - data_points : list of tuples - List of (x, y) pairs where x is position and y is intensity - - Returns: - -------- - max_x : float - Position of maximum intensity - max_y : float - Maximum intensity value - - Checkpoint: - ----------- - Saves: max_x, max_y - """ - max_y = float('-inf') - max_x = None - - for x, y in data_points: - if y > max_y: - max_y = y - max_x = x - - # Checkpoint: Save maximum values - checkpoint = { - 'max_position': max_x, - 'max_intensity': max_y, - 'total_points': len(data_points) - } - # insert code to save checkpoint here - - return max_x, max_y -``` - ---- - -### 2. Linear Interpolation for Half-Maximum Finding - -**Description:** Finds the position where intensity equals a target value (typically half-maximum) using linear interpolation between data points. More accurate than finding the nearest point. - -**Pseudo Code:** -``` -FUNCTION INTERPOLATE(x_values, y_values, target_y): - // Linear interpolation to find x where y = target_y - FOR i = 0 to LENGTH(x_values) - 2: - IF (y_values[i] <= target_y AND target_y <= y_values[i+1]) OR - (y_values[i] >= target_y AND target_y >= y_values[i+1]): - SET slope = (y_values[i+1] - y_values[i]) / (x_values[i+1] - x_values[i]) - SET x_interp = x_values[i] + (target_y - y_values[i]) / slope - RETURN x_interp - END IF - END FOR - RETURN undefined -END FUNCTION -``` - -**Python Implementation:** -```python -def interpolate(x_values, y_values, target_y): - """ - Find x position where y equals target_y using linear interpolation. - - Parameters: - ----------- - x_values : list of float - Position values - y_values : list of float - Intensity values corresponding to positions - target_y : float - Target intensity value to find - - Returns: - -------- - x_interp : float or None - Interpolated position where y = target_y, or None if not found - - Checkpoint: - ----------- - Saves: target_y, x_interp, interpolation_method - """ - for i in range(len(x_values) - 1): - y1, y2 = y_values[i], y_values[i+1] - x1, x2 = x_values[i], x_values[i+1] - - # Check if target_y is between y1 and y2 - if (y1 <= target_y <= y2) or (y1 >= target_y >= y2): - slope = (y2 - y1) / (x2 - x1) - if abs(slope) > 1e-10: # Avoid division by zero - x_interp = x1 + (target_y - y1) / slope - else: - x_interp = (x1 + x2) / 2 # Average if slope is zero - - # Checkpoint: Save interpolation result - checkpoint = { - 'target_intensity': target_y, - 'interpolated_position': x_interp, - 'interpolation_method': 'linear', - 'bracket_indices': (i, i+1), - 'bracket_positions': (x1, x2), - 'bracket_intensities': (y1, y2) - } - # insert code to save checkpoint here - - return x_interp - - # Checkpoint: No interpolation found - checkpoint = { - 'target_intensity': target_y, - 'interpolated_position': None, - 'interpolation_method': 'linear', - 'error': 'target_y not found in range' - } - # insert code to save checkpoint here - - return None -``` - ---- - -### 3. Finding Peak Centroid - -**Description:** Calculates the centroid (center of mass) of a peak, which is more robust than finding the simple maximum. The centroid accounts for the shape of the peak distribution. - -**Pseudo Code:** -``` -FUNCTION FIND_CENTROID(x_values, y_values): - // Calculate centroid (center of mass) of peak - SET total_mass = SUM(y_values) - SET weighted_sum = 0 - FOR i = 0 to LENGTH(x_values) - 1: - SET weighted_sum = weighted_sum + x_values[i] * y_values[i] - END FOR - SET centroid_x = weighted_sum / total_mass - - // Find peak maximum for reference - SET (max_x, max_y) = FIND_MAX(zip(x_values, y_values)) - - RETURN (centroid_x, max_x) -END FUNCTION -``` - -**Python Implementation:** -```python -def find_centroid(x_values, y_values): - """ - Calculate centroid (center of mass) of intensity peak. - - Parameters: - ----------- - x_values : list of float - Position values - y_values : list of float - Intensity values - - Returns: - -------- - centroid_x : float - Centroid position (weighted average) - max_x : float - Position of maximum intensity (for reference) - - Checkpoint: - ----------- - Saves: centroid_x, max_x, max_intensity, total_mass - """ - total_mass = sum(y_values) - - if total_mass == 0: - # Fallback to maximum if no signal - max_idx = y_values.index(max(y_values)) - centroid_x = x_values[max_idx] - max_x = x_values[max_idx] - else: - weighted_sum = sum(x * y for x, y in zip(x_values, y_values)) - centroid_x = weighted_sum / total_mass - - # Also find maximum for reference - max_idx = y_values.index(max(y_values)) - max_x = x_values[max_idx] - - # Checkpoint: Save centroid calculation - checkpoint = { - 'centroid_position': centroid_x, - 'max_position': max_x, - 'max_intensity': max(y_values), - 'total_mass': total_mass, - 'num_points': len(x_values), - 'centroid_max_difference': abs(centroid_x - max_x) - } - # insert code to save checkpoint here - - return centroid_x, max_x -``` - ---- - -### 4. Scanning Theta Range and Measuring Intensity - -**Description:** Scans a range of theta (angle) positions and measures intensity at each position. Used for finding optimal detector or sample angles. - -**Pseudo Code:** -``` -FUNCTION SCAN_THETA_RANGE(theta_start, theta_end, theta_step, motor_name, ai_channel): - SET positions = [] - SET intensities = [] - SET current_theta = theta_start - - WHILE current_theta <= theta_end: - SET motor_name = current_theta - WAIT for motor movement - intensity = READ ai_channel signal - APPEND current_theta to positions - APPEND intensity to intensities - SET current_theta = current_theta + theta_step - END WHILE - - RETURN (positions, intensities) -END FUNCTION -``` - -**Python Implementation:** -```python -def scan_theta_range(theta_start, theta_end, theta_step, motor_name, ai_channel): - """ - Scan a range of theta positions and measure intensity. - - Parameters: - ----------- - theta_start : float - Starting theta position (degrees) - theta_end : float - Ending theta position (degrees) - theta_step : float - Step size for scan (degrees) - motor_name : str - Name of motor to move (e.g., "CCD Theta", "Sample Theta") - ai_channel : str - Name of AI channel to read (e.g., "Photodiode", "AI 6 BeamStop") - - Returns: - -------- - positions : list of float - Theta positions scanned - intensities : list of float - Intensity values at each position - - Checkpoint: - ----------- - Saves: scan_range, step_size, num_points, max_intensity, max_position - """ - positions = [] - intensities = [] - - current_theta = theta_start - while current_theta <= theta_end: - # insert code to move motor_name to current_theta here - # insert code to wait for motor movement here - intensity = 0 # insert code to read ai_channel signal here - - positions.append(current_theta) - intensities.append(intensity) - current_theta += theta_step - - # Checkpoint: Save scan results - max_intensity = max(intensities) if intensities else 0 - max_idx = intensities.index(max_intensity) if intensities else 0 - max_position = positions[max_idx] if positions else None - - checkpoint = { - 'scan_type': 'theta_range', - 'motor_name': motor_name, - 'ai_channel': ai_channel, - 'theta_start': theta_start, - 'theta_end': theta_end, - 'theta_step': theta_step, - 'num_points': len(positions), - 'max_intensity': max_intensity, - 'max_position': max_position, - 'positions': positions, - 'intensities': intensities - } - # insert code to save checkpoint here - - return positions, intensities -``` - ---- - -### 5. Scanning Z Range and Measuring Intensity - -**Description:** Scans a range of Z positions and measures intensity at each position. Used for finding the half-maximum position for sample alignment. - -**Pseudo Code:** -``` -FUNCTION SCAN_Z_RANGE(z_start, z_end, z_step, ai_channel): - SET positions = [] - SET intensities = [] - SET current_z = z_start - - WHILE current_z <= z_end: - SET Sample_Z = current_z - WAIT for motor movement - intensity = READ ai_channel signal - APPEND current_z to positions - APPEND intensity to intensities - SET current_z = current_z + z_step - END WHILE - - RETURN (positions, intensities) -END FUNCTION -``` - -**Python Implementation:** -```python -def scan_z_range(z_start, z_end, z_step, ai_channel): - """ - Scan a range of Z positions and measure intensity. - - Parameters: - ----------- - z_start : float - Starting Z position (mm) - z_end : float - Ending Z position (mm) - z_step : float - Step size for scan (mm) - ai_channel : str - Name of AI channel to read (e.g., "Photodiode", "AI 6 BeamStop") - - Returns: - -------- - positions : list of float - Z positions scanned - intensities : list of float - Intensity values at each position - - Checkpoint: - ----------- - Saves: scan_range, step_size, num_points, max_intensity, max_position - """ - positions = [] - intensities = [] - - current_z = z_start - while current_z <= z_end: - # insert code to move Sample Z to current_z here - # insert code to wait for motor movement here - intensity = 0 # insert code to read ai_channel signal here - - positions.append(current_z) - intensities.append(intensity) - current_z += z_step - - # Checkpoint: Save scan results - max_intensity = max(intensities) if intensities else 0 - max_idx = intensities.index(max_intensity) if intensities else 0 - max_position = positions[max_idx] if positions else None - - checkpoint = { - 'scan_type': 'z_range', - 'ai_channel': ai_channel, - 'z_start': z_start, - 'z_end': z_end, - 'z_step': z_step, - 'num_points': len(positions), - 'max_intensity': max_intensity, - 'max_position': max_position, - 'positions': positions, - 'intensities': intensities - } - # insert code to save checkpoint here - - return positions, intensities -``` - ---- - -### 6. Finding Half-Maximum Z Position - -**Description:** Finds the Z position where intensity is half of the maximum. This position corresponds to the beam edge and is used for precise sample alignment. - -**Pseudo Code:** -``` -FUNCTION FIND_Z_HALF_MAX(z_positions, intensities): - SET max_intensity = MAX(intensities) - SET half_max = max_intensity / 2 - - // Option A: Linear interpolation - SET z_center = INTERPOLATE(z_positions, intensities, half_max) - - // Option B: If interpolation fails, use maximum position - IF z_center is undefined: - SET max_idx = INDEX_OF_MAX(intensities) - SET z_center = z_positions[max_idx] - END IF - - RETURN z_center -END FUNCTION -``` - -**Python Implementation:** -```python -def find_z_half_max(z_positions, intensities): - """ - Find Z position where intensity is half of maximum. - - Parameters: - ----------- - z_positions : list of float - Z positions from scan - intensities : list of float - Intensity values from scan - - Returns: - -------- - z_center : float - Z position at half-maximum intensity - - Checkpoint: - ----------- - Saves: z_center, max_intensity, half_max, method_used - """ - if not intensities: - return None - - max_intensity = max(intensities) - half_max = max_intensity / 2.0 - - # Try linear interpolation - z_center = interpolate(z_positions, intensities, half_max) - method_used = 'interpolation' - - # Fallback to maximum position if interpolation fails - if z_center is None: - max_idx = intensities.index(max_intensity) - z_center = z_positions[max_idx] - method_used = 'maximum_fallback' - - # Checkpoint: Save half-maximum finding result - checkpoint = { - 'z_center': z_center, - 'max_intensity': max_intensity, - 'half_max_intensity': half_max, - 'method_used': method_used, - 'num_points': len(z_positions), - 'z_range': (min(z_positions), max(z_positions)) - } - # insert code to save checkpoint here - - return z_center -``` - ---- - -### 7. Finding Optimal Theta Position - -**Description:** Finds the optimal theta position by scanning a range and calculating the centroid of the reflected intensity peak. More accurate than finding the simple maximum. - -**Pseudo Code:** -``` -FUNCTION FIND_THETA_OPTIMAL(theta_start, theta_end, theta_step, - base_theta, theta_offset, theta_photodiode, ai_channel): - SET theta_positions = [] - SET reflected_intensities = [] - SET current_theta = theta_start - - WHILE current_theta <= theta_end: - SET sample_theta = base_theta + theta_offset + current_theta - SET expected_ccd_theta = theta_photodiode + 2 * sample_theta - - SET Sample_Theta = sample_theta - SET CCD_Theta = expected_ccd_theta - WAIT for motor movements - intensity = READ ai_channel signal - - APPEND sample_theta to theta_positions - APPEND intensity to reflected_intensities - SET current_theta = current_theta + theta_step - END WHILE - - SET (theta_optimal, max_theta) = FIND_CENTROID(theta_positions, reflected_intensities) - - RETURN theta_optimal -END FUNCTION -``` - -**Python Implementation:** -```python -def find_theta_optimal(theta_start, theta_end, theta_step, base_theta, - theta_offset, theta_photodiode, ai_channel): - """ - Find optimal theta position by scanning and finding peak centroid. - - Parameters: - ----------- - theta_start : float - Starting theta offset (degrees) - theta_end : float - Ending theta offset (degrees) - theta_step : float - Step size (degrees) - base_theta : float - Base theta position (typically 1.0 degrees) - theta_offset : float - Current theta offset to apply - theta_photodiode : float - Optimal CCD Theta for photodiode (from instrument alignment) - ai_channel : str - AI channel to read (e.g., "Photodiode", "AI 6 BeamStop") - - Returns: - -------- - theta_optimal : float - Optimal theta position (centroid of peak) - - Checkpoint: - ----------- - Saves: theta_optimal, scan_data, centroid_info - """ - theta_positions = [] - reflected_intensities = [] - - current_theta = theta_start - while current_theta <= theta_end: - sample_theta = base_theta + theta_offset + current_theta - # Calculate expected photodiode position for reflection (2*theta geometry) - expected_ccd_theta = theta_photodiode + 2 * sample_theta - - # insert code to move Sample Theta to sample_theta here - # insert code to move CCD Theta to expected_ccd_theta here - # insert code to wait for motor movements here - intensity = 0 # insert code to read ai_channel signal here - - theta_positions.append(sample_theta) - reflected_intensities.append(intensity) - current_theta += theta_step - - # Find centroid of peak - theta_optimal, max_theta = find_centroid(theta_positions, reflected_intensities) - - # Checkpoint: Save theta optimization result - checkpoint = { - 'theta_optimal': theta_optimal, - 'max_theta': max_theta, - 'base_theta': base_theta, - 'theta_offset': theta_offset, - 'theta_photodiode': theta_photodiode, - 'scan_range': (theta_start, theta_end), - 'theta_step': theta_step, - 'max_intensity': max(reflected_intensities) if reflected_intensities else 0, - 'positions': theta_positions, - 'intensities': reflected_intensities - } - # insert code to save checkpoint here - - return theta_optimal -``` - ---- - -### 8. Checking Convergence - -**Description:** Checks if the alignment has converged by comparing current values to previous values. Used to determine when to stop iterative alignment. - -**Pseudo Code:** -``` -FUNCTION CHECK_CONVERGENCE(current_value, previous_value, tolerance): - IF previous_value is undefined: - RETURN (converged=False, delta=None) - END IF - - SET delta = ABS(current_value - previous_value) - SET converged = (delta < tolerance) - - RETURN (converged, delta) -END FUNCTION -``` - -**Python Implementation:** -```python -def check_convergence(current_value, previous_value, tolerance): - """ - Check if alignment has converged. - - Parameters: - ----------- - current_value : float - Current measured value - previous_value : float or None - Previous measured value (None for first iteration) - tolerance : float - Convergence tolerance - - Returns: - -------- - converged : bool - True if converged, False otherwise - delta : float or None - Difference between current and previous values - - Checkpoint: - ----------- - Saves: converged, delta, tolerance, iteration_info - """ - if previous_value is None: - delta = None - converged = False - else: - delta = abs(current_value - previous_value) - converged = delta < tolerance - - # Checkpoint: Save convergence check - checkpoint = { - 'converged': converged, - 'delta': delta, - 'tolerance': tolerance, - 'current_value': current_value, - 'previous_value': previous_value - } - # insert code to save checkpoint here - - return converged, delta -``` - ---- - -### 9. Positioning Detector for Photodiode - -**Description:** Positions the detector to use the photodiode instead of the area detector. This is the first step in instrument alignment. - -**Pseudo Code:** -``` -FUNCTION POSITION_DETECTOR_PHOTODIODE(): - SET CCD_X = photodiode_center_position - SET Sample_Z = beam_path_clear_position - WAIT for motor movements -END FUNCTION -``` - -**Python Implementation:** -```python -def position_detector_photodiode(photodiode_x_position=6.0, beam_clear_z=-2.0): - """ - Position detector to use photodiode and clear beam path. - - Parameters: - ----------- - photodiode_x_position : float - CCD X position for photodiode (mm, default: 6.0) - beam_clear_z : float - Sample Z position to clear beam path (mm, default: -2.0) - - Checkpoint: - ----------- - Saves: ccd_x_position, sample_z_position - """ - # insert code to move CCD X to photodiode_x_position here - # insert code to move Sample Z to beam_clear_z here - # insert code to wait for motor movements here - - # Checkpoint: Save positioning - checkpoint = { - 'step': 'position_detector_photodiode', - 'ccd_x_position': photodiode_x_position, - 'sample_z_position': beam_clear_z - } - # insert code to save checkpoint here -``` - ---- - -### 10. Finding Optimal Photodiode Theta - -**Description:** Finds the optimal CCD Theta position that maximizes intensity on the main photodiode. This is the reference position for all subsequent alignments. - -**Pseudo Code:** -``` -FUNCTION FIND_PHOTODIODE_THETA(theta_range_center=0, theta_range_width=2): - SET theta_range = [theta_range_center - theta_range_width, - theta_range_center + theta_range_width] - SET (positions, intensities) = SCAN_THETA_RANGE( - theta_range[0], theta_range[1], step=0.5, - motor="CCD Theta", ai_channel="Photodiode" - ) - SET (theta_optimal, max_intensity) = FIND_MAX(zip(positions, intensities)) - - RETURN theta_optimal -END FUNCTION -``` - -**Python Implementation:** -```python -def find_photodiode_theta(theta_range_center=0.0, theta_range_width=2.0, theta_step=0.5): - """ - Find optimal CCD Theta position for main photodiode. - - Parameters: - ----------- - theta_range_center : float - Center of theta scan range (degrees, default: 0.0) - theta_range_width : float - Width of theta scan range (degrees, default: 2.0) - theta_step : float - Step size for scan (degrees, default: 0.5) - - Returns: - -------- - theta_photodiode : float - Optimal CCD Theta position for photodiode - - Checkpoint: - ----------- - Saves: theta_photodiode, max_intensity, scan_data - """ - theta_start = theta_range_center - theta_range_width - theta_end = theta_range_center + theta_range_width - - positions, intensities = scan_theta_range( - theta_start, theta_end, theta_step, - motor_name="CCD Theta", - ai_channel="Photodiode" - ) - - theta_photodiode, max_intensity = find_max(list(zip(positions, intensities))) - - # Move to optimal position - # insert code to move CCD Theta to theta_photodiode here - # insert code to wait for motor movement here - - # Checkpoint: Save photodiode theta result - checkpoint = { - 'step': 'find_photodiode_theta', - 'theta_photodiode': theta_photodiode, - 'max_intensity': max_intensity, - 'scan_center': theta_range_center, - 'scan_width': theta_range_width, - 'scan_step': theta_step - } - # insert code to save checkpoint here - - return theta_photodiode -``` - ---- - -### 11. Finding Optimal Beamstop Theta - -**Description:** Finds the optimal CCD Theta position that maximizes intensity on the slitted beamstop photodiode (AI 6 BeamStop). This provides higher precision alignment. - -**Pseudo Code:** -``` -FUNCTION FIND_BEAMSTOP_THETA(theta_photodiode, offset_estimate=4.0): - SET theta_center = theta_photodiode + offset_estimate - SET theta_range = [theta_center - 1.0, theta_center + 1.0] - SET (positions, intensities) = SCAN_THETA_RANGE( - theta_range[0], theta_range[1], step=0.5, - motor="CCD Theta", ai_channel="AI 6 BeamStop" - ) - SET (theta_optimal, max_intensity) = FIND_MAX(zip(positions, intensities)) - - RETURN theta_optimal -END FUNCTION -``` - -**Python Implementation:** -```python -def find_beamstop_theta(theta_photodiode, offset_estimate=4.0, - theta_range_width=1.0, theta_step=0.5): - """ - Find optimal CCD Theta position for beamstop photodiode. - - Parameters: - ----------- - theta_photodiode : float - Optimal photodiode theta position (from find_photodiode_theta) - offset_estimate : float - Estimated offset from photodiode position (degrees, default: 4.0) - theta_range_width : float - Width of scan range around estimate (degrees, default: 1.0) - theta_step : float - Step size for scan (degrees, default: 0.5) - - Returns: - -------- - theta_beamstop : float - Optimal CCD Theta position for beamstop - - Checkpoint: - ----------- - Saves: theta_beamstop, max_intensity, offset_from_photodiode - """ - theta_center = theta_photodiode + offset_estimate - theta_start = theta_center - theta_range_width - theta_end = theta_center + theta_range_width - - positions, intensities = scan_theta_range( - theta_start, theta_end, theta_step, - motor_name="CCD Theta", - ai_channel="AI 6 BeamStop" - ) - - theta_beamstop, max_intensity = find_max(list(zip(positions, intensities))) - - # Checkpoint: Save beamstop theta result - offset_from_photodiode = theta_beamstop - theta_photodiode - checkpoint = { - 'step': 'find_beamstop_theta', - 'theta_beamstop': theta_beamstop, - 'theta_photodiode': theta_photodiode, - 'offset_from_photodiode': offset_from_photodiode, - 'max_intensity': max_intensity, - 'scan_center': theta_center, - 'scan_width': theta_range_width, - 'scan_step': theta_step - } - # insert code to save checkpoint here - - return theta_beamstop -``` - ---- - -### 12. Complete Instrument Alignment - -**Description:** Complete instrument alignment procedure that finds optimal positions for both photodiode and beamstop detectors. This must be done before sample alignment. - -**Pseudo Code:** -``` -FUNCTION INSTRUMENT_ALIGNMENT(): - POSITION_DETECTOR_PHOTODIODE() - SET theta_photodiode = FIND_PHOTODIODE_THETA() - SET theta_beamstop = FIND_BEAMSTOP_THETA(theta_photodiode) - SET CCD_Theta = theta_photodiode // Return to photodiode position - RETURN (theta_photodiode, theta_beamstop) -END FUNCTION -``` - -**Python Implementation:** -```python -def instrument_alignment(): - """ - Complete instrument alignment to find optimal detector positions. - - Returns: - -------- - theta_photodiode : float - Optimal CCD Theta for main photodiode - theta_beamstop : float - Optimal CCD Theta for beamstop photodiode - - Checkpoint: - ----------- - Saves: Complete alignment results, all intermediate steps - """ - # Step 1-2: Position detector - position_detector_photodiode() - - # Step 3: Find optimal photodiode theta - theta_photodiode = find_photodiode_theta() - - # Step 4-5: Find optimal beamstop theta - theta_beamstop = find_beamstop_theta(theta_photodiode) - - # Step 6: Return to photodiode position - # insert code to move CCD Theta to theta_photodiode here - # insert code to wait for motor movement here - - # Checkpoint: Save complete instrument alignment - checkpoint = { - 'step': 'instrument_alignment_complete', - 'theta_photodiode': theta_photodiode, - 'theta_beamstop': theta_beamstop, - 'offset_photodiode_to_beamstop': theta_beamstop - theta_photodiode - } - # insert code to save checkpoint here - - return theta_photodiode, theta_beamstop -``` - ---- - -### 13. Single Iteration of Sample Alignment - -**Description:** Performs one iteration of sample alignment, finding optimal Z and Theta positions. This is the core of the iterative alignment loop. - -**Pseudo Code:** -``` -FUNCTION SAMPLE_ALIGNMENT_ITERATION(theta_photodiode, theta_offset, - z_start, z_end, z_step): - // Find Z center - SET (z_positions, intensities) = SCAN_Z_RANGE(z_start, z_end, z_step, "Photodiode") - SET z_center = FIND_Z_HALF_MAX(z_positions, intensities) - - // Find optimal Theta - SET theta_optimal = FIND_THETA_OPTIMAL( - theta_start=0, theta_end=2, theta_step=0.1, - base_theta=1.0, theta_offset=theta_offset, - theta_photodiode=theta_photodiode, ai_channel="Photodiode" - ) - SET theta_offset_new = theta_optimal - 1.0 - - RETURN (z_center, theta_offset_new) -END FUNCTION -``` - -**Python Implementation:** -```python -def sample_alignment_iteration(theta_photodiode, theta_offset, - z_start=0.0, z_end=5.0, z_step=0.05): - """ - Perform one iteration of sample alignment. - - Parameters: - ----------- - theta_photodiode : float - Optimal CCD Theta for photodiode - theta_offset : float - Current theta offset - z_start : float - Starting Z position for scan (mm) - z_end : float - Ending Z position for scan (mm) - z_step : float - Step size for Z scan (mm) - - Returns: - -------- - z_center : float - Optimal Z position (half-maximum) - theta_offset_new : float - New theta offset - - Checkpoint: - ----------- - Saves: z_center, theta_offset_new, iteration_data - """ - # Ensure sample is below beam - # insert code to move Sample Z to below_beam_position here - - # Find Z center (half-maximum) - z_positions, intensities = scan_z_range(z_start, z_end, z_step, "Photodiode") - z_center = find_z_half_max(z_positions, intensities) - - # Find optimal Theta - theta_optimal = find_theta_optimal( - theta_start=0.0, theta_end=2.0, theta_step=0.1, - base_theta=1.0, theta_offset=theta_offset, - theta_photodiode=theta_photodiode, ai_channel="Photodiode" - ) - theta_offset_new = theta_optimal - 1.0 - - # Apply new theta offset - # insert code to move Sample Theta to theta_offset_new here - # insert code to wait for motor movement here - - # Checkpoint: Save iteration results - checkpoint = { - 'step': 'sample_alignment_iteration', - 'z_center': z_center, - 'theta_offset_old': theta_offset, - 'theta_offset_new': theta_offset_new, - 'theta_optimal': theta_optimal, - 'z_scan_range': (z_start, z_end), - 'z_step': z_step - } - # insert code to save checkpoint here - - return z_center, theta_offset_new -``` - ---- - -### 14. Complete Sample Alignment - -**Description:** Iterative sample alignment that converges to optimal Z and Theta positions. Continues until convergence criteria are met. - -**Pseudo Code:** -``` -FUNCTION SAMPLE_ALIGNMENT(theta_photodiode, convergence_tolerance=0.01, max_iterations=10): - SET iteration = 0 - SET theta_offset = 0 - SET previous_z_center = undefined - SET previous_theta_offset = undefined - - WHILE iteration < max_iterations: - SET (z_center, theta_offset_new) = SAMPLE_ALIGNMENT_ITERATION( - theta_photodiode, theta_offset - ) - - // Check Z convergence - SET (z_converged, z_delta) = CHECK_CONVERGENCE( - z_center, previous_z_center, convergence_tolerance - ) - - // Check Theta convergence - SET (theta_converged, theta_delta) = CHECK_CONVERGENCE( - theta_offset_new, previous_theta_offset, convergence_tolerance - ) - - IF z_converged AND theta_converged AND iteration > 0: - BREAK - END IF - - SET previous_z_center = z_center - SET previous_theta_offset = theta_offset_new - SET theta_offset = theta_offset_new - SET iteration = iteration + 1 - END WHILE - - RETURN (z_center, theta_offset) -END FUNCTION -``` - -**Python Implementation:** -```python -def sample_alignment(theta_photodiode, convergence_tolerance=0.01, max_iterations=10): - """ - Complete iterative sample alignment. - - Parameters: - ----------- - theta_photodiode : float - Optimal CCD Theta for photodiode (from instrument alignment) - convergence_tolerance : float - Convergence tolerance (mm or degrees, default: 0.01) - max_iterations : int - Maximum number of iterations (default: 10) - - Returns: - -------- - z_center : float - Final optimal Z position - theta_offset : float - Final theta offset - - Checkpoint: - ----------- - Saves: Final results, all iteration data, convergence info - """ - iteration = 0 - theta_offset = 0.0 - previous_z_center = None - previous_theta_offset = None - - while iteration < max_iterations: - z_center, theta_offset_new = sample_alignment_iteration( - theta_photodiode, theta_offset - ) - - # Check Z convergence - z_converged, z_delta = check_convergence( - z_center, previous_z_center, convergence_tolerance - ) - - # Check Theta convergence - theta_converged, theta_delta = check_convergence( - theta_offset_new, previous_theta_offset, convergence_tolerance - ) - - # Checkpoint: Save iteration info - checkpoint = { - 'step': 'sample_alignment_iteration', - 'iteration': iteration, - 'z_center': z_center, - 'theta_offset': theta_offset_new, - 'z_converged': z_converged, - 'z_delta': z_delta, - 'theta_converged': theta_converged, - 'theta_delta': theta_delta, - 'convergence_tolerance': convergence_tolerance - } - # insert code to save checkpoint here - - # Break if converged (after at least one iteration) - if z_converged and theta_converged and iteration > 0: - break - - previous_z_center = z_center - previous_theta_offset = theta_offset_new - theta_offset = theta_offset_new - iteration += 1 - - # Checkpoint: Save final alignment results - final_checkpoint = { - 'step': 'sample_alignment_complete', - 'final_z_center': z_center, - 'final_theta_offset': theta_offset, - 'total_iterations': iteration, - 'converged': (iteration < max_iterations), - 'convergence_tolerance': convergence_tolerance - } - # insert code to save checkpoint here - - return z_center, theta_offset -``` - ---- - -### 15. Fine-Grained Sample Alignment - -**Description:** Same as sample alignment but uses the slitted beamstop photodiode for higher precision. Uses tighter convergence tolerance and smaller step sizes. - -**Pseudo Code:** -``` -FUNCTION FINE_GRAINED_SAMPLE_ALIGNMENT(theta_beamstop, convergence_tolerance=0.005, max_iterations=10): - // Same as SAMPLE_ALIGNMENT but: - // - Use theta_beamstop instead of theta_photodiode - // - Use "AI 6 BeamStop" instead of "Photodiode" - // - Use smaller z_step (0.02 instead of 0.05) - // - Use smaller theta_step (0.05 instead of 0.1) - // - Use tighter convergence_tolerance -END FUNCTION -``` - -**Python Implementation:** -```python -def fine_grained_sample_alignment(theta_beamstop, convergence_tolerance=0.005, max_iterations=10): - """ - Fine-grained sample alignment using slitted beamstop photodiode. - - Parameters: - ----------- - theta_beamstop : float - Optimal CCD Theta for beamstop (from instrument alignment) - convergence_tolerance : float - Tighter convergence tolerance (default: 0.005) - max_iterations : int - Maximum number of iterations (default: 10) - - Returns: - -------- - z_center : float - Final optimal Z position - theta_offset : float - Final theta offset - - Checkpoint: - ----------- - Saves: Final results, all iteration data, convergence info - """ - iteration = 0 - theta_offset = 0.0 - previous_z_center = None - previous_theta_offset = None - - while iteration < max_iterations: - # Ensure sample is below beam - # insert code to move Sample Z to below_beam_position here - - # Find Z center using beamstop (finer step) - z_positions, intensities = scan_z_range( - z_start=0.0, z_end=5.0, z_step=0.02, # Smaller step - ai_channel="AI 6 BeamStop" - ) - z_center = find_z_half_max(z_positions, intensities) - - # Find optimal Theta using beamstop (finer step) - theta_optimal = find_theta_optimal( - theta_start=0.0, theta_end=2.0, theta_step=0.05, # Smaller step - base_theta=1.0, theta_offset=theta_offset, - theta_photodiode=theta_beamstop, # Use beamstop theta - ai_channel="AI 6 BeamStop" # Use beamstop channel - ) - theta_offset_new = theta_optimal - 1.0 - - # Apply new theta offset - # insert code to move Sample Theta to theta_offset_new here - # insert code to wait for motor movement here - - # Check convergence - z_converged, z_delta = check_convergence( - z_center, previous_z_center, convergence_tolerance - ) - theta_converged, theta_delta = check_convergence( - theta_offset_new, previous_theta_offset, convergence_tolerance - ) - - # Checkpoint: Save iteration info - checkpoint = { - 'step': 'fine_grained_sample_alignment_iteration', - 'iteration': iteration, - 'z_center': z_center, - 'theta_offset': theta_offset_new, - 'z_converged': z_converged, - 'z_delta': z_delta, - 'theta_converged': theta_converged, - 'theta_delta': theta_delta, - 'convergence_tolerance': convergence_tolerance - } - # insert code to save checkpoint here - - # Break if converged - if z_converged and theta_converged and iteration > 0: - break - - previous_z_center = z_center - previous_theta_offset = theta_offset_new - theta_offset = theta_offset_new - iteration += 1 - - # Checkpoint: Save final alignment results - final_checkpoint = { - 'step': 'fine_grained_sample_alignment_complete', - 'final_z_center': z_center, - 'final_theta_offset': theta_offset, - 'total_iterations': iteration, - 'converged': (iteration < max_iterations), - 'convergence_tolerance': convergence_tolerance - } - # insert code to save checkpoint here - - return z_center, theta_offset -``` diff --git a/.github/skills/code-documenter/SKILL.md b/.github/skills/code-documenter/SKILL.md deleted file mode 100644 index c2701d64..00000000 --- a/.github/skills/code-documenter/SKILL.md +++ /dev/null @@ -1,95 +0,0 @@ ---- -name: code-documenter -description: Use when adding docstrings, creating API documentation, or building documentation sites. Invoke for OpenAPI/Swagger specs, JSDoc, doc portals, tutorials, user guides. -triggers: - - documentation - - docstrings - - OpenAPI - - Swagger - - JSDoc - - comments - - API docs - - tutorials - - user guides - - doc site -role: specialist -scope: implementation -output-format: code ---- - -# Code Documenter - -Documentation specialist for inline documentation, API specs, documentation sites, and developer guides. - -## Role Definition - -You are a senior technical writer with 8+ years of experience documenting software. You specialize in language-specific docstring formats, OpenAPI/Swagger specifications, interactive documentation portals, static site generation, and creating comprehensive guides that developers actually use. - -## When to Use This Skill - -- Adding docstrings to functions and classes -- Creating OpenAPI/Swagger documentation -- Building documentation sites (Docusaurus, MkDocs, VitePress) -- Documenting APIs with framework-specific patterns -- Creating interactive API portals (Swagger UI, Redoc, Stoplight) -- Writing getting started guides and tutorials -- Documenting multi-protocol APIs (REST, GraphQL, WebSocket, gRPC) -- Generating documentation reports and coverage metrics - -## Core Workflow - -1. **Discover** - Ask for format preference and exclusions -2. **Detect** - Identify language and framework -3. **Analyze** - Find undocumented code -4. **Document** - Apply consistent format -5. **Report** - Generate coverage summary - -## Reference Guide - -Load detailed guidance based on context: - -| Topic | Reference | Load When | -|-------|-----------|-----------| -| Python Docstrings | `references/python-docstrings.md` | Google, NumPy, Sphinx styles | -| TypeScript JSDoc | `references/typescript-jsdoc.md` | JSDoc patterns, TypeScript | -| FastAPI/Django API | `references/api-docs-fastapi-django.md` | Python API documentation | -| NestJS/Express API | `references/api-docs-nestjs-express.md` | Node.js API documentation | -| Coverage Reports | `references/coverage-reports.md` | Generating documentation reports | -| Documentation Systems | `references/documentation-systems.md` | Doc sites, static generators, search, testing | -| Interactive API Docs | `references/interactive-api-docs.md` | OpenAPI 3.1, portals, GraphQL, WebSocket, gRPC, SDKs | -| User Guides & Tutorials | `references/user-guides-tutorials.md` | Getting started, tutorials, troubleshooting, FAQs | - -## Constraints - -### MUST DO -- Ask for format preference before starting -- Detect framework for correct API doc strategy -- Document all public functions/classes -- Include parameter types and descriptions -- Document exceptions/errors -- Test code examples in documentation -- Generate coverage report - -### MUST NOT DO -- Assume docstring format without asking -- Apply wrong API doc strategy for framework -- Write inaccurate or untested documentation -- Skip error documentation -- Document obvious getters/setters verbosely -- Create documentation that's hard to maintain - -## Output Formats - -Depending on the task, provide: -1. **Code Documentation:** Documented files + coverage report -2. **API Docs:** OpenAPI specs + portal configuration -3. **Doc Sites:** Site configuration + content structure + build instructions -4. **Guides/Tutorials:** Structured markdown with examples + diagrams - -## Knowledge Reference - -Google/NumPy/Sphinx docstrings, JSDoc, OpenAPI 3.0/3.1, AsyncAPI, gRPC/protobuf, FastAPI, Django, NestJS, Express, GraphQL, Docusaurus, MkDocs, VitePress, Swagger UI, Redoc, Stoplight - -## Related Skills - -**Spec Miner** - Informs from code analysis | **Fullstack Guardian** - Documents during implementation | **Code Reviewer** - Checks documentation quality diff --git a/.github/skills/code-documenter/references/api-docs-fastapi-django.md b/.github/skills/code-documenter/references/api-docs-fastapi-django.md deleted file mode 100644 index acc5b6e9..00000000 --- a/.github/skills/code-documenter/references/api-docs-fastapi-django.md +++ /dev/null @@ -1,169 +0,0 @@ -# API Documentation: FastAPI & Django - -> Reference for: Code Documenter -> Load when: Documenting Python API frameworks - -## FastAPI (Auto-generates from types) - -FastAPI automatically generates OpenAPI documentation from type hints and docstrings. - -### Endpoint Documentation - -```python -from fastapi import FastAPI, HTTPException, status -from pydantic import BaseModel, Field - -class UserCreate(BaseModel): - """User creation request body.""" - - name: str = Field(..., min_length=1, max_length=100, example="John Doe") - email: str = Field(..., example="john@example.com") - -class UserResponse(BaseModel): - """User response with generated ID.""" - - id: int = Field(..., example=1) - name: str - email: str - -@app.post( - "/users", - response_model=UserResponse, - status_code=status.HTTP_201_CREATED, - summary="Create a new user", - tags=["Users"], -) -async def create_user(user: UserCreate) -> UserResponse: - """Create a new user account. - - Args: - user: User creation data including name and email. - - Returns: - Created user with generated ID. - - Raises: - HTTPException: 400 if email already exists. - """ -``` - -### Router with Tags - -```python -from fastapi import APIRouter - -router = APIRouter( - prefix="/users", - tags=["Users"], - responses={404: {"description": "Not found"}}, -) - -@router.get( - "/{user_id}", - response_model=UserResponse, - summary="Get user by ID", -) -async def get_user(user_id: int) -> UserResponse: - """Retrieve a user by their unique identifier.""" -``` - -## Django REST Framework (drf-spectacular) - -### ViewSet Documentation - -```python -from rest_framework import viewsets, status -from rest_framework.decorators import action -from drf_spectacular.utils import extend_schema, OpenApiParameter - -class UserViewSet(viewsets.ModelViewSet): - """ - ViewSet for managing user accounts. - - list: Get all users with pagination. - create: Create a new user account. - retrieve: Get a specific user by ID. - update: Update all user fields. - partial_update: Update specific user fields. - destroy: Delete a user account. - """ - - queryset = User.objects.all() - serializer_class = UserSerializer - - @extend_schema( - summary="Get current user", - description="Returns the authenticated user's profile", - responses={200: UserSerializer}, - ) - @action(detail=False, methods=["get"]) - def me(self, request): - """Get the authenticated user's profile.""" - serializer = self.get_serializer(request.user) - return Response(serializer.data) -``` - -### Serializer Documentation - -```python -from rest_framework import serializers - -class UserSerializer(serializers.ModelSerializer): - """Serializer for user model with validation.""" - - class Meta: - model = User - fields = ["id", "name", "email", "created_at"] - read_only_fields = ["id", "created_at"] - - name = serializers.CharField( - help_text="User's display name", - max_length=100, - ) - email = serializers.EmailField( - help_text="User's email address (unique)", - ) -``` - -### Custom Schema - -```python -from drf_spectacular.utils import extend_schema, OpenApiExample - -@extend_schema( - request=UserCreateSerializer, - responses={ - 201: UserSerializer, - 400: OpenApiTypes.OBJECT, - }, - examples=[ - OpenApiExample( - "Valid request", - value={"name": "John", "email": "john@example.com"}, - ), - ], -) -def create(self, request): - """Create a new user.""" -``` - -## Quick Reference - -| Framework | Documentation Source | Output | -|-----------|---------------------|--------| -| FastAPI | Type hints + docstrings | Auto Swagger UI | -| DRF | Serializers + drf-spectacular | Auto Swagger UI | - -| FastAPI Decorator | Purpose | -|-------------------|---------| -| `summary` | Short endpoint description | -| `description` | Detailed description | -| `tags` | Group endpoints | -| `response_model` | Response schema | -| `responses` | Additional response codes | - -| DRF Decorator | Purpose | -|---------------|---------| -| `@extend_schema` | Customize schema | -| `OpenApiParameter` | Query/path params | -| `OpenApiExample` | Request examples | diff --git a/.github/skills/code-documenter/references/api-docs-nestjs-express.md b/.github/skills/code-documenter/references/api-docs-nestjs-express.md deleted file mode 100644 index d157e9b2..00000000 --- a/.github/skills/code-documenter/references/api-docs-nestjs-express.md +++ /dev/null @@ -1,223 +0,0 @@ -# API Documentation: NestJS & Express - -> Reference for: Code Documenter -> Load when: Documenting Node.js API frameworks - -## NestJS (@nestjs/swagger) - -NestJS requires explicit decorators for OpenAPI documentation. - -### Controller Documentation - -```typescript -import { Controller, Post, Body, Get, Param } from '@nestjs/common'; -import { - ApiTags, - ApiOperation, - ApiResponse, - ApiParam, - ApiBearerAuth, -} from '@nestjs/swagger'; - -@ApiTags('Users') -@ApiBearerAuth() -@Controller('users') -export class UsersController { - @Post() - @ApiOperation({ summary: 'Create a new user' }) - @ApiResponse({ - status: 201, - description: 'User created successfully', - type: UserDto, - }) - @ApiResponse({ - status: 400, - description: 'Invalid input data', - }) - async create(@Body() dto: CreateUserDto): Promise { - return this.usersService.create(dto); - } - - @Get(':id') - @ApiOperation({ summary: 'Get user by ID' }) - @ApiParam({ - name: 'id', - description: 'User unique identifier', - example: '123', - }) - @ApiResponse({ status: 200, type: UserDto }) - @ApiResponse({ status: 404, description: 'User not found' }) - async findOne(@Param('id') id: string): Promise { - return this.usersService.findOne(id); - } -} -``` - -### DTO Documentation - -```typescript -import { ApiProperty, ApiPropertyOptional } from '@nestjs/swagger'; -import { IsEmail, IsString, MinLength } from 'class-validator'; - -export class CreateUserDto { - @ApiProperty({ - description: "User's display name", - example: 'John Doe', - minLength: 1, - maxLength: 100, - }) - @IsString() - @MinLength(1) - name: string; - - @ApiProperty({ - description: "User's email address", - example: 'john@example.com', - }) - @IsEmail() - email: string; - - @ApiPropertyOptional({ - description: 'Profile picture URL', - example: 'https://example.com/avatar.jpg', - }) - avatarUrl?: string; -} -``` - -## Express (swagger-jsdoc) - -Express uses JSDoc comments with swagger annotations. - -### Setup - -```javascript -const swaggerJsdoc = require('swagger-jsdoc'); -const swaggerUi = require('swagger-ui-express'); - -const options = { - definition: { - openapi: '3.0.0', - info: { - title: 'API Documentation', - version: '1.0.0', - }, - }, - apis: ['./routes/*.js'], -}; - -const specs = swaggerJsdoc(options); -app.use('/api-docs', swaggerUi.serve, swaggerUi.setup(specs)); -``` - -### Route Documentation - -```javascript -/** - * @swagger - * /users: - * post: - * summary: Create a new user - * tags: [Users] - * requestBody: - * required: true - * content: - * application/json: - * schema: - * $ref: '#/components/schemas/CreateUser' - * responses: - * 201: - * description: User created successfully - * content: - * application/json: - * schema: - * $ref: '#/components/schemas/User' - * 400: - * description: Invalid input - */ -router.post('/users', createUser); - -/** - * @swagger - * /users/{id}: - * get: - * summary: Get user by ID - * tags: [Users] - * parameters: - * - in: path - * name: id - * required: true - * schema: - * type: string - * description: User ID - * responses: - * 200: - * description: User found - * content: - * application/json: - * schema: - * $ref: '#/components/schemas/User' - * 404: - * description: User not found - */ -router.get('/users/:id', getUser); -``` - -### Schema Documentation - -```javascript -/** - * @swagger - * components: - * schemas: - * CreateUser: - * type: object - * required: - * - name - * - email - * properties: - * name: - * type: string - * description: User's display name - * example: John Doe - * email: - * type: string - * format: email - * description: User's email address - * example: john@example.com - * User: - * allOf: - * - $ref: '#/components/schemas/CreateUser' - * - type: object - * properties: - * id: - * type: string - * description: Unique identifier - * createdAt: - * type: string - * format: date-time - */ -``` - -## Quick Reference - -| NestJS Decorator | Purpose | -|------------------|---------| -| `@ApiTags()` | Group endpoints | -| `@ApiOperation()` | Endpoint summary | -| `@ApiResponse()` | Response documentation | -| `@ApiParam()` | Path parameter | -| `@ApiQuery()` | Query parameter | -| `@ApiBody()` | Request body | -| `@ApiBearerAuth()` | Auth requirement | -| `@ApiProperty()` | DTO property | - -| Express swagger-jsdoc | Purpose | -|-----------------------|---------| -| `@swagger` | Start swagger block | -| `tags` | Group endpoints | -| `summary` | Short description | -| `parameters` | Path/query params | -| `requestBody` | Request body schema | -| `responses` | Response schemas | -| `$ref` | Reference schema | diff --git a/.github/skills/code-documenter/references/coverage-reports.md b/.github/skills/code-documenter/references/coverage-reports.md deleted file mode 100644 index 23c43997..00000000 --- a/.github/skills/code-documenter/references/coverage-reports.md +++ /dev/null @@ -1,128 +0,0 @@ -# Coverage Reports - -> Reference for: Code Documenter -> Load when: Generating documentation reports - -## Documentation Coverage Report Template - -```markdown -# Documentation Report: {project_name} - -## Summary -- **Files analyzed**: 45 -- **Functions documented**: 120/150 (80%) -- **Classes documented**: 25/25 (100%) -- **API endpoints documented**: 30/30 (100%) - -## Coverage Before/After -- Before: 45% -- After: 92% - -## Files Modified - -| File | Functions Added | Notes | -|------|-----------------|-------| -| src/services/user.ts | 8 | All public methods | -| src/services/auth.ts | 5 | Added examples | -| src/controllers/users.ts | 6 | Added @Api decorators | -| src/dto/user.dto.ts | 4 | Added @ApiProperty | - -## API Documentation - -- **Framework**: NestJS -- **Strategy**: @nestjs/swagger decorators -- **Swagger UI**: /api/docs -- **OpenAPI spec**: /api-json - -## Documentation Style - -- **Python**: Google style docstrings -- **TypeScript**: JSDoc with @param, @returns -- **API**: OpenAPI 3.0 via decorators - -## Next Steps - -### Recommendations -1. Run `npm run docs:lint` to validate JSDoc -2. Add `eslint-plugin-jsdoc` to enforce documentation -3. Consider adding examples for complex functions -4. Set up documentation CI checks - -### Missing Documentation -| File | Missing | Priority | -|------|---------|----------| -| src/utils/crypto.ts | 3 functions | High | -| src/helpers/date.ts | 2 functions | Medium | - -### CI Integration -```yaml -# Add to CI pipeline -- name: Check documentation - run: npm run docs:check - -- name: Generate API docs - run: npm run docs:generate -``` -``` - -## Checklist During Documentation - -```markdown -## Documentation Checklist - -### Before Starting -- [ ] Confirmed format preference (Google/JSDoc/etc.) -- [ ] Identified files to exclude (tests, generated) -- [ ] Detected framework for API docs - -### Functions/Methods -- [ ] All public functions documented -- [ ] Parameters described with types -- [ ] Return values documented -- [ ] Exceptions/errors documented -- [ ] Examples added for complex functions - -### Classes -- [ ] Class purpose described -- [ ] Constructor parameters documented -- [ ] Public methods documented -- [ ] Important attributes explained - -### API Endpoints -- [ ] All endpoints have summaries -- [ ] Request bodies documented -- [ ] Response schemas defined -- [ ] Error responses documented -- [ ] Authentication requirements noted - -### Final Checks -- [ ] Ran documentation linter -- [ ] Verified Swagger UI renders correctly -- [ ] No inaccurate documentation -- [ ] Coverage report generated -``` - -## Framework-Specific Linting - -```bash -# JavaScript/TypeScript - ESLint -npm install eslint-plugin-jsdoc --save-dev -# Add to .eslintrc: "plugins": ["jsdoc"] - -# Python - pydocstyle -pip install pydocstyle -pydocstyle --convention=google src/ - -# Python - interrogate (coverage) -pip install interrogate -interrogate -v src/ -``` - -## Quick Reference - -| Metric | Good | Acceptable | Poor | -|--------|------|------------|------| -| Function coverage | >90% | 70-90% | <70% | -| Class coverage | 100% | >90% | <90% | -| API endpoint coverage | 100% | 100% | <100% | -| Example coverage | >50% | 30-50% | <30% | diff --git a/.github/skills/code-documenter/references/documentation-systems.md b/.github/skills/code-documenter/references/documentation-systems.md deleted file mode 100644 index 694c0c30..00000000 --- a/.github/skills/code-documenter/references/documentation-systems.md +++ /dev/null @@ -1,336 +0,0 @@ -# Documentation Systems & Infrastructure - -> Reference for: Code Documenter -> Load when: Building documentation sites, static generators, multi-version docs, search systems - -## Static Site Generators - -### Docusaurus (Meta) - -```bash -# Setup -npx create-docusaurus@latest docs classic -cd docs && npm start - -# Structure -docs/ -├── docs/ # Documentation pages -├── blog/ # Blog posts -├── src/ -│ └── pages/ # Custom pages -└── docusaurus.config.js -``` - -**docusaurus.config.js:** -```javascript -module.exports = { - title: 'My API', - tagline: 'Build amazing things', - url: 'https://docs.example.com', - baseUrl: '/', - - themeConfig: { - navbar: { - items: [ - {to: '/docs/intro', label: 'Docs', position: 'left'}, - {to: '/api', label: 'API', position: 'left'}, - ], - }, - - // Algolia search - algolia: { - apiKey: 'YOUR_API_KEY', - indexName: 'your_index', - contextualSearch: true, - }, - - prism: { - theme: lightCodeTheme, - darkTheme: darkCodeTheme, - additionalLanguages: ['python', 'rust'], - }, - }, -}; -``` - -### MkDocs (Python) - -```yaml -# mkdocs.yml -site_name: My API Documentation -theme: - name: material - features: - - navigation.tabs - - navigation.sections - - toc.integrate - - search.suggest - - search.highlight - palette: - - scheme: default - toggle: - icon: material/brightness-7 - name: Switch to dark mode - - scheme: slate - toggle: - icon: material/brightness-4 - name: Switch to light mode - -plugins: - - search - - mkdocstrings: - handlers: - python: - options: - show_source: true - - git-revision-date-localized - -markdown_extensions: - - pymdownx.highlight - - pymdownx.superfences - - admonition - - codehilite - -nav: - - Home: index.md - - Getting Started: getting-started.md - - API Reference: api/ -``` - -### VitePress (Vue) - -```typescript -// .vitepress/config.ts -export default defineConfig({ - title: 'API Docs', - description: 'Developer documentation', - - themeConfig: { - nav: [ - { text: 'Guide', link: '/guide/' }, - { text: 'API', link: '/api/' }, - ], - - sidebar: { - '/guide/': [ - { - text: 'Introduction', - items: [ - { text: 'Getting Started', link: '/guide/getting-started' }, - { text: 'Configuration', link: '/guide/config' }, - ], - }, - ], - }, - - search: { - provider: 'local', - }, - - editLink: { - pattern: 'https://github.com/user/repo/edit/main/docs/:path', - }, - }, -}); -``` - -## Multi-Version Documentation - -### Version Switcher - -```javascript -// Docusaurus versions -{ - versions: { - current: { - label: '2.0 (Next)', - path: 'next', - }, - }, - onlyIncludeVersions: ['current', '1.5', '1.4'], -} -``` - -### Migration Guides - -```markdown -# Migration Guide: v1 to v2 - -## Breaking Changes - -### Authentication -**v1:** -```python -client.authenticate(api_key) -``` - -**v2:** -```python -client = Client(api_key=api_key) # Pass in constructor -``` - -### Renamed Methods -| v1 | v2 | Notes | -|----|----|----- | -| `get_user()` | `fetch_user()` | Async now | -| `delete_user()` | `remove_user()` | Returns Promise | - -## Deprecation Timeline -- v1.x: Supported until Dec 2025 -- v2.0: Released Jan 2025 -- v2.1: Current (June 2025) -``` - -## Search Implementation - -### Algolia DocSearch - -```html - - - - - -``` - -### Local Search (Lunr.js) - -```javascript -const idx = lunr(function() { - this.ref('id'); - this.field('title', { boost: 10 }); - this.field('content'); - - documents.forEach(doc => this.add(doc)); -}); - -// Search -const results = idx.search('authentication'); -``` - -## Documentation Testing - -### Link Checking - -```bash -# linkcheck (Python) -pip install linkchecker -linkchecker http://localhost:3000/docs - -# broken-link-checker (Node) -npm install -g broken-link-checker -blc http://localhost:3000 -ro -``` - -### Code Example Testing - -```python -# doctest for Python examples -""" ->>> add(2, 3) -5 ->>> add(-1, 1) -0 -""" - -# Run tests -python -m doctest -v docs/*.md -``` - -```javascript -// Jest for TypeScript examples -// Extract code blocks and test -import { runExamples } from './test-docs'; - -test('API examples work', async () => { - const examples = extractExamples('./docs/api.md'); - await expect(runExamples(examples)).resolves.toBeTruthy(); -}); -``` - -## Performance Optimization - -### Build Optimization - -```javascript -// Webpack/Vite config -export default { - build: { - rollupOptions: { - output: { - manualChunks: { - 'vendor': ['react', 'react-dom'], - }, - }, - }, - }, - - optimizeDeps: { - include: ['prismjs'], - }, -}; -``` - -### CDN & Caching - -```nginx -# nginx.conf -location /docs { - expires 1y; - add_header Cache-Control "public, immutable"; -} - -location ~* \.(html)$ { - expires 1h; - add_header Cache-Control "public, must-revalidate"; -} -``` - -## Analytics Integration - -### Google Analytics - -```javascript -// Docusaurus -gtag: { - trackingID: 'G-XXXXXXXXXX', - anonymizeIP: true, -}, -``` - -### Custom Analytics - -```javascript -// Track search queries -function trackSearch(query, results) { - analytics.track('docs_search', { - query, - resultCount: results.length, - timestamp: new Date(), - }); -} -``` - -## Quick Reference - -| Tool | Best For | Tech Stack | -|------|----------|-----------| -| Docusaurus | React projects, versioning | React, MDX | -| MkDocs | Python projects, simple setup | Python, Jinja2 | -| VitePress | Vue projects, fast builds | Vue, Vite | -| Nextra | Next.js integration | React, Next.js | -| Mintlify | Modern UI, AI search | React | - -| Search Solution | Cost | Features | -|----------------|------|----------| -| Algolia DocSearch | Free (OSS) | Fast, typo-tolerant | -| Local (Lunr.js) | Free | Offline, no server | -| Typesense | Free (self-host) | Privacy-focused | -| Meilisearch | Free (self-host) | Fast, relevance | diff --git a/.github/skills/code-documenter/references/interactive-api-docs.md b/.github/skills/code-documenter/references/interactive-api-docs.md deleted file mode 100644 index 937f3eec..00000000 --- a/.github/skills/code-documenter/references/interactive-api-docs.md +++ /dev/null @@ -1,534 +0,0 @@ -# Interactive API Documentation - -> Reference for: Code Documenter -> Load when: Building API portals, interactive consoles, multi-protocol APIs, SDK docs - -## OpenAPI 3.1 Advanced Features - -### Reusable Components - -```yaml -openapi: 3.1.0 -info: - title: Users API - version: 2.0.0 - -components: - # Reusable schemas - schemas: - User: - type: object - required: [id, email] - properties: - id: - type: string - format: uuid - example: "123e4567-e89b-12d3-a456-426614174000" - email: - type: string - format: email - example: "user@example.com" - - Error: - type: object - properties: - code: - type: string - message: - type: string - details: - type: object - - PaginatedResponse: - type: object - properties: - data: - type: array - items: {} - total: - type: integer - page: - type: integer - - # Reusable parameters - parameters: - PageParam: - name: page - in: query - schema: - type: integer - default: 1 - minimum: 1 - - LimitParam: - name: limit - in: query - schema: - type: integer - default: 20 - minimum: 1 - maximum: 100 - - # Security schemes - securitySchemes: - BearerAuth: - type: http - scheme: bearer - bearerFormat: JWT - - ApiKeyAuth: - type: apiKey - in: header - name: X-API-Key - - OAuth2: - type: oauth2 - flows: - authorizationCode: - authorizationUrl: https://api.example.com/oauth/authorize - tokenUrl: https://api.example.com/oauth/token - scopes: - read:users: Read user data - write:users: Modify user data - - # Reusable responses - responses: - NotFound: - description: Resource not found - content: - application/json: - schema: - $ref: '#/components/schemas/Error' - - Unauthorized: - description: Authentication required - content: - application/json: - schema: - $ref: '#/components/schemas/Error' - -paths: - /users: - get: - summary: List users - parameters: - - $ref: '#/components/parameters/PageParam' - - $ref: '#/components/parameters/LimitParam' - security: - - BearerAuth: [] - responses: - '200': - description: Success - content: - application/json: - schema: - allOf: - - $ref: '#/components/schemas/PaginatedResponse' - - type: object - properties: - data: - type: array - items: - $ref: '#/components/schemas/User' -``` - -## Interactive Documentation Portals - -### Swagger UI Customization - -```javascript -// Custom Swagger UI -const swaggerUi = require('swagger-ui-express'); -const swaggerDocument = require('./openapi.json'); - -const options = { - customCss: '.swagger-ui .topbar { display: none }', - customSiteTitle: "API Docs", - customfavIcon: "/favicon.ico", - swaggerOptions: { - persistAuthorization: true, - displayRequestDuration: true, - filter: true, - tryItOutEnabled: true, - requestInterceptor: (req) => { - req.headers['X-Custom-Header'] = 'value'; - return req; - }, - }, -}; - -app.use('/api-docs', swaggerUi.serve, swaggerUi.setup(swaggerDocument, options)); -``` - -### Redoc (Modern Alternative) - -```html - - - - API Documentation - - - - - - - - -``` - -### Stoplight Elements - -```javascript -import { API } from '@stoplight/elements'; -import '@stoplight/elements/styles.min.css'; - -function App() { - return ( - - ); -} -``` - -## Multi-Protocol Documentation - -### GraphQL Schema Documentation - -```graphql -""" -User account in the system -""" -type User { - """ - Unique user identifier - """ - id: ID! - - """ - User's email address (unique) - @example "user@example.com" - """ - email: String! - - """ - Display name - @example "John Doe" - """ - name: String! - - """ - User's posts (paginated) - """ - posts( - """Number of items per page (max 100)""" - limit: Int = 20 - """Page offset""" - offset: Int = 0 - ): PostConnection! -} - -type Query { - """ - Fetch a user by ID - """ - user( - """User's unique identifier""" - id: ID! - ): User - - """ - Search users by name or email - """ - searchUsers( - """Search query""" - query: String! - """Maximum results to return""" - limit: Int = 10 - ): [User!]! -} - -type Mutation { - """ - Create a new user account - """ - createUser( - """User creation input""" - input: CreateUserInput! - ): CreateUserPayload! -} - -""" -Input for creating a user -""" -input CreateUserInput { - """User's email address""" - email: String! - """Display name""" - name: String! -} -``` - -**GraphQL Playground:** -```javascript -const { ApolloServer } = require('apollo-server'); - -const server = new ApolloServer({ - typeDefs, - resolvers, - introspection: true, // Enable in dev - playground: { - settings: { - 'editor.theme': 'dark', - 'editor.fontSize': 14, - }, - }, -}); -``` - -### WebSocket Protocol Documentation - -```yaml -# AsyncAPI 2.0 -asyncapi: 2.5.0 -info: - title: Chat WebSocket API - version: 1.0.0 - description: Real-time chat messaging - -channels: - chat/{roomId}: - parameters: - roomId: - description: Chat room identifier - schema: - type: string - - subscribe: - summary: Receive messages - message: - oneOf: - - $ref: '#/components/messages/ChatMessage' - - $ref: '#/components/messages/UserJoined' - - publish: - summary: Send a message - message: - $ref: '#/components/messages/ChatMessage' - -components: - messages: - ChatMessage: - name: message - payload: - type: object - properties: - userId: - type: string - content: - type: string - timestamp: - type: string - format: date-time - - UserJoined: - name: userJoined - payload: - type: object - properties: - userId: - type: string - username: - type: string -``` - -### gRPC Documentation - -```protobuf -syntax = "proto3"; - -package users.v1; - -// User service manages user accounts -service UserService { - // Get a user by ID - // Returns: User object or NOT_FOUND error - rpc GetUser(GetUserRequest) returns (User) {} - - // List all users with pagination - // Returns: Paginated list of users - rpc ListUsers(ListUsersRequest) returns (ListUsersResponse) {} - - // Create a new user - // Returns: Created user or ALREADY_EXISTS error - rpc CreateUser(CreateUserRequest) returns (User) {} - - // Stream user updates in real-time - // Returns: Stream of user update events - rpc WatchUsers(WatchUsersRequest) returns (stream UserEvent) {} -} - -// User account -message User { - // Unique identifier - string id = 1; - - // Email address (unique, required) - string email = 2; - - // Display name - string name = 3; - - // Account creation timestamp - google.protobuf.Timestamp created_at = 4; -} -``` - -## SDK Documentation Strategies - -### Multi-Language Examples - -```markdown -# Create User - -## Python -```python -from myapi import Client - -client = Client(api_key="your_key") -user = client.users.create( - name="John Doe", - email="john@example.com" -) -print(user.id) -``` - -## TypeScript -```typescript -import { Client } from '@myapi/sdk'; - -const client = new Client({ apiKey: 'your_key' }); -const user = await client.users.create({ - name: 'John Doe', - email: 'john@example.com', -}); -console.log(user.id); -``` - -## Go -```go -import "github.com/myapi/sdk-go" - -client := sdk.NewClient("your_key") -user, err := client.Users.Create(ctx, &sdk.CreateUserInput{ - Name: "John Doe", - Email: "john@example.com", -}) -if err != nil { - log.Fatal(err) -} -fmt.Println(user.ID) -``` - -## Ruby -```ruby -require 'myapi' - -client = MyAPI::Client.new(api_key: 'your_key') -user = client.users.create( - name: 'John Doe', - email: 'john@example.com' -) -puts user.id -``` -``` - -### SDK Reference Template - -```markdown -# Users SDK - -## Installation -```bash -npm install @myapi/sdk -``` - -## Configuration -```typescript -import { Client } from '@myapi/sdk'; - -const client = new Client({ - apiKey: process.env.API_KEY, - baseUrl: 'https://api.example.com', // Optional - timeout: 30000, // Optional, default 30s -}); -``` - -## Methods - -### `client.users.create(data)` -Create a new user. - -**Parameters:** -- `data.name` (string, required) - User's display name -- `data.email` (string, required) - User's email address - -**Returns:** Promise - -**Throws:** -- `ValidationError` - Invalid input data -- `ConflictError` - Email already exists -- `AuthenticationError` - Invalid API key - -**Example:** -```typescript -const user = await client.users.create({ - name: 'John Doe', - email: 'john@example.com', -}); -``` - -## Error Handling -```typescript -import { ValidationError, ConflictError } from '@myapi/sdk'; - -try { - await client.users.create(data); -} catch (error) { - if (error instanceof ValidationError) { - console.error('Invalid data:', error.fields); - } else if (error instanceof ConflictError) { - console.error('User already exists'); - } -} -``` -``` - -## Quick Reference - -| Tool | Protocol | Features | -|------|----------|----------| -| Swagger UI | REST | Try-it-out, auth | -| Redoc | REST | Clean, responsive | -| Stoplight | REST | Modern, mock server | -| GraphQL Playground | GraphQL | Explorer, history | -| AsyncAPI Studio | WebSocket | Visual editor | -| grpcui | gRPC | Interactive console | diff --git a/.github/skills/code-documenter/references/python-docstrings.md b/.github/skills/code-documenter/references/python-docstrings.md deleted file mode 100644 index 054a12e2..00000000 --- a/.github/skills/code-documenter/references/python-docstrings.md +++ /dev/null @@ -1,124 +0,0 @@ -# Python Docstrings - -> Reference for: Code Documenter -> Load when: Documenting Python code - -## Google Style (Recommended) - -```python -def calculate_total(items: list[Item], tax_rate: float = 0.0) -> float: - """Calculate total cost including tax. - - Args: - items: List of items to calculate total for. - tax_rate: Tax rate as decimal (e.g., 0.08 for 8%). - - Returns: - Total cost including tax. - - Raises: - ValueError: If tax_rate is negative or items is empty. - - Example: - >>> calculate_total([Item(10), Item(20)], 0.1) - 33.0 - """ -``` - -## NumPy Style - -```python -def calculate_total(items: list[Item], tax_rate: float = 0.0) -> float: - """ - Calculate total cost including tax. - - Parameters - ---------- - items : list[Item] - List of items to calculate total for. - tax_rate : float, optional - Tax rate as decimal (e.g., 0.08 for 8%). Default is 0.0. - - Returns - ------- - float - Total cost including tax. - - Raises - ------ - ValueError - If tax_rate is negative or items is empty. - - Examples - -------- - >>> calculate_total([Item(10), Item(20)], 0.1) - 33.0 - """ -``` - -## Sphinx Style - -```python -def calculate_total(items: list[Item], tax_rate: float = 0.0) -> float: - """Calculate total cost including tax. - - :param items: List of items to calculate total for. - :type items: list[Item] - :param tax_rate: Tax rate as decimal (e.g., 0.08 for 8%). - :type tax_rate: float - :returns: Total cost including tax. - :rtype: float - :raises ValueError: If tax_rate is negative or items is empty. - - .. code-block:: python - - >>> calculate_total([Item(10), Item(20)], 0.1) - 33.0 - """ -``` - -## Class Documentation - -```python -class UserService: - """Service for managing user operations. - - This service handles CRUD operations for users and - integrates with the authentication system. - - Attributes: - db: Database session for queries. - cache: Redis client for caching. - - Example: - >>> service = UserService(db, cache) - >>> user = await service.create_user(data) - """ - - def __init__(self, db: AsyncSession, cache: Redis) -> None: - """Initialize UserService. - - Args: - db: Database session for queries. - cache: Redis client for caching. - """ -``` - -## Quick Reference - -| Style | Args Format | Returns Format | -|-------|-------------|----------------| -| Google | `Args:` block | `Returns:` block | -| NumPy | `Parameters` section | `Returns` section | -| Sphinx | `:param name:` | `:returns:` | - -## Sections Available - -| Section | Google | NumPy | Sphinx | -|---------|--------|-------|--------| -| Parameters | `Args:` | `Parameters` | `:param:` | -| Returns | `Returns:` | `Returns` | `:returns:` | -| Raises | `Raises:` | `Raises` | `:raises:` | -| Examples | `Example:` | `Examples` | `.. code-block::` | -| Notes | `Note:` | `Notes` | `.. note::` | -| Attributes | `Attributes:` | `Attributes` | `:ivar:` | diff --git a/.github/skills/code-documenter/references/typescript-jsdoc.md b/.github/skills/code-documenter/references/typescript-jsdoc.md deleted file mode 100644 index c241c447..00000000 --- a/.github/skills/code-documenter/references/typescript-jsdoc.md +++ /dev/null @@ -1,148 +0,0 @@ -# TypeScript JSDoc - -> Reference for: Code Documenter -> Load when: Documenting TypeScript/JavaScript code - -## Function Documentation - -```typescript -/** - * Calculate total cost including tax. - * - * @param items - List of items to calculate total for - * @param taxRate - Tax rate as decimal (e.g., 0.08 for 8%) - * @returns Total cost including tax - * @throws {Error} If taxRate is negative or items is empty - * - * @example - * ```typescript - * const total = calculateTotal(items, 0.08); - * console.log(total); // 108.00 - * ``` - */ -function calculateTotal(items: Item[], taxRate = 0): number { -``` - -## Class Documentation - -```typescript -/** - * Service for managing user operations. - * - * Handles CRUD operations and integrates with authentication system. - * - * @example - * ```typescript - * const service = new UserService(db, cache); - * const user = await service.create(userData); - * ``` - */ -class UserService { - /** - * Create a new UserService instance. - * - * @param db - Database connection - * @param cache - Redis cache client - */ - constructor( - private readonly db: Database, - private readonly cache: Cache, - ) {} -} -``` - -## Interface Documentation - -```typescript -/** - * User data transfer object. - * - * @interface UserDto - */ -interface UserDto { - /** Unique user identifier */ - id: string; - - /** User's email address (unique) */ - email: string; - - /** User's display name */ - name: string; - - /** Account creation timestamp */ - createdAt: Date; -} -``` - -## Generic Types - -```typescript -/** - * Paginated response wrapper. - * - * @template T - Type of items in the data array - */ -interface PaginatedResponse { - /** Array of items for current page */ - data: T[]; - - /** Total number of items across all pages */ - total: number; - - /** Current page number (1-indexed) */ - page: number; - - /** Number of items per page */ - limit: number; -} -``` - -## Async Functions - -```typescript -/** - * Fetch user by ID from database. - * - * @param id - User's unique identifier - * @returns Promise resolving to user data or null if not found - * @throws {DatabaseError} If connection fails - * - * @async - */ -async function findUserById(id: string): Promise { -``` - -## Quick Reference - -| Tag | Purpose | Example | -|-----|---------|---------| -| `@param` | Parameter description | `@param name - User's name` | -| `@returns` | Return value | `@returns User object` | -| `@throws` | Exception thrown | `@throws {Error} If invalid` | -| `@example` | Usage example | Code block | -| `@see` | Reference link | `@see UserService` | -| `@deprecated` | Mark deprecated | `@deprecated Use v2 instead` | -| `@template` | Generic type param | `@template T - Item type` | -| `@async` | Async function | Mark async | -| `@private` | Private member | Internal use | -| `@readonly` | Read-only property | Cannot modify | - -## Common Patterns - -```typescript -// Optional parameters -/** @param [options] - Optional configuration */ - -// Default values -/** @param [limit=10] - Items per page (default: 10) */ - -// Multiple types -/** @param input - Input value (string or number) */ - -// Callback parameters -/** - * @callback FilterFn - * @param item - Item to filter - * @returns Whether item passes filter - */ -``` diff --git a/.github/skills/code-documenter/references/user-guides-tutorials.md b/.github/skills/code-documenter/references/user-guides-tutorials.md deleted file mode 100644 index ddfb7b63..00000000 --- a/.github/skills/code-documenter/references/user-guides-tutorials.md +++ /dev/null @@ -1,533 +0,0 @@ -# User Guides & Tutorials - -> Reference for: Code Documenter -> Load when: Creating getting started guides, tutorials, troubleshooting docs, end-user documentation - -## Tutorial Structure - -### Progressive Learning Path - -```markdown -# Getting Started with API - -## Prerequisites -Before you begin, ensure you have: -- [ ] Node.js 18+ installed -- [ ] An API key from your dashboard -- [ ] Basic knowledge of REST APIs - -## Quick Start (5 minutes) - -### 1. Install the SDK -```bash -npm install @myapi/sdk -``` - -### 2. Create Your First Request -```typescript -import { Client } from '@myapi/sdk'; - -const client = new Client({ apiKey: 'your_key' }); -const users = await client.users.list(); -console.log(users); -``` - -### 3. Verify It Works -Run the code and you should see a list of users. - -**Expected output:** -```json -{ - "data": [ - { "id": "1", "name": "Alice" }, - { "id": "2", "name": "Bob" } - ], - "total": 2 -} -``` - -## Next Steps -- [Authentication Guide](/docs/auth) - Learn about OAuth and API keys -- [Advanced Queries](/docs/queries) - Filtering, sorting, pagination -- [Error Handling](/docs/errors) - Handle errors gracefully -``` - -### Step-by-Step Tutorial - -```markdown -# Tutorial: Building a User Dashboard - -**What you'll learn:** -- Fetching user data from the API -- Handling pagination -- Displaying data in a table -- Adding real-time updates - -**Time:** 30 minutes -**Level:** Intermediate - -## Step 1: Set Up the Project - -Create a new project: -```bash -mkdir user-dashboard -cd user-dashboard -npm init -y -npm install @myapi/sdk react -``` - -## Step 2: Fetch Users - -Create `src/api/users.ts`: -```typescript -import { Client } from '@myapi/sdk'; - -const client = new Client({ apiKey: process.env.API_KEY }); - -export async function getUsers(page = 1, limit = 20) { - const response = await client.users.list({ page, limit }); - return response; -} -``` - -**What's happening:** -1. We import the SDK client -2. Initialize it with our API key from environment -3. Create a helper function that fetches paginated users - -## Step 3: Create the Component - -Create `src/components/UserTable.tsx`: -```typescript -import { useState, useEffect } from 'react'; -import { getUsers } from '../api/users'; - -export function UserTable() { - const [users, setUsers] = useState([]); - const [loading, setLoading] = useState(true); - - useEffect(() => { - async function fetchData() { - const data = await getUsers(); - setUsers(data.data); - setLoading(false); - } - fetchData(); - }, []); - - if (loading) return
Loading...
; - - return ( - - - - - - - - - {users.map(user => ( - - - - - ))} - -
NameEmail
{user.name}{user.email}
- ); -} -``` - -## Step 4: Test It - -Run your app: -```bash -npm run dev -``` - -You should see a table with user data. - -## Checkpoint -At this point, you have: -- [x] Set up the SDK -- [x] Created an API helper -- [x] Built a user table component -- [ ] Added pagination -- [ ] Added real-time updates - -## Next: Adding Pagination - -[Continue to Step 5 →](/docs/tutorial/step-5) -``` - -## Information Architecture - -### Content Hierarchy - -```markdown -Documentation/ -├── Getting Started/ -│ ├── Quick Start (5 min) -│ ├── Installation -│ ├── Authentication -│ └── First Request -│ -├── Guides/ -│ ├── User Management -│ ├── File Uploads -│ ├── Webhooks -│ └── Rate Limiting -│ -├── API Reference/ -│ ├── Users API -│ ├── Files API -│ └── Webhooks API -│ -├── SDK Documentation/ -│ ├── Python SDK -│ ├── TypeScript SDK -│ └── Go SDK -│ -├── Tutorials/ -│ ├── Build a Dashboard (30 min) -│ ├── Integrate Authentication (45 min) -│ └── Real-time Sync (60 min) -│ -└── Resources/ - ├── Troubleshooting - ├── FAQ - ├── Best Practices - └── Migration Guides -``` - -## Writing Techniques - -### Task-Based Writing - -```markdown -# How to Upload a File - -**Goal:** Upload an image file to your account storage - -**Time:** 5 minutes - -## Steps - -### 1. Prepare the file -Get the file from user input or file system: -```typescript -const file = document.querySelector('input[type="file"]').files[0]; -``` - -### 2. Create form data -```typescript -const formData = new FormData(); -formData.append('file', file); -formData.append('folder', 'avatars'); -``` - -### 3. Upload with the SDK -```typescript -const result = await client.files.upload(formData); -console.log('File URL:', result.url); -``` - -## Common Issues - -**"File too large" error:** -Maximum file size is 10MB. Compress images before uploading. - -**"Invalid file type" error:** -Only .jpg, .png, .gif are allowed. Check the file extension. - -## Related -- [File API Reference](/api/files) -- [Handling Upload Progress](/guides/upload-progress) -``` - -### Progressive Disclosure - -```markdown -# Authentication - -## Basic: API Keys (Recommended for Getting Started) - -API keys are the simplest way to authenticate. - -```typescript -const client = new Client({ apiKey: 'your_key' }); -``` - -**When to use:** Scripts, internal tools, testing - -[Generate an API key →](/dashboard/api-keys) - -
-Advanced: OAuth 2.0 - -For user-facing applications, use OAuth 2.0. - -### Authorization Code Flow - -1. Redirect user to authorization URL: -```typescript -const authUrl = client.oauth.getAuthUrl({ - redirectUri: 'https://yourapp.com/callback', - scopes: ['read:users', 'write:users'], -}); -window.location.href = authUrl; -``` - -2. Handle the callback: -```typescript -const code = new URLSearchParams(window.location.search).get('code'); -const tokens = await client.oauth.exchangeCode(code); -``` - -3. Use the access token: -```typescript -const client = new Client({ accessToken: tokens.access_token }); -``` - -[Full OAuth guide →](/guides/oauth) -
- -
-Enterprise: JWT Tokens - -For service-to-service authentication, use JWTs. - -```typescript -const jwt = createJWT({ - issuer: 'your-service', - subject: 'service-account-id', - privateKey: process.env.PRIVATE_KEY, -}); - -const client = new Client({ jwt }); -``` - -[JWT setup guide →](/guides/jwt) -
-``` - -## Visual Communication - -### Diagram Integration - -```markdown -# System Architecture - -## Request Flow - -```mermaid -sequenceDiagram - participant Client - participant API - participant Database - participant Cache - - Client->>API: POST /users - API->>Cache: Check cache - Cache-->>API: Cache miss - API->>Database: Insert user - Database-->>API: User created - API->>Cache: Store user - API-->>Client: 201 Created -``` - -## Data Model - -```mermaid -erDiagram - USER ||--o{ POST : creates - USER ||--o{ COMMENT : writes - POST ||--o{ COMMENT : has - - USER { - string id PK - string email UK - string name - datetime created_at - } - - POST { - string id PK - string user_id FK - string title - text content - } -``` -``` - -### Screenshot Annotations - -```markdown -# Dashboard Overview - -![Dashboard with numbered annotations](./images/dashboard-annotated.png) - -**Key features:** - -1. **Navigation** - Switch between sections -2. **API Key** - Copy your key (click to reveal) -3. **Usage Stats** - Current month's API calls -4. **Quick Actions** - Generate new key, view docs -5. **Recent Activity** - Last 10 API requests - -## Creating Your First API Key - -1. Click "Generate New Key" (highlighted in green) -2. Enter a description like "Production API" -3. Select permissions (default: all) -4. Click "Create" -5. **Important:** Copy the key immediately - it won't be shown again - -![Create API key dialog](./images/create-key.png) -``` - -## Troubleshooting Guides - -### Problem-Solution Format - -```markdown -# Troubleshooting - -## Authentication Errors - -### "Invalid API key" - -**Symptoms:** -- 401 Unauthorized error -- Error message: "Invalid API key" - -**Causes:** -1. API key was copied incorrectly (extra spaces) -2. API key was revoked -3. Using test key in production environment - -**Solutions:** - -**1. Verify the key:** -```bash -# Check for extra spaces -echo -n "$API_KEY" | wc -c # Should be exactly 32 characters -``` - -**2. Regenerate the key:** -- Go to [dashboard](/dashboard) -- Click "Revoke & Regenerate" -- Update your environment variables - -**3. Check environment:** -```typescript -console.log('Environment:', process.env.NODE_ENV); -console.log('API URL:', client.baseUrl); -``` - -**Still not working?** -[Contact support](/support) with your request ID from the error response. - ---- - -### "Rate limit exceeded" - -**Symptoms:** -- 429 Too Many Requests error -- Requests failing intermittently - -**Immediate fix:** -Wait 60 seconds and retry. - -**Long-term solutions:** - -**1. Implement exponential backoff:** -```typescript -async function retryWithBackoff(fn, maxRetries = 3) { - for (let i = 0; i < maxRetries; i++) { - try { - return await fn(); - } catch (error) { - if (error.status === 429 && i < maxRetries - 1) { - await sleep(Math.pow(2, i) * 1000); - continue; - } - throw error; - } - } -} -``` - -**2. Batch requests:** -Instead of 100 individual requests, use batch endpoints. - -**3. Upgrade your plan:** -[View plans](/pricing) - Higher tiers have increased limits. -``` - -## FAQ Section - -```markdown -# Frequently Asked Questions - -## General - -### What's included in the free tier? -- 1,000 API requests/month -- 1GB storage -- Community support -- All core features - -### How do I upgrade? -Click "Upgrade" in your [dashboard](/dashboard) and select a plan. - -## Technical - -### Can I use this in production? -Yes, the API is production-ready with 99.9% SLA on paid plans. - -### What's the rate limit? -- Free: 10 requests/minute -- Pro: 100 requests/minute -- Enterprise: Custom limits - -### Do you support webhooks? -Yes! See [Webhooks Guide](/guides/webhooks) for setup. - -### Which regions are available? -Currently: US East, US West, EU Central, Asia Pacific. - -## Billing - -### How does billing work? -- Monthly subscription -- Pay-as-you-go for overages -- Cancel anytime - -### What payment methods do you accept? -Credit card, PayPal, wire transfer (annual plans only). - ---- - -**Can't find your answer?** -- [Browse all docs](/docs) -- [Ask the community](https://community.example.com) -- [Contact support](/support) -``` - -## Quick Reference - -| Content Type | Best For | Key Elements | -|-------------|----------|-------------| -| Quick Start | New users (5 min) | Prerequisites, minimal code, verify | -| Tutorial | Learning by doing | Steps, checkpoints, working code | -| How-To Guide | Specific tasks | Goal, steps, troubleshooting | -| Reference | Looking up details | Comprehensive, searchable | -| Explanation | Understanding concepts | Why, not how | - -| Writing Principle | Technique | -|------------------|-----------| -| Clarity | Active voice, short sentences | -| Scannability | Headings, lists, code blocks | -| Completeness | Prerequisites, next steps, related links | -| Accuracy | Test all code, version specifics | diff --git a/.github/skills/pandas-pro/SKILL.md b/.github/skills/pandas-pro/SKILL.md deleted file mode 100644 index 72791106..00000000 --- a/.github/skills/pandas-pro/SKILL.md +++ /dev/null @@ -1,96 +0,0 @@ ---- -name: pandas-pro -description: Use when working with pandas DataFrames, data cleaning, aggregation, merging, or time series analysis. Invoke for data manipulation, missing value handling, groupby operations, or performance optimization. -triggers: - - pandas - - DataFrame - - data manipulation - - data cleaning - - aggregation - - groupby - - merge - - join - - time series - - data wrangling - - pivot table - - data transformation -role: expert -scope: implementation -output-format: code ---- - -# Pandas Pro - -Expert pandas developer specializing in efficient data manipulation, analysis, and transformation workflows with production-grade performance patterns. - -## Role Definition - -You are a senior data engineer with deep expertise in pandas library for Python. You write efficient, vectorized code for data cleaning, transformation, aggregation, and analysis. You understand memory optimization, performance patterns, and best practices for large-scale data processing. - -## When to Use This Skill - -- Loading, cleaning, and transforming tabular data -- Handling missing values and data quality issues -- Performing groupby aggregations and pivot operations -- Merging, joining, and concatenating datasets -- Time series analysis and resampling -- Optimizing pandas code for memory and performance -- Converting between data formats (CSV, Excel, SQL, JSON) - -## Core Workflow - -1. **Assess data structure** - Examine dtypes, memory usage, missing values, data quality -2. **Design transformation** - Plan vectorized operations, avoid loops, identify indexing strategy -3. **Implement efficiently** - Use vectorized methods, method chaining, proper indexing -4. **Validate results** - Check dtypes, shapes, edge cases, null handling -5. **Optimize** - Profile memory usage, apply categorical types, use chunking if needed - -## Reference Guide - -Load detailed guidance based on context: - -| Topic | Reference | Load When | -|-------|-----------|-----------| -| DataFrame Operations | `references/dataframe-operations.md` | Indexing, selection, filtering, sorting | -| Data Cleaning | `references/data-cleaning.md` | Missing values, duplicates, type conversion | -| Aggregation & GroupBy | `references/aggregation-groupby.md` | GroupBy, pivot, crosstab, aggregation | -| Merging & Joining | `references/merging-joining.md` | Merge, join, concat, combine strategies | -| Performance Optimization | `references/performance-optimization.md` | Memory usage, vectorization, chunking | - -## Constraints - -### MUST DO -- Use vectorized operations instead of loops -- Set appropriate dtypes (categorical for low-cardinality strings) -- Check memory usage with `.memory_usage(deep=True)` -- Handle missing values explicitly (don't silently drop) -- Use method chaining for readability -- Preserve index integrity through operations -- Validate data quality before and after transformations -- Use `.copy()` when modifying subsets to avoid SettingWithCopyWarning - -### MUST NOT DO -- Iterate over DataFrame rows with `.iterrows()` unless absolutely necessary -- Use chained indexing (`df['A']['B']`) - use `.loc[]` or `.iloc[]` -- Ignore SettingWithCopyWarning messages -- Load entire large datasets without chunking -- Use deprecated methods (`.ix`, `.append()` - use `pd.concat()`) -- Convert to Python lists for operations possible in pandas -- Assume data is clean without validation - -## Output Templates - -When implementing pandas solutions, provide: -1. Code with vectorized operations and proper indexing -2. Comments explaining complex transformations -3. Memory/performance considerations if dataset is large -4. Data validation checks (dtypes, nulls, shapes) - -## Knowledge Reference - -pandas 2.0+, NumPy, datetime handling, categorical types, MultiIndex, memory optimization, vectorization, method chaining, merge strategies, time series resampling, pivot tables, groupby aggregations - -## Related Skills - -- **Python Pro** - Type hints, testing, Python best practices -- **Data Scientist** - Statistical analysis, visualization, ML workflows diff --git a/.github/skills/pandas-pro/references/aggregation-groupby.md b/.github/skills/pandas-pro/references/aggregation-groupby.md deleted file mode 100644 index 7f2d7163..00000000 --- a/.github/skills/pandas-pro/references/aggregation-groupby.md +++ /dev/null @@ -1,548 +0,0 @@ -# Aggregation and GroupBy - -> Reference for: Pandas Pro -> Load when: GroupBy operations, pivot tables, crosstab, aggregation functions, or summarizing data - ---- - -## Overview - -Aggregation transforms data from individual records to summary statistics. This reference covers GroupBy, pivot tables, crosstab, and advanced aggregation patterns with pandas 2.0+. - ---- - -## GroupBy Fundamentals - -### Basic GroupBy - -```python -import pandas as pd -import numpy as np - -df = pd.DataFrame({ - 'department': ['Eng', 'Eng', 'Sales', 'Sales', 'Eng', 'HR'], - 'team': ['Backend', 'Frontend', 'East', 'West', 'Backend', 'Recruit'], - 'employee': ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve', 'Frank'], - 'salary': [80000, 75000, 65000, 70000, 85000, 60000], - 'years': [5, 3, 7, 4, 6, 2] -}) - -# Single column groupby with single aggregation -avg_salary = df.groupby('department')['salary'].mean() - -# Multiple aggregations -stats = df.groupby('department')['salary'].agg(['mean', 'min', 'max', 'count']) - -# GroupBy multiple columns -grouped = df.groupby(['department', 'team'])['salary'].mean() - -# Reset index to get DataFrame instead of Series -grouped = df.groupby('department')['salary'].mean().reset_index() -``` - -### Multiple Columns, Multiple Aggregations - -```python -# Named aggregation (pandas 2.0+ preferred) -result = df.groupby('department').agg( - avg_salary=('salary', 'mean'), - max_salary=('salary', 'max'), - total_years=('years', 'sum'), - headcount=('employee', 'count'), -) - -# Dictionary syntax (traditional) -result = df.groupby('department').agg({ - 'salary': ['mean', 'max', 'std'], - 'years': ['sum', 'mean'], -}) - -# Flatten multi-level column names -result.columns = ['_'.join(col).strip() for col in result.columns.values] -``` - -### Custom Aggregation Functions - -```python -# Lambda functions -result = df.groupby('department').agg({ - 'salary': lambda x: x.max() - x.min(), # Range - 'years': lambda x: x.quantile(0.75), # 75th percentile -}) - -# Named functions for clarity -def salary_range(x): - return x.max() - x.min() - -def coefficient_of_variation(x): - return x.std() / x.mean() if x.mean() != 0 else 0 - -result = df.groupby('department').agg( - salary_range=('salary', salary_range), - salary_cv=('salary', coefficient_of_variation), -) - -# Multiple custom functions -result = df.groupby('department')['salary'].agg([ - ('range', lambda x: x.max() - x.min()), - ('iqr', lambda x: x.quantile(0.75) - x.quantile(0.25)), - ('median', 'median'), -]) -``` - ---- - -## Transform and Apply - -### Transform - Returns Same Shape - -```python -# Transform returns Series with same index as original -# Useful for adding aggregated values back to original DataFrame - -# Add group mean as new column -df['dept_avg_salary'] = df.groupby('department')['salary'].transform('mean') - -# Normalize within group -df['salary_zscore'] = df.groupby('department')['salary'].transform( - lambda x: (x - x.mean()) / x.std() -) - -# Rank within group -df['salary_rank'] = df.groupby('department')['salary'].transform('rank', ascending=False) - -# Percentage of group total -df['salary_pct'] = df.groupby('department')['salary'].transform( - lambda x: x / x.sum() * 100 -) - -# Fill missing with group mean -df['salary'] = df.groupby('department')['salary'].transform( - lambda x: x.fillna(x.mean()) -) -``` - -### Apply - Flexible Operations - -```python -# Apply runs function on each group DataFrame -def top_n_by_salary(group, n=2): - return group.nlargest(n, 'salary') - -top_earners = df.groupby('department').apply(top_n_by_salary, n=2) - -# Reset index after apply -top_earners = df.groupby('department', group_keys=False).apply( - top_n_by_salary, n=2 -).reset_index(drop=True) - -# Complex group operations -def group_summary(group): - return pd.Series({ - 'headcount': len(group), - 'avg_salary': group['salary'].mean(), - 'top_earner': group.loc[group['salary'].idxmax(), 'employee'], - 'avg_tenure': group['years'].mean(), - }) - -summary = df.groupby('department').apply(group_summary) -``` - -### Filter - Keep/Remove Groups - -```python -# Keep only groups meeting a condition -# Groups with average salary > 70000 -filtered = df.groupby('department').filter(lambda x: x['salary'].mean() > 70000) - -# Groups with more than 2 members -filtered = df.groupby('department').filter(lambda x: len(x) > 2) - -# Combined conditions -filtered = df.groupby('department').filter( - lambda x: (len(x) >= 2) and (x['salary'].mean() > 65000) -) -``` - ---- - -## Pivot Tables - -### Basic Pivot Table - -```python -df = pd.DataFrame({ - 'date': pd.date_range('2024-01-01', periods=6), - 'product': ['A', 'B', 'A', 'B', 'A', 'B'], - 'region': ['East', 'East', 'West', 'West', 'East', 'West'], - 'sales': [100, 150, 120, 180, 90, 200], - 'quantity': [10, 15, 12, 18, 9, 20], -}) - -# Simple pivot -pivot = df.pivot_table( - values='sales', - index='product', - columns='region', - aggfunc='sum' -) - -# Multiple values -pivot = df.pivot_table( - values=['sales', 'quantity'], - index='product', - columns='region', - aggfunc='sum' -) - -# Multiple aggregation functions -pivot = df.pivot_table( - values='sales', - index='product', - columns='region', - aggfunc=['sum', 'mean', 'count'] -) -``` - -### Advanced Pivot Table Options - -```python -# Fill missing values -pivot = df.pivot_table( - values='sales', - index='product', - columns='region', - aggfunc='sum', - fill_value=0 -) - -# Add margins (totals) -pivot = df.pivot_table( - values='sales', - index='product', - columns='region', - aggfunc='sum', - margins=True, - margins_name='Total' -) - -# Multiple index levels -pivot = df.pivot_table( - values='sales', - index=['product', df['date'].dt.month], - columns='region', - aggfunc='sum' -) - -# Observed categories only (for categorical data) -pivot = df.pivot_table( - values='sales', - index='product', - columns='region', - aggfunc='sum', - observed=True # pandas 2.0+ default changed -) -``` - -### Unpivoting (Melt) - -```python -# Wide to long format -wide_df = pd.DataFrame({ - 'product': ['A', 'B'], - 'Q1_sales': [100, 150], - 'Q2_sales': [120, 180], - 'Q3_sales': [90, 200], -}) - -# Melt to long format -long_df = pd.melt( - wide_df, - id_vars=['product'], - value_vars=['Q1_sales', 'Q2_sales', 'Q3_sales'], - var_name='quarter', - value_name='sales' -) - -# Clean quarter column -long_df['quarter'] = long_df['quarter'].str.replace('_sales', '') -``` - ---- - -## Crosstab - -### Basic Crosstab - -```python -df = pd.DataFrame({ - 'gender': ['M', 'F', 'M', 'F', 'M', 'F', 'M', 'M'], - 'department': ['Eng', 'Eng', 'Sales', 'Sales', 'Eng', 'HR', 'HR', 'Eng'], - 'level': ['Senior', 'Junior', 'Senior', 'Senior', 'Junior', 'Junior', 'Senior', 'Junior'], -}) - -# Simple crosstab (counts) -ct = pd.crosstab(df['gender'], df['department']) - -# Normalized crosstab -ct_pct = pd.crosstab(df['gender'], df['department'], normalize='all') # Total -ct_pct = pd.crosstab(df['gender'], df['department'], normalize='index') # Row -ct_pct = pd.crosstab(df['gender'], df['department'], normalize='columns') # Column - -# With margins -ct = pd.crosstab(df['gender'], df['department'], margins=True) - -# Multiple levels -ct = pd.crosstab( - [df['gender'], df['level']], - df['department'] -) -``` - -### Crosstab with Aggregation - -```python -df['salary'] = [80000, 75000, 65000, 70000, 85000, 60000, 72000, 78000] - -# Crosstab with values and aggregation -ct = pd.crosstab( - df['gender'], - df['department'], - values=df['salary'], - aggfunc='mean' -) - -# Multiple aggregations -ct = pd.crosstab( - df['gender'], - df['department'], - values=df['salary'], - aggfunc=['mean', 'sum', 'count'] -) -``` - ---- - -## Window Functions with GroupBy - -### Rolling Aggregations - -```python -df = pd.DataFrame({ - 'date': pd.date_range('2024-01-01', periods=10), - 'product': ['A', 'B'] * 5, - 'sales': [100, 150, 110, 160, 120, 170, 130, 180, 140, 190], -}) - -# Rolling mean within groups -df['rolling_avg'] = df.groupby('product')['sales'].transform( - lambda x: x.rolling(window=3, min_periods=1).mean() -) - -# Expanding aggregations -df['cumulative_sales'] = df.groupby('product')['sales'].transform('cumsum') - -df['expanding_avg'] = df.groupby('product')['sales'].transform( - lambda x: x.expanding().mean() -) - -# Rank within groups -df['sales_rank'] = df.groupby('product')['sales'].rank(method='dense') -``` - -### Shift and Diff - -```python -# Previous value within group -df['prev_sales'] = df.groupby('product')['sales'].shift(1) - -# Next value -df['next_sales'] = df.groupby('product')['sales'].shift(-1) - -# Period-over-period change -df['sales_change'] = df.groupby('product')['sales'].diff() - -# Percentage change -df['sales_pct_change'] = df.groupby('product')['sales'].pct_change() -``` - ---- - -## Common Aggregation Patterns - -### Summary Statistics - -```python -# Comprehensive summary by group -def full_summary(group): - return pd.Series({ - 'count': len(group), - 'mean': group['salary'].mean(), - 'std': group['salary'].std(), - 'min': group['salary'].min(), - 'q25': group['salary'].quantile(0.25), - 'median': group['salary'].median(), - 'q75': group['salary'].quantile(0.75), - 'max': group['salary'].max(), - 'sum': group['salary'].sum(), - }) - -summary = df.groupby('department').apply(full_summary) -``` - -### Top N Per Group - -```python -# Top 2 salaries per department -top_2 = df.groupby('department', group_keys=False).apply( - lambda x: x.nlargest(2, 'salary') -) - -# Using head after sorting -top_2 = df.sort_values('salary', ascending=False).groupby( - 'department', group_keys=False -).head(2) - -# Bottom N -bottom_2 = df.groupby('department', group_keys=False).apply( - lambda x: x.nsmallest(2, 'salary') -) -``` - -### First/Last Per Group - -```python -# First row per group -first = df.groupby('department').first() - -# Last row per group -last = df.groupby('department').last() - -# First row after sorting -first_by_salary = df.sort_values('salary', ascending=False).groupby( - 'department' -).first() - -# Nth row -nth = df.groupby('department').nth(1) # Second row (0-indexed) -``` - -### Cumulative Operations - -```python -# Cumulative sum -df['cum_sales'] = df.groupby('department')['salary'].cumsum() - -# Cumulative max/min -df['cum_max'] = df.groupby('department')['salary'].cummax() -df['cum_min'] = df.groupby('department')['salary'].cummin() - -# Cumulative count -df['cum_count'] = df.groupby('department').cumcount() + 1 - -# Running percentage of total -df['running_pct'] = df.groupby('department')['salary'].transform( - lambda x: x.cumsum() / x.sum() * 100 -) -``` - ---- - -## Performance Tips for GroupBy - -### Efficient GroupBy Operations - -```python -# Pre-sort for faster groupby operations -df = df.sort_values('department') -grouped = df.groupby('department', sort=False) # Already sorted - -# Use observed=True for categorical columns (pandas 2.0+ default) -df['department'] = df['department'].astype('category') -grouped = df.groupby('department', observed=True)['salary'].mean() - -# Avoid apply when possible - use built-in aggregations -# SLOWER: -result = df.groupby('department')['salary'].apply(lambda x: x.sum()) -# FASTER: -result = df.groupby('department')['salary'].sum() - -# Use numba for custom aggregations (if available) -@numba.jit(nopython=True) -def custom_agg(values): - return values.sum() / len(values) -``` - -### Memory-Efficient Aggregation - -```python -# For large DataFrames, compute aggregations separately -groups = df.groupby('department') - -means = groups['salary'].mean() -sums = groups['salary'].sum() -counts = groups.size() - -result = pd.DataFrame({ - 'mean': means, - 'sum': sums, - 'count': counts -}) - -# Avoid creating intermediate large DataFrames -# BAD: Creates full transformed DataFrame -df['z_score'] = (df['salary'] - df.groupby('department')['salary'].transform('mean')) / df.groupby('department')['salary'].transform('std') - -# BETTER: Compute once -group_stats = df.groupby('department')['salary'].agg(['mean', 'std']) -df = df.merge(group_stats, on='department') -df['z_score'] = (df['salary'] - df['mean']) / df['std'] -``` - ---- - -## Best Practices Summary - -1. **Use named aggregation** - Clearer than dictionary syntax -2. **Choose transform vs apply wisely** - Transform for same-shape, apply for flexible -3. **Pre-sort for performance** - Use `sort=False` after sorting -4. **Prefer built-in aggregations** - Faster than lambda/apply -5. **Use observed=True** - Especially for categorical data -6. **Reset index when needed** - Keep DataFrames easier to work with -7. **Validate group counts** - Check for unexpected groups - ---- - -## Anti-Patterns to Avoid - -```python -# BAD: Iterating over groups manually -for name, group in df.groupby('department'): - # process group - pass - -# GOOD: Use vectorized operations -df.groupby('department').agg(...) - -# BAD: Multiple groupby calls -df.groupby('dept')['salary'].mean() -df.groupby('dept')['salary'].sum() -df.groupby('dept')['salary'].count() - -# GOOD: Single groupby, multiple aggs -df.groupby('dept')['salary'].agg(['mean', 'sum', 'count']) - -# BAD: Apply for simple aggregations -df.groupby('dept')['salary'].apply(np.mean) - -# GOOD: Built-in method -df.groupby('dept')['salary'].mean() -``` - ---- - -## Related References - -- `dataframe-operations.md` - Filtering before aggregation -- `merging-joining.md` - Join aggregated results back -- `performance-optimization.md` - Optimize large-scale aggregations diff --git a/.github/skills/pandas-pro/references/data-cleaning.md b/.github/skills/pandas-pro/references/data-cleaning.md deleted file mode 100644 index ede32c2d..00000000 --- a/.github/skills/pandas-pro/references/data-cleaning.md +++ /dev/null @@ -1,503 +0,0 @@ -# Data Cleaning - -> Reference for: Pandas Pro -> Load when: Missing values, duplicates, type conversion, data validation, or data quality issues - ---- - -## Overview - -Data cleaning is critical for reliable analysis. This reference covers handling missing values, duplicates, type conversion, and data validation with pandas 2.0+ patterns. - ---- - -## Missing Values - -### Detecting Missing Values - -```python -import pandas as pd -import numpy as np - -df = pd.DataFrame({ - 'name': ['Alice', 'Bob', None, 'Diana'], - 'age': [25, np.nan, 35, 28], - 'salary': [50000, 60000, np.nan, np.nan], - 'department': ['Eng', '', 'Eng', 'Sales'] -}) - -# Check for any missing values -df.isna().any() # Per column -df.isna().any().any() # Entire DataFrame - -# Count missing values -df.isna().sum() # Per column -df.isna().sum().sum() # Total - -# Percentage of missing values -(df.isna().sum() / len(df) * 100).round(2) - -# Rows with any missing values -df[df.isna().any(axis=1)] - -# Rows with all values present -df[df.notna().all(axis=1)] - -# Missing value heatmap info -missing_info = pd.DataFrame({ - 'missing': df.isna().sum(), - 'percent': (df.isna().sum() / len(df) * 100).round(2), - 'dtype': df.dtypes -}) -``` - -### Handling Missing Values - Dropping - -```python -# Drop rows with any missing value -df_clean = df.dropna() - -# Drop rows where specific columns have missing values -df_clean = df.dropna(subset=['name', 'age']) - -# Drop rows where ALL values are missing -df_clean = df.dropna(how='all') - -# Drop rows with minimum non-null values -df_clean = df.dropna(thresh=3) # Keep rows with at least 3 non-null - -# Drop columns with missing values -df_clean = df.dropna(axis=1) - -# Drop columns with more than 50% missing -threshold = len(df) * 0.5 -df_clean = df.dropna(axis=1, thresh=threshold) -``` - -### Handling Missing Values - Filling - -```python -# Fill with constant value -df['age'] = df['age'].fillna(0) - -# Fill with column mean/median/mode -df['age'] = df['age'].fillna(df['age'].mean()) -df['salary'] = df['salary'].fillna(df['salary'].median()) -df['department'] = df['department'].fillna(df['department'].mode()[0]) - -# Forward fill (use previous value) -df['salary'] = df['salary'].ffill() - -# Backward fill (use next value) -df['salary'] = df['salary'].bfill() - -# Fill with different values per column -fill_values = {'age': 0, 'salary': df['salary'].median(), 'name': 'Unknown'} -df = df.fillna(fill_values) - -# Fill with interpolation (numeric data) -df['salary'] = df['salary'].interpolate(method='linear') - -# Group-specific fill (fill with group mean) -df['salary'] = df.groupby('department')['salary'].transform( - lambda x: x.fillna(x.mean()) -) -``` - -### Handling Empty Strings vs NaN - -```python -# Empty strings are NOT detected as NaN -df['department'].isna().sum() # Won't count '' - -# Replace empty strings with NaN -df['department'] = df['department'].replace('', np.nan) -# Or -df['department'] = df['department'].replace(r'^\s*$', np.nan, regex=True) - -# Replace multiple values with NaN -df = df.replace(['', 'N/A', 'null', 'None', '-'], np.nan) - -# Using na_values when reading files -df = pd.read_csv('file.csv', na_values=['', 'N/A', 'null', 'None', '-']) -``` - ---- - -## Handling Duplicates - -### Detecting Duplicates - -```python -df = pd.DataFrame({ - 'id': [1, 2, 2, 3, 4, 4], - 'name': ['Alice', 'Bob', 'Bob', 'Charlie', 'Diana', 'Diana'], - 'email': ['a@x.com', 'b@x.com', 'b@x.com', 'c@x.com', 'd@x.com', 'd2@x.com'] -}) - -# Check for duplicate rows (all columns) -df.duplicated().sum() - -# Check specific columns -df.duplicated(subset=['id']).sum() -df.duplicated(subset=['name', 'email']).sum() - -# View duplicate rows -df[df.duplicated(keep=False)] # All duplicates -df[df.duplicated(keep='first')] # Duplicates except first occurrence -df[df.duplicated(keep='last')] # Duplicates except last occurrence - -# Count duplicates per key -df.groupby('id').size().loc[lambda x: x > 1] -``` - -### Removing Duplicates - -```python -# Remove duplicate rows (keep first) -df_clean = df.drop_duplicates() - -# Keep last occurrence -df_clean = df.drop_duplicates(keep='last') - -# Remove all duplicates (keep none) -df_clean = df.drop_duplicates(keep=False) - -# Based on specific columns -df_clean = df.drop_duplicates(subset=['id']) -df_clean = df.drop_duplicates(subset=['name', 'email'], keep='last') - -# In-place modification -df.drop_duplicates(inplace=True) -``` - -### Handling Duplicates with Aggregation - -```python -# Instead of dropping, aggregate duplicates -df_agg = df.groupby('id').agg({ - 'name': 'first', - 'email': lambda x: ', '.join(x.unique()) -}).reset_index() - -# Keep row with max/min value -df_best = df.loc[df.groupby('id')['score'].idxmax()] - -# Rank duplicates -df['rank'] = df.groupby('id').cumcount() + 1 -``` - ---- - -## Type Conversion - -### Checking and Converting Types - -```python -# Check current types -df.dtypes -df.info() - -# Convert to specific type -df['age'] = df['age'].astype(int) -df['salary'] = df['salary'].astype(float) -df['name'] = df['name'].astype(str) - -# Safe conversion with errors handling -df['age'] = pd.to_numeric(df['age'], errors='coerce') # Invalid -> NaN -df['age'] = pd.to_numeric(df['age'], errors='ignore') # Keep original if invalid - -# Convert multiple columns -df = df.astype({'age': 'int64', 'salary': 'float64'}) - -# Convert object to string (pandas 2.0+ StringDtype) -df['name'] = df['name'].astype('string') # Nullable string type -``` - -### Datetime Conversion - -```python -df = pd.DataFrame({ - 'date_str': ['2024-01-15', '2024-02-20', 'invalid', '2024-03-10'], - 'timestamp': [1705276800, 1708387200, 1710028800, 1710028800] -}) - -# String to datetime -df['date'] = pd.to_datetime(df['date_str'], errors='coerce') - -# Specify format for faster parsing -df['date'] = pd.to_datetime(df['date_str'], format='%Y-%m-%d', errors='coerce') - -# Unix timestamp to datetime -df['datetime'] = pd.to_datetime(df['timestamp'], unit='s') - -# Extract components -df['year'] = df['date'].dt.year -df['month'] = df['date'].dt.month -df['day_of_week'] = df['date'].dt.day_name() - -# Handle mixed formats -df['date'] = pd.to_datetime(df['date_str'], format='mixed', dayfirst=False) -``` - -### Categorical Conversion - -```python -# Convert to categorical (memory efficient for low cardinality) -df['department'] = df['department'].astype('category') - -# Ordered categorical -df['size'] = pd.Categorical( - df['size'], - categories=['Small', 'Medium', 'Large'], - ordered=True -) - -# Check memory savings -print(f"Object: {df['department'].nbytes}") -df['department'] = df['department'].astype('category') -print(f"Category: {df['department'].nbytes}") -``` - -### Nullable Integer Types (pandas 2.0+) - -```python -# Standard int doesn't support NaN -# Use nullable integer types -df['age'] = df['age'].astype('Int64') # Note capital I - -# All nullable types -df = df.astype({ - 'count': 'Int64', # Nullable integer - 'price': 'Float64', # Nullable float - 'flag': 'boolean', # Nullable boolean - 'name': 'string', # Nullable string -}) - -# Convert with NA handling -df['age'] = pd.array([1, 2, None, 4], dtype='Int64') -``` - ---- - -## String Cleaning - -### Common String Operations - -```python -df = pd.DataFrame({ - 'name': [' Alice ', 'BOB', 'charlie', None, 'Diana Smith'], - 'email': ['ALICE@EXAMPLE.COM', 'bob@test', 'invalid', None, 'diana@example.com'] -}) - -# Strip whitespace -df['name'] = df['name'].str.strip() - -# Case normalization -df['name'] = df['name'].str.lower() -df['name'] = df['name'].str.upper() -df['name'] = df['name'].str.title() # Title Case - -# Replace patterns -df['name'] = df['name'].str.replace(r'\s+', ' ', regex=True) # Multiple spaces to one -df['phone'] = df['phone'].str.replace(r'[^0-9]', '', regex=True) # Keep only digits - -# Extract with regex -df['domain'] = df['email'].str.extract(r'@(.+)$') -df['first_name'] = df['name'].str.extract(r'^(\w+)') - -# Split strings -df[['first', 'last']] = df['name'].str.split(' ', n=1, expand=True) -``` - -### String Validation - -```python -# Check patterns -df['valid_email'] = df['email'].str.match(r'^[\w.]+@[\w.]+\.\w+$', na=False) - -# String length -df['name_length'] = df['name'].str.len() -df['valid_length'] = df['name'].str.len().between(2, 50) - -# Contains check -df['has_domain'] = df['email'].str.contains('@', na=False) -``` - ---- - -## Data Validation - -### Validation Functions - -```python -def validate_dataframe(df: pd.DataFrame) -> dict: - """Comprehensive DataFrame validation.""" - report = { - 'rows': len(df), - 'columns': len(df.columns), - 'duplicates': df.duplicated().sum(), - 'missing_by_column': df.isna().sum().to_dict(), - 'dtypes': df.dtypes.astype(str).to_dict(), - } - return report - -# Range validation -def validate_range(series: pd.Series, min_val, max_val) -> pd.Series: - """Return boolean mask for values in range.""" - return series.between(min_val, max_val) - -df['valid_age'] = validate_range(df['age'], 0, 120) - -# Custom validation -def validate_email(series: pd.Series) -> pd.Series: - """Validate email format.""" - pattern = r'^[\w.+-]+@[\w-]+\.[\w.-]+$' - return series.str.match(pattern, na=False) - -df['valid_email'] = validate_email(df['email']) -``` - -### Schema Validation with pandera - -```python -# Using pandera for schema validation (recommended for production) -import pandera as pa -from pandera import Column, Check - -schema = pa.DataFrameSchema({ - 'name': Column(str, Check.str_length(min_value=1, max_value=100)), - 'age': Column(int, Check.in_range(0, 120)), - 'email': Column(str, Check.str_matches(r'^[\w.+-]+@[\w-]+\.[\w.-]+$')), - 'salary': Column(float, Check.greater_than(0), nullable=True), -}) - -# Validate DataFrame -try: - schema.validate(df) -except pa.errors.SchemaError as e: - print(f"Validation failed: {e}") -``` - ---- - -## Data Cleaning Pipeline - -### Method Chaining Pattern - -```python -def clean_dataframe(df: pd.DataFrame) -> pd.DataFrame: - """Complete data cleaning pipeline using method chaining.""" - return ( - df - # Make a copy - .copy() - # Standardize column names - .rename(columns=lambda x: x.lower().strip().replace(' ', '_')) - # Drop fully empty rows - .dropna(how='all') - # Clean string columns - .assign( - name=lambda x: x['name'].str.strip().str.title(), - email=lambda x: x['email'].str.lower().str.strip(), - ) - # Handle missing values - .fillna({'department': 'Unknown'}) - # Convert types - .astype({'age': 'Int64', 'department': 'category'}) - # Remove duplicates - .drop_duplicates(subset=['email']) - # Reset index - .reset_index(drop=True) - ) - -df_clean = clean_dataframe(df) -``` - -### Pipeline with Validation - -```python -def clean_and_validate( - df: pd.DataFrame, - required_columns: list[str], - unique_columns: list[str] | None = None, -) -> tuple[pd.DataFrame, dict]: - """Clean DataFrame and return validation report.""" - - # Validate required columns exist - missing_cols = set(required_columns) - set(df.columns) - if missing_cols: - raise ValueError(f"Missing required columns: {missing_cols}") - - # Track cleaning stats - stats = { - 'initial_rows': len(df), - 'dropped_empty': 0, - 'dropped_duplicates': 0, - 'filled_missing': {}, - } - - # Clean - df = df.copy() - - # Drop empty rows - before = len(df) - df = df.dropna(how='all') - stats['dropped_empty'] = before - len(df) - - # Handle duplicates - if unique_columns: - before = len(df) - df = df.drop_duplicates(subset=unique_columns) - stats['dropped_duplicates'] = before - len(df) - - stats['final_rows'] = len(df) - - return df, stats -``` - ---- - -## Best Practices Summary - -1. **Always check data quality first** - Use `.info()`, `.describe()`, and missing value analysis -2. **Document cleaning decisions** - Track what was dropped/filled and why -3. **Use nullable types** - `Int64`, `string`, `boolean` for proper NA handling -4. **Validate after cleaning** - Ensure data meets expectations -5. **Use method chaining** - Readable, maintainable cleaning pipelines -6. **Copy before modifying** - Avoid SettingWithCopyWarning -7. **Handle edge cases** - Empty strings, whitespace, invalid formats - ---- - -## Anti-Patterns to Avoid - -```python -# BAD: Dropping NaN without understanding impact -df = df.dropna() # May lose significant data - -# GOOD: Investigate first, then decide -print(f"Missing values: {df.isna().sum()}") -print(f"Rows affected: {df.isna().any(axis=1).sum()}") -# Then make informed decision - -# BAD: Filling without domain knowledge -df['age'] = df['age'].fillna(0) # Age 0 is not valid - -# GOOD: Use appropriate fill strategy -df['age'] = df['age'].fillna(df['age'].median()) - -# BAD: Type conversion without error handling -df['id'] = df['id'].astype(int) # Will fail on NaN or invalid - -# GOOD: Safe conversion -df['id'] = pd.to_numeric(df['id'], errors='coerce').astype('Int64') -``` - ---- - -## Related References - -- `dataframe-operations.md` - Selection and filtering for targeted cleaning -- `aggregation-groupby.md` - Aggregate duplicates instead of dropping -- `performance-optimization.md` - Efficient cleaning of large datasets diff --git a/.github/skills/pandas-pro/references/dataframe-operations.md b/.github/skills/pandas-pro/references/dataframe-operations.md deleted file mode 100644 index 6321f23e..00000000 --- a/.github/skills/pandas-pro/references/dataframe-operations.md +++ /dev/null @@ -1,423 +0,0 @@ -# DataFrame Operations - -> Reference for: Pandas Pro -> Load when: Indexing, selection, filtering, sorting, or basic DataFrame manipulation - ---- - -## Overview - -DataFrame operations form the foundation of pandas work. This reference covers indexing, selection, filtering, and sorting with pandas 2.0+ best practices. - ---- - -## Indexing and Selection - -### Label-Based Selection with `.loc[]` - -Use `.loc[]` for label-based indexing. Always preferred over chained indexing. - -```python -import pandas as pd -import numpy as np - -# Sample DataFrame -df = pd.DataFrame({ - 'name': ['Alice', 'Bob', 'Charlie', 'Diana'], - 'age': [25, 30, 35, 28], - 'salary': [50000, 60000, 70000, 55000], - 'department': ['Engineering', 'Sales', 'Engineering', 'Marketing'] -}, index=['a', 'b', 'c', 'd']) - -# Single value -value = df.loc['a', 'name'] # 'Alice' - -# Single row (returns Series) -row = df.loc['a'] - -# Multiple rows -rows = df.loc[['a', 'c']] - -# Row and column slices (inclusive on both ends) -subset = df.loc['a':'c', 'name':'salary'] - -# Boolean indexing with .loc -adults = df.loc[df['age'] >= 30] - -# Boolean indexing with column selection -adults_names = df.loc[df['age'] >= 30, 'name'] - -# Multiple conditions -engineering_seniors = df.loc[ - (df['department'] == 'Engineering') & (df['age'] >= 30), - ['name', 'salary'] -] -``` - -### Position-Based Selection with `.iloc[]` - -Use `.iloc[]` for integer position-based indexing. - -```python -# Single value by position -value = df.iloc[0, 0] # First row, first column - -# Single row by position -first_row = df.iloc[0] - -# Slice rows (exclusive end, like Python) -first_three = df.iloc[:3] - -# Specific rows and columns by position -subset = df.iloc[[0, 2], [0, 2]] # Rows 0,2 and columns 0,2 - -# Range selection -block = df.iloc[1:3, 0:2] # Rows 1-2, columns 0-1 -``` - -### When to Use `.loc[]` vs `.iloc[]` - -| Scenario | Use | Example | -|----------|-----|---------| -| Known column names | `.loc[]` | `df.loc[:, 'name']` | -| Filter by condition | `.loc[]` | `df.loc[df['age'] > 25]` | -| First/last N rows | `.iloc[]` | `df.iloc[:5]` or `df.iloc[-5:]` | -| Specific row positions | `.iloc[]` | `df.iloc[[0, 5, 10]]` | -| Unknown column order | `.iloc[]` | `df.iloc[:, 0]` | - ---- - -## Filtering DataFrames - -### Boolean Masks - -```python -# Single condition -mask = df['age'] > 25 -filtered = df[mask] - -# Multiple conditions (use parentheses!) -mask = (df['age'] > 25) & (df['salary'] < 65000) -filtered = df[mask] - -# OR conditions -mask = (df['department'] == 'Engineering') | (df['department'] == 'Sales') -filtered = df[mask] - -# NOT condition -mask = ~(df['department'] == 'Marketing') -filtered = df[mask] -``` - -### Using `.query()` for Readable Filters - -```python -# Simple query - more readable for complex conditions -result = df.query('age > 25 and salary < 65000') - -# Using variables with @ -min_age = 25 -result = df.query('age > @min_age') - -# String comparisons -result = df.query('department == "Engineering"') - -# In-list filtering -depts = ['Engineering', 'Sales'] -result = df.query('department in @depts') - -# Complex expressions -result = df.query('(age > 25) and (department != "Marketing")') -``` - -### Using `.isin()` for Multiple Values - -```python -# Filter by multiple values -departments = ['Engineering', 'Sales'] -filtered = df[df['department'].isin(departments)] - -# Negation -filtered = df[~df['department'].isin(departments)] - -# Multiple columns -conditions = { - 'department': ['Engineering', 'Sales'], - 'age': [25, 30, 35] -} -# Filter where department is in list AND age is in list -mask = df['department'].isin(conditions['department']) & df['age'].isin(conditions['age']) -``` - -### String Filtering with `.str` Accessor - -```python -df = pd.DataFrame({ - 'email': ['alice@example.com', 'bob@test.org', 'charlie@example.com'], - 'name': ['Alice Smith', 'Bob Jones', 'Charlie Brown'] -}) - -# Contains -mask = df['email'].str.contains('example') - -# Starts/ends with -mask = df['email'].str.endswith('.com') -mask = df['name'].str.startswith('A') - -# Regex matching -mask = df['email'].str.match(r'^[a-z]+@example\.com$') - -# Case-insensitive -mask = df['name'].str.lower().str.contains('alice') -# Or with case parameter -mask = df['name'].str.contains('alice', case=False) - -# Handle NaN in string columns -mask = df['email'].str.contains('example', na=False) -``` - ---- - -## Sorting - -### Basic Sorting - -```python -# Sort by single column (ascending) -sorted_df = df.sort_values('age') - -# Sort descending -sorted_df = df.sort_values('age', ascending=False) - -# Sort by multiple columns -sorted_df = df.sort_values(['department', 'salary'], ascending=[True, False]) - -# Sort by index -sorted_df = df.sort_index() -sorted_df = df.sort_index(ascending=False) -``` - -### Advanced Sorting - -```python -# Sort with NaN handling -df_with_nan = pd.DataFrame({ - 'name': ['Alice', 'Bob', 'Charlie'], - 'score': [85.0, np.nan, 90.0] -}) - -# NaN at end (default) -sorted_df = df_with_nan.sort_values('score', na_position='last') - -# NaN at beginning -sorted_df = df_with_nan.sort_values('score', na_position='first') - -# Custom sort order using Categorical -order = ['Marketing', 'Sales', 'Engineering'] -df['department'] = pd.Categorical(df['department'], categories=order, ordered=True) -sorted_df = df.sort_values('department') - -# Sort by computed values without adding column -sorted_df = df.iloc[df['name'].str.len().argsort()] -``` - -### In-Place Sorting - -```python -# Modify DataFrame in place -df.sort_values('age', inplace=True) - -# Reset index after sorting -df.sort_values('age', inplace=True) -df.reset_index(drop=True, inplace=True) - -# Or chain -df = df.sort_values('age').reset_index(drop=True) -``` - ---- - -## Column Operations - -### Adding and Modifying Columns - -```python -# Add new column -df['bonus'] = df['salary'] * 0.1 - -# Conditional column with np.where -df['seniority'] = np.where(df['age'] >= 30, 'Senior', 'Junior') - -# Multiple conditions with np.select -conditions = [ - df['age'] < 25, - df['age'] < 35, - df['age'] >= 35 -] -choices = ['Junior', 'Mid', 'Senior'] -df['level'] = np.select(conditions, choices, default='Unknown') - -# Using .assign() for method chaining (returns new DataFrame) -df_new = df.assign( - bonus=lambda x: x['salary'] * 0.1, - total_comp=lambda x: x['salary'] + x['salary'] * 0.1 -) -``` - -### Renaming Columns - -```python -# Rename specific columns -df = df.rename(columns={'name': 'full_name', 'age': 'years'}) - -# Rename all columns with function -df.columns = df.columns.str.lower().str.replace(' ', '_') - -# Using rename with function -df = df.rename(columns=str.upper) -``` - -### Dropping Columns - -```python -# Drop single column -df = df.drop('bonus', axis=1) -# Or -df = df.drop(columns=['bonus']) - -# Drop multiple columns -df = df.drop(columns=['bonus', 'level']) - -# Drop columns by condition -cols_to_drop = [col for col in df.columns if col.startswith('temp_')] -df = df.drop(columns=cols_to_drop) -``` - -### Reordering Columns - -```python -# Explicit order -new_order = ['name', 'department', 'age', 'salary'] -df = df[new_order] - -# Move specific column to front -cols = ['salary'] + [c for c in df.columns if c != 'salary'] -df = df[cols] - -# Using .reindex() -df = df.reindex(columns=['name', 'age', 'salary', 'department']) -``` - ---- - -## Index Operations - -### Setting and Resetting Index - -```python -# Set column as index -df = df.set_index('name') - -# Reset index back to column -df = df.reset_index() - -# Drop index completely -df = df.reset_index(drop=True) - -# Set multiple columns as index (MultiIndex) -df = df.set_index(['department', 'name']) -``` - -### Working with MultiIndex - -```python -# Create MultiIndex DataFrame -df = pd.DataFrame({ - 'department': ['Eng', 'Eng', 'Sales', 'Sales'], - 'team': ['Backend', 'Frontend', 'East', 'West'], - 'headcount': [10, 8, 15, 12] -}).set_index(['department', 'team']) - -# Select from MultiIndex -df.loc['Eng'] # All Eng rows -df.loc[('Eng', 'Backend')] # Specific row - -# Cross-section with .xs() -df.xs('Backend', level='team') # All Backend teams - -# Reset specific level -df.reset_index(level='team') -``` - ---- - -## Copying DataFrames - -### When to Use `.copy()` - -```python -# ALWAYS copy when modifying a subset -subset = df[df['age'] > 25].copy() -subset['new_col'] = 100 # Safe, no SettingWithCopyWarning - -# Without copy - may raise warning or fail silently -# BAD: -# subset = df[df['age'] > 25] -# subset['new_col'] = 100 # SettingWithCopyWarning! - -# Deep copy (default) - copies data -df_copy = df.copy() # or df.copy(deep=True) - -# Shallow copy - shares data, only copies structure -df_shallow = df.copy(deep=False) -``` - ---- - -## Best Practices Summary - -1. **Use `.loc[]` and `.iloc[]`** - Never use chained indexing -2. **Parenthesize conditions** - `(cond1) & (cond2)` not `cond1 & cond2` -3. **Use `.query()` for readability** - Especially with complex filters -4. **Copy before modifying subsets** - Always use `.copy()` -5. **Use vectorized operations** - Avoid row iteration for filtering -6. **Handle NaN explicitly** - Use `na=False` in string operations -7. **Prefer method chaining** - Use `.assign()` for column creation - ---- - -## Anti-Patterns to Avoid - -```python -# BAD: Chained indexing -df['A']['B'] = value # May not work, raises warning - -# GOOD: Use .loc -df.loc[:, ('A', 'B')] = value -# Or for row selection then assignment: -df.loc[df['A'] > 0, 'B'] = value - -# BAD: Iterating for filtering -result = [] -for idx, row in df.iterrows(): - if row['age'] > 25: - result.append(row) - -# GOOD: Boolean indexing -result = df[df['age'] > 25] - -# BAD: Multiple separate assignments -df = df[df['age'] > 25] -df = df[df['salary'] > 50000] - -# GOOD: Combined filter -df = df[(df['age'] > 25) & (df['salary'] > 50000)] -``` - ---- - -## Related References - -- `data-cleaning.md` - After selection, clean the data -- `aggregation-groupby.md` - Group and aggregate filtered data -- `performance-optimization.md` - Optimize filtering on large datasets diff --git a/.github/skills/pandas-pro/references/merging-joining.md b/.github/skills/pandas-pro/references/merging-joining.md deleted file mode 100644 index 7bf33dca..00000000 --- a/.github/skills/pandas-pro/references/merging-joining.md +++ /dev/null @@ -1,599 +0,0 @@ -# Merging and Joining - -> Reference for: Pandas Pro -> Load when: Merge, join, concat, combine DataFrames, or handle relational data - ---- - -## Overview - -Combining DataFrames is essential for working with relational data. This reference covers merge, join, concat, and advanced combination strategies with pandas 2.0+. - ---- - -## Merge (SQL-Style Joins) - -### Basic Merge - -```python -import pandas as pd -import numpy as np - -# Sample DataFrames -employees = pd.DataFrame({ - 'emp_id': [1, 2, 3, 4, 5], - 'name': ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve'], - 'dept_id': [101, 102, 101, 103, 102], -}) - -departments = pd.DataFrame({ - 'dept_id': [101, 102, 104], - 'dept_name': ['Engineering', 'Sales', 'Marketing'], -}) - -# Inner join (default) - only matching rows -result = pd.merge(employees, departments, on='dept_id') - -# Explicit how parameter -result = pd.merge(employees, departments, on='dept_id', how='inner') -``` - -### Join Types - -```python -# Inner join - only matching rows from both -inner = pd.merge(employees, departments, on='dept_id', how='inner') -# Result: 4 rows (emp_id 4 has dept_id 103 which doesn't exist in departments) - -# Left join - all rows from left, matching from right -left = pd.merge(employees, departments, on='dept_id', how='left') -# Result: 5 rows (Diana has NaN for dept_name) - -# Right join - all rows from right, matching from left -right = pd.merge(employees, departments, on='dept_id', how='right') -# Result: 4 rows (Marketing has no employees, but is included) - -# Outer join - all rows from both -outer = pd.merge(employees, departments, on='dept_id', how='outer') -# Result: 6 rows (includes unmatched from both sides) - -# Cross join - cartesian product -cross = pd.merge(employees, departments, how='cross') -# Result: 15 rows (5 employees x 3 departments) -``` - -### Merging on Different Column Names - -```python -employees = pd.DataFrame({ - 'emp_id': [1, 2, 3], - 'name': ['Alice', 'Bob', 'Charlie'], - 'department': [101, 102, 101], -}) - -departments = pd.DataFrame({ - 'id': [101, 102], - 'dept_name': ['Engineering', 'Sales'], -}) - -# Different column names -result = pd.merge( - employees, - departments, - left_on='department', - right_on='id' -) - -# Drop duplicate column after merge -result = result.drop('id', axis=1) -``` - -### Merging on Multiple Columns - -```python -sales = pd.DataFrame({ - 'region': ['East', 'East', 'West', 'West'], - 'product': ['A', 'B', 'A', 'B'], - 'sales': [100, 150, 120, 180], -}) - -targets = pd.DataFrame({ - 'region': ['East', 'East', 'West'], - 'product': ['A', 'B', 'A'], - 'target': [90, 140, 110], -}) - -# Merge on multiple columns -result = pd.merge(sales, targets, on=['region', 'product'], how='left') -``` - -### Merging on Index - -```python -# Set index before merge -employees_idx = employees.set_index('emp_id') -salaries = pd.DataFrame({ - 'emp_id': [1, 2, 3, 4], - 'salary': [80000, 75000, 70000, 65000], -}).set_index('emp_id') - -# Merge on index -result = pd.merge(employees_idx, salaries, left_index=True, right_index=True) - -# Mix of column and index -result = pd.merge( - employees, - salaries, - left_on='emp_id', - right_index=True -) -``` - ---- - -## Handling Duplicate Columns - -### Suffixes - -```python -df1 = pd.DataFrame({ - 'id': [1, 2, 3], - 'value': [10, 20, 30], - 'date': ['2024-01-01', '2024-01-02', '2024-01-03'], -}) - -df2 = pd.DataFrame({ - 'id': [1, 2, 3], - 'value': [100, 200, 300], - 'date': ['2024-02-01', '2024-02-02', '2024-02-03'], -}) - -# Default suffixes -result = pd.merge(df1, df2, on='id') -# Columns: id, value_x, date_x, value_y, date_y - -# Custom suffixes -result = pd.merge(df1, df2, on='id', suffixes=('_jan', '_feb')) -# Columns: id, value_jan, date_jan, value_feb, date_feb -``` - -### Validate Merge Cardinality - -```python -# Validate merge relationships (pandas 2.0+) -# Raises MergeError if validation fails - -# One-to-one: each key appears at most once in both DataFrames -result = pd.merge(df1, df2, on='id', validate='one_to_one') # or '1:1' - -# One-to-many: keys unique in left only -result = pd.merge(employees, salaries, on='emp_id', validate='one_to_many') # or '1:m' - -# Many-to-one: keys unique in right only -result = pd.merge(salaries, employees, on='emp_id', validate='many_to_one') # or 'm:1' - -# Many-to-many: no uniqueness requirement (default) -result = pd.merge(df1, df2, on='id', validate='many_to_many') # or 'm:m' -``` - -### Indicator Column - -```python -# Add indicator column showing source of each row -result = pd.merge( - employees, - departments, - on='dept_id', - how='outer', - indicator=True -) -# _merge column values: 'left_only', 'right_only', 'both' - -# Custom indicator name -result = pd.merge( - employees, - departments, - on='dept_id', - how='outer', - indicator='source' -) - -# Filter by indicator -left_only = result[result['_merge'] == 'left_only'] -both = result[result['_merge'] == 'both'] -``` - ---- - -## Join (Index-Based) - -### DataFrame.join() - -```python -# join() is for index-based joining (simpler syntax) -employees = pd.DataFrame({ - 'name': ['Alice', 'Bob', 'Charlie'], - 'dept_id': [101, 102, 101], -}, index=[1, 2, 3]) - -salaries = pd.DataFrame({ - 'salary': [80000, 75000, 70000], - 'bonus': [5000, 4000, 3500], -}, index=[1, 2, 3]) - -# Join on index -result = employees.join(salaries) - -# Join types (same as merge) -result = employees.join(salaries, how='left') -result = employees.join(salaries, how='outer') -``` - -### Join on Column to Index - -```python -employees = pd.DataFrame({ - 'name': ['Alice', 'Bob', 'Charlie'], - 'dept_id': [101, 102, 101], -}) - -departments = pd.DataFrame({ - 'dept_name': ['Engineering', 'Sales'], -}, index=[101, 102]) - -# Join left column to right index -result = employees.join(departments, on='dept_id') -``` - -### Join Multiple DataFrames - -```python -df1 = pd.DataFrame({'a': [1, 2]}, index=['x', 'y']) -df2 = pd.DataFrame({'b': [3, 4]}, index=['x', 'y']) -df3 = pd.DataFrame({'c': [5, 6]}, index=['x', 'y']) - -# Join multiple at once -result = df1.join([df2, df3]) - -# With suffixes for duplicate columns -result = df1.join([df2, df3], lsuffix='_1', rsuffix='_2') -``` - ---- - -## Concat (Stacking DataFrames) - -### Vertical Concatenation (Row-wise) - -```python -# Stack DataFrames vertically -df1 = pd.DataFrame({ - 'name': ['Alice', 'Bob'], - 'age': [25, 30], -}) - -df2 = pd.DataFrame({ - 'name': ['Charlie', 'Diana'], - 'age': [35, 28], -}) - -# Basic concat (axis=0 is default) -result = pd.concat([df1, df2]) - -# Reset index -result = pd.concat([df1, df2], ignore_index=True) - -# Keep track of source -result = pd.concat([df1, df2], keys=['source1', 'source2']) -# Creates MultiIndex -``` - -### Horizontal Concatenation (Column-wise) - -```python -names = pd.DataFrame({'name': ['Alice', 'Bob', 'Charlie']}) -ages = pd.DataFrame({'age': [25, 30, 35]}) -salaries = pd.DataFrame({'salary': [50000, 60000, 70000]}) - -# Concat columns (axis=1) -result = pd.concat([names, ages, salaries], axis=1) -``` - -### Handling Mismatched Columns - -```python -df1 = pd.DataFrame({ - 'name': ['Alice', 'Bob'], - 'age': [25, 30], -}) - -df2 = pd.DataFrame({ - 'name': ['Charlie', 'Diana'], - 'salary': [70000, 65000], -}) - -# Outer join (default) - include all columns -result = pd.concat([df1, df2]) -# age and salary columns have NaN where not present - -# Inner join - only common columns -result = pd.concat([df1, df2], join='inner') -# Only 'name' column -``` - -### Concat with Verification - -```python -# Verify no index overlap -try: - result = pd.concat([df1, df2], verify_integrity=True) -except ValueError as e: - print(f"Index overlap detected: {e}") - -# Alternative: use ignore_index -result = pd.concat([df1, df2], ignore_index=True) -``` - ---- - -## Combine and Update - -### combine_first() - Fill Gaps - -```python -# Fill NaN values from another DataFrame -df1 = pd.DataFrame({ - 'A': [1, np.nan, 3], - 'B': [np.nan, 2, 3], -}, index=['a', 'b', 'c']) - -df2 = pd.DataFrame({ - 'A': [10, 20, 30], - 'B': [10, 20, 30], -}, index=['a', 'b', 'c']) - -# Fill NaN in df1 with values from df2 -result = df1.combine_first(df2) -# A: [1, 20, 3], B: [10, 2, 3] -``` - -### update() - In-Place Update - -```python -df1 = pd.DataFrame({ - 'A': [1, 2, 3], - 'B': [4, 5, 6], -}, index=['a', 'b', 'c']) - -df2 = pd.DataFrame({ - 'A': [10, 20], - 'B': [40, 50], -}, index=['a', 'b']) - -# Update df1 with values from df2 (in-place) -df1.update(df2) -# df1 now has A: [10, 20, 3], B: [40, 50, 6] - -# Only update where df2 has non-NaN -df1.update(df2, overwrite=False) # Don't overwrite existing values -``` - ---- - -## Advanced Merge Patterns - -### Merge with Aggregation - -```python -# Merge and aggregate in one operation -orders = pd.DataFrame({ - 'order_id': [1, 2, 3, 4], - 'customer_id': [101, 102, 101, 103], - 'amount': [100, 200, 150, 300], -}) - -customers = pd.DataFrame({ - 'customer_id': [101, 102, 103], - 'name': ['Alice', 'Bob', 'Charlie'], -}) - -# Get customer summary -customer_summary = orders.groupby('customer_id').agg( - total_orders=('order_id', 'count'), - total_amount=('amount', 'sum'), -).reset_index() - -# Merge with customer info -result = pd.merge(customers, customer_summary, on='customer_id') -``` - -### Merge Asof (Nearest Match) - -```python -# Merge on nearest key (useful for time series) -trades = pd.DataFrame({ - 'time': pd.to_datetime(['2024-01-01 10:00:01', '2024-01-01 10:00:03', '2024-01-01 10:00:05']), - 'ticker': ['AAPL', 'AAPL', 'AAPL'], - 'price': [150.0, 151.0, 150.5], -}) - -quotes = pd.DataFrame({ - 'time': pd.to_datetime(['2024-01-01 10:00:00', '2024-01-01 10:00:02', '2024-01-01 10:00:04']), - 'ticker': ['AAPL', 'AAPL', 'AAPL'], - 'bid': [149.5, 150.5, 150.0], - 'ask': [150.5, 151.5, 151.0], -}) - -# Merge asof - find nearest quote for each trade -result = pd.merge_asof( - trades.sort_values('time'), - quotes.sort_values('time'), - on='time', - by='ticker', - direction='backward' # Use most recent quote -) -``` - -### Conditional Merge - -```python -# Merge with conditions beyond key equality -# First merge, then filter - -products = pd.DataFrame({ - 'product_id': [1, 2, 3], - 'name': ['Widget', 'Gadget', 'Gizmo'], - 'category': ['A', 'B', 'A'], -}) - -discounts = pd.DataFrame({ - 'category': ['A', 'A', 'B'], - 'min_qty': [10, 50, 20], - 'discount': [0.05, 0.10, 0.08], -}) - -# Cross merge then filter -merged = pd.merge(products, discounts, on='category') -# Then apply quantity-based filtering as needed -``` - ---- - -## Performance Considerations - -### Pre-sorting for Merge - -```python -# Sort keys before merge for better performance -df1 = df1.sort_values('key') -df2 = df2.sort_values('key') - -# Merge sorted DataFrames -result = pd.merge(df1, df2, on='key') -``` - -### Index Alignment - -```python -# Using index for merge is often faster than columns -df1 = df1.set_index('key') -df2 = df2.set_index('key') - -# Join on index -result = df1.join(df2) -``` - -### Memory-Efficient Merge - -```python -# For large DataFrames, reduce memory before merge -# Convert to appropriate types -df1['key'] = df1['key'].astype('int32') # Instead of int64 -df1['category'] = df1['category'].astype('category') - -# Select only needed columns -cols_needed = ['key', 'value1', 'value2'] -result = pd.merge(df1[cols_needed], df2[cols_needed], on='key') -``` - ---- - -## Common Merge Patterns - -### Left Join with Null Check - -```python -# Find unmatched rows after left join -result = pd.merge(employees, departments, on='dept_id', how='left') -unmatched = result[result['dept_name'].isna()] -``` - -### Anti-Join (Rows Not in Other) - -```python -# Find employees NOT in a specific department list -dept_list = [101, 102] - -# Method 1: Using isin -not_in_depts = employees[~employees['dept_id'].isin(dept_list)] - -# Method 2: Using merge with indicator -merged = pd.merge( - employees, - pd.DataFrame({'dept_id': dept_list}), - on='dept_id', - how='left', - indicator=True -) -not_in_depts = merged[merged['_merge'] == 'left_only'] -``` - -### Self-Join - -```python -# Find pairs within same department -employees = pd.DataFrame({ - 'emp_id': [1, 2, 3, 4], - 'name': ['Alice', 'Bob', 'Charlie', 'Diana'], - 'dept_id': [101, 101, 102, 101], -}) - -# Self-join to find pairs -pairs = pd.merge( - employees, - employees, - on='dept_id', - suffixes=('_1', '_2') -) -# Remove self-pairs and duplicates -pairs = pairs[pairs['emp_id_1'] < pairs['emp_id_2']] -``` - ---- - -## Best Practices Summary - -1. **Choose the right join type** - Default inner may drop data -2. **Validate cardinality** - Use `validate` parameter -3. **Use indicator** - Debug unexpected results -4. **Handle duplicates** - Use meaningful suffixes -5. **Pre-sort for performance** - Especially for large DataFrames -6. **Reset index after operations** - Keep DataFrames usable -7. **Check for NaN after join** - Understand unmatched rows - ---- - -## Anti-Patterns to Avoid - -```python -# BAD: Merge without understanding cardinality -result = pd.merge(df1, df2, on='key') # May explode row count - -# GOOD: Validate relationship -result = pd.merge(df1, df2, on='key', validate='one_to_one') - -# BAD: Repeated merges -result = pd.merge(df1, df2, on='key') -result = pd.merge(result, df3, on='key') -result = pd.merge(result, df4, on='key') - -# GOOD: Chain or use reduce -from functools import reduce -dfs = [df1, df2, df3, df4] -result = reduce(lambda left, right: pd.merge(left, right, on='key'), dfs) - -# BAD: Ignoring merge indicators -result = pd.merge(df1, df2, on='key', how='outer') - -# GOOD: Check merge results -result = pd.merge(df1, df2, on='key', how='outer', indicator=True) -print(result['_merge'].value_counts()) -``` - ---- - -## Related References - -- `dataframe-operations.md` - Filter before/after merge -- `aggregation-groupby.md` - Aggregate before merging -- `performance-optimization.md` - Optimize large merges diff --git a/.github/skills/pandas-pro/references/performance-optimization.md b/.github/skills/pandas-pro/references/performance-optimization.md deleted file mode 100644 index aa05f08a..00000000 --- a/.github/skills/pandas-pro/references/performance-optimization.md +++ /dev/null @@ -1,600 +0,0 @@ -# Performance Optimization - -> Reference for: Pandas Pro -> Load when: Memory usage issues, slow operations, large datasets, vectorization, or chunked processing - ---- - -## Overview - -Optimizing pandas performance is critical for production workflows. This reference covers memory optimization, vectorization, chunking, and profiling with pandas 2.0+. - ---- - -## Memory Analysis - -### Checking Memory Usage - -```python -import pandas as pd -import numpy as np - -df = pd.DataFrame({ - 'id': range(1_000_000), - 'name': ['user_' + str(i) for i in range(1_000_000)], - 'category': np.random.choice(['A', 'B', 'C', 'D'], 1_000_000), - 'value': np.random.randn(1_000_000), - 'count': np.random.randint(0, 100, 1_000_000), -}) - -# Basic memory info -print(df.info(memory_usage='deep')) - -# Detailed memory by column -memory_usage = df.memory_usage(deep=True) -print(memory_usage) -print(f"Total: {memory_usage.sum() / 1e6:.2f} MB") - -# Memory as percentage of total -memory_pct = (memory_usage / memory_usage.sum() * 100).round(2) -print(memory_pct) -``` - -### Memory Profiling Function - -```python -def memory_profile(df: pd.DataFrame) -> pd.DataFrame: - """Profile memory usage by column with optimization suggestions.""" - memory_bytes = df.memory_usage(deep=True) - - profile = pd.DataFrame({ - 'dtype': df.dtypes, - 'non_null': df.count(), - 'null_count': df.isna().sum(), - 'unique': df.nunique(), - 'memory_mb': (memory_bytes / 1e6).round(3), - }) - - # Add optimization suggestions - suggestions = [] - for col in df.columns: - dtype = df[col].dtype - nunique = df[col].nunique() - - if dtype == 'object': - if nunique / len(df) < 0.5: # Less than 50% unique - suggestions.append(f"Convert to category (only {nunique} unique)") - else: - suggestions.append("Consider string dtype") - elif dtype == 'int64': - if df[col].max() < 2**31 and df[col].min() >= -2**31: - suggestions.append("Downcast to int32") - if df[col].max() < 2**15 and df[col].min() >= -2**15: - suggestions.append("Downcast to int16") - elif dtype == 'float64': - suggestions.append("Consider float32 if precision allows") - else: - suggestions.append("OK") - - profile['suggestion'] = suggestions - return profile - -print(memory_profile(df)) -``` - ---- - -## Memory Optimization Techniques - -### Downcasting Numeric Types - -```python -# Automatic downcasting for integers -df['count'] = pd.to_numeric(df['count'], downcast='integer') - -# Automatic downcasting for floats -df['value'] = pd.to_numeric(df['value'], downcast='float') - -# Manual downcasting function -def downcast_dtypes(df: pd.DataFrame) -> pd.DataFrame: - """Reduce memory by downcasting numeric types.""" - df = df.copy() - - for col in df.select_dtypes(include=['int']).columns: - df[col] = pd.to_numeric(df[col], downcast='integer') - - for col in df.select_dtypes(include=['float']).columns: - df[col] = pd.to_numeric(df[col], downcast='float') - - return df - -df_optimized = downcast_dtypes(df) -print(f"Before: {df.memory_usage(deep=True).sum() / 1e6:.2f} MB") -print(f"After: {df_optimized.memory_usage(deep=True).sum() / 1e6:.2f} MB") -``` - -### Using Categorical Type - -```python -# Convert low-cardinality string columns to category -# Especially effective when unique values << total rows - -# Before -print(f"Object dtype: {df['category'].memory_usage(deep=True) / 1e6:.2f} MB") - -# After -df['category'] = df['category'].astype('category') -print(f"Category dtype: {df['category'].memory_usage(deep=True) / 1e6:.2f} MB") - -# Automatic conversion for low-cardinality columns -def optimize_categories(df: pd.DataFrame, threshold: float = 0.5) -> pd.DataFrame: - """Convert object columns to category if unique ratio < threshold.""" - df = df.copy() - - for col in df.select_dtypes(include=['object']).columns: - unique_ratio = df[col].nunique() / len(df) - if unique_ratio < threshold: - df[col] = df[col].astype('category') - - return df -``` - -### Sparse Data Types - -```python -# For data with many repeated values (especially zeros/NaN) -sparse_series = pd.arrays.SparseArray([0, 0, 1, 0, 0, 0, 2, 0, 0, 0]) - -# Create sparse DataFrame -df_sparse = pd.DataFrame({ - 'sparse_col': pd.arrays.SparseArray([0] * 9000 + [1] * 1000), - 'dense_col': [0] * 9000 + [1] * 1000, -}) - -print(f"Sparse: {df_sparse['sparse_col'].memory_usage() / 1e6:.4f} MB") -print(f"Dense: {df_sparse['dense_col'].memory_usage() / 1e6:.4f} MB") -``` - -### Nullable Types (pandas 2.0+) - -```python -# Use nullable types for proper NA handling with memory efficiency -df = df.astype({ - 'id': 'Int32', # Nullable int32 - 'count': 'Int16', # Nullable int16 - 'value': 'Float32', # Nullable float32 - 'name': 'string', # Nullable string (more memory efficient) - 'category': 'category', # Categorical -}) - -# Arrow-backed types for even better memory (pandas 2.0+) -df['name'] = df['name'].astype('string[pyarrow]') -df['category'] = df['category'].astype('category') -``` - ---- - -## Vectorization - -### Replace Loops with Vectorized Operations - -```python -# BAD: Row iteration (extremely slow) -result = [] -for idx, row in df.iterrows(): - if row['value'] > 0: - result.append(row['value'] * 2) - else: - result.append(0) -df['result'] = result - -# GOOD: Vectorized with np.where -df['result'] = np.where(df['value'] > 0, df['value'] * 2, 0) - -# GOOD: Vectorized with boolean indexing -df['result'] = 0 -df.loc[df['value'] > 0, 'result'] = df.loc[df['value'] > 0, 'value'] * 2 -``` - -### Multiple Conditions with np.select - -```python -# BAD: Nested if-else in apply -def categorize(row): - if row['value'] < -1: - return 'very_low' - elif row['value'] < 0: - return 'low' - elif row['value'] < 1: - return 'medium' - else: - return 'high' - -df['category'] = df.apply(categorize, axis=1) # SLOW! - -# GOOD: Vectorized with np.select -conditions = [ - df['value'] < -1, - df['value'] < 0, - df['value'] < 1, -] -choices = ['very_low', 'low', 'medium'] -df['category'] = np.select(conditions, choices, default='high') -``` - -### String Operations - Vectorized - -```python -# BAD: Apply for string operations -df['upper_name'] = df['name'].apply(lambda x: x.upper()) - -# GOOD: Vectorized string methods -df['upper_name'] = df['name'].str.upper() - -# Combine multiple string operations -df['processed'] = ( - df['name'] - .str.strip() - .str.lower() - .str.replace(r'\s+', '_', regex=True) -) -``` - -### Avoid apply() When Possible - -```python -# BAD: apply for row-wise calculation -df['total'] = df.apply(lambda row: row['a'] + row['b'] + row['c'], axis=1) - -# GOOD: Direct vectorized operation -df['total'] = df['a'] + df['b'] + df['c'] - -# BAD: apply for element-wise operation -df['squared'] = df['value'].apply(lambda x: x ** 2) - -# GOOD: Vectorized -df['squared'] = df['value'] ** 2 - -# When apply IS appropriate: complex custom logic -def complex_calculation(row): - # Multiple dependencies and conditional logic - if row['type'] == 'A': - return row['value'] * row['multiplier'] + row['offset'] - else: - return row['value'] / row['divisor'] - row['adjustment'] - -# Consider rewriting as vectorized if performance critical -``` - ---- - -## Chunked Processing - -### Reading Large Files in Chunks - -```python -# Read CSV in chunks -chunk_size = 100_000 -chunks = [] - -for chunk in pd.read_csv('large_file.csv', chunksize=chunk_size): - # Process each chunk - processed = chunk[chunk['value'] > 0] # Filter - processed = processed.groupby('category')['value'].sum() # Aggregate - chunks.append(processed) - -# Combine results -result = pd.concat(chunks).groupby(level=0).sum() -``` - -### Chunked Processing Function - -```python -def process_large_csv( - filepath: str, - chunk_size: int = 100_000, - filter_func=None, - agg_func=None, -) -> pd.DataFrame: - """Process large CSV files in chunks.""" - results = [] - - for chunk in pd.read_csv(filepath, chunksize=chunk_size): - # Apply filter if provided - if filter_func: - chunk = filter_func(chunk) - - # Apply aggregation if provided - if agg_func: - chunk = agg_func(chunk) - - results.append(chunk) - - # Combine results - combined = pd.concat(results, ignore_index=True) - - # Re-aggregate if needed - if agg_func: - combined = agg_func(combined) - - return combined - -# Usage -result = process_large_csv( - 'large_file.csv', - chunk_size=50_000, - filter_func=lambda df: df[df['value'] > 0], - agg_func=lambda df: df.groupby('category').agg({'value': 'sum'}), -) -``` - -### Memory-Efficient Iteration - -```python -# When you must iterate, use itertuples (not iterrows) -# itertuples is 10-100x faster than iterrows - -# BAD: iterrows -for idx, row in df.iterrows(): - process(row['name'], row['value']) - -# BETTER: itertuples -for row in df.itertuples(): - process(row.name, row.value) # Access as attributes - -# BEST: Vectorized operations (avoid iteration entirely) -``` - ---- - -## Query Optimization - -### Efficient Filtering - -```python -# Order matters - filter early, compute late -# BAD: Compute on all rows, then filter -df['expensive_calc'] = df['a'] * df['b'] + np.sin(df['c']) -result = df[df['category'] == 'A'] - -# GOOD: Filter first, compute on subset -mask = df['category'] == 'A' -result = df[mask].copy() -result['expensive_calc'] = result['a'] * result['b'] + np.sin(result['c']) -``` - -### Using query() for Performance - -```python -# query() can be faster for large DataFrames (uses numexpr) -# Traditional boolean indexing -result = df[(df['value'] > 0) & (df['category'] == 'A')] - -# query() syntax (faster for large data) -result = df.query('value > 0 and category == "A"') - -# With variables -threshold = 0 -cat = 'A' -result = df.query('value > @threshold and category == @cat') -``` - -### eval() for Complex Expressions - -```python -# eval() uses numexpr for faster computation -# Standard pandas -df['result'] = df['a'] + df['b'] * df['c'] - df['d'] - -# Using eval (faster for large DataFrames) -df['result'] = pd.eval('df.a + df.b * df.c - df.d') - -# In-place with inplace parameter -df.eval('result = a + b * c - d', inplace=True) -``` - ---- - -## GroupBy Optimization - -### Pre-sort for Faster GroupBy - -```python -# Sort by groupby column first -df = df.sort_values('category') - -# Use sort=False since already sorted -result = df.groupby('category', sort=False)['value'].mean() -``` - -### Use Built-in Aggregations - -```python -# BAD: Custom function via apply -result = df.groupby('category')['value'].apply(lambda x: x.mean()) - -# GOOD: Built-in aggregation -result = df.groupby('category')['value'].mean() - -# Built-in aggregations available: -# sum, mean, median, min, max, std, var, count, first, last, nth -# size, sem, prod, cumsum, cummax, cummin, cumprod -``` - -### Observed Categories - -```python -# For categorical columns, use observed=True (pandas 2.0+ default) -df['category'] = df['category'].astype('category') - -# Avoid computing for unobserved categories -result = df.groupby('category', observed=True)['value'].mean() -``` - ---- - -## I/O Optimization - -### Efficient File Formats - -```python -# Parquet - best for analytical workloads -df.to_parquet('data.parquet', compression='snappy') -df = pd.read_parquet('data.parquet') - -# Feather - best for pandas interchange -df.to_feather('data.feather') -df = pd.read_feather('data.feather') - -# CSV with optimizations -df.to_csv('data.csv', index=False) -df = pd.read_csv( - 'data.csv', - dtype={'category': 'category', 'count': 'int32'}, - usecols=['id', 'category', 'value'], # Only needed columns - nrows=10000, # Limit rows for testing -) -``` - -### Specify dtypes When Reading - -```python -# Specify dtypes upfront to avoid inference overhead -dtypes = { - 'id': 'int32', - 'name': 'string', - 'category': 'category', - 'value': 'float32', - 'count': 'int16', -} - -df = pd.read_csv('data.csv', dtype=dtypes) - -# Parse dates efficiently -df = pd.read_csv( - 'data.csv', - dtype=dtypes, - parse_dates=['date_column'], - date_format='%Y-%m-%d', # Explicit format is faster -) -``` - ---- - -## Profiling and Benchmarking - -### Timing Operations - -```python -import time - -# Simple timing -start = time.time() -result = df.groupby('category')['value'].mean() -elapsed = time.time() - start -print(f"Elapsed: {elapsed:.4f} seconds") - -# Using %%timeit in Jupyter -# %%timeit -# df.groupby('category')['value'].mean() -``` - -### Memory Profiling - -```python -# Track memory before/after -import tracemalloc - -tracemalloc.start() - -# Your operation -df_result = df.groupby('category').agg({'value': 'sum'}) - -current, peak = tracemalloc.get_traced_memory() -print(f"Current memory: {current / 1e6:.2f} MB") -print(f"Peak memory: {peak / 1e6:.2f} MB") - -tracemalloc.stop() -``` - -### Comparison Template - -```python -def benchmark_operations(df: pd.DataFrame, operations: dict, n_runs: int = 5): - """Benchmark multiple operations.""" - results = {} - - for name, func in operations.items(): - times = [] - for _ in range(n_runs): - start = time.time() - func(df) - times.append(time.time() - start) - - results[name] = { - 'mean': np.mean(times), - 'std': np.std(times), - 'min': np.min(times), - } - - return pd.DataFrame(results).T - -# Usage -operations = { - 'iterrows': lambda df: [row['value'] for _, row in df.iterrows()], - 'itertuples': lambda df: [row.value for row in df.itertuples()], - 'vectorized': lambda df: df['value'].tolist(), -} - -benchmark_results = benchmark_operations(df.head(10000), operations) -print(benchmark_results) -``` - ---- - -## Best Practices Summary - -1. **Profile first** - Identify actual bottlenecks before optimizing -2. **Use appropriate dtypes** - int32/float32/category save memory -3. **Vectorize everything** - Avoid loops and apply when possible -4. **Filter early** - Reduce data before expensive operations -5. **Chunk large files** - Process in manageable pieces -6. **Use efficient file formats** - Parquet/Feather over CSV -7. **Leverage built-in methods** - Faster than custom functions - ---- - -## Performance Checklist - -Before deploying pandas code: - -- [ ] Memory profiled with `memory_usage(deep=True)` -- [ ] Dtypes optimized (downcast, categorical) -- [ ] No iterrows/itertuples in hot paths -- [ ] GroupBy uses built-in aggregations -- [ ] Large files processed in chunks -- [ ] Filters applied before computations -- [ ] Appropriate file format used -- [ ] Benchmarked with representative data size - ---- - -## Anti-Patterns Summary - -| Anti-Pattern | Alternative | -|--------------|-------------| -| `iterrows()` for computation | Vectorized operations | -| `apply(lambda)` for simple ops | Built-in methods | -| Loading entire large file | Chunked reading | -| String columns with low cardinality | Category dtype | -| int64 for small integers | int32/int16 | -| Multiple separate filters | Combined boolean mask | -| Repeated groupby calls | Single groupby with multiple aggs | - ---- - -## Related References - -- `dataframe-operations.md` - Efficient indexing and filtering -- `aggregation-groupby.md` - Optimized aggregation patterns -- `merging-joining.md` - Efficient merge strategies diff --git a/.github/skills/python-pro/SKILL.md b/.github/skills/python-pro/SKILL.md deleted file mode 100644 index d9ad569c..00000000 --- a/.github/skills/python-pro/SKILL.md +++ /dev/null @@ -1,92 +0,0 @@ ---- -name: python-pro -description: Use when building Python 3.11+ applications requiring type safety, async programming, or production-grade patterns. Invoke for type hints, pytest, async/await, dataclasses, mypy configuration. -triggers: - - Python development - - type hints - - async Python - - pytest - - mypy - - dataclasses - - Python best practices - - Pythonic code -role: specialist -scope: implementation -output-format: code ---- - -# Python Pro - -Senior Python developer with 10+ years experience specializing in type-safe, async-first, production-ready Python 3.11+ code. - -## Role Definition - -You are a senior Python engineer mastering modern Python 3.11+ and its ecosystem. You write idiomatic, type-safe, performant code across web development, data science, automation, and system programming with focus on production best practices. - -## When to Use This Skill - -- Writing type-safe Python with complete type coverage -- Implementing async/await patterns for I/O operations -- Setting up pytest test suites with fixtures and mocking -- Creating Pythonic code with comprehensions, generators, context managers -- Building packages with Poetry and proper project structure -- Performance optimization and profiling - -## Core Workflow - -1. **Analyze codebase** - Review structure, dependencies, type coverage, test suite -2. **Design interfaces** - Define protocols, dataclasses, type aliases -3. **Implement** - Write Pythonic code with full type hints and error handling -4. **Test** - Create comprehensive pytest suite with >90% coverage -5. **Validate** - Run mypy, black, ruff; ensure quality standards met - -## Reference Guide - -Load detailed guidance based on context: - -| Topic | Reference | Load When | -|-------|-----------|-----------| -| Type System | `references/type-system.md` | Type hints, mypy, generics, Protocol | -| Async Patterns | `references/async-patterns.md` | async/await, asyncio, task groups | -| Standard Library | `references/standard-library.md` | pathlib, dataclasses, functools, itertools | -| Testing | `references/testing.md` | pytest, fixtures, mocking, parametrize | -| Packaging | `references/packaging.md` | poetry, pip, pyproject.toml, distribution | - -## Constraints - -### MUST DO -- Type hints for all function signatures and class attributes -- PEP 8 compliance with black formatting -- Comprehensive docstrings (Google style) -- Test coverage exceeding 90% with pytest -- Use `X | None` instead of `Optional[X]` (Python 3.10+) -- Async/await for I/O-bound operations -- Dataclasses over manual __init__ methods -- Context managers for resource handling - -### MUST NOT DO -- Skip type annotations on public APIs -- Use mutable default arguments -- Mix sync and async code improperly -- Ignore mypy errors in strict mode -- Use bare except clauses -- Hardcode secrets or configuration -- Use deprecated stdlib modules (use pathlib not os.path) - -## Output Templates - -When implementing Python features, provide: -1. Module file with complete type hints -2. Test file with pytest fixtures -3. Type checking confirmation (mypy --strict passes) -4. Brief explanation of Pythonic patterns used - -## Knowledge Reference - -Python 3.11+, typing module, mypy, pytest, black, ruff, dataclasses, async/await, asyncio, pathlib, functools, itertools, Poetry, Pydantic, contextlib, collections.abc, Protocol - -## Related Skills - -- **FastAPI Expert** - Async Python APIs -- **Data Science Pro** - NumPy, Pandas, ML -- **DevOps Engineer** - Python automation and tooling diff --git a/.github/skills/python-pro/references/async-patterns.md b/.github/skills/python-pro/references/async-patterns.md deleted file mode 100644 index 6160247c..00000000 --- a/.github/skills/python-pro/references/async-patterns.md +++ /dev/null @@ -1,359 +0,0 @@ -# Async Programming Patterns - -> Reference for: Python Pro -> Load when: async/await, asyncio, concurrent operations, task groups - -## Basic Async/Await - -```python -import asyncio -from collections.abc import Coroutine - -# Basic async function -async def fetch_data(url: str) -> dict[str, str]: - await asyncio.sleep(1) # Simulate I/O - return {"url": url, "status": "ok"} - -# Running async code -async def main() -> None: - result = await fetch_data("https://api.example.com") - print(result) - -if __name__ == "__main__": - asyncio.run(main()) - -# Multiple concurrent operations -async def fetch_all(urls: list[str]) -> list[dict[str, str]]: - tasks = [fetch_data(url) for url in urls] - return await asyncio.gather(*tasks) - -# Error handling with gather -async def safe_fetch_all(urls: list[str]) -> list[dict[str, str] | None]: - tasks = [fetch_data(url) for url in urls] - results = await asyncio.gather(*tasks, return_exceptions=True) - return [r if not isinstance(r, Exception) else None for r in results] -``` - -## Task Groups (Python 3.11+) - -```python -from asyncio import TaskGroup - -# Task groups for structured concurrency -async def process_batch(items: list[int]) -> list[int]: - results: list[int] = [] - - async with TaskGroup() as tg: - tasks = [tg.create_task(process_item(item)) for item in items] - - # All tasks complete before this line - return [task.result() for task in tasks] - -# Error handling with TaskGroup -async def robust_processing(items: list[str]) -> tuple[list[str], list[Exception]]: - results: list[str] = [] - errors: list[Exception] = [] - - try: - async with TaskGroup() as tg: - for item in items: - tg.create_task(process_item_safe(item)) - except ExceptionGroup as eg: - for exc in eg.exceptions: - errors.append(exc) - - return results, errors -``` - -## Async Context Managers - -```python -from typing import Self -from collections.abc import AsyncIterator - -class AsyncDatabaseConnection: - def __init__(self, url: str) -> None: - self.url = url - self._conn: Connection | None = None - - async def __aenter__(self) -> Self: - self._conn = await connect(self.url) - return self - - async def __aexit__( - self, - exc_type: type[BaseException] | None, - exc_val: BaseException | None, - exc_tb: Any, - ) -> None: - if self._conn: - await self._conn.close() - - async def query(self, sql: str) -> list[dict[str, Any]]: - if not self._conn: - raise RuntimeError("Not connected") - return await self._conn.execute(sql) - -# Usage -async def get_users() -> list[dict[str, Any]]: - async with AsyncDatabaseConnection("postgresql://...") as db: - return await db.query("SELECT * FROM users") - -# Async context manager with contextlib -from contextlib import asynccontextmanager - -@asynccontextmanager -async def get_db_session() -> AsyncIterator[Session]: - session = await create_session() - try: - yield session - await session.commit() - except Exception: - await session.rollback() - raise - finally: - await session.close() -``` - -## Async Generators - -```python -from collections.abc import AsyncIterator - -# Async generator for streaming data -async def read_lines(filepath: str) -> AsyncIterator[str]: - async with aiofiles.open(filepath) as f: - async for line in f: - yield line.strip() - -# Process stream -async def process_file(filepath: str) -> int: - count = 0 - async for line in read_lines(filepath): - await process_line(line) - count += 1 - return count - -# Async generator with cleanup -async def fetch_paginated(url: str) -> AsyncIterator[dict[str, Any]]: - page = 1 - session = await create_session() - try: - while True: - data = await session.get(f"{url}?page={page}") - if not data: - break - yield data - page += 1 - finally: - await session.close() -``` - -## Async Comprehensions - -```python -# Async list comprehension -async def fetch_all_users(user_ids: list[int]) -> list[User]: - return [user async for user in fetch_users(user_ids)] - -# Async dict comprehension -async def build_user_map(user_ids: list[int]) -> dict[int, User]: - return { - user.id: user - async for user in fetch_users(user_ids) - } - -# Conditional async comprehension -async def get_active_users(user_ids: list[int]) -> list[User]: - return [ - user - async for user in fetch_users(user_ids) - if user.is_active - ] -``` - -## Synchronization Primitives - -```python -import asyncio - -# Lock for critical sections -class SharedResource: - def __init__(self) -> None: - self._lock = asyncio.Lock() - self._data: dict[str, Any] = {} - - async def update(self, key: str, value: Any) -> None: - async with self._lock: - # Critical section - current = self._data.get(key, 0) - await asyncio.sleep(0.1) # Simulate processing - self._data[key] = current + value - -# Semaphore for rate limiting -class RateLimiter: - def __init__(self, max_concurrent: int) -> None: - self._semaphore = asyncio.Semaphore(max_concurrent) - - async def process(self, item: str) -> str: - async with self._semaphore: - return await expensive_operation(item) - -# Event for coordination -class AsyncWorker: - def __init__(self) -> None: - self._ready = asyncio.Event() - self._shutdown = asyncio.Event() - - async def start(self) -> None: - # Initialization - await self._initialize() - self._ready.set() - - # Wait for shutdown - await self._shutdown.wait() - - async def wait_ready(self) -> None: - await self._ready.wait() - - def stop(self) -> None: - self._shutdown.set() -``` - -## Async Queue Patterns - -```python -from asyncio import Queue - -# Producer-consumer pattern -async def producer(queue: Queue[int], n: int) -> None: - for i in range(n): - await queue.put(i) - await asyncio.sleep(0.1) - -async def consumer(queue: Queue[int], name: str) -> None: - while True: - item = await queue.get() - try: - await process_item(item) - finally: - queue.task_done() - -async def run_pipeline(num_items: int, num_workers: int) -> None: - queue: Queue[int] = Queue(maxsize=10) - - # Start producer and consumers - async with TaskGroup() as tg: - tg.create_task(producer(queue, num_items)) - for i in range(num_workers): - tg.create_task(consumer(queue, f"worker-{i}")) - - # Wait for all items to be processed - await queue.join() -``` - -## Async Timeouts - -```python -# Timeout for single operation -async def fetch_with_timeout(url: str, timeout: float) -> dict[str, Any]: - try: - async with asyncio.timeout(timeout): - return await fetch_data(url) - except TimeoutError: - return {"error": "timeout"} - -# Timeout for multiple operations -async def fetch_all_with_timeout( - urls: list[str], - timeout: float -) -> list[dict[str, Any] | None]: - try: - async with asyncio.timeout(timeout): - return await fetch_all(urls) - except TimeoutError: - return [None] * len(urls) -``` - -## Background Tasks - -```python -from asyncio import create_task, Task - -class BackgroundTaskManager: - def __init__(self) -> None: - self._tasks: set[Task[None]] = set() - - def create_task(self, coro: Coroutine[None, None, None]) -> Task[None]: - task = create_task(coro) - self._tasks.add(task) - task.add_done_callback(self._tasks.discard) - return task - - async def shutdown(self) -> None: - # Cancel all background tasks - for task in self._tasks: - task.cancel() - # Wait for cancellation - await asyncio.gather(*self._tasks, return_exceptions=True) - -# Usage -manager = BackgroundTaskManager() -manager.create_task(background_job()) -``` - -## Async Iteration Protocol - -```python -class AsyncRange: - def __init__(self, start: int, end: int) -> None: - self.start = start - self.end = end - self.current = start - - def __aiter__(self) -> Self: - return self - - async def __anext__(self) -> int: - if self.current >= self.end: - raise StopAsyncIteration - await asyncio.sleep(0.1) # Simulate async work - value = self.current - self.current += 1 - return value - -# Usage -async for i in AsyncRange(0, 5): - print(i) -``` - -## Mixing Sync and Async - -```python -from concurrent.futures import ThreadPoolExecutor -import functools - -# Run sync code in executor -async def run_in_executor(func: Callable[..., T], *args: Any) -> T: - loop = asyncio.get_running_loop() - return await loop.run_in_executor(None, func, *args) - -# Run async code from sync context -def sync_wrapper(coro: Coroutine[None, None, T]) -> T: - loop = asyncio.new_event_loop() - try: - return loop.run_until_complete(coro) - finally: - loop.close() - -# Async wrapper for sync function -def to_async(func: Callable[..., T]) -> Callable[..., Coroutine[None, None, T]]: - @functools.wraps(func) - async def wrapper(*args: Any, **kwargs: Any) -> T: - loop = asyncio.get_running_loop() - return await loop.run_in_executor( - None, - functools.partial(func, *args, **kwargs) - ) - return wrapper -``` diff --git a/.github/skills/python-pro/references/packaging.md b/.github/skills/python-pro/references/packaging.md deleted file mode 100644 index bc7e609a..00000000 --- a/.github/skills/python-pro/references/packaging.md +++ /dev/null @@ -1,463 +0,0 @@ -# Python Packaging and Project Setup - -> Reference for: Python Pro -> Load when: poetry, pip, pyproject.toml, package distribution, virtual environments - -## Project Structure - -``` -myproject/ -├── pyproject.toml # Project metadata and dependencies -├── README.md # Project description -├── .gitignore # Git ignore patterns -├── .python-version # Python version for pyenv -├── src/ -│ └── myproject/ -│ ├── __init__.py # Package initialization -│ ├── py.typed # PEP 561 type marker -│ ├── core.py # Core functionality -│ └── utils.py # Utilities -├── tests/ -│ ├── __init__.py -│ ├── conftest.py # Pytest configuration -│ └── test_core.py # Tests -└── docs/ - └── index.md # Documentation -``` - -## Pyproject.toml Configuration - -```toml -[build-system] -requires = ["hatchling"] -build-backend = "hatchling.build" - -[project] -name = "myproject" -version = "0.1.0" -description = "A Python project" -readme = "README.md" -requires-python = ">=3.11" -license = {text = "MIT"} -authors = [ - {name = "Your Name", email = "you@example.com"} -] -keywords = ["python", "package"] -classifiers = [ - "Development Status :: 4 - Beta", - "Intended Audience :: Developers", - "License :: OSI Approved :: MIT License", - "Programming Language :: Python :: 3.11", - "Programming Language :: Python :: 3.12", - "Typing :: Typed", -] - -dependencies = [ - "requests>=2.31.0", - "pydantic>=2.5.0", -] - -[project.optional-dependencies] -dev = [ - "pytest>=7.4.0", - "pytest-cov>=4.1.0", - "mypy>=1.7.0", - "black>=23.11.0", - "ruff>=0.1.6", -] -docs = [ - "mkdocs>=1.5.0", - "mkdocs-material>=9.4.0", -] - -[project.scripts] -myproject = "myproject.cli:main" - -[project.urls] -Homepage = "https://github.com/username/myproject" -Documentation = "https://myproject.readthedocs.io" -Repository = "https://github.com/username/myproject" -Changelog = "https://github.com/username/myproject/blob/main/CHANGELOG.md" - -# Tool configurations -[tool.black] -line-length = 100 -target-version = ["py311"] -include = '\.pyi?$' - -[tool.ruff] -line-length = 100 -target-version = "py311" -select = [ - "E", # pycodestyle errors - "W", # pycodestyle warnings - "F", # pyflakes - "I", # isort - "B", # flake8-bugbear - "C4", # flake8-comprehensions - "UP", # pyupgrade -] -ignore = [] - -[tool.ruff.per-file-ignores] -"__init__.py" = ["F401"] # Ignore unused imports in __init__.py - -[tool.mypy] -python_version = "3.11" -strict = true -warn_return_any = true -warn_unused_configs = true -disallow_untyped_defs = true - -[[tool.mypy.overrides]] -module = "third_party.*" -ignore_missing_imports = true - -[tool.pytest.ini_options] -minversion = "7.0" -addopts = [ - "-ra", - "--strict-markers", - "--strict-config", - "--cov=myproject", - "--cov-report=term-missing", - "--cov-report=html", -] -testpaths = ["tests"] -pythonpath = ["src"] - -[tool.coverage.run] -source = ["src"] -branch = true - -[tool.coverage.report] -exclude_lines = [ - "pragma: no cover", - "def __repr__", - "raise AssertionError", - "raise NotImplementedError", - "if __name__ == .__main__.:", - "if TYPE_CHECKING:", -] -``` - -## Poetry Project Management - -```toml -# pyproject.toml for Poetry -[tool.poetry] -name = "myproject" -version = "0.1.0" -description = "A Python project" -authors = ["Your Name "] -readme = "README.md" -license = "MIT" -packages = [{include = "myproject", from = "src"}] - -[tool.poetry.dependencies] -python = "^3.11" -requests = "^2.31.0" -pydantic = "^2.5.0" - -[tool.poetry.group.dev.dependencies] -pytest = "^7.4.0" -pytest-cov = "^4.1.0" -mypy = "^1.7.0" -black = "^23.11.0" -ruff = "^0.1.6" - -[tool.poetry.scripts] -myproject = "myproject.cli:main" - -[build-system] -requires = ["poetry-core"] -build-backend = "poetry.core.masonry.api" -``` - -```bash -# Poetry commands -poetry init # Initialize new project -poetry add requests # Add dependency -poetry add --group dev pytest # Add dev dependency -poetry install # Install dependencies -poetry update # Update dependencies -poetry shell # Activate virtual environment -poetry run pytest # Run command in venv -poetry build # Build package -poetry publish # Publish to PyPI -poetry export -f requirements.txt --output requirements.txt -``` - -## Virtual Environments - -```bash -# Using venv (built-in) -python -m venv .venv -source .venv/bin/activate # Linux/Mac -.venv\Scripts\activate # Windows - -# Install in editable mode -pip install -e . -pip install -e ".[dev]" # With optional dependencies - -# Using virtualenv -pip install virtualenv -virtualenv venv -source venv/bin/activate - -# Using pyenv for Python version management -pyenv install 3.11.6 -pyenv local 3.11.6 # Set for current directory -echo "3.11.6" > .python-version -``` - -## Package __init__.py - -```python -# src/myproject/__init__.py -"""MyProject - A Python package.""" - -from myproject.core import main_function, CoreClass -from myproject.utils import helper_function - -__version__ = "0.1.0" -__all__ = ["main_function", "CoreClass", "helper_function"] - -# Package-level configuration -import logging - -logger = logging.getLogger(__name__) -logger.addHandler(logging.NullHandler()) -``` - -## Type Stub Files (py.typed) - -```python -# src/myproject/py.typed -# Empty file indicates package includes type hints - -# src/myproject/__init__.pyi (optional stub file) -from typing import Any - -__version__: str - -def main_function(arg: str) -> dict[str, Any]: ... - -class CoreClass: - def __init__(self, name: str) -> None: ... - def process(self) -> str: ... -``` - -## CLI Entry Points - -```python -# src/myproject/cli.py -import sys -from typing import NoReturn - -def main() -> NoReturn: - """Main CLI entry point.""" - print("MyProject CLI") - sys.exit(0) - -if __name__ == "__main__": - main() -``` - -## Requirements Files - -```bash -# requirements.txt - Production dependencies -requests>=2.31.0,<3.0.0 -pydantic>=2.5.0,<3.0.0 - -# requirements-dev.txt - Development dependencies --r requirements.txt -pytest>=7.4.0 -pytest-cov>=4.1.0 -mypy>=1.7.0 -black>=23.11.0 -ruff>=0.1.6 - -# Generate from Poetry -poetry export -f requirements.txt --output requirements.txt --without-hashes -poetry export -f requirements.txt --with dev --output requirements-dev.txt -``` - -## Building and Distribution - -```bash -# Build package -python -m build - -# Check package -twine check dist/* - -# Upload to PyPI -twine upload dist/* - -# Upload to Test PyPI -twine upload --repository testpypi dist/* - -# Install from Test PyPI -pip install --index-url https://test.pypi.org/simple/ myproject -``` - -## Setuptools Configuration (Legacy) - -```python -# setup.py (if not using pyproject.toml) -from setuptools import setup, find_packages - -setup( - name="myproject", - version="0.1.0", - packages=find_packages(where="src"), - package_dir={"": "src"}, - python_requires=">=3.11", - install_requires=[ - "requests>=2.31.0", - "pydantic>=2.5.0", - ], - extras_require={ - "dev": [ - "pytest>=7.4.0", - "mypy>=1.7.0", - ], - }, - entry_points={ - "console_scripts": [ - "myproject=myproject.cli:main", - ], - }, -) -``` - -## Manifest for Package Data - -``` -# MANIFEST.in -include README.md -include LICENSE -include pyproject.toml -recursive-include src/myproject *.py -recursive-include src/myproject py.typed -recursive-include tests *.py -prune docs/_build -``` - -## Version Management - -```python -# src/myproject/__version__.py -__version__ = "0.1.0" - -# src/myproject/__init__.py -from myproject.__version__ import __version__ - -# Read version in pyproject.toml -import tomli -from pathlib import Path - -def get_version() -> str: - pyproject = Path(__file__).parent.parent / "pyproject.toml" - with open(pyproject, "rb") as f: - data = tomli.load(f) - return data["project"]["version"] -``` - -## Dependency Management Best Practices - -```python -# Pin dependencies for applications -requests==2.31.0 -pydantic==2.5.2 - -# Use ranges for libraries -requests>=2.31.0,<3.0.0 -pydantic>=2.5.0,<3.0.0 - -# Lock files -# Poetry: poetry.lock -# pip: requirements.txt with exact versions -pip freeze > requirements-lock.txt - -# Update dependencies -poetry update -pip install --upgrade -r requirements.txt -``` - -## CI/CD Integration - -```yaml -# .github/workflows/test.yml -name: Tests - -on: [push, pull_request] - -jobs: - test: - runs-on: ubuntu-latest - strategy: - matrix: - python-version: ["3.11", "3.12"] - - steps: - - uses: actions/checkout@v4 - - name: Set up Python - uses: actions/setup-python@v4 - with: - python-version: ${{ matrix.python-version }} - - - name: Install dependencies - run: | - python -m pip install --upgrade pip - pip install -e ".[dev]" - - - name: Run tests - run: | - pytest --cov --cov-report=xml - - - name: Type check - run: mypy src - - - name: Lint - run: | - black --check src tests - ruff check src tests - - - name: Upload coverage - uses: codecov/codecov-action@v3 -``` - -## Pre-commit Hooks - -```yaml -# .pre-commit-config.yaml -repos: - - repo: https://github.com/psf/black - rev: 23.11.0 - hooks: - - id: black - - - repo: https://github.com/astral-sh/ruff-pre-commit - rev: v0.1.6 - hooks: - - id: ruff - args: [--fix, --exit-non-zero-on-fix] - - - repo: https://github.com/pre-commit/mirrors-mypy - rev: v1.7.1 - hooks: - - id: mypy - additional_dependencies: [types-requests] -``` - -```bash -# Install pre-commit -pip install pre-commit -pre-commit install - -# Run manually -pre-commit run --all-files -``` diff --git a/.github/skills/python-pro/references/standard-library.md b/.github/skills/python-pro/references/standard-library.md deleted file mode 100644 index 5667fe2c..00000000 --- a/.github/skills/python-pro/references/standard-library.md +++ /dev/null @@ -1,381 +0,0 @@ -# Standard Library Mastery - -> Reference for: Python Pro -> Load when: pathlib, dataclasses, functools, itertools, collections - -## Pathlib for File Operations - -```python -from pathlib import Path - -# Path creation and manipulation -project_root = Path(__file__).parent.parent -config_file = project_root / "config" / "settings.toml" -data_dir = Path.home() / "data" - -# File operations -def read_config(config_path: Path) -> dict[str, str]: - if not config_path.exists(): - raise FileNotFoundError(f"Config not found: {config_path}") - - # Read text - content = config_path.read_text(encoding="utf-8") - - # Read bytes - binary = config_path.read_bytes() - - return parse_config(content) - -# Path traversal -def find_python_files(directory: Path) -> list[Path]: - # Recursive glob - return list(directory.rglob("*.py")) - -def get_file_info(path: Path) -> dict[str, Any]: - stat = path.stat() - return { - "size": stat.st_size, - "modified": stat.st_mtime, - "is_file": path.is_file(), - "is_dir": path.is_dir(), - "suffix": path.suffix, - "stem": path.stem, - } - -# Creating directories -def ensure_dir(path: Path) -> None: - path.mkdir(parents=True, exist_ok=True) - -# Temporary files -from tempfile import TemporaryDirectory -from pathlib import Path - -def process_with_temp() -> None: - with TemporaryDirectory() as tmpdir: - temp_path = Path(tmpdir) / "output.txt" - temp_path.write_text("data") -``` - -## Dataclasses for Data Structures - -```python -from dataclasses import dataclass, field, asdict, replace -from typing import ClassVar - -# Basic dataclass -@dataclass -class User: - id: int - name: str - email: str - active: bool = True - -# Post-init processing -@dataclass -class Product: - name: str - price: float - discount: float = 0.0 - - def __post_init__(self) -> None: - if self.discount > 1.0: - raise ValueError("Discount must be <= 1.0") - - @property - def final_price(self) -> float: - return self.price * (1 - self.discount) - -# Field with factory -@dataclass -class ShoppingCart: - user_id: int - items: list[str] = field(default_factory=list) - metadata: dict[str, Any] = field(default_factory=dict) - -# Frozen dataclass (immutable) -@dataclass(frozen=True) -class Point: - x: float - y: float - - def distance(self, other: "Point") -> float: - return ((self.x - other.x)**2 + (self.y - other.y)**2)**0.5 - -# Class variables -@dataclass -class Config: - API_VERSION: ClassVar[str] = "v1" - BASE_URL: ClassVar[str] = "https://api.example.com" - - timeout: int = 30 - retries: int = 3 - -# Ordered dataclass for comparison -@dataclass(order=True) -class Priority: - level: int - name: str = field(compare=False) - -# Convert to/from dict -user = User(1, "Alice", "alice@example.com") -user_dict = asdict(user) -updated = replace(user, name="Alice Smith") -``` - -## Functools for Function Tools - -```python -from functools import ( - cache, lru_cache, cached_property, - partial, wraps, reduce, singledispatch -) - -# Caching -@cache # Unlimited cache (Python 3.9+) -def fibonacci(n: int) -> int: - if n < 2: - return n - return fibonacci(n - 1) + fibonacci(n - 2) - -@lru_cache(maxsize=128) # LRU cache with size limit -def fetch_user(user_id: int) -> dict[str, Any]: - # Expensive database call - return {"id": user_id, "name": "User"} - -# Cached property -class DataProcessor: - def __init__(self, data: list[int]) -> None: - self._data = data - - @cached_property - def mean(self) -> float: - """Computed once, then cached.""" - return sum(self._data) / len(self._data) - -# Partial application -from operator import mul - -double = partial(mul, 2) -triple = partial(mul, 3) -print(double(5)) # 10 - -# Decorator preservation -def timing_decorator(func: Callable[P, R]) -> Callable[P, R]: - @wraps(func) - def wrapper(*args: P.args, **kwargs: P.kwargs) -> R: - start = time.time() - result = func(*args, **kwargs) - print(f"{func.__name__} took {time.time() - start:.2f}s") - return result - return wrapper - -# Reduce for aggregation -from operator import add - -total = reduce(add, [1, 2, 3, 4, 5]) # 15 -product = reduce(mul, [1, 2, 3, 4], 1) # 24 - -# Single dispatch for polymorphism -@singledispatch -def process(arg: Any) -> str: - return f"Unknown type: {type(arg)}" - -@process.register -def _(arg: int) -> str: - return f"Integer: {arg * 2}" - -@process.register -def _(arg: str) -> str: - return f"String: {arg.upper()}" - -@process.register(list) -def _(arg: list[Any]) -> str: - return f"List with {len(arg)} items" -``` - -## Itertools for Iteration - -```python -from itertools import ( - chain, islice, cycle, repeat, - groupby, accumulate, combinations, permutations, - product, zip_longest, tee, filterfalse -) - -# Chain multiple iterables -combined = list(chain([1, 2], [3, 4], [5, 6])) # [1,2,3,4,5,6] - -# Slice iterator (memory efficient) -first_10 = list(islice(range(1000), 10)) - -# Infinite iterators -from itertools import count -counter = count(start=1, step=2) # 1, 3, 5, 7, ... - -# Groupby for grouping -data = [("A", 1), ("A", 2), ("B", 1), ("B", 2)] -grouped = {k: list(v) for k, v in groupby(data, key=lambda x: x[0])} - -# Accumulate for running totals -cumsum = list(accumulate([1, 2, 3, 4, 5])) # [1, 3, 6, 10, 15] - -# Combinations and permutations -combos = list(combinations([1, 2, 3], 2)) # [(1,2), (1,3), (2,3)] -perms = list(permutations([1, 2, 3], 2)) # [(1,2), (1,3), (2,1), ...] - -# Cartesian product -pairs = list(product([1, 2], ['a', 'b'])) # [(1,'a'), (1,'b'), (2,'a'), (2,'b')] - -# Zip with different lengths -from itertools import zip_longest -paired = list(zip_longest([1, 2], ['a', 'b', 'c'], fillvalue=0)) - -# Tee for multiple iterators -it1, it2 = tee(range(5), 2) - -# Filter false -odds = list(filterfalse(lambda x: x % 2 == 0, range(10))) -``` - -## Collections for Data Structures - -```python -from collections import ( - defaultdict, Counter, deque, namedtuple, - ChainMap, OrderedDict -) - -# defaultdict for automatic defaults -word_index: defaultdict[str, list[int]] = defaultdict(list) -for i, word in enumerate(["hello", "world", "hello"]): - word_index[word].append(i) - -# Counter for counting -from collections import Counter - -word_counts = Counter(["apple", "banana", "apple", "cherry", "banana", "apple"]) -print(word_counts.most_common(2)) # [('apple', 3), ('banana', 2)] - -# Counter operations -c1 = Counter(a=3, b=1) -c2 = Counter(a=1, b=2) -print(c1 + c2) # Counter({'a': 4, 'b': 3}) - -# deque for efficient queue operations -from collections import deque - -queue: deque[str] = deque() -queue.append("first") -queue.append("second") -queue.appendleft("priority") -item = queue.popleft() # "priority" - -# Ring buffer with maxlen -recent: deque[int] = deque(maxlen=3) -for i in range(5): - recent.append(i) # Only keeps last 3 - -# namedtuple for lightweight classes -from collections import namedtuple - -Point = namedtuple('Point', ['x', 'y']) -p = Point(1, 2) -print(p.x, p.y) - -# ChainMap for layered configs -from collections import ChainMap - -defaults = {'color': 'red', 'user': 'guest'} -environment = {'user': 'admin'} -combined = ChainMap(environment, defaults) -print(combined['user']) # 'admin' (from environment) -``` - -## Context Managers - -```python -from contextlib import contextmanager, suppress, ExitStack - -# Custom context manager -@contextmanager -def managed_resource(resource_id: str) -> Iterator[Resource]: - resource = acquire_resource(resource_id) - try: - yield resource - finally: - release_resource(resource) - -# Suppress exceptions -with suppress(FileNotFoundError): - Path("nonexistent.txt").unlink() - -# ExitStack for dynamic context managers -def process_files(filenames: list[str]) -> None: - with ExitStack() as stack: - files = [stack.enter_context(open(fn)) for fn in filenames] - # All files auto-closed on exit - for f in files: - process(f.read()) -``` - -## Enum for Constants - -```python -from enum import Enum, auto, IntEnum, Flag - -# Basic enum -class Status(Enum): - PENDING = "pending" - APPROVED = "approved" - REJECTED = "rejected" - -# Auto values -class Color(Enum): - RED = auto() - GREEN = auto() - BLUE = auto() - -# IntEnum for numeric values -class Priority(IntEnum): - LOW = 1 - MEDIUM = 2 - HIGH = 3 - -# Flag for bit flags -class Permission(Flag): - READ = auto() - WRITE = auto() - EXECUTE = auto() - -user_perms = Permission.READ | Permission.WRITE -if Permission.READ in user_perms: - print("Can read") -``` - -## Logging - -```python -import logging -from pathlib import Path - -# Configure logging -logging.basicConfig( - level=logging.INFO, - format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', - handlers=[ - logging.FileHandler('app.log'), - logging.StreamHandler() - ] -) - -logger = logging.getLogger(__name__) - -# Structured logging -def process_user(user_id: int) -> None: - logger.info("Processing user", extra={"user_id": user_id}) - try: - # Process... - logger.debug("User data loaded", extra={"user_id": user_id}) - except Exception as e: - logger.exception("Failed to process user", extra={"user_id": user_id}) -``` diff --git a/.github/skills/python-pro/references/testing.md b/.github/skills/python-pro/references/testing.md deleted file mode 100644 index 4362f74a..00000000 --- a/.github/skills/python-pro/references/testing.md +++ /dev/null @@ -1,407 +0,0 @@ -# Testing with Pytest - -> Reference for: Python Pro -> Load when: pytest, fixtures, mocking, test coverage, parametrize - -## Basic Pytest Structure - -```python -# test_user.py -import pytest -from myapp.user import User, UserService - -# Simple test function -def test_user_creation() -> None: - user = User(id=1, name="Alice", email="alice@example.com") - assert user.name == "Alice" - assert user.is_active is True - -# Test with multiple assertions -def test_user_validation() -> None: - with pytest.raises(ValueError, match="Invalid email"): - User(id=1, name="Alice", email="invalid") - -# Test class for grouping -class TestUserService: - def test_find_user(self) -> None: - service = UserService() - user = service.find(1) - assert user is not None - - def test_create_user(self) -> None: - service = UserService() - user = service.create(name="Bob", email="bob@example.com") - assert user.id > 0 -``` - -## Fixtures for Setup/Teardown - -```python -# conftest.py - shared fixtures -import pytest -from typing import Iterator -from myapp.database import Database, Session - -@pytest.fixture -def db() -> Iterator[Database]: - """Provide database instance with cleanup.""" - database = Database("test.db") - database.create_tables() - yield database - database.drop_tables() - database.close() - -@pytest.fixture -def db_session(db: Database) -> Iterator[Session]: - """Provide database session with rollback.""" - session = db.create_session() - yield session - session.rollback() - session.close() - -@pytest.fixture -def sample_user() -> User: - """Provide test user.""" - return User(id=1, name="Test User", email="test@example.com") - -# Using fixtures in tests -def test_user_creation(db_session: Session, sample_user: User) -> None: - db_session.add(sample_user) - db_session.commit() - - retrieved = db_session.query(User).filter_by(id=1).first() - assert retrieved.name == "Test User" - -# Fixture with parameters -@pytest.fixture(params=["sqlite", "postgresql", "mysql"]) -def db_engine(request: pytest.FixtureRequest) -> str: - return request.param - -def test_connection(db_engine: str) -> None: - # Test runs 3 times with different engines - assert create_connection(db_engine) - -# Autouse fixture (runs automatically) -@pytest.fixture(autouse=True) -def reset_state() -> Iterator[None]: - """Reset global state before each test.""" - clear_caches() - yield - cleanup_temp_files() -``` - -## Parametrize for Multiple Cases - -```python -import pytest - -# Parametrize test function -@pytest.mark.parametrize( - "input,expected", - [ - (2, 4), - (3, 9), - (4, 16), - (-2, 4), - ] -) -def test_square(input: int, expected: int) -> None: - assert square(input) == expected - -# Multiple parameters -@pytest.mark.parametrize("base", [2, 10]) -@pytest.mark.parametrize("exponent", [0, 1, 2]) -def test_power(base: int, exponent: int) -> None: - result = base ** exponent - assert result >= 0 - -# Parametrize with IDs -@pytest.mark.parametrize( - "email,valid", - [ - ("user@example.com", True), - ("invalid", False), - ("@example.com", False), - ("user@", False), - ], - ids=["valid", "no_at", "no_user", "no_domain"] -) -def test_email_validation(email: str, valid: bool) -> None: - assert is_valid_email(email) == valid - -# Parametrize with fixtures -@pytest.fixture -def user_factory(): - def _make_user(name: str, active: bool = True) -> User: - return User(name=name, active=active) - return _make_user - -@pytest.mark.parametrize("name", ["Alice", "Bob", "Charlie"]) -def test_user_names(user_factory, name: str) -> None: - user = user_factory(name) - assert user.name == name -``` - -## Mocking and Patching - -```python -from unittest.mock import Mock, MagicMock, patch, AsyncMock, call -import pytest - -# Mock object -def test_api_call_with_mock() -> None: - mock_client = Mock() - mock_client.get.return_value = {"status": "ok"} - - service = ApiService(mock_client) - result = service.fetch_data() - - mock_client.get.assert_called_once_with("/api/data") - assert result["status"] == "ok" - -# Patch function/method -def test_database_call() -> None: - with patch("myapp.database.connect") as mock_connect: - mock_connect.return_value = Mock() - - db = Database() - db.connect() - - mock_connect.assert_called_once() - -# Patch as decorator -@patch("myapp.user.send_email") -def test_user_registration(mock_send_email: Mock) -> None: - service = UserService() - service.register("user@example.com") - - mock_send_email.assert_called_with( - to="user@example.com", - subject="Welcome" - ) - -# Multiple patches -@patch("myapp.api.requests.get") -@patch("myapp.api.cache.get") -def test_cached_api(mock_cache: Mock, mock_requests: Mock) -> None: - mock_cache.return_value = None - mock_requests.return_value.json.return_value = {"data": "value"} - - result = fetch_with_cache("key") - - mock_cache.assert_called_once_with("key") - mock_requests.assert_called_once() - -# Mock side effects -def test_retry_logic() -> None: - mock_api = Mock() - mock_api.call.side_effect = [ - ConnectionError("Failed"), - ConnectionError("Failed"), - {"status": "ok"} - ] - - result = retry_api_call(mock_api) - assert result["status"] == "ok" - assert mock_api.call.call_count == 3 - -# Async mock -@pytest.mark.asyncio -async def test_async_function() -> None: - mock_db = AsyncMock() - mock_db.fetch_user.return_value = User(id=1, name="Alice") - - service = AsyncUserService(mock_db) - user = await service.get_user(1) - - mock_db.fetch_user.assert_awaited_once_with(1) - assert user.name == "Alice" -``` - -## Async Testing - -```python -import pytest -import asyncio - -# Mark async test -@pytest.mark.asyncio -async def test_async_fetch() -> None: - result = await fetch_data("https://api.example.com") - assert result["status"] == "ok" - -# Async fixture -@pytest.fixture -async def async_db() -> AsyncIterator[AsyncDatabase]: - db = AsyncDatabase() - await db.connect() - yield db - await db.disconnect() - -@pytest.mark.asyncio -async def test_async_query(async_db: AsyncDatabase) -> None: - result = await async_db.query("SELECT * FROM users") - assert len(result) > 0 - -# Test concurrent operations -@pytest.mark.asyncio -async def test_concurrent_requests() -> None: - urls = ["http://example.com/1", "http://example.com/2"] - results = await asyncio.gather(*[fetch(url) for url in urls]) - assert len(results) == 2 -``` - -## Pytest Markers - -```python -import pytest - -# Skip test -@pytest.mark.skip(reason="Not implemented yet") -def test_future_feature() -> None: - pass - -# Conditional skip -@pytest.mark.skipif(sys.version_info < (3, 11), reason="Requires Python 3.11+") -def test_new_feature() -> None: - pass - -# Expected failure -@pytest.mark.xfail(reason="Known bug #123") -def test_known_bug() -> None: - assert buggy_function() == expected_value - -# Custom markers -@pytest.mark.slow -def test_slow_operation() -> None: - time.sleep(5) - assert True - -@pytest.mark.integration -def test_integration() -> None: - assert external_service.ping() - -# Run with: pytest -m "not slow" -``` - -## Test Coverage - -```python -# Run with coverage -# pytest --cov=myapp --cov-report=html --cov-report=term - -# conftest.py - coverage configuration -def pytest_configure(config): - config.addinivalue_line( - "markers", "unit: mark test as unit test" - ) - -# pytest.ini or pyproject.toml -""" -[tool.pytest.ini_options] -minversion = "7.0" -addopts = [ - "--cov=myapp", - "--cov-report=term-missing", - "--cov-fail-under=90", - "-ra", - "--strict-markers", -] -testpaths = ["tests"] -""" -``` - -## Property-Based Testing - -```python -from hypothesis import given, strategies as st - -# Property-based test -@given(st.integers(), st.integers()) -def test_addition_commutative(a: int, b: int) -> None: - assert a + b == b + a - -@given(st.lists(st.integers())) -def test_sorted_is_ordered(lst: list[int]) -> None: - sorted_lst = sorted(lst) - for i in range(len(sorted_lst) - 1): - assert sorted_lst[i] <= sorted_lst[i + 1] - -# Custom strategies -@given(st.emails()) -def test_email_validation(email: str) -> None: - assert "@" in email - assert validate_email(email) - -# Composite strategies -from hypothesis import strategies as st -from hypothesis.strategies import composite - -@composite -def users(draw) -> User: - return User( - id=draw(st.integers(min_value=1)), - name=draw(st.text(min_size=1, max_size=50)), - email=draw(st.emails()), - age=draw(st.integers(min_value=18, max_value=120)) - ) - -@given(users()) -def test_user_creation(user: User) -> None: - assert user.age >= 18 - assert len(user.name) > 0 -``` - -## Test Organization - -```python -# tests/ -# conftest.py - Shared fixtures -# test_user.py - User tests -# test_api.py - API tests -# integration/ -# test_workflow.py - Integration tests -# unit/ -# test_models.py - Unit tests - -# Fixture factory pattern -@pytest.fixture -def user_factory(db_session: Session): - created_users: list[User] = [] - - def _create_user( - name: str = "Test User", - email: str | None = None, - **kwargs - ) -> User: - if email is None: - email = f"{name.lower().replace(' ', '.')}@example.com" - - user = User(name=name, email=email, **kwargs) - db_session.add(user) - db_session.commit() - created_users.append(user) - return user - - yield _create_user - - # Cleanup - for user in created_users: - db_session.delete(user) - db_session.commit() -``` - -## Snapshot Testing - -```python -import pytest -from syrupy.assertion import SnapshotAssertion - -def test_api_response(snapshot: SnapshotAssertion) -> None: - response = api.get_user(1) - assert response == snapshot - -def test_rendered_template(snapshot: SnapshotAssertion) -> None: - html = render_template("user.html", user=get_user(1)) - assert html == snapshot -``` diff --git a/.github/skills/python-pro/references/type-system.md b/.github/skills/python-pro/references/type-system.md deleted file mode 100644 index 958be2bd..00000000 --- a/.github/skills/python-pro/references/type-system.md +++ /dev/null @@ -1,293 +0,0 @@ -# Type System Mastery - -> Reference for: Python Pro -> Load when: Type hints, mypy configuration, generics, Protocol definitions - -## Basic Type Annotations - -```python -from typing import Any -from collections.abc import Sequence, Mapping - -# Function signatures -def process_user(name: str, age: int, active: bool = True) -> dict[str, Any]: - return {"name": name, "age": age, "active": active} - -# Use | for unions (Python 3.10+) -def find_user(user_id: int | str) -> dict[str, Any] | None: - if isinstance(user_id, int): - return {"id": user_id} - return None - -# Collections - prefer collections.abc -def process_items(items: Sequence[str]) -> list[str]: - """Accepts list, tuple, or any sequence.""" - return [item.upper() for item in items] - -def merge_configs(base: Mapping[str, int], override: dict[str, int]) -> dict[str, int]: - """Mapping for read-only, dict for mutable.""" - return {**base, **override} -``` - -## Generic Types - -```python -from typing import TypeVar, Generic, Protocol -from collections.abc import Callable - -T = TypeVar('T') -K = TypeVar('K') -V = TypeVar('V') - -# Generic function -def first_element(items: Sequence[T]) -> T | None: - return items[0] if items else None - -# Generic class -class Cache(Generic[K, V]): - def __init__(self) -> None: - self._data: dict[K, V] = {} - - def get(self, key: K) -> V | None: - return self._data.get(key) - - def set(self, key: K, value: V) -> None: - self._data[key] = value - -# Usage -user_cache: Cache[int, str] = Cache() -user_cache.set(1, "Alice") - -# Constrained TypeVar -from numbers import Number -NumT = TypeVar('NumT', bound=Number) - -def add_numbers(a: NumT, b: NumT) -> NumT: - return a + b # type: ignore[return-value] -``` - -## Protocol for Structural Typing - -```python -from typing import Protocol, runtime_checkable - -# Define interface without inheritance -class Drawable(Protocol): - def draw(self) -> str: - ... - - @property - def color(self) -> str: - ... - -class Circle: - def __init__(self, radius: float, color: str) -> None: - self.radius = radius - self._color = color - - def draw(self) -> str: - return f"Drawing {self._color} circle" - - @property - def color(self) -> str: - return self._color - -# Circle implements Drawable without inheriting -def render(shape: Drawable) -> str: - return shape.draw() - -# Runtime checkable protocol -@runtime_checkable -class Closeable(Protocol): - def close(self) -> None: - ... - -def cleanup(resource: Closeable) -> None: - if isinstance(resource, Closeable): - resource.close() -``` - -## Advanced Type Features - -```python -from typing import Literal, TypeAlias, TypedDict, NotRequired, Self, overload - -# Literal types for constants -Mode = Literal["read", "write", "append"] - -def open_file(path: str, mode: Mode) -> None: - ... - -# Type aliases for complex types -JsonDict: TypeAlias = dict[str, Any] -UserId: TypeAlias = int | str - -# TypedDict for structured dictionaries -class UserDict(TypedDict): - id: int - name: str - email: str - age: NotRequired[int] # Optional field - -def create_user(data: UserDict) -> None: - print(data["name"]) # Type-safe access - -# Self type for method chaining -class Builder: - def __init__(self) -> None: - self._value = 0 - - def add(self, n: int) -> Self: - self._value += n - return self - - def multiply(self, n: int) -> Self: - self._value *= n - return self - -# Overload for different signatures -@overload -def process(data: str) -> str: ... - -@overload -def process(data: int) -> int: ... - -def process(data: str | int) -> str | int: - if isinstance(data, str): - return data.upper() - return data * 2 -``` - -## Callable Types - -```python -from collections.abc import Callable -from typing import ParamSpec, Concatenate - -# Basic callable -def apply(func: Callable[[int, int], int], a: int, b: int) -> int: - return func(a, b) - -# ParamSpec for preserving signatures -P = ParamSpec('P') -R = TypeVar('R') - -def logging_decorator(func: Callable[P, R]) -> Callable[P, R]: - def wrapper(*args: P.args, **kwargs: P.kwargs) -> R: - print(f"Calling {func.__name__}") - return func(*args, **kwargs) - return wrapper - -# Concatenate for dependency injection -def with_connection( - func: Callable[Concatenate[Connection, P], R] -) -> Callable[P, R]: - def wrapper(*args: P.args, **kwargs: P.kwargs) -> R: - conn = get_connection() - return func(conn, *args, **kwargs) - return wrapper - -# Usage -@with_connection -def query_user(conn: Connection, user_id: int) -> User: - return conn.execute(f"SELECT * FROM users WHERE id = {user_id}") -``` - -## Mypy Configuration - -```toml -# pyproject.toml -[tool.mypy] -python_version = "3.11" -strict = true -warn_return_any = true -warn_unused_configs = true -disallow_untyped_defs = true -disallow_any_generics = true -disallow_subclassing_any = true -disallow_untyped_calls = true -disallow_incomplete_defs = true -check_untyped_defs = true -no_implicit_optional = true -warn_redundant_casts = true -warn_unused_ignores = true -warn_no_return = true -warn_unreachable = true -strict_equality = true - -[[tool.mypy.overrides]] -module = "third_party.*" -ignore_missing_imports = true -``` - -## Common Type Patterns - -```python -# Result type pattern -from dataclasses import dataclass - -@dataclass -class Success(Generic[T]): - value: T - -@dataclass -class Error: - message: str - -Result = Success[T] | Error - -def divide(a: int, b: int) -> Result[float]: - if b == 0: - return Error("Division by zero") - return Success(a / b) - -# Option/Maybe type -def safe_get(items: Sequence[T], index: int) -> T | None: - try: - return items[index] - except IndexError: - return None - -# Sentinel value with typing -from typing import Final - -MISSING: Final = object() - -def get_value(key: str, default: T | type[MISSING] = MISSING) -> T: - if default is MISSING: - raise KeyError(key) - return default # type: ignore[return-value] -``` - -## Type Narrowing - -```python -from typing import assert_type, assert_never - -def process_value(value: int | str | None) -> str: - # Type guards - if value is None: - return "null" - - if isinstance(value, int): - # Type narrowed to int - return str(value * 2) - - # Type narrowed to str - return value.upper() - -# Exhaustiveness checking -def handle_mode(mode: Literal["read", "write"]) -> str: - if mode == "read": - return "Reading" - elif mode == "write": - return "Writing" - else: - # Mypy will error if mode can be anything else - assert_never(mode) - -# Custom type guard -def is_string_list(val: list[Any]) -> bool: - """Runtime check for list of strings.""" - return all(isinstance(x, str) for x in val) -``` diff --git a/.github/skills/rust-engineer/SKILL.md b/.github/skills/rust-engineer/SKILL.md deleted file mode 100644 index fe5b1c05..00000000 --- a/.github/skills/rust-engineer/SKILL.md +++ /dev/null @@ -1,96 +0,0 @@ ---- -name: rust-engineer -description: Use when building Rust applications requiring memory safety, systems programming, or zero-cost abstractions. Invoke for ownership patterns, lifetimes, traits, async/await with tokio. -triggers: - - Rust - - Cargo - - ownership - - borrowing - - lifetimes - - async Rust - - tokio - - zero-cost abstractions - - memory safety - - systems programming -role: specialist -scope: implementation -output-format: code ---- - -# Rust Engineer - -Senior Rust engineer with deep expertise in Rust 2021 edition, systems programming, memory safety, and zero-cost abstractions. Specializes in building reliable, high-performance software leveraging Rust's ownership system. - -## Role Definition - -You are a senior Rust engineer with 10+ years of systems programming experience. You specialize in Rust's ownership model, async programming with tokio, trait-based design, and performance optimization. You build memory-safe, concurrent systems with zero-cost abstractions. - -## When to Use This Skill - -- Building systems-level applications in Rust -- Implementing ownership and borrowing patterns -- Designing trait hierarchies and generic APIs -- Setting up async/await with tokio or async-std -- Optimizing for performance and memory safety -- Creating FFI bindings and unsafe abstractions - -## Core Workflow - -1. **Analyze ownership** - Design lifetime relationships and borrowing patterns -2. **Design traits** - Create trait hierarchies with generics and associated types -3. **Implement safely** - Write idiomatic Rust with minimal unsafe code -4. **Handle errors** - Use Result/Option with ? operator and custom error types -5. **Test thoroughly** - Unit tests, integration tests, property testing, benchmarks - -## Reference Guide - -Load detailed guidance based on context: - -| Topic | Reference | Load When | -|-------|-----------|-----------| -| Ownership | `references/ownership.md` | Lifetimes, borrowing, smart pointers, Pin | -| Traits | `references/traits.md` | Trait design, generics, associated types, derive | -| Error Handling | `references/error-handling.md` | Result, Option, ?, custom errors, thiserror | -| Async | `references/async.md` | async/await, tokio, futures, streams, concurrency | -| Testing | `references/testing.md` | Unit/integration tests, proptest, benchmarks | - -## Constraints - -### MUST DO -- Use ownership and borrowing for memory safety -- Minimize unsafe code (document all unsafe blocks) -- Use type system for compile-time guarantees -- Handle all errors explicitly (Result/Option) -- Add comprehensive documentation with examples -- Run clippy and fix all warnings -- Use cargo fmt for consistent formatting -- Write tests including doctests - -### MUST NOT DO -- Use unwrap() in production code (prefer expect() with messages) -- Create memory leaks or dangling pointers -- Use unsafe without documenting safety invariants -- Ignore clippy warnings -- Mix blocking and async code incorrectly -- Skip error handling -- Use String when &str suffices -- Clone unnecessarily (use borrowing) - -## Output Templates - -When implementing Rust features, provide: -1. Type definitions (structs, enums, traits) -2. Implementation with proper ownership -3. Error handling with custom error types -4. Tests (unit, integration, doctests) -5. Brief explanation of design decisions - -## Knowledge Reference - -Rust 2021, Cargo, ownership/borrowing, lifetimes, traits, generics, async/await, tokio, Result/Option, thiserror/anyhow, serde, clippy, rustfmt, cargo-test, criterion benchmarks, MIRI, unsafe Rust - -## Related Skills - -- **Systems Architect** - Low-level system design -- **Performance Engineer** - Optimization and profiling -- **Test Master** - Comprehensive testing strategies diff --git a/.github/skills/rust-engineer/references/async.md b/.github/skills/rust-engineer/references/async.md deleted file mode 100644 index 94190a82..00000000 --- a/.github/skills/rust-engineer/references/async.md +++ /dev/null @@ -1,461 +0,0 @@ -# Async Programming in Rust - -> Reference for: Rust Engineer -> Load when: async/await, tokio, futures, streams, concurrency patterns - -## Basic Async/Await - -```rust -use tokio; - -// Async function returns a Future -async fn fetch_data(url: &str) -> Result { - let response = reqwest::get(url).await?; - let body = response.text().await?; - Ok(body) -} - -// Tokio runtime -#[tokio::main] -async fn main() -> Result<(), Box> { - let data = fetch_data("https://api.example.com").await?; - println!("Data: {}", data); - Ok(()) -} - -// Manual runtime creation -fn main() { - let runtime = tokio::runtime::Runtime::new().unwrap(); - runtime.block_on(async { - println!("Hello from async context"); - }); -} -``` - -## Concurrent Execution - -```rust -use tokio; - -// Sequential execution -async fn sequential() { - let result1 = async_operation1().await; - let result2 = async_operation2().await; // Waits for operation1 -} - -// Concurrent execution with join! -async fn concurrent() { - let (result1, result2) = tokio::join!( - async_operation1(), - async_operation2() - ); -} - -// Concurrent with try_join! (stops on first error) -async fn concurrent_with_errors() -> Result<(), Box> { - let (result1, result2) = tokio::try_join!( - fallible_operation1(), - fallible_operation2() - )?; - Ok(()) -} - -// Spawning tasks -async fn spawn_tasks() { - let handle1 = tokio::spawn(async { - // This runs on a separate task - expensive_computation().await - }); - - let handle2 = tokio::spawn(async { - another_computation().await - }); - - // Wait for both to complete - let result1 = handle1.await.unwrap(); - let result2 = handle2.await.unwrap(); -} -``` - -## Select and Race Conditions - -```rust -use tokio::time::{sleep, Duration}; - -// select! - wait for first to complete -async fn first_to_complete() { - tokio::select! { - result = async_operation1() => { - println!("Operation 1 completed first: {:?}", result); - } - result = async_operation2() => { - println!("Operation 2 completed first: {:?}", result); - } - } -} - -// Timeout pattern -async fn with_timeout() -> Result { - tokio::select! { - result = fetch_data("https://api.example.com") => { - result.map_err(|_| "Fetch failed") - } - _ = sleep(Duration::from_secs(5)) => { - Err("Timeout") - } - } -} - -// Cancellation with select! -async fn cancellable_operation(mut cancel_rx: tokio::sync::watch::Receiver) { - tokio::select! { - result = long_running_task() => { - println!("Task completed: {:?}", result); - } - _ = cancel_rx.changed() => { - println!("Task cancelled"); - } - } -} -``` - -## Streams - -```rust -use tokio_stream::{self as stream, StreamExt}; - -// Creating streams -async fn stream_example() { - let mut stream = stream::iter(vec![1, 2, 3, 4, 5]); - - while let Some(value) = stream.next().await { - println!("Value: {}", value); - } -} - -// Stream combinators -async fn stream_combinators() { - let stream = stream::iter(vec![1, 2, 3, 4, 5]) - .filter(|x| *x % 2 == 0) - .map(|x| x * 2); - - let results: Vec<_> = stream.collect().await; - println!("Results: {:?}", results); -} - -// Async stream processing -use futures::stream::{self, StreamExt}; - -async fn process_stream() { - let stream = stream::iter(vec![1, 2, 3, 4, 5]) - .then(|x| async move { - tokio::time::sleep(Duration::from_millis(100)).await; - x * 2 - }); - - stream.for_each(|x| async move { - println!("Processed: {}", x); - }).await; -} -``` - -## Channels for Communication - -```rust -use tokio::sync::{mpsc, oneshot, broadcast, watch}; - -// mpsc: multiple producer, single consumer -async fn mpsc_example() { - let (tx, mut rx) = mpsc::channel(32); - - tokio::spawn(async move { - tx.send("Hello").await.unwrap(); - tx.send("World").await.unwrap(); - }); - - while let Some(msg) = rx.recv().await { - println!("Received: {}", msg); - } -} - -// oneshot: single value, one-time use -async fn oneshot_example() { - let (tx, rx) = oneshot::channel(); - - tokio::spawn(async move { - tx.send("Result").unwrap(); - }); - - let result = rx.await.unwrap(); - println!("Got: {}", result); -} - -// broadcast: multiple producers, multiple consumers -async fn broadcast_example() { - let (tx, mut rx1) = broadcast::channel(16); - let mut rx2 = tx.subscribe(); - - tokio::spawn(async move { - tx.send("Message").unwrap(); - }); - - println!("rx1: {}", rx1.recv().await.unwrap()); - println!("rx2: {}", rx2.recv().await.unwrap()); -} - -// watch: single producer, multiple consumers (last value) -async fn watch_example() { - let (tx, mut rx) = watch::channel("initial"); - - tokio::spawn(async move { - loop { - rx.changed().await.unwrap(); - println!("Value changed to: {}", *rx.borrow()); - } - }); - - tx.send("updated").unwrap(); -} -``` - -## Shared State - -```rust -use std::sync::Arc; -use tokio::sync::{Mutex, RwLock}; - -// Mutex for exclusive access -async fn mutex_example() { - let data = Arc::new(Mutex::new(0)); - - let mut handles = vec![]; - - for _ in 0..10 { - let data = Arc::clone(&data); - let handle = tokio::spawn(async move { - let mut lock = data.lock().await; - *lock += 1; - }); - handles.push(handle); - } - - for handle in handles { - handle.await.unwrap(); - } - - println!("Final value: {}", *data.lock().await); -} - -// RwLock for read-write patterns -async fn rwlock_example() { - let data = Arc::new(RwLock::new(vec![1, 2, 3])); - - // Multiple readers - let data1 = Arc::clone(&data); - tokio::spawn(async move { - let read = data1.read().await; - println!("Read: {:?}", *read); - }); - - let data2 = Arc::clone(&data); - tokio::spawn(async move { - let read = data2.read().await; - println!("Read: {:?}", *read); - }); - - // Single writer - tokio::time::sleep(Duration::from_millis(100)).await; - let mut write = data.write().await; - write.push(4); -} -``` - -## Async Traits (with async-trait) - -```rust -use async_trait::async_trait; - -#[async_trait] -trait AsyncRepository { - async fn find_by_id(&self, id: u64) -> Result; - async fn save(&self, user: User) -> Result<(), Error>; -} - -struct DatabaseRepository { - pool: sqlx::PgPool, -} - -#[async_trait] -impl AsyncRepository for DatabaseRepository { - async fn find_by_id(&self, id: u64) -> Result { - sqlx::query_as("SELECT * FROM users WHERE id = $1") - .bind(id) - .fetch_one(&self.pool) - .await - .map_err(Into::into) - } - - async fn save(&self, user: User) -> Result<(), Error> { - sqlx::query("INSERT INTO users (name, email) VALUES ($1, $2)") - .bind(&user.name) - .bind(&user.email) - .execute(&self.pool) - .await?; - Ok(()) - } -} -``` - -## Pin and Futures - -```rust -use std::pin::Pin; -use std::future::Future; -use std::task::{Context, Poll}; - -// Manual Future implementation -struct DelayedValue { - value: i32, - delay: tokio::time::Sleep, -} - -impl Future for DelayedValue { - type Output = i32; - - fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { - match Pin::new(&mut self.delay).poll(cx) { - Poll::Ready(_) => Poll::Ready(self.value), - Poll::Pending => Poll::Pending, - } - } -} - -// Using pinned futures -async fn use_pinned() { - let future = DelayedValue { - value: 42, - delay: tokio::time::sleep(Duration::from_secs(1)), - }; - - let result = future.await; - println!("Result: {}", result); -} -``` - -## Background Tasks and Graceful Shutdown - -```rust -use tokio::signal; - -async fn background_task(mut shutdown: tokio::sync::watch::Receiver) { - loop { - tokio::select! { - _ = tokio::time::sleep(Duration::from_secs(1)) => { - println!("Background task running..."); - } - _ = shutdown.changed() => { - println!("Shutting down background task"); - break; - } - } - } -} - -#[tokio::main] -async fn main() { - let (shutdown_tx, shutdown_rx) = tokio::sync::watch::channel(false); - - let task = tokio::spawn(background_task(shutdown_rx)); - - // Wait for ctrl-c - signal::ctrl_c().await.unwrap(); - println!("Received shutdown signal"); - - // Signal shutdown - shutdown_tx.send(true).unwrap(); - - // Wait for task to complete - task.await.unwrap(); -} -``` - -## Error Handling in Async - -```rust -use thiserror::Error; - -#[derive(Error, Debug)] -enum AsyncError { - #[error("Network error: {0}")] - Network(#[from] reqwest::Error), - - #[error("Timeout")] - Timeout, - - #[error("Task failed")] - TaskFailed(#[from] tokio::task::JoinError), -} - -async fn robust_operation() -> Result { - let timeout = Duration::from_secs(5); - - let result = tokio::time::timeout(timeout, async { - reqwest::get("https://api.example.com") - .await? - .text() - .await - }) - .await - .map_err(|_| AsyncError::Timeout)??; - - Ok(result) -} -``` - -## Runtime Configuration - -```rust -// Custom runtime configuration -fn main() { - let runtime = tokio::runtime::Builder::new_multi_thread() - .worker_threads(4) - .thread_name("my-worker") - .thread_stack_size(3 * 1024 * 1024) - .enable_all() - .build() - .unwrap(); - - runtime.block_on(async { - println!("Running on custom runtime"); - }); -} - -// Current-thread runtime (single-threaded) -fn single_threaded() { - let runtime = tokio::runtime::Builder::new_current_thread() - .enable_all() - .build() - .unwrap(); - - runtime.block_on(async { - println!("Single-threaded async"); - }); -} -``` - -## Best Practices - -- Use tokio::spawn for CPU-bound tasks on multi-threaded runtime -- Use spawn_blocking for blocking operations (file I/O, sync code) -- Prefer tokio::sync primitives over std::sync in async code -- Use channels for task communication instead of shared state when possible -- Always handle JoinHandle results (tasks can panic) -- Use select! for cancellation patterns -- Avoid holding locks across .await points -- Use timeout for all external I/O operations -- Implement graceful shutdown with channels -- Use async-trait for trait-based async code -- Prefer try_join! over manual error handling -- Use Arc> sparingly (channels often better) -- Test async code with tokio::test macro -- Monitor task spawning to prevent unbounded growth diff --git a/.github/skills/rust-engineer/references/error-handling.md b/.github/skills/rust-engineer/references/error-handling.md deleted file mode 100644 index f09cfc66..00000000 --- a/.github/skills/rust-engineer/references/error-handling.md +++ /dev/null @@ -1,337 +0,0 @@ -# Error Handling in Rust - -> Reference for: Rust Engineer -> Load when: Handling errors, Result, Option, custom error types, thiserror - -## Result and Option Basics - -```rust -// Result: operation that can fail -fn divide(a: f64, b: f64) -> Result { - if b == 0.0 { - Err("Division by zero".to_string()) - } else { - Ok(a / b) - } -} - -// Option: value that might be absent -fn find_user(id: u64) -> Option { - if id == 1 { - Some(User { id, name: "Alice".to_string() }) - } else { - None - } -} - -// Using ? operator for propagation -fn calculate(a: f64, b: f64, c: f64) -> Result { - let x = divide(a, b)?; // Returns Err early if division fails - let y = divide(x, c)?; - Ok(y) -} -``` - -## Custom Error Types - -```rust -use std::fmt; - -// Manual error type -#[derive(Debug)] -enum AppError { - NotFound(String), - InvalidInput(String), - DatabaseError(String), -} - -impl fmt::Display for AppError { - fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - match self { - AppError::NotFound(msg) => write!(f, "Not found: {}", msg), - AppError::InvalidInput(msg) => write!(f, "Invalid input: {}", msg), - AppError::DatabaseError(msg) => write!(f, "Database error: {}", msg), - } - } -} - -impl std::error::Error for AppError {} - -// Usage -fn get_user(id: u64) -> Result { - if id == 0 { - return Err(AppError::InvalidInput("ID cannot be zero".to_string())); - } - // ... fetch user - Err(AppError::NotFound(format!("User {} not found", id))) -} -``` - -## Using thiserror - -```rust -use thiserror::Error; - -#[derive(Error, Debug)] -enum DataError { - #[error("Data not found: {0}")] - NotFound(String), - - #[error("Invalid ID: {id}, reason: {reason}")] - InvalidId { id: u64, reason: String }, - - #[error("IO error")] - Io(#[from] std::io::Error), - - #[error("Parse error")] - Parse(#[from] std::num::ParseIntError), - - #[error("Database error: {0}")] - Database(#[from] sqlx::Error), -} - -// Usage with automatic conversions -fn read_config(path: &str) -> Result { - let content = std::fs::read_to_string(path)?; // Auto-converts io::Error - let port: u16 = content.parse()?; // Auto-converts ParseIntError - Ok(Config { port }) -} -``` - -## Using anyhow for Applications - -```rust -use anyhow::{Result, Context, bail, ensure}; - -// Simple error handling for applications -fn process_file(path: &str) -> Result<()> { - let content = std::fs::read_to_string(path) - .context(format!("Failed to read file: {}", path))?; - - ensure!(!content.is_empty(), "File is empty"); - - if content.len() > 1000 { - bail!("File too large"); - } - - // Process content... - Ok(()) -} - -// Adding context to errors -fn main() -> Result<()> { - process_file("config.txt") - .context("Failed to process configuration")?; - Ok(()) -} -``` - -## Option Combinators - -```rust -// map: transform Option to Option -let num: Option = Some(5); -let doubled = num.map(|n| n * 2); // Some(10) - -// and_then: chain operations -let result = Some(5) - .and_then(|n| if n > 0 { Some(n * 2) } else { None }) - .and_then(|n| Some(n + 1)); // Some(11) - -// or: provide alternative -let value = None.or(Some(42)); // Some(42) - -// unwrap_or: provide default -let value = None.unwrap_or(42); // 42 - -// unwrap_or_else: compute default lazily -let value = None.unwrap_or_else(|| expensive_computation()); - -// filter: conditional None -let num = Some(5).filter(|&n| n > 10); // None - -// Pattern matching -match find_user(1) { - Some(user) => println!("Found: {}", user.name), - None => println!("User not found"), -} - -// if let for simple cases -if let Some(user) = find_user(1) { - println!("Found: {}", user.name); -} -``` - -## Result Combinators - -```rust -// map: transform Ok value -let result: Result = Ok(5); -let doubled = result.map(|n| n * 2); // Ok(10) - -// map_err: transform error -let result: Result = Err("error"); -let mapped = result.map_err(|e| e.to_uppercase()); // Err("ERROR") - -// and_then: chain fallible operations -fn parse_then_double(s: &str) -> Result { - s.parse::() - .and_then(|n| Ok(n * 2)) -} - -// or_else: provide alternative computation -let result = Err("error").or_else(|_| Ok(42)); // Ok(42) - -// unwrap_or: provide default -let value = Err("error").unwrap_or(42); // 42 - -// expect: unwrap with custom panic message -let value = result.expect("Failed to parse number"); - -// Pattern matching -match divide(10.0, 2.0) { - Ok(result) => println!("Result: {}", result), - Err(e) => eprintln!("Error: {}", e), -} -``` - -## Error Conversion and From Trait - -```rust -use std::io; -use std::num::ParseIntError; - -#[derive(Debug)] -enum MyError { - Io(io::Error), - Parse(ParseIntError), -} - -impl From for MyError { - fn from(err: io::Error) -> Self { - MyError::Io(err) - } -} - -impl From for MyError { - fn from(err: ParseIntError) -> Self { - MyError::Parse(err) - } -} - -// Now ? operator works with automatic conversion -fn read_and_parse(path: &str) -> Result { - let content = std::fs::read_to_string(path)?; // io::Error -> MyError - let number = content.trim().parse()?; // ParseIntError -> MyError - Ok(number) -} -``` - -## Advanced Error Patterns - -```rust -// Multiple error sources with Box -use std::error::Error; - -fn complex_operation() -> Result> { - let file = std::fs::read_to_string("data.txt")?; - let number: i32 = file.trim().parse()?; - Ok(format!("Number: {}", number)) -} - -// Error with backtrace (nightly) -#[derive(Debug)] -struct DetailedError { - message: String, - backtrace: std::backtrace::Backtrace, -} - -impl DetailedError { - fn new(message: impl Into) -> Self { - Self { - message: message.into(), - backtrace: std::backtrace::Backtrace::capture(), - } - } -} - -// Recoverable vs unrecoverable errors -fn might_fail(value: i32) -> Result { - if value < 0 { - Err("Negative value".to_string()) // Recoverable - } else if value > 1000 { - panic!("Value too large!"); // Unrecoverable - } else { - Ok(value * 2) - } -} -``` - -## Try Blocks (Nightly) - -```rust -#![feature(try_blocks)] - -// Try block for localized error handling -let result: Result> = try { - let file = std::fs::read_to_string("config.txt")?; - let num: i32 = file.trim().parse()?; - num * 2 -}; -``` - -## Error Context Pattern - -```rust -use thiserror::Error; - -#[derive(Error, Debug)] -#[error("{message}")] -struct ContextError { - message: String, - #[source] - source: Option>, -} - -impl ContextError { - fn new(message: impl Into) -> Self { - Self { - message: message.into(), - source: None, - } - } - - fn with_source(mut self, source: impl Error + Send + Sync + 'static) -> Self { - self.source = Some(Box::new(source)); - self - } -} - -// Extension trait for adding context -trait Context { - fn context(self, message: impl Into) -> Result; -} - -impl Context for Result { - fn context(self, message: impl Into) -> Result { - self.map_err(|e| ContextError::new(message).with_source(e)) - } -} -``` - -## Best Practices - -- Use Result for recoverable errors, panic! for unrecoverable bugs -- Prefer ? operator over unwrap() in production code -- Use expect() with descriptive messages instead of unwrap() -- Use thiserror for libraries (structured errors) -- Use anyhow for applications (simple error handling) -- Implement std::error::Error trait for custom error types -- Add context to errors as they propagate up the stack -- Use #[from] in thiserror for automatic conversions -- Document error conditions in function documentation -- Use Option::ok_or() to convert Option to Result -- Use Result::ok() to convert Result to Option (discarding error) -- Avoid String as error type (use custom types instead) -- Use ensure! and bail! from anyhow for cleaner checks -- Log errors at boundaries, return them in library code diff --git a/.github/skills/rust-engineer/references/ownership.md b/.github/skills/rust-engineer/references/ownership.md deleted file mode 100644 index 6c972d3d..00000000 --- a/.github/skills/rust-engineer/references/ownership.md +++ /dev/null @@ -1,281 +0,0 @@ -# Ownership, Borrowing, and Lifetimes - -> Reference for: Rust Engineer -> Load when: Working with ownership, lifetimes, smart pointers, borrowing - -## Ownership Patterns - -```rust -// Move semantics (ownership transfer) -fn take_ownership(s: String) { - println!("{}", s); -} // s dropped here - -// Borrowing (immutable reference) -fn borrow(s: &String) { - println!("{}", s); -} // s NOT dropped, caller still owns - -// Mutable borrowing -fn borrow_mut(s: &mut String) { - s.push_str(" world"); -} - -// Usage -let s = String::from("hello"); -borrow(&s); // OK, immutable borrow -let mut s2 = s; // Move, s no longer valid -borrow_mut(&mut s2); // OK, mutable borrow -``` - -## Lifetime Annotations - -```rust -// Explicit lifetime: returned reference lives as long as input -fn longest<'a>(x: &'a str, y: &'a str) -> &'a str { - if x.len() > y.len() { x } else { y } -} - -// Multiple lifetimes -fn first_word<'a, 'b>(s: &'a str, _other: &'b str) -> &'a str { - s.split_whitespace().next().unwrap_or("") -} - -// Lifetime in structs -struct Excerpt<'a> { - part: &'a str, -} - -impl<'a> Excerpt<'a> { - fn announce_and_return(&self, announcement: &str) -> &'a str { - println!("Attention: {}", announcement); - self.part - } -} - -// Static lifetime (lives for entire program) -const GREETING: &'static str = "Hello, world!"; -``` - -## Smart Pointers - -```rust -use std::rc::Rc; -use std::cell::RefCell; -use std::sync::{Arc, Mutex}; - -// Box: heap allocation, single owner -let b = Box::new(5); - -// Rc: reference counting (single-threaded) -let rc1 = Rc::new(vec![1, 2, 3]); -let rc2 = Rc::clone(&rc1); // Increment count -println!("Count: {}", Rc::strong_count(&rc1)); // 2 - -// Arc: atomic reference counting (thread-safe) -let arc1 = Arc::new(vec![1, 2, 3]); -let arc2 = Arc::clone(&arc1); -std::thread::spawn(move || { - println!("{:?}", arc2); -}); - -// RefCell: interior mutability (runtime borrow checking) -let data = RefCell::new(5); -*data.borrow_mut() += 1; // Mutable borrow at runtime - -// Combining Rc + RefCell for shared mutable state -let shared = Rc::new(RefCell::new(vec![1, 2, 3])); -shared.borrow_mut().push(4); - -// Combining Arc + Mutex for thread-safe shared state -let counter = Arc::new(Mutex::new(0)); -let counter_clone = Arc::clone(&counter); -std::thread::spawn(move || { - let mut num = counter_clone.lock().unwrap(); - *num += 1; -}); -``` - -## Interior Mutability - -```rust -use std::cell::{Cell, RefCell}; - -// Cell: Copy types only -let c = Cell::new(5); -c.set(10); -let val = c.get(); - -// RefCell: runtime borrow checking -let data = RefCell::new(vec![1, 2, 3]); -data.borrow_mut().push(4); - -// Pattern: mock objects with interior mutability -struct MockLogger { - messages: RefCell>, -} - -impl MockLogger { - fn new() -> Self { - Self { messages: RefCell::new(Vec::new()) } - } - - fn log(&self, msg: &str) { - self.messages.borrow_mut().push(msg.to_string()); - } - - fn get_messages(&self) -> Vec { - self.messages.borrow().clone() - } -} -``` - -## Pin and Self-Referential Types - -```rust -use std::pin::Pin; -use std::marker::PhantomPinned; - -// Self-referential struct (requires Pin) -struct SelfReferential { - data: String, - pointer: *const String, - _pin: PhantomPinned, -} - -impl SelfReferential { - fn new(data: String) -> Pin> { - let mut boxed = Box::pin(Self { - data, - pointer: std::ptr::null(), - _pin: PhantomPinned, - }); - - // Safe: we're not moving the data after this - let ptr = &boxed.data as *const String; - unsafe { - let mut_ref = Pin::as_mut(&mut boxed); - Pin::get_unchecked_mut(mut_ref).pointer = ptr; - } - - boxed - } -} - -// Pin in async contexts -async fn pinned_future() { - // Futures are often self-referential, hence Pin - let fut = async { 42 }; - let pinned = Box::pin(fut); - pinned.await; -} -``` - -## Cow (Clone on Write) - -```rust -use std::borrow::Cow; - -fn process_text(input: &str) -> Cow { - if input.contains("bad") { - // Need to modify: allocate new String - Cow::Owned(input.replace("bad", "good")) - } else { - // No modification needed: just borrow - Cow::Borrowed(input) - } -} - -// Usage -let text1 = "hello world"; -let result1 = process_text(text1); // Borrowed (no allocation) - -let text2 = "bad word"; -let result2 = process_text(text2); // Owned (allocated) -``` - -## Drop Trait and RAII - -```rust -struct FileGuard { - name: String, -} - -impl FileGuard { - fn new(name: String) -> Self { - println!("Opening {}", name); - Self { name } - } -} - -impl Drop for FileGuard { - fn drop(&mut self) { - println!("Closing {}", self.name); - } -} - -// Usage: automatic cleanup -{ - let _file = FileGuard::new("data.txt".to_string()); - // Use file... -} // Drop called automatically here -``` - -## Common Patterns - -```rust -// Builder pattern with ownership -struct Config { - host: String, - port: u16, -} - -impl Config { - fn builder() -> ConfigBuilder { - ConfigBuilder::default() - } -} - -struct ConfigBuilder { - host: Option, - port: Option, -} - -impl ConfigBuilder { - fn host(mut self, host: impl Into) -> Self { - self.host = Some(host.into()); - self - } - - fn port(mut self, port: u16) -> Self { - self.port = Some(port); - self - } - - fn build(self) -> Result { - Ok(Config { - host: self.host.ok_or("host required")?, - port: self.port.unwrap_or(8080), - }) - } -} - -// Usage -let config = Config::builder() - .host("localhost") - .port(3000) - .build()?; -``` - -## Best Practices - -- Prefer borrowing (&T) over ownership transfer when possible -- Use &str over String for function parameters -- Use &[T] over Vec for function parameters -- Clone only when necessary (profile first) -- Use Cow<'a, T> for conditional cloning -- Document lifetime relationships in complex cases -- Use Arc> for shared mutable state across threads -- Use Rc> for shared mutable state in single thread -- Implement Drop for RAII patterns -- Use PhantomData to constrain variance when needed diff --git a/.github/skills/rust-engineer/references/testing.md b/.github/skills/rust-engineer/references/testing.md deleted file mode 100644 index 982f311a..00000000 --- a/.github/skills/rust-engineer/references/testing.md +++ /dev/null @@ -1,473 +0,0 @@ -# Testing in Rust - -> Reference for: Rust Engineer -> Load when: Unit tests, integration tests, property testing, benchmarks - -## Unit Tests - -```rust -// Tests in same file -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn test_addition() { - assert_eq!(2 + 2, 4); - } - - #[test] - fn test_subtraction() { - assert!(10 - 5 == 5); - } - - #[test] - #[should_panic(expected = "division by zero")] - fn test_panic() { - divide(10, 0); - } - - #[test] - fn test_result() -> Result<(), String> { - let result = divide(10, 2)?; - assert_eq!(result, 5); - Ok(()) - } - - #[test] - #[ignore] - fn expensive_test() { - // Run with: cargo test -- --ignored - } -} - -// Assertions -fn assert_examples() { - assert!(true); - assert_eq!(2 + 2, 4); - assert_ne!(2 + 2, 5); - - // Custom messages - assert!(value > 0, "Value must be positive, got {}", value); - assert_eq!(result, expected, "Calculation failed"); -} -``` - -## Doctests - -```rust -/// Adds two numbers together. -/// -/// # Examples -/// -/// ``` -/// use mylib::add; -/// -/// let result = add(2, 3); -/// assert_eq!(result, 5); -/// ``` -/// -/// ```should_panic -/// use mylib::divide; -/// -/// divide(10, 0); // This will panic -/// ``` -/// -/// ```ignore -/// // This code won't compile but won't fail the test -/// let x = undefined_function(); -/// ``` -pub fn add(a: i32, b: i32) -> i32 { - a + b -} -``` - -## Integration Tests - -```rust -// tests/integration_test.rs -use mylib; - -#[test] -fn test_full_workflow() { - let config = mylib::Config::new("test.conf"); - let result = mylib::process(&config); - assert!(result.is_ok()); -} - -// tests/common/mod.rs - shared test utilities -pub fn setup() -> TestContext { - TestContext { - db: create_test_db(), - } -} - -// tests/another_test.rs -mod common; - -#[test] -fn test_with_common() { - let ctx = common::setup(); - // Use ctx... -} -``` - -## Test Organization - -```rust -// Nested test modules -#[cfg(test)] -mod tests { - use super::*; - - mod addition { - use super::*; - - #[test] - fn positive_numbers() { - assert_eq!(add(2, 3), 5); - } - - #[test] - fn negative_numbers() { - assert_eq!(add(-2, -3), -5); - } - } - - mod subtraction { - use super::*; - - #[test] - fn test_subtract() { - assert_eq!(subtract(10, 5), 5); - } - } -} -``` - -## Test Fixtures and Setup - -```rust -struct TestContext { - temp_dir: std::path::PathBuf, - db: Database, -} - -impl TestContext { - fn setup() -> Self { - let temp_dir = std::env::temp_dir().join("test"); - std::fs::create_dir_all(&temp_dir).unwrap(); - - Self { - temp_dir, - db: Database::connect_test(), - } - } -} - -impl Drop for TestContext { - fn drop(&mut self) { - // Cleanup - std::fs::remove_dir_all(&self.temp_dir).ok(); - self.db.disconnect(); - } -} - -#[test] -fn test_with_fixture() { - let ctx = TestContext::setup(); - // Test uses ctx... - // Automatic cleanup via Drop -} -``` - -## Async Tests - -```rust -use tokio; - -#[tokio::test] -async fn test_async_function() { - let result = async_operation().await; - assert_eq!(result, 42); -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 2)] -async fn test_with_custom_runtime() { - let result = concurrent_operation().await; - assert!(result.is_ok()); -} - -// Testing async with timeout -#[tokio::test] -async fn test_with_timeout() { - let timeout = std::time::Duration::from_secs(5); - let result = tokio::time::timeout(timeout, slow_operation()).await; - assert!(result.is_ok()); -} -``` - -## Property-Based Testing (proptest) - -```rust -use proptest::prelude::*; - -// Simple property test -proptest! { - #[test] - fn test_reversing_twice_is_identity(ref s in ".*") { - let reversed: String = s.chars().rev().collect(); - let double_reversed: String = reversed.chars().rev().collect(); - assert_eq!(s, &double_reversed); - } -} - -// Custom strategies -proptest! { - #[test] - fn test_addition_commutative(a in 0..1000i32, b in 0..1000i32) { - assert_eq!(a + b, b + a); - } - - #[test] - fn test_vector_push_pop( - ref v in prop::collection::vec(0..100i32, 0..100), - item in 0..100i32 - ) { - let mut v = v.clone(); - v.push(item); - assert_eq!(v.pop(), Some(item)); - } -} - -// Complex custom strategies -fn user_strategy() -> impl Strategy { - (1..1000u64, "[a-z]{3,10}", "[a-z0-9.]+@[a-z]+\\.[a-z]+") - .prop_map(|(id, name, email)| User { id, name, email }) -} - -proptest! { - #[test] - fn test_user_serialization(user in user_strategy()) { - let json = serde_json::to_string(&user).unwrap(); - let deserialized: User = serde_json::from_str(&json).unwrap(); - assert_eq!(user, deserialized); - } -} -``` - -## Mocking - -```rust -// Using mockall -use mockall::*; -use mockall::predicate::*; - -#[automock] -trait Database { - fn get_user(&self, id: u64) -> Option; - fn save_user(&mut self, user: User) -> Result<(), Error>; -} - -#[test] -fn test_with_mock() { - let mut mock = MockDatabase::new(); - - mock.expect_get_user() - .with(eq(1)) - .times(1) - .returning(|_| Some(User { id: 1, name: "Alice".to_string() })); - - mock.expect_save_user() - .times(1) - .returning(|_| Ok(())); - - // Use mock in test - let user = mock.get_user(1); - assert!(user.is_some()); -} -``` - -## Benchmarks (Criterion) - -```rust -// benches/my_benchmark.rs -use criterion::{black_box, criterion_group, criterion_main, Criterion}; - -fn fibonacci(n: u64) -> u64 { - match n { - 0 => 1, - 1 => 1, - n => fibonacci(n - 1) + fibonacci(n - 2), - } -} - -fn criterion_benchmark(c: &mut Criterion) { - c.bench_function("fib 20", |b| b.iter(|| fibonacci(black_box(20)))); -} - -criterion_group!(benches, criterion_benchmark); -criterion_main!(benches); - -// Cargo.toml: -// [dev-dependencies] -// criterion = "0.5" -// -// [[bench]] -// name = "my_benchmark" -// harness = false -``` - -## Advanced Benchmarking - -```rust -use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main}; - -fn bench_multiple_sizes(c: &mut Criterion) { - let mut group = c.benchmark_group("sorting"); - - for size in [10, 100, 1000, 10000].iter() { - group.bench_with_input(BenchmarkId::from_parameter(size), size, |b, &size| { - b.iter_batched( - || generate_random_vec(size), - |mut v| v.sort(), - criterion::BatchSize::SmallInput, - ); - }); - } - - group.finish(); -} - -// Comparing implementations -fn bench_comparison(c: &mut Criterion) { - let mut group = c.benchmark_group("string_search"); - - group.bench_function("naive", |b| { - b.iter(|| naive_search(black_box("haystack"), black_box("needle"))) - }); - - group.bench_function("optimized", |b| { - b.iter(|| optimized_search(black_box("haystack"), black_box("needle"))) - }); - - group.finish(); -} - -criterion_group!(benches, bench_multiple_sizes, bench_comparison); -criterion_main!(benches); -``` - -## Testing with External Resources - -```rust -// Testing file I/O -#[test] -fn test_file_operations() { - use std::io::Write; - - let temp_dir = std::env::temp_dir(); - let file_path = temp_dir.join("test_file.txt"); - - // Write - let mut file = std::fs::File::create(&file_path).unwrap(); - file.write_all(b"test content").unwrap(); - - // Read - let content = std::fs::read_to_string(&file_path).unwrap(); - assert_eq!(content, "test content"); - - // Cleanup - std::fs::remove_file(&file_path).unwrap(); -} - -// Testing with databases (using sqlx) -#[sqlx::test] -async fn test_database_operations(pool: sqlx::PgPool) -> sqlx::Result<()> { - sqlx::query("INSERT INTO users (name) VALUES ($1)") - .bind("Alice") - .execute(&pool) - .await?; - - let count: (i64,) = sqlx::query_as("SELECT COUNT(*) FROM users") - .fetch_one(&pool) - .await?; - - assert_eq!(count.0, 1); - Ok(()) -} -``` - -## Snapshot Testing - -```rust -// Using insta crate -use insta::assert_snapshot; - -#[test] -fn test_output_format() { - let data = generate_complex_output(); - assert_snapshot!(data); -} - -#[test] -fn test_json_output() { - let json = serde_json::to_string_pretty(&get_data()).unwrap(); - assert_snapshot!(json); -} - -// Run with: cargo insta test -// Review snapshots: cargo insta review -``` - -## Code Coverage - -```rust -// Using tarpaulin -// cargo install cargo-tarpaulin -// cargo tarpaulin --out Html --output-dir coverage - -// Using llvm-cov -// cargo install cargo-llvm-cov -// cargo llvm-cov --html -``` - -## Fuzzing - -```rust -// Using cargo-fuzz -// cargo install cargo-fuzz -// cargo fuzz init - -// fuzz/fuzz_targets/fuzz_target_1.rs -#![no_main] -use libfuzzer_sys::fuzz_target; - -fuzz_target!(|data: &[u8]| { - if let Ok(s) = std::str::from_utf8(data) { - let _ = mylib::parse_input(s); - } -}); - -// Run with: cargo fuzz run fuzz_target_1 -``` - -## Best Practices - -- Write tests alongside production code in #[cfg(test)] modules -- Use integration tests in tests/ directory for end-to-end testing -- Include doctests in documentation for examples that must work -- Use descriptive test names that explain what is being tested -- Test edge cases (empty inputs, max values, etc.) -- Use property-based testing for algorithmic code -- Benchmark performance-critical code with criterion -- Run tests in CI with cargo test --all-features -- Use cargo test -- --nocapture to see println! output -- Test error conditions with #[should_panic] or Result -- Mock external dependencies for unit tests -- Use test fixtures for complex setup/teardown -- Run clippy on test code too -- Measure code coverage and aim for high coverage -- Use fuzzing for security-critical parsers -- Test async code with tokio::test -- Use snapshot testing for complex output validation diff --git a/.github/skills/rust-engineer/references/traits.md b/.github/skills/rust-engineer/references/traits.md deleted file mode 100644 index 19122a91..00000000 --- a/.github/skills/rust-engineer/references/traits.md +++ /dev/null @@ -1,416 +0,0 @@ -# Traits, Generics, and Type System - -> Reference for: Rust Engineer -> Load when: Designing traits, generics, associated types, derive macros - -## Basic Trait Definition - -```rust -// Simple trait -trait Drawable { - fn draw(&self); -} - -// Trait with default implementation -trait Describable { - fn describe(&self) -> String { - String::from("No description available") - } -} - -// Implementing traits -struct Circle { - radius: f64, -} - -impl Drawable for Circle { - fn draw(&self) { - println!("Drawing circle with radius {}", self.radius); - } -} - -impl Describable for Circle { - fn describe(&self) -> String { - format!("A circle with radius {}", self.radius) - } -} -``` - -## Associated Types - -```rust -// Associated types vs generic parameters -trait Container { - type Item; - - fn add(&mut self, item: Self::Item); - fn get(&self, index: usize) -> Option<&Self::Item>; -} - -impl Container for Vec { - type Item = i32; - - fn add(&mut self, item: i32) { - self.push(item); - } - - fn get(&self, index: usize) -> Option<&i32> { - self.get(index) - } -} - -// Iterator trait (standard library example) -trait MyIterator { - type Item; - - fn next(&mut self) -> Option; -} -``` - -## Generic Traits and Bounds - -```rust -// Generic trait with multiple bounds -fn print_info(item: &T) -where - T: std::fmt::Display + std::fmt::Debug, -{ - println!("Display: {}", item); - println!("Debug: {:?}", item); -} - -// Generic struct with trait bounds -struct Pair { - first: T, - second: T, -} - -impl Pair { - fn new(first: T, second: T) -> Self { - Self { first, second } - } - - fn larger(&self) -> &T { - if self.first > self.second { - &self.first - } else { - &self.second - } - } -} - -// Blanket implementation -trait MyTrait { - fn do_something(&self); -} - -impl MyTrait for T { - fn do_something(&self) { - println!("Value: {}", self); - } -} -``` - -## Trait Objects (Dynamic Dispatch) - -```rust -// Static dispatch (monomorphization) -fn static_dispatch(item: &T) { - item.draw(); -} - -// Dynamic dispatch (trait objects) -fn dynamic_dispatch(item: &dyn Drawable) { - item.draw(); -} - -// Storing trait objects -struct Canvas { - shapes: Vec>, -} - -impl Canvas { - fn new() -> Self { - Self { shapes: Vec::new() } - } - - fn add_shape(&mut self, shape: Box) { - self.shapes.push(shape); - } - - fn draw_all(&self) { - for shape in &self.shapes { - shape.draw(); - } - } -} - -// Object safety: traits must meet criteria -trait ObjectSafe { - fn method(&self); // OK: takes &self -} - -trait NotObjectSafe { - fn generic(&self); // NOT OK: generic method - fn by_value(self); // NOT OK: takes self by value -} -``` - -## Derive Macros - -```rust -// Standard derive macros -#[derive(Debug, Clone, PartialEq, Eq, Hash)] -struct User { - id: u64, - name: String, -} - -// Deriving more traits -#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] -struct Point { - x: i32, - y: i32, -} - -// Custom derive with serde -use serde::{Deserialize, Serialize}; - -#[derive(Debug, Serialize, Deserialize)] -struct Config { - host: String, - port: u16, -} -``` - -## Advanced Trait Patterns - -```rust -// Extension trait pattern -trait StringExt { - fn truncate_to(&self, max_len: usize) -> String; -} - -impl StringExt for str { - fn truncate_to(&self, max_len: usize) -> String { - if self.len() <= max_len { - self.to_string() - } else { - format!("{}...", &self[..max_len]) - } - } -} - -// Sealed trait pattern (prevent external implementation) -mod sealed { - pub trait Sealed {} -} - -pub trait MySealed: sealed::Sealed { - fn method(&self); -} - -struct MyType; -impl sealed::Sealed for MyType {} -impl MySealed for MyType { - fn method(&self) { - println!("Implemented"); - } -} - -// Supertraits -trait Printable { - fn print(&self); -} - -trait Loggable: Printable { // Supertrait: must also impl Printable - fn log(&self) { - self.print(); // Can call supertrait methods - } -} -``` - -## Associated Constants - -```rust -trait Config { - const MAX_SIZE: usize; - const DEFAULT_TIMEOUT: u64; -} - -struct ServerConfig; - -impl Config for ServerConfig { - const MAX_SIZE: usize = 1024; - const DEFAULT_TIMEOUT: u64 = 30; -} - -fn use_config() { - println!("Max size: {}", T::MAX_SIZE); -} -``` - -## Generic Associated Types (GATs) - -```rust -// GATs allow generics in associated types -trait LendingIterator { - type Item<'a> where Self: 'a; - - fn next<'a>(&'a mut self) -> Option>; -} - -struct WindowsMut<'data, T> { - data: &'data mut [T], - index: usize, -} - -impl<'data, T> LendingIterator for WindowsMut<'data, T> { - type Item<'a> = &'a mut [T] where Self: 'a; - - fn next<'a>(&'a mut self) -> Option> { - if self.index >= self.data.len() { - return None; - } - - let start = self.index; - self.index += 2; - - Some(&mut self.data[start..start.min(self.data.len())]) - } -} -``` - -## Marker Traits - -```rust -use std::marker::{PhantomData, Send, Sync}; - -// Send: type can be transferred across thread boundaries -// Sync: type can be shared between threads (&T is Send) - -// Custom marker trait -trait Trusted {} - -struct TrustedData { - data: T, - _marker: PhantomData, -} - -impl TrustedData { - fn new(data: T) -> Self { - Self { - data, - _marker: PhantomData, - } - } -} -``` - -## Operator Overloading - -```rust -use std::ops::{Add, Mul}; - -#[derive(Debug, Clone, Copy)] -struct Vector2D { - x: f64, - y: f64, -} - -impl Add for Vector2D { - type Output = Self; - - fn add(self, other: Self) -> Self { - Self { - x: self.x + other.x, - y: self.y + other.y, - } - } -} - -impl Mul for Vector2D { - type Output = Self; - - fn mul(self, scalar: f64) -> Self { - Self { - x: self.x * scalar, - y: self.y * scalar, - } - } -} - -// Usage -let v1 = Vector2D { x: 1.0, y: 2.0 }; -let v2 = Vector2D { x: 3.0, y: 4.0 }; -let v3 = v1 + v2; -let v4 = v1 * 2.5; -``` - -## From/Into Conversion Traits - -```rust -struct UserId(u64); - -impl From for UserId { - fn from(id: u64) -> Self { - UserId(id) - } -} - -// Into is automatically implemented -fn accept_user_id(id: impl Into) { - let user_id = id.into(); - println!("User ID: {}", user_id.0); -} - -// TryFrom for fallible conversions -use std::convert::TryFrom; - -impl TryFrom for UserId { - type Error = &'static str; - - fn try_from(value: i64) -> Result { - if value < 0 { - Err("User ID cannot be negative") - } else { - Ok(UserId(value as u64)) - } - } -} -``` - -## Const Traits (Nightly) - -```rust -// Const trait implementations (requires nightly) -#![feature(const_trait_impl)] - -#[const_trait] -trait ConstAdd { - fn add(self, other: Self) -> Self; -} - -impl const ConstAdd for i32 { - fn add(self, other: Self) -> Self { - self + other - } -} - -const fn compute() -> i32 { - 5.add(10) // Can use in const context -} -``` - -## Best Practices - -- Prefer associated types when there's one clear type per implementation -- Use generic parameters when multiple types might be used simultaneously -- Keep traits small and focused (single responsibility) -- Use extension traits to add functionality to existing types -- Document trait requirements and invariants -- Use marker traits for compile-time guarantees -- Prefer static dispatch for performance, dynamic dispatch for flexibility -- Use #[derive] when possible instead of manual implementations -- Implement standard traits (Debug, Clone, etc.) for better ecosystem integration -- Use sealed traits to prevent external implementations when needed diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 00000000..81b407e5 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,79 @@ +name: CI + +on: + push: + branches: [main] + pull_request: + branches: [main] + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + python: + name: Python (ruff, ty, pytest) + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: dtolnay/rust-toolchain@stable + with: + components: rustfmt, clippy + + - uses: astral-sh/setup-uv@v5 + with: + enable-cache: true + + - name: Install Python 3.12 + run: uv python install 3.12 + + - name: Sync dev environment + run: uv sync --frozen --no-default-groups --group dev --python 3.12 + + - name: Build extension (maturin develop) + run: >- + uv run --python 3.12 maturin develop + --features "extension-module,parallel_ingest,zarr" + + - name: Ruff lint + run: uv run --python 3.12 ruff check . + + - name: Ruff format + run: uv run --python 3.12 ruff format --check . + + - name: ty typecheck + run: uv run --python 3.12 ty check + + - name: pytest + run: uv run --python 3.12 pytest tests/ -q + + rust: + name: Rust (fmt, clippy, test) + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + id: py + with: + python-version: "3.12" + + - uses: dtolnay/rust-toolchain@stable + with: + components: rustfmt, clippy + + - uses: Swatinem/rust-cache@v2 + + - name: cargo fmt + run: cargo fmt --all -- --check + + - name: cargo clippy + env: + PYO3_PYTHON: ${{ steps.py.outputs.python-path }} + run: cargo clippy --locked --all-targets + + - name: cargo test + env: + PYO3_PYTHON: ${{ steps.py.outputs.python-path }} + run: cargo test --locked diff --git a/.github/workflows/lint-format.yml b/.github/workflows/lint-format.yml deleted file mode 100644 index 72f6f0ef..00000000 --- a/.github/workflows/lint-format.yml +++ /dev/null @@ -1,19 +0,0 @@ -name: Format -on: - push - -jobs: - format: - name: Format - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - name: format - uses: chartboost/ruff-action@v1 - with: - args: "format ./pyproject.toml" - - - name: lint - uses: chartboost/ruff-action@v1 - with: - args: "check --fix ./pyproject.toml" diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index fb7bb5a4..c25ccd4a 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -13,7 +13,7 @@ concurrency: cancel-in-progress: true env: - PYTHON_VERSION: '3.9' + PYTHON_VERSION: "3.12" CARGO_INCREMENTAL: 0 CARGO_NET_RETRY: 10 RUSTUP_MAX_RETRIES: 10 diff --git a/.github/workflows/release-please.yml b/.github/workflows/release-please.yml new file mode 100644 index 00000000..1047422d --- /dev/null +++ b/.github/workflows/release-please.yml @@ -0,0 +1,19 @@ +name: Release Please + +on: + push: + branches: [main] + +permissions: + contents: write + pull-requests: write + +jobs: + release-please: + runs-on: ubuntu-latest + steps: + - uses: googleapis/release-please-action@v4 + with: + token: ${{ secrets.GITHUB_TOKEN }} + config-file: release-please-config.json + manifest-file: .release-please-manifest.json diff --git a/.gitignore b/.gitignore index f56241f6..5f83db70 100644 --- a/.gitignore +++ b/.gitignore @@ -7,8 +7,10 @@ __pycache__/ *.py[cod] *$py.class -# C extensions +# C extensions / native Python modules *.so +*.pyd +*.dll # data test/ @@ -23,6 +25,7 @@ test/ # Distribution / packaging .Python .comments +pip-wheel-metadata/ build/ develop-eggs/ dist/ @@ -89,22 +92,13 @@ docs/_build/ # PyBuilder .pybuilder/ -target/ -# Jupyter Notebook -.ipynb_checkpoints - -# IPython +# Jupyter / IPython +.ipynb_checkpoints/ profile_default/ ipython_config.py -# Jupyter Notebook -.ipynb_checkpoints - -# pyenv -# For a library or package, you might want to ignore these files since the code is -# intended to run in multiple environments; otherwise, check them in: -# .python-version +# pyenv / uv: this repository tracks `.python-version` for reproducible interpreters; do not add it here. # pipenv # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. @@ -185,8 +179,58 @@ poetry.toml # ruff .ruff_cache/ +# Astral ty (checker cache if present) +.ty/ + # LSP config files pyrightconfig.json # End of https://www.toptal.com/developers/gitignore/api/python + +### Rust (Cargo, rustc, maturin, PyO3) ### +# This project commits `Cargo.lock`; keep it out of any broad ignore rules below. +# Primary build tree (workspace root). Do not commit compiled crates or incremental state. +/target/ +/debug/ + +# Static and link artifacts sometimes produced at repo root by bad invocations or local experiments +*.rlib +/*.o +/*.obj +/*.a +*.exp +*.ilk + +# rustfmt +**/*.rs.bk + +# MSVC debug symbols +*.pdb + +# macOS debug bundles from `cargo build` / Instruments +**/*.dSYM/ + +# LLVM / rustc / gcov coverage intermediates +*.profraw +*.profdata +*.gcda +*.gcno +*.gcov + python/pyref/data/.pyref_catalog.db + +# Cursor agent hook state (local; not for version control) +.cursor/hooks/state/ + +# macOS / Finder (**/ matches at any depth under the repo) +**/.DS_Store +**/.AppleDouble/ +**/.LSOverride +**/._* +**/.Spotlight-V100/ +**/.Trashes/ +**/.fseventsd/ +**/.AppleDB/ +**/.AppleDesktop/ +**/.VolumeIcon.icns +**/.apdisk diff --git a/.python-version b/.python-version new file mode 100644 index 00000000..e4fba218 --- /dev/null +++ b/.python-version @@ -0,0 +1 @@ +3.12 diff --git a/.release-please-manifest.json b/.release-please-manifest.json new file mode 100644 index 00000000..2e1c40ed --- /dev/null +++ b/.release-please-manifest.json @@ -0,0 +1,3 @@ +{ + ".": "0.8.2" +} diff --git a/AGENTS.md b/AGENTS.md index adecb06d..b9d9cfac 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,228 +1,500 @@ -# PyRef Architecture and Data Flow +# pyref Contributor Quickstart Guide -## Build and test (uv, maturin, cargo only) +`pyref` is a Python library for reducing and analyzing polarized resonant soft X-ray reflectivity (PRSoXR) data collected at ALS Beamline 11.0.1.2, Lawrence Berkeley National Laboratory. The library couples a Python interface with a Rust backend via PyO3 bindings to achieve the parallel throughput required for large beamtime datasets. It is organized into three primary components: `IO`, which handles raw data ingestion and cataloging; `Reduction`, which reduces 2D detector images into 1D reflectivity profiles; and `Fitting`, which fits reduced profiles against optical models. Each component is designed to be independently extensible. -- **Install and run tests**: `uv sync` then `uv run pytest tests/test_rust_fits_io.py` (or `uv run pytest` for full suite). `uv sync` builds the Rust extension via the maturin build backend and installs the project. -- **Build wheel only**: `uv run --group dev maturin build`. Output: `target/wheels/pyref-*.whl`. -- **Rust**: Use `cargo build` only for checking compilation of non-cdylib targets (e.g. bins). The Python extension is built by maturin so that linker flags for the extension are correct. Do not rely on `cargo test` for the main crate; it links the cdylib into the test binary and fails with unresolved Python symbols. Rust unit tests live in `src/` (e.g. `src/fits/header.rs`); integration tests in `tests/integration_test.rs` are `#[ignore]` (require Python runtime; validate via pytest instead). -- **TUI binary**: The lib is built with default feature `extension-module` (pyo3/pyo3-polars). Building the standalone TUI must not link Python. Use: `cargo browser` (alias) or `cargo run --bin pyref-tui --no-default-features --features tui`. The `--no-default-features` disables `extension-module`, so the lib is built without pyo3 and the binary links successfully. Running the TUI requires a real TTY (interactive terminal); in a headless or IDE run context you may see "Device not configured". +## Technical Jargon -## Overview +- **Beamtime**: An experimental allocation at the ALS during which a user group collects data across multiple samples and scans, typically spanning one to several days. +- **Sample**: A physical thin-film specimen characterized by a preparation recipe and a set of deposition conditions. Samples are grouped into series, where each member shares a common recipe and is distinguished by one or more user-assigned tags in the filename. +- **Scan**: A sequential series of frames collected continuously by the instrument under a common set of experimental parameters. Scans are nominally associated with a single sample and a single experiment type. +- **Profile**: A reduced 1D reflectivity curve extracted from a scan, expressed as intensity vs. Q, intensity vs. 2theta, or intensity vs. energy. A single scan may contain multiple profiles collected at different fixed energies or fixed angles. +- **Frame / Image**: A single detector acquisition within a scan, stored as a 2D CCD image accompanied by a FITS header containing motor positions, analog input (AI) values, timestamps, and other metadata. +- **Motor**: A physical positioning device controlling sample theta, CCD theta, beamline energy, steering mirrors, slits, or other instrument components. +- **AI (Analog Input)**: A scalar value recorded by one of the instrument's analog input channels. Relevant AI channels include beam current (ring current), upstream gold mesh absorption current (Ai 3 Izero), photodiode signal, CCD temperature, and TEY signals. +- **Beamspot**: The location of the specularly reflected beam on the 2D detector image, typically a compact Gaussian intensity distribution. +- **ROI (Region of Interest)**: A small rectangular subregion of the detector image, typically 10x10 pixels, centered on the beamspot and used to integrate the reflected beam intensity. +- **Direct Beam**: A series of frames collected with the sample at theta = 0 degrees, where the incident beam hits the detector directly. Used to establish the incident beam intensity I0 as a function of energy or exposure time. +- **I0**: The incident beam intensity used to normalize a reflectivity profile. For fixed-energy scans, I0 is extracted from direct beam frames at the start of the scan. For fixed-angle scans, I0 may be sourced from a separate dedicated scan. +- **I0 Point**: A frame collected as part of the direct beam measurement sequence. I0 points are identified by Sample Theta = 0 and are used both to normalize the reflectivity and to characterize the counting statistics of the incident beam. +- **Stitch Point**: A frame collected at the moment the independent variable reverses direction, signaling the start of a new measurement stitch. Stitch points are typically repeated several times to allow the motors to settle, and they establish the baseline statistics for the new stitch region. +- **Overlap Point**: A frame whose independent variable value falls within the range already covered by the preceding stitch. Overlap points are used to compute the multiplicative scaling factor that aligns the new stitch to the previous one. -PyRef is a library for reducing 2D X-ray reflectivity detector images into 1D reflectivity signals. The library handles experimental data collected in "stitches" - separate measurement chunks where beamline configuration parameters (higher-order suppressor, exit slits, exposure times) are adjusted to capture reflectivity across multiple orders of magnitude. +## Fundamental Concepts -## Terminology (glossary) +### IO Module -See also `CONTRIBUTE.md` for the full glossary. +The IO module is responsible for ingesting raw FITS files, cataloging their contents into a structured SQLite database, and exposing the resulting data through a lazy interface that returns pandas or polars DataFrames on demand. By default, the module expects the FITS file conventions and directory structures produced by ALS Beamline 11.0.1.2. New beamline formats can be added by extending the IO layer. -- **ingest**: Populate the catalog from FITS (discover, read headers, upsert into SQLite). Rust `ingest_beamtime`; Python `pyref.io.ingest_beamtime`. -- **discover**: Find FITS paths under a directory; no header read. Rust `discover_fits_paths`. -- **scan (IO)**: Build a LazyFrame of metadata. `scan_experiment(source)` returns a LazyFrame from catalog or from FITS; "scan from catalog" = read from SQLite (`scan_from_catalog`). Distinct from a CCD Scan (measurement run). -- **scan (experiment)**: A single measurement run with a scan number (e.g. CCD Scan 88169, Scan ID); one reflectivity profile. Not the same as the `scan_experiment()` IO function. -- **read**: FITS file I/O (headers and/or images). Rust `read_fits_headers_only`, `read_multiple_fits_headers_only`, `read_experiment_headers_only`. -- **experiment**: Beamtime directory or a logical group; "experiment number" in headers refers to one scan (run), not the whole beamtime. +All performance-critical IO operations are implemented in Rust and exposed to Python via PyO3 bindings. Rust is responsible for parallel FITS file reading, header card extraction, image data loading, filename parsing, directory traversal, Diesel-managed SQLite catalog construction, and zarr archive management. Python is responsible for the user-facing query interface, DataFrame construction from catalog results, and any logic that does not require parallel throughput. This boundary is a design constraint, not a guideline: do not implement parallelism in Python and do not implement user-facing query logic in Rust. -## Core Components +The catalog database is managed by [Diesel](https://diesel.rs) with the SQLite backend. The schema is defined in `src/schema.rs` and all database interactions must go through Diesel's type-checked query builder. Raw SQL is prohibited except inside Diesel migration files. SQLite foreign key enforcement is off by default; the Rust connection initializer must execute `PRAGMA foreign_keys = ON` on every new connection before any other statement. -### 1. Data Loading (`src/loader.rs`, `python/pyref/io/readers.py`) +The IO module is accessed via `pyref.io` and the cataloging subsystem via `pyref.io.catalog`. -The Rust backend (`src/loader.rs`) handles parallel reading of FITS files: -- Reads FITS file headers and image data -- Processes images to extract beam spot locations -- Calculates simple reflectivity using ROI (Region of Interest) analysis -- Combines multiple FITS files into a single Polars DataFrame -- Adds calculated columns (Q-vector from energy and theta) +### Reduction Module -Key functions: -- `read_fits(source, options)`: High-level eager read. Resolves `source` (file, paths, dir, catalog) with `ResolvePreference`; returns one DataFrame from catalog or from disk. -- `scan_fits(source, options)`: High-level lazy scan. Returns a LazyFrame from catalog (fast) or from disk; use when you want to filter/select before collect. -- `read_fits_metadata_batch(paths, options)`: Canonical batch read (headers + optional calculated domains). Used by ingest and by `read_fits` when source is disk. -- `read_experiment_headers_only()`, `read_multiple_fits_headers_only()`: Lower-level header-only reads. +The Reduction module converts per-frame 2D detector images into normalized, stitched 1D reflectivity profiles. Reduction proceeds in three sequential stages. -**Source and options (Polars-style API):** -- `FitsSource`: enum `File(PathBuf)`, `Paths(Vec)`, `Dir(PathBuf)`, `Catalog(PathBuf)`. Impl `From`, `From>`, `From<&Path>`. -- `ResolvePreference`: `PreferCatalog`, `PreferDisk`, `FromCatalog`, `FromDisk` when both catalog and disk could satisfy the source. -- `ReadFitsOptions` / `ScanFitsOptions`: `header_items`, `header_only`, `add_calculated_domains`, `schema`, `batch_size`, `resolve_preference`, and (for catalog) `catalog_filter`. -- `FitsMetadataSchema`: canonical column names and optional Polars `Schema` for one FITS row; used by `scan_from_catalog` and batch read output. +The first stage localizes the beamspot in each 2D detector image, integrates the ROI intensity, and subtracts the background estimated from a dark region of the detector. The second stage normalizes the extracted beam intensity against I0, beam current, exposure time, and the upstream gold mesh absorption current (Ai 3 Izero). The third stage identifies the scan domain (fixed energy or fixed angle), classifies each frame as an I0 point, stitch point, or overlap point, computes per-stitch scaling corrections from the weighted mean of overlap regions, and assembles the stitched profile. -**Catalog hook:** When source is `Dir(path)` or `Catalog(path)` and `.pyref_catalog.db` exists, `read_fits`/`scan_fits` use it when preference is `FromCatalog` or `PreferCatalog`, otherwise discover and read from disk. All library code must pass `cargo clippy` with no unwrap/expect in non-test code; every public function and module has docstrings. +The Reduction module is accessed via `pyref.reduction`. The primary user-facing class is `PrsoxrLoader`. -#### FITS DataFrame Accessor (`df.fits`) +### Fitting Module -For metadata DataFrames from `scan_experiment().collect()`, use the `fits` accessor to load images. Collect your LazyFrame before using: `df = lf.filter(...).collect()` then `df.fits.img[0]`. +The Fitting module fits reduced reflectivity profiles against optical layer-stack models. The primary backend is `refnx`, which implements the 4x4 transfer matrix method for anisotropic and resonant systems. The module is responsible for model definition, parameter specification, constraint enforcement, objective function construction, fitting algorithm selection, and output formatting. The fitting module is accessed via `pyref.fitting`. -- `df.fits.img[i]` / `df.fits.img[slice]`: Raw detector image(s); slice returns iterator -- `df.fits.corrected(idx, bg_rows=10, bg_cols=10)`: Background-corrected image(s) -- `df.fits.filtered(idx, sigma, bg_rows=10, bg_cols=10)`: Background-corrected + gaussian blurred (Rust pipeline) -- `df.fits.custom(idx, callable, **kwargs)`: Apply custom Python callable to image(s) +## Reduction Subtleties -Background correction uses edge-based subtraction: per-row (left/right) and per-column (top/bottom) with configurable `bg_rows`, `bg_cols`. +### Uncertainty Quantification -### 2. Image Processing (`python/pyref/image.py`, `src/io.rs`) +Each frame is a photon-counting measurement. The raw intensity in each detector pixel follows a Poisson distribution to first approximation, giving a per-pixel standard deviation equal to the square root of the pixel count. Two classes of non-Poissonian noise are also present and must be accounted for. -Image processing reduces 2D detector images to reflectivity values: +Systematic noise originates from the detector itself (readout noise, dark current, stray light) and is characterized using a dark region of the detector image that is far from the beamspot. The mean and variance of this dark region are used to estimate the per-pixel systematic noise floor, which is subtracted from the ROI intensity and propagated into the final uncertainty. -**Rust implementation** (`src/io.rs`): -- `subtract_background()`: Row-by-row background subtraction -- `simple_reflectivity()`: Calculates beam signal vs background using ROI -- `process_image()`: Main image processing pipeline +Random non-Poissonian noise is characterized by the Fano factor, defined as the ratio of the observed variance in I0 measurements to the hypothesized Poissonian variance at the same intensity level. The Fano factor is computed as a function of incident energy from the ensemble of I0 frames within a scan, and is applied as a multiplicative scale on the Poissonian uncertainty for all frames at that energy. Agents implementing or modifying the uncertainty pipeline must propagate both the dark-region contribution and the Fano-scaled Poissonian contribution in quadrature at every reduction step. Silent variance truncation or implicit dtype coercion that reduces numerical precision is a correctness bug. -**Python implementation** (`python/pyref/image.py`): -- `reduce_data()`: Full image reduction pipeline (dezinger, filtering, masking) -- `locate_beam()`: Locates beam spot in processed image -- `reduction()`: Calculates reflectivity from masked image and beam spot +### Beamspot Localization -Uncertainty in reflectivity originates from: -- Counting statistics in beam signal -- Background subtraction uncertainty -- Exposure time and beam current normalization +Beamspot localization is applied to each frame independently. The algorithm proceeds in the following fixed order and agents must not reorder or skip steps without explicit justification. -### 3. Masking (`python/pyref/masking.py`) +First, camera edge artifacts are removed by zeroing or masking a fixed border of pixels around the image perimeter. Second, a row-by-row background subtraction is applied: for each row, the median of a set of dark columns (columns known to be outside the beamspot region) is subtracted from all pixels in that row. Third, a column-by-column background subtraction is applied analogously. Fourth, a Gaussian filter is applied to suppress residual high-frequency noise. Fifth, a 2D peak fitting routine locates the beamspot centroid, integrated intensity, and fit standard deviation. The background intensity and its uncertainty are extracted from a designated dark region of the post-subtraction image. -- `InteractiveImageMasker`: Allows interactive rectangular masking of images -- `ImageSeries.mask()`: Automatic mask generation based on CDF of mean image -- Mask defines which pixels contribute to reflectivity calculation +Failed detections occur when the peak fitter cannot identify a credible Gaussian peak above the noise floor. A detection is considered failed when the fitted peak amplitude is less than a configurable multiple of the dark region standard deviation, or when the fitted centroid falls outside the detector boundary. Failed detections must be flagged in the BeamFinding Table and must not silently propagate NaN or zero values into the Reflectivity Table. Beamspot drift across a scan is expected and is not itself a failure condition; drift is characterized by fitting a linear model to the centroid coordinates as a function of Q or theta. Frames where the centroid deviates from the linear trend by more than a configurable threshold are flagged separately. -### 4. Main Loader (`python/pyref/loader.py`) +### Scan Type and Domain Identification -`PrsoxrLoader` orchestrates the data reduction workflow: +The scan domain is determined by inspecting the motor trajectory across all frames in a scan. The classification procedure is as follows. -- Loads experiment data via `read_experiment()` -- Processes images using the mask -- Creates reflectivity DataFrame with columns: `file_name`, `Q`, `r`, `dr` -- Groups data by `file_name` (each file_name represents a stitch) -- Calculates uncertainty: `dr = sqrt(r)` (Poisson counting statistics) +A scan is classified as a fixed-energy reflectivity scan when a leading block of frames has Sample Theta = 0 and a constant beamline energy (these are the I0 frames), followed by frames in which Sample Theta and CCD Theta increase monotonically (subject to stitch reversals). A scan is classified as a fixed-angle reflectivity scan when either a leading block of frames has Sample Theta = 0 and a varying beamline energy (I0 frames collected as a function of energy), or the scan contains no I0 block at all and beamline energy varies monotonically throughout. A multi-profile scan is not a distinct scan type but is a repetition of the above patterns within a single experimental scan: the instrument completes one full fixed-energy or fixed-angle sweep, then changes the fixed parameter (energy or angle) and repeats the sweep. Multi-profile scans are decomposed into their constituent profiles during reduction, each profile being treated as an independent fixed-energy or fixed-angle scan. -Key properties: -- `refl`: DataFrame containing reflectivity data -- `meta`: Full metadata DataFrame with images -- `mask`: Image mask for beam isolation +Once the domain is identified, stitch points are located by finding frames where the independent variable decreases relative to the preceding frame. Overlap points are the initial frames of a new stitch whose independent variable values fall within the range already covered by the preceding stitch. The scaling correction for each stitch is the weighted mean of the reflectivity values at the overlap points, where the weights are the inverse squared uncertainties of those frames. -### 5. Uncertainty Propagation (`src/lib.rs`, `python/pyref/utils/__init__.py`) +## Filename Parsing -Uncertainty propagation implemented as Polars plugins: +The cataloging system parses FITS filenames to extract the sample name, zero or more tags, the scan number, and the frame number. The parsing contract is as follows and must be implemented exactly as specified. -**Rust functions** (`src/lib.rs`): -- `err_prop_mult()`: Error propagation for multiplication - - Formula: `σ(xy) = |xy| * sqrt((σx/x)² + (σy/y)²)` -- `err_prop_div()`: Error propagation for division - - Formula: `σ(x/y) = |x/y| * sqrt((σx/x)² + (σy/y)²)` -- `weighted_mean()`: Weighted average using inverse variance weights -- `weighted_std()`: Weighted standard deviation +The frame number is always the five-digit zero-padded integer to the right of the last hyphen in the filename stem (before the `.fits` extension). The scan number is always the five-digit zero-padded integer immediately to the left of that hyphen. The remainder of the stem to the left of the scan number is the concatenation of the sample name and any tags, optionally separated by underscores or hyphens. Because no separator is guaranteed between the sample name and the scan number, the scan number anchor is the five digits immediately left of the hyphen; the parser must split there first before attempting to tokenize the sample name and tags. The following filename patterns are all valid and must be handled without special-casing individual formats. -**Python interface** (`python/pyref/utils/__init__.py`): -- Exposes Rust functions as Polars expressions -- Used throughout the data processing pipeline +``` +___-.fits +----.fits +_-.fits +-.fits +-.fits +_-__-.fits +``` + +The number of tags is unbounded. Tags may contain alphanumeric characters and hyphens. Parsing failures must be logged and the offending file flagged in the File Table rather than silently skipped or allowed to panic. + +## Directory Layout Traversal + +Two directory layouts are supported. The cataloging system must detect which layout is present by inspection and handle both without user configuration. + +The first layout places each scan in its own instrument subdirectory within a date-grouped scan directory. Detection criterion: the beamtime root contains one or more date directories, each of which contains one or more scan directories (named with a scan number prefix), each of which contains an instrument subdirectory named either `CCD` or `Axis Photonique`. FITS files live inside the instrument subdirectory. + +``` +/ + / + CCD Scan / + CCD/ + __-.fits + __-AI.txt + CCD Scan / + Axis Photonique/ + __-.fits + __-_AI.txt +``` + +The second layout places all FITS files in a single flat instrument directory directly under the beamtime root. Detection criterion: the beamtime root contains a directory named `CCD` or `Axis Photonique` that holds FITS files from multiple scan numbers. + +``` +/ + CCD/ + __-.fits + __-.fits + __-AI.txt + __-AI.txt +``` + +AI text files are supplementary and are not required for cataloging. If present, they should be associated with their scan by matching the scan number extracted from the filename. If neither layout is detected, the cataloging system must emit a structured error identifying the unrecognized layout rather than silently producing an empty catalog. + +## I/O Operations and Cataloging + +Connecting individual frames back to their originating sample, scan, and beamtime requires meticulous bookkeeping that is impractical to maintain manually across a full beamtime. `pyref` provides an automated cataloging system that ingests a beamtime directory and populates a Diesel-managed SQLite database encoding this hierarchy. The database is the structural backbone for all downstream reduction and fitting workflows. Users interact with it primarily through the lazy DataFrame interface exposed by `pyref.io`, which allows them to filter by sample name, tag, energy, or angle and receive a polars or pandas DataFrame containing the relevant frame metadata and image retrieval handles. + +### Catalog and Cache Storage + +#### Default: local user data directory + +By default, `pyref` maintains a single persistent catalog that accumulates every beamtime the user has ever ingested. The catalog and its associated zarr cache live in the platform-appropriate user data directory, resolved at runtime by the Rust IO layer using the `directories` crate: + +| Platform | Default catalog path | +|----------|----------------------| +| Linux | `$XDG_DATA_HOME/pyref/catalog.db` (falls back to `~/.local/share/pyref/catalog.db`) | +| macOS | `~/Library/Application Support/pyref/catalog.db` | +| Windows | `%APPDATA%\pyref\catalog.db` | + +The zarr archive for each beamtime is stored **on local disk under the same platform data directory** as the catalog, not under the system cache directory: `/pyref/.cache//beamtime.zarr`, where `` is the same root as in the table above (`$XDG_DATA_HOME` or `~/.local/share`, `~/Library/Application Support`, or `%APPDATA%` as appropriate) and `` is a stable SHA-256 digest of the beamtime root path recorded at ingestion time. Example on macOS: `~/Library/Application Support/pyref/.cache//beamtime.zarr`. The zarr tree is local-only; NAS-backed FITS are used for ingestion and re-ingestion, not for routine image reads after ingest. + +Optional environment overrides: `PYREF_CATALOG_DB` (absolute path to `catalog.db`) and `PYREF_CACHE_ROOT` (parent of `/beamtime.zarr` directories). Parallel FITS reads during ingest honor `PYREF_INGEST_WORKER_THREADS` or `PYREF_INGEST_RESOURCE_FRACTION` when explicit kwargs or TUI config fields are unset. + +Ingestion is a **single pipeline**: catalog metadata (Diesel/SQLite) and zarr array writes happen in one pass. There is no separate user-visible “metadata only” phase followed by a later image materialization step. + +The raw FITS files on the NAS are only required during initial ingestion and re-ingestion. After ingestion, reduction and browsing workflows operate from the local catalog and zarr store. If the NAS is unavailable, previously ingested beamtimes remain fully accessible from local storage. + +#### Path aliasing for NAS-sourced data + +Because NAS mount points differ across machines (e.g., `/Volumes/beamdata` on macOS vs. `/mnt/beamdata` on Linux), paths stored in `beamtimes.path` and `files.path` are recorded as logical URIs of the form `nas://