From 25fbc287371225ccec7cb2f5912d8a3fe999b475 Mon Sep 17 00:00:00 2001 From: Heather Patrick Date: Fri, 14 Aug 2026 12:07:02 -0400 Subject: [PATCH 01/13] Added a Troubleshooting Guide Troubleshooting Guide added to enhance usability by providing fixes to common issues users may encounter using cfa-dataops. --- docs/Troubleshooting Guide | 291 +++++++++++++++++++++++++++++++++++++ 1 file changed, 291 insertions(+) create mode 100644 docs/Troubleshooting Guide diff --git a/docs/Troubleshooting Guide b/docs/Troubleshooting Guide new file mode 100644 index 0000000..8527f07 --- /dev/null +++ b/docs/Troubleshooting Guide @@ -0,0 +1,291 @@ +# CFA DataOps — Troubleshooting Guide + +## Purpose + +This guide is intended to improve the user experience with cfa-dataops client capabilities. It provides guidance on troubleshooting common issues that CFA users may encounter when using **CFA dataops** toolkit, catalogs, and reporting utilities. + + +## How to Use This Guide + +1. **Start with a quick health check** (environment, access). +2. **Find your issue** in the sections below. +3. **Implement a solution**, then re‑run the minimal example(s). +4. If the issue persists, collect logs and **open a GitHub issue** in the [Repo](https://github.com/CDCgov/cfa-dataops) + + +## Quick Health Check + +- **Python environment** + + - Confirm Python 3.10+ is active and dependencies installed (pandas, polars, duckdb, papermill, etc.). + +- **Import the core namespaces** + + ```Python``` + + `from cfa.dataops import datacat, reportcat` + + `print("Datasets:", datacat.__namespace_list__)` + + `print("Reports:", reportcat.__namespace_list__)` + + If imports fail, see **Environment & Installation** below. + +- **Azure Cloud access** + + If your workflow requires blob access, ensure you’re logged in, using the appropriate authorization for your context. + + `az login --identity` + +## Common Issues & Solutions + +### 1. Environment & Installation + + **Issue** + + `ModuleNotFoundError` for cfa.dataops, polars, duckdb, or pandera + +**Common causes** + + - Virtual environment not activated. + - Installed with incompatible Python version. + +**Solution** + + - Verify Python listed in pyproject.toml and install/upgrade accordingly. + - Re‑install the project using your team’s standard (e.g., uv, pip, or poetry) and re‑activate the venv. + - Re‑try minimal imports (see Quick Health Check). + + +### 2. Accessing Data — get_dataframe() Errors + + **Issue** + + Errors when loading dataframes (e.g., “no matching version”, “cannot resolve selection”). + +**Common causes** + + - No dataset versions meet your version-spec. + - Using default selection where multiple matches exist. + - Attempting to load large outputs into pandas that exceed memory. + +**Solution** + + - Preview the version that would be used before loading: + + `from cfa.dataops import datacat` + + `resolved = datacat.private.scenarios.covid19vax_trends.load.resolve_version( version_spec=">=2025-05-01,<2025-06-01", selection="newest",)` + + `print(resolved.version)` + + `print(resolved.blob_url)` + + Then pass the same arguments to: + + `get_dataframe()` + + + - List available versions to confirm your constraints + + `datacat.private.scenarios.covid19vax_trends.load.get_versions()` + + If empty or unexpected, re‑run ETL or relax version-spec. + + - Choose appropriate output for size/performance + + \# pandas DataFrame + + `df = datacat.private.scenarios.covid19vax_trends.load.get_dataframe(output="pandas")` + + \# polars DataFrame + + `df_pl = datacat.private.scenarios.covid19vax_trends.load.get_dataframe(output="polars")` + + \# lazy polars (defer materialization) + + `df_lazy = datacat.private.scenarios.covid19vax_trends.load.get_dataframe(output="pl_lazy")` + + Use polars/pl_lazy for large datasets to mitigate memory pressure. + + + - Advanced version filters + + - If you rely on range or pattern filters (e.g., >=..., <..., latest), confirm your string syntax matches the project’s conventions noted in release notes. + + +### 3. Authentication / Azure Blob Access + +**Issue** + + Read/write fails to blob storage; “permission denied” or timeouts. + +**Common causes** + + - Not authenticated in the current shell/session. + - Missing role or wrong subscription/tenant. + - Network constraints. + +**Solution** + + - Log in using your approved method (e.g., az login --identity) and verify subscription. + - If you use helper utilities from the companion cfa-cloudops library for authentication/workflows, ensure it’s correctly configured and you’re on a supported Python version. + - Re‑run a small read_blobs/get_versions check to validate access via the catalog (see **Data User Guide**). + + +### 4. Catalog Creation & Management + + **Issue** + + New datasets or catalogs don’t appear under datacat.__namespace_list__. + +**Common causes** + + - Catalog not installed/registered in the environment. + - Misconfigured dataset definition (TOML). + - Namespace conflicts. + +**Solution** + + - Use the catalog initialization CLI to create a standards‑compliant structure: + + `dataops_catalog_init --help` + + + Then follow the catalog creation and managing guides in docs/. + + - Validate dataset configuration (paths, names, and extract/load endpoints) and re‑install the catalog if needed. (See Managing Catalogs and Catalog Creation pages). + - Restart your Python session to refresh namespace discovery and re‑check: + + `from cfa.dataops import datacat` + + `print(datacat.__namespace_list__)` + + +### 5. Schema Validation Failures + + **Issue** + + Errors citing required columns, types, or value ranges. + + **Common causes** + + - Source feed changed upstream. + - Transform altered types unexpectedly. + - Misaligned schema definitions. + + **Solution** + + - Review the dataset’s schema expectations and correct the ETL or input source. (See Data Validation in the Data User Guide). + + - If recent changes impacted schemas, consult **Release Notes** for updates and migrate accordingly. + + +### 6. Reporting / Notebook Conversion (Reportcat) + + **Issue** + + Notebook→HTML export fails; missing assets or runtime errors. + + **Common causes** + + - Missing notebook dependencies (e.g., papermill, ipykernel, ipywidgets). + - Incorrect report namespace or path. + +**Solution** + + - Verify report namespaces + + `from cfa.dataops import reportcat` + + `print("Reports:", reportcat.__namespace_list__)` + + + - Try a minimal conversion + + `html = reportcat.private.examples.basics_ipynb.nb_to_html_str()` + + If this works, the issue is likely report‑specific. + + - Confirm report dependencies are installed; pyproject.toml lists required packages (e.g., papermill, ipykernel, ipywidgets). + + - Check the **Report Generation** docs for patterns & publishing steps. + + +### 7. Performance & Memory + + **Issue** + + Slow dataframe operations; process killed due to memory. + + **Common causes** + + - Loading large datasets into pandas. + - Inefficient transformations and eager evaluation. + + **Solution** + + - Use Polars or DuckDB for large joins/aggregations + - Use lazy operations via output="pl_lazy" and materialize only the final result. + - Filter and project early (column & row pruning) before joins; verify results on a small sample. + + +### 8. Versioning & Reproducibility + + **Issue** + + Analyses aren’t reproducible; different runs return different data. + + **Common causes** + + - Implicit latest version selection. + - Ambiguous version-spec ranges. + + **Solution** + + - Pin exact versions using timestamp equality: + + `df = datacat.private.scenarios.covid19vax_trends.load.get_dataframe(version_spec="==2025-06-03T17-59-16")` + + - Document your version-spec and selection and store them with the analysis for auditability. + + +## Minimal Working Examples (MWE) + + - List datasets & load one + + `from cfa.dataops import datacat` + + `print(datacat.__namespace_list__)` + + `df = datacat.private.scenarios.covid19vax_trends.load.get_dataframe()` + + - Preview version before load + + `from cfa.dataops import datacatresolved = datacat.private.scenarios.covid19vax_trends.load.resolve_version(version_spec=">=2025-05-01,<2025-06-01", selection="newest",)` + + `print(resolved.version, resolved.blob_url)` + + - Notebook → HTML (quick check) + + `from cfa.dataops import reportcat` + + `html = reportcat.private.examples.basics_ipynb.nb_to_html_str()` + + +## When to Open a GitHub Issue + +Open an issue when: + + - You can reproduce a failure using an MWE above. + - Behavior contradicts documented APIs or release notes. + - A dataset’s schema or versioning appears inconsistent with docs. + +Provide: + + - Python version, package versions (pip list/uv pip list), OS. + - Exact code snippet and traceback. + - Dataset/catalog names and version-spec. + - Whether Azure auth was active (az login), if relevant. + +Use the repo’s [issue tracker](CDCgov/cfa-dataops) From 1f512da802c36a98c71d0238abc51eb97fbc6355 Mon Sep 17 00:00:00 2001 From: Heather Patrick Date: Thu, 20 Aug 2026 08:23:19 -0400 Subject: [PATCH 02/13] Removed reportcat import from troubleshooting guide --- docs/Troubleshooting Guide | 47 ++++---------------------------------- 1 file changed, 5 insertions(+), 42 deletions(-) diff --git a/docs/Troubleshooting Guide b/docs/Troubleshooting Guide index 8527f07..d141c54 100644 --- a/docs/Troubleshooting Guide +++ b/docs/Troubleshooting Guide @@ -23,12 +23,10 @@ This guide is intended to improve the user experience with cfa-dataops client ca ```Python``` - `from cfa.dataops import datacat, reportcat` + `from cfa.dataops import datacat` `print("Datasets:", datacat.__namespace_list__)` - `print("Reports:", reportcat.__namespace_list__)` - If imports fail, see **Environment & Installation** below. - **Azure Cloud access** @@ -181,37 +179,6 @@ This guide is intended to improve the user experience with cfa-dataops client ca - If recent changes impacted schemas, consult **Release Notes** for updates and migrate accordingly. -### 6. Reporting / Notebook Conversion (Reportcat) - - **Issue** - - Notebook→HTML export fails; missing assets or runtime errors. - - **Common causes** - - - Missing notebook dependencies (e.g., papermill, ipykernel, ipywidgets). - - Incorrect report namespace or path. - -**Solution** - - - Verify report namespaces - - `from cfa.dataops import reportcat` - - `print("Reports:", reportcat.__namespace_list__)` - - - - Try a minimal conversion - - `html = reportcat.private.examples.basics_ipynb.nb_to_html_str()` - - If this works, the issue is likely report‑specific. - - - Confirm report dependencies are installed; pyproject.toml lists required packages (e.g., papermill, ipykernel, ipywidgets). - - - Check the **Report Generation** docs for patterns & publishing steps. - - ### 7. Performance & Memory **Issue** @@ -262,17 +229,13 @@ This guide is intended to improve the user experience with cfa-dataops client ca - Preview version before load - `from cfa.dataops import datacatresolved = datacat.private.scenarios.covid19vax_trends.load.resolve_version(version_spec=">=2025-05-01,<2025-06-01", selection="newest",)` + `from cfa.dataops import datacat` - `print(resolved.version, resolved.blob_url)` - - - Notebook → HTML (quick check) + `resolved = datacat.private.scenarios.covid19vax_trends.load.resolve_version(version_spec=">=2025-05-01,<2025-06-01", selection="newest",)` - `from cfa.dataops import reportcat` + `print(resolved.version, resolved.blob_url)` - `html = reportcat.private.examples.basics_ipynb.nb_to_html_str()` - - + ## When to Open a GitHub Issue Open an issue when: From 2100cf81cccb0f46ff5c71886e165eaa43d6cdaf Mon Sep 17 00:00:00 2001 From: Heather Patrick Date: Tue, 25 Aug 2026 22:38:53 +0000 Subject: [PATCH 03/13] removed references to reportcat --- docs/{Troubleshooting Guide => troubleshooting-guide.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename docs/{Troubleshooting Guide => troubleshooting-guide.md} (99%) diff --git a/docs/Troubleshooting Guide b/docs/troubleshooting-guide.md similarity index 99% rename from docs/Troubleshooting Guide rename to docs/troubleshooting-guide.md index d141c54..7ba8d39 100644 --- a/docs/Troubleshooting Guide +++ b/docs/troubleshooting-guide.md @@ -10,7 +10,7 @@ This guide is intended to improve the user experience with cfa-dataops client ca 1. **Start with a quick health check** (environment, access). 2. **Find your issue** in the sections below. 3. **Implement a solution**, then re‑run the minimal example(s). -4. If the issue persists, collect logs and **open a GitHub issue** in the [Repo](https://github.com/CDCgov/cfa-dataops) +4. If the issue persists, collect logs and **open a GitHub issue** in the [Repo](https://github.com/CDCgov/cfa-dataops). ## Quick Health Check From 0ad9b99d63f793a2469de23830922108e9684d5a Mon Sep 17 00:00:00 2001 From: Heather Patrick Date: Tue, 25 Aug 2026 23:27:07 +0000 Subject: [PATCH 04/13] re-checking pre-commit status --- docs/troubleshooting-guide.md | 84 +++++++++++++++++------------------ 1 file changed, 42 insertions(+), 42 deletions(-) diff --git a/docs/troubleshooting-guide.md b/docs/troubleshooting-guide.md index 7ba8d39..8113781 100644 --- a/docs/troubleshooting-guide.md +++ b/docs/troubleshooting-guide.md @@ -3,7 +3,7 @@ ## Purpose This guide is intended to improve the user experience with cfa-dataops client capabilities. It provides guidance on troubleshooting common issues that CFA users may encounter when using **CFA dataops** toolkit, catalogs, and reporting utilities. - + ## How to Use This Guide @@ -16,11 +16,11 @@ This guide is intended to improve the user experience with cfa-dataops client ca ## Quick Health Check - **Python environment** - - - Confirm Python 3.10+ is active and dependencies installed (pandas, polars, duckdb, papermill, etc.). + + - Confirm Python 3.10+ is active and dependencies installed (pandas, polars, duckdb, papermill, etc.). - **Import the core namespaces** - + ```Python``` `from cfa.dataops import datacat` @@ -38,7 +38,7 @@ This guide is intended to improve the user experience with cfa-dataops client ca ## Common Issues & Solutions ### 1. Environment & Installation - + **Issue** `ModuleNotFoundError` for cfa.dataops, polars, duckdb, or pandera @@ -52,7 +52,7 @@ This guide is intended to improve the user experience with cfa-dataops client ca - Verify Python listed in pyproject.toml and install/upgrade accordingly. - Re‑install the project using your team’s standard (e.g., uv, pip, or poetry) and re‑activate the venv. - - Re‑try minimal imports (see Quick Health Check). + - Re‑try minimal imports (see Quick Health Check). ### 2. Accessing Data — get_dataframe() Errors @@ -70,45 +70,45 @@ This guide is intended to improve the user experience with cfa-dataops client ca **Solution** - Preview the version that would be used before loading: - + `from cfa.dataops import datacat` - + `resolved = datacat.private.scenarios.covid19vax_trends.load.resolve_version( version_spec=">=2025-05-01,<2025-06-01", selection="newest",)` - + `print(resolved.version)` - + `print(resolved.blob_url)` - + Then pass the same arguments to: - `get_dataframe()` + `get_dataframe()` - List available versions to confirm your constraints `datacat.private.scenarios.covid19vax_trends.load.get_versions()` - - If empty or unexpected, re‑run ETL or relax version-spec. + + If empty or unexpected, re‑run ETL or relax version-spec. - Choose appropriate output for size/performance - + \# pandas DataFrame - + `df = datacat.private.scenarios.covid19vax_trends.load.get_dataframe(output="pandas")` - + \# polars DataFrame - + `df_pl = datacat.private.scenarios.covid19vax_trends.load.get_dataframe(output="polars")` - + \# lazy polars (defer materialization) - + `df_lazy = datacat.private.scenarios.covid19vax_trends.load.get_dataframe(output="pl_lazy")` - - Use polars/pl_lazy for large datasets to mitigate memory pressure. + + Use polars/pl_lazy for large datasets to mitigate memory pressure. - Advanced version filters - + - If you rely on range or pattern filters (e.g., >=..., <..., latest), confirm your string syntax matches the project’s conventions noted in release notes. @@ -127,14 +127,14 @@ This guide is intended to improve the user experience with cfa-dataops client ca **Solution** - Log in using your approved method (e.g., az login --identity) and verify subscription. - - If you use helper utilities from the companion cfa-cloudops library for authentication/workflows, ensure it’s correctly configured and you’re on a supported Python version. + - If you use helper utilities from the companion cfa-cloudops library for authentication/workflows, ensure it’s correctly configured and you’re on a supported Python version. - Re‑run a small read_blobs/get_versions check to validate access via the catalog (see **Data User Guide**). ### 4. Catalog Creation & Management **Issue** - + New datasets or catalogs don’t appear under datacat.__namespace_list__. **Common causes** @@ -146,24 +146,24 @@ This guide is intended to improve the user experience with cfa-dataops client ca **Solution** - Use the catalog initialization CLI to create a standards‑compliant structure: - + `dataops_catalog_init --help` - - + + Then follow the catalog creation and managing guides in docs/. - Validate dataset configuration (paths, names, and extract/load endpoints) and re‑install the catalog if needed. (See Managing Catalogs and Catalog Creation pages). - Restart your Python session to refresh namespace discovery and re‑check: - + `from cfa.dataops import datacat` - + `print(datacat.__namespace_list__)` - - + + ### 5. Schema Validation Failures **Issue** - + Errors citing required columns, types, or value ranges. **Common causes** @@ -175,14 +175,14 @@ This guide is intended to improve the user experience with cfa-dataops client ca **Solution** - Review the dataset’s schema expectations and correct the ETL or input source. (See Data Validation in the Data User Guide). - + - If recent changes impacted schemas, consult **Release Notes** for updates and migrate accordingly. ### 7. Performance & Memory **Issue** - + Slow dataframe operations; process killed due to memory. **Common causes** @@ -193,14 +193,14 @@ This guide is intended to improve the user experience with cfa-dataops client ca **Solution** - Use Polars or DuckDB for large joins/aggregations - - Use lazy operations via output="pl_lazy" and materialize only the final result. + - Use lazy operations via output="pl_lazy" and materialize only the final result. - Filter and project early (column & row pruning) before joins; verify results on a small sample. ### 8. Versioning & Reproducibility **Issue** - + Analyses aren’t reproducible; different runs return different data. **Common causes** @@ -211,9 +211,9 @@ This guide is intended to improve the user experience with cfa-dataops client ca **Solution** - Pin exact versions using timestamp equality: - + `df = datacat.private.scenarios.covid19vax_trends.load.get_dataframe(version_spec="==2025-06-03T17-59-16")` - + - Document your version-spec and selection and store them with the analysis for auditability. @@ -226,7 +226,7 @@ This guide is intended to improve the user experience with cfa-dataops client ca `print(datacat.__namespace_list__)` `df = datacat.private.scenarios.covid19vax_trends.load.get_dataframe()` - + - Preview version before load `from cfa.dataops import datacat` @@ -234,8 +234,8 @@ This guide is intended to improve the user experience with cfa-dataops client ca `resolved = datacat.private.scenarios.covid19vax_trends.load.resolve_version(version_spec=">=2025-05-01,<2025-06-01", selection="newest",)` `print(resolved.version, resolved.blob_url)` - - + + ## When to Open a GitHub Issue Open an issue when: From 41bd722bb43ea6ad3a70eb0b4816b59665f553ad Mon Sep 17 00:00:00 2001 From: Heather Patrick Date: Wed, 26 Aug 2026 21:08:27 +0000 Subject: [PATCH 05/13] new README file for dataops/tests directory added --- tests/dataops-tests.md | 94 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644 tests/dataops-tests.md diff --git a/tests/dataops-tests.md b/tests/dataops-tests.md new file mode 100644 index 0000000..4904469 --- /dev/null +++ b/tests/dataops-tests.md @@ -0,0 +1,94 @@ +# CFA DataOps Tests + +## Overview +The cfa-dataops/tests directory contains automated checks to help ensure the reliability of **cfa-dataops** library and its supporting utilities. The suite is designed to run locally and in CI, emphasizing fast unit tests while allowing (optional) integration tests that touch cloud resources used by CFA DataOps (e.g. Azure Blob Storage) + + +## Quick Start Checklist +1. Python: install Python 3.10 or newer. + +2. Clone the repo + + `git clone https://github.com/CDCgov/cfa-dataops.git` + + `cd cfa-dataops` + +3. Set up the environment (recommended: uv) + + \# Install project dependencies using uv + + `uv sync` + +4. Authenticate to Azure (Optional) + + if you will run integration tests that touch cloud resources: + + `az login –identity` + +5. Run the tests + + \# All tests (recommended) + + `uv run pytest` + + +## Getting Started +1. Install & Setup + + #### With uv (recommended) + + \# from the repository root + + `uv sync` + + `uv run pytest` + + #### With pip (alternative) + + `python -m venv .venv` + + `source .venv/bin/activate' + + \# Windows: + + `.venv\Scripts\activate` + + `python -m pip install --upgrade pip` + + `pip install -e .` + + `pytest` + +2. Running Specific Tests + + #### Single file or node ID (pytest standard) + + `uv run pytest tests/path/to/testmodule.py::TestClass::testmethod` + + + Selecting tests via node IDs is a standard pytest feature. + - Show detailed output + + `uv run pytest -vv` + +3. Coverage (optional) + + If you’d like coverage reports: + + `uv run pytest --cov=cfa.dataops --cov-report=term-missing` + +4. Cloud-Dependent Tests (optional) + + Some tests may rely on access to CDC cloud resources. + + \# Authenticate (if applicable) + + `az login –identity` + + +## Docs for developers + Project documentation explains how data catalogs and ETL/reporting components work: + - Project documentation + - Data User Guide + - Data Developer Guide + - CLI Tools Reference From dfb3be8ff36a4c5a715029e1f9750bf9a084df1e Mon Sep 17 00:00:00 2001 From: Heather Patrick Date: Thu, 27 Aug 2026 13:16:13 +0000 Subject: [PATCH 06/13] added a key features section --- tests/dataops-tests.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/tests/dataops-tests.md b/tests/dataops-tests.md index 4904469..60a2c28 100644 --- a/tests/dataops-tests.md +++ b/tests/dataops-tests.md @@ -1,8 +1,14 @@ # CFA DataOps Tests ## Overview -The cfa-dataops/tests directory contains automated checks to help ensure the reliability of **cfa-dataops** library and its supporting utilities. The suite is designed to run locally and in CI, emphasizing fast unit tests while allowing (optional) integration tests that touch cloud resources used by CFA DataOps (e.g. Azure Blob Storage) - +The cfa-dataops/tests directory contains automated checks to help ensure the reliability of **cfa-dataops** library and its supporting utilities. The suite is designed to run locally and in CI, emphasizing fast unit tests while allowing (optional) integration tests that touch cloud resources used by CFA DataOps (e.g. Azure Blob Storage). + +## Key Features of Tests Directory +**Pytest-based suite:** Leverages pytest for discovery and execution. +**Mocking support:** Uses pytest-mock to isolate external dependencies during unit testing. +**Property-based tests:** Optionally uses hypothesis to validate invariants across randomized inputs. +**Coverage instrumentation:** Configurable via .coveragerc and pytest-cov. +**Works with uv:** The ecosystem commonly runs commands through uv (e.g., uv run pytest) for consistent environments. ## Quick Start Checklist 1. Python: install Python 3.10 or newer. From d82dacdd5e7ba84106283c5c3ebd5c1b091a6a20 Mon Sep 17 00:00:00 2001 From: Heather Patrick Date: Thu, 27 Aug 2026 13:19:08 +0000 Subject: [PATCH 07/13] added a key features section --- tests/dataops-tests.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tests/dataops-tests.md b/tests/dataops-tests.md index 60a2c28..84b369e 100644 --- a/tests/dataops-tests.md +++ b/tests/dataops-tests.md @@ -4,10 +4,10 @@ The cfa-dataops/tests directory contains automated checks to help ensure the reliability of **cfa-dataops** library and its supporting utilities. The suite is designed to run locally and in CI, emphasizing fast unit tests while allowing (optional) integration tests that touch cloud resources used by CFA DataOps (e.g. Azure Blob Storage). ## Key Features of Tests Directory -**Pytest-based suite:** Leverages pytest for discovery and execution. -**Mocking support:** Uses pytest-mock to isolate external dependencies during unit testing. -**Property-based tests:** Optionally uses hypothesis to validate invariants across randomized inputs. -**Coverage instrumentation:** Configurable via .coveragerc and pytest-cov. +**Pytest-based suite:** Leverages pytest for discovery and execution. +**Mocking support:** Uses pytest-mock to isolate external dependencies during unit testing. +**Property-based tests:** Optionally uses hypothesis to validate invariants across randomized inputs. +**Coverage instrumentation:** Configurable via .coveragerc and pytest-cov. **Works with uv:** The ecosystem commonly runs commands through uv (e.g., uv run pytest) for consistent environments. ## Quick Start Checklist From 95603ee3c76f04dd444ba9d117ceb7ace2f74d59 Mon Sep 17 00:00:00 2001 From: Heather Patrick Date: Thu, 27 Aug 2026 17:59:57 +0000 Subject: [PATCH 08/13] new glossary created --- docs/glossary.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 docs/glossary.md diff --git a/docs/glossary.md b/docs/glossary.md new file mode 100644 index 0000000..e69de29 From 9ffd64eebbb8c743bdcb51fd723c6b89a891af75 Mon Sep 17 00:00:00 2001 From: Heather Patrick Date: Thu, 27 Aug 2026 18:12:35 +0000 Subject: [PATCH 09/13] made a few file edits --- docs/glossary.md | 76 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/docs/glossary.md b/docs/glossary.md index e69de29..864cae4 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -0,0 +1,76 @@ +# CFA DataOps Glossary +This glossary provides clear, CDC-context definiitons of key technologies, tools and concepts frequently used in the **cfa-dataops** environment. It is intended to support new developers onboarding into CFA DataOps workflows. + + +## Azure Blob Storage +**Azure Blob Storage** is Microsoft Azure's cloud object storage solution used for storing large volumes of unstructured data such as CSV files, Parquet datasets, model outputs, logs, and other artifacts. + +### Why it matters in cfa-dataops + - It provides secure, scablable storage for ingestion pipelines, cleaned datasets, and analytical outputs used in CFA modeling and analytics + - Many cfa-dataops integration tests rely on Blob Storage access, which requires authenticating with 'az login --identity` + - Enables cloub-based pipelines that mirror production environments, making local-to-cloud reproducibility easier + + +## Catalog (CFA Catalog) +The **CFA Catalog** is a central structured repository of datasets used by CFA modeling teams. It provides metadata, versioning, provenance, and standardized accessibility, enabling discoverability, reproducibility, and governance. + +### Why it matters in cfa-dataops + - Ensures datasets are well-documented and versioned + - Allows analytics teams to locate authoratative ("source of truth") datasets quickly + - Supports publication workflows for modeling and public-facing data products + - Ensure reproducible analytics across CFA teams. + + +## DuckDB +DuckDB is an in-process OLAP (analytical) database designed for fast, local analytical queries. It runs inside Python and supports fast SQL queries on large data files without requiring a server. + +### Why it matters in cfa-dataops + - Supports SQL, making transformations readable and standardized + - Enables reproducible local pipelines before cloud publication + - Ideal for rapid local development and reproducible ETL workflows + - Efficient for working with large CSV/Parquet datasets locally + + +## Hypothesis +Hypothesis is a property-based testing framework for Python. Instead of manually specifying inputs, Hypothesis automatially generates input data to explore edge cases. + +### Why it matters in cfa-dataops + - Helps ensure reliability of ingestion and transformation functions + - Useful for validating data schemas or catalog consistency rules + - Integrated into cfa-dataops testing alongside pytest (unit + property-based tests, unit + randomized checks) + + +## Polars +Polars is a high-performance DataFrame library for Rust and Python, optimized for tabular data processing. + +### Why it matters in cfa-dataops + - Extremely fast for cleaning, filtering, merging, and reshaping datasets + - Offers better performance compared to pandas for large datasets + - Works seamlessly with DuckDB to deliver flexible, efficient ETL patterns + - Offers declarative query patterns and efficient lazy computation + + +## Pytest +**pytest** is a Python testing framework used to write and execute test suites, including unit tests, integration tests, and property-based tests. + +### Why it matters in cfa-dataops + - CFA DataOps uses pytest as its primary test runner, including support for: + - Discovery of test files + - Mocking with pytest-mock + - Coverage reporting + - Property-based tests via Hypothesis + - Unit tests, + - Integration tests + - pytest integrates seamlessly with uv (uv run pytest) + - supports node ID selection for running specific tests. + + +## UV +`uv` is a fast, modern Python package environment manager designed to replace slower and heavier tools sucha as pip and virtualenv. It ensures reproducible environments and predictable dependency resolution. + +### Why it matters in cfa-dataops + - uv provides reliable installs and consistent execution environments across developer machines and CI + - In cfa-dataops, uv is the recommended setup tool for running tests and syncing dependencies (uv sync, uv run pytest) + - It improves the stability of pipelines and reduces environment drift + + From def90cc66282442f6e81a715ff7c83a01d096620 Mon Sep 17 00:00:00 2001 From: Heather Patrick Date: Thu, 27 Aug 2026 18:17:20 +0000 Subject: [PATCH 10/13] Re-run pre-commit --- docs/glossary.md | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/docs/glossary.md b/docs/glossary.md index 864cae4..a875ef2 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -9,7 +9,7 @@ This glossary provides clear, CDC-context definiitons of key technologies, tools - It provides secure, scablable storage for ingestion pipelines, cleaned datasets, and analytical outputs used in CFA modeling and analytics - Many cfa-dataops integration tests rely on Blob Storage access, which requires authenticating with 'az login --identity` - Enables cloub-based pipelines that mirror production environments, making local-to-cloud reproducibility easier - + ## Catalog (CFA Catalog) The **CFA Catalog** is a central structured repository of datasets used by CFA modeling teams. It provides metadata, versioning, provenance, and standardized accessibility, enabling discoverability, reproducibility, and governance. @@ -54,14 +54,14 @@ Polars is a high-performance DataFrame library for Rust and Python, optimized fo **pytest** is a Python testing framework used to write and execute test suites, including unit tests, integration tests, and property-based tests. ### Why it matters in cfa-dataops - - CFA DataOps uses pytest as its primary test runner, including support for: + - CFA DataOps uses pytest as its primary test runner, including support for: - Discovery of test files - Mocking with pytest-mock - Coverage reporting - - Property-based tests via Hypothesis - - Unit tests, + - Property-based tests via Hypothesis + - Unit tests, - Integration tests - - pytest integrates seamlessly with uv (uv run pytest) + - pytest integrates seamlessly with uv (uv run pytest) - supports node ID selection for running specific tests. @@ -72,5 +72,3 @@ Polars is a high-performance DataFrame library for Rust and Python, optimized fo - uv provides reliable installs and consistent execution environments across developer machines and CI - In cfa-dataops, uv is the recommended setup tool for running tests and syncing dependencies (uv sync, uv run pytest) - It improves the stability of pipelines and reduces environment drift - - From 7f43bbd2df5817c0bd9b4f4fbd35fd87ef90aee0 Mon Sep 17 00:00:00 2001 From: Heather Patrick Date: Wed, 2 Sep 2026 13:43:25 +0000 Subject: [PATCH 11/13] updated data_user_guide to support Polars LazyFrame --- docs/data_user_guide.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/docs/data_user_guide.md b/docs/data_user_guide.md index ff7e417..5bdfb0e 100644 --- a/docs/data_user_guide.md +++ b/docs/data_user_guide.md @@ -18,7 +18,7 @@ df = datacat.private.scenarios.covid19vax_trends.load.get_dataframe() ## Accessing Data -When the ETL pipelines are run, the data sources (raw and/or transformed) are stored into Azure Blob Storage. You can access these datasets directly using the `datacat` interface: +Raw and transformed data produced by ETL pipelines are stored in Azure Blob Storage. You can access these datasets directly using the `datacat` interface: ```python from cfa.dataops import datacat @@ -27,12 +27,18 @@ from cfa.dataops import datacat df = datacat.private.scenarios.covid19vax_trends.load.get_dataframe() # Get raw data as polars DataFrame -df = datacat.private.scenarios.seroprevalence.extract.get_dataframe(output="polars") +raw_df = datacat.private.scenarios.seroprevalence.extract.get_dataframe(output="polars") # Get specific version -df = datacat.private.scenarios.covid19vax_trends.load.get_dataframe( +version_df = datacat.private.scenarios.covid19vax_trends.load.get_dataframe( version_spec="==2025-06-03T17-56-50" ) + +# Get raw or transformed data as Polars Lazyframe +lazy_df = datacat.private.scenarios.seropervalence.extract.get_dataframe(output="pl_lazy") + +# Get reference datasets +ref_df = datacat.reference.my_reference_dataset.get_dataframe() ``` ### Dataset Access Methods @@ -138,6 +144,9 @@ vax_df = datacat.private.scenarios.covid19vax_trends.load.get_dataframe() # Get raw data for analysis raw_vax = datacat.private.scenarios.covid19vax_trends.extract.get_dataframe() + +# Get raw or transformed data as LazyFrame +lazy_vax = datacat.private.scenarios.covid19vax_trends.load.get_dataframe(output="pl_lazy") ``` ### Fetching Versions within a Range From 95810c30e72df8d1507221c1e5cdc3aab87ab809 Mon Sep 17 00:00:00 2001 From: Heather Patrick Date: Fri, 4 Sep 2026 21:14:07 +0000 Subject: [PATCH 12/13] new content added to the file --- docs/data_developer_guide.md | 74 +++++++++++++++++++++++++++++++++++- 1 file changed, 72 insertions(+), 2 deletions(-) diff --git a/docs/data_developer_guide.md b/docs/data_developer_guide.md index e15dee8..648985c 100644 --- a/docs/data_developer_guide.md +++ b/docs/data_developer_guide.md @@ -1,6 +1,6 @@ # Data Developer Guide -This guide explains how to add new datasets and ETL processes to your catalog repositories. +This guide explains how to create, maintain, add update datasets within a **CFA DataOps** catalog repository. A catalog is a Python package that contains one or more datasets, each with configurable TOML-based ETL workflows. This guide is intended for **dataset developers** who author ETL pipelines, add new dataset versions, and manage catalog content, schemas, and validation logic. > **Prerequisites**: You need to have a catalog repository created and installed. See [Managing Catalogs](managing_catalogs.md) for setup instructions. @@ -23,6 +23,22 @@ The ETL pipeline system is built around: - Python ETL scripts that handle extraction, transformation and loading - SQL templates for transformations (optional) - Schema validation using Pandera +- Catalog repository content (datasets/, reports/, workflows/, etc.) +- datacat; the runtime dataset interface used to inspect, validate, and load dataset versions + +## Key directories for developers: + +### `datasets/` +contains TOML files defining dataset ETL pipelines, metadata, validation rules, and staging behaviours. + +### `workflows/` +contains reusable Python modules or workflow scripts supporting ETL. + +### `reports/` +Contains notebook templates or report-genrating logic tied to datasets (optional). + +### `catalog_defaults.toml` +Defines common config shared by all datasets in the catalog (e.g. blob paths, validation defaults). ## Update an existing dataset @@ -52,7 +68,61 @@ To add a new dataset to your catalog repository: 3. Create a new ETL script in `{your_catalog}/workflows/{workflow_type}/` 4. Add SQL transformation templates if using SQL for transforms (these are [Mako templates](https://www.makotemplates.org/)) -### Configuration file +## Versioning Behavior + +Dataset versions are typially timestamped (e.g. 2025-10-31). Developers can: + +Inspect versions + +```python +from cfa.dataops import datacat + +datacat.my_project.my_dataset.load.get_versions() +``` + +Load a version + +```python +df = datacat.my_project.mydataset.load.get_dataframe() +``` + +Load with a version filter + +```python +df = datacat.my_project.my_dataset.load.get_dataframe(version=">2024.12.01,<2025.08") +``` + +See which version would be chosen + +```python +v = datacat.my_project.my_dataset.load.resolve_versions(version="latest") +``` + + + +## Configuration file + +Configuration sections typyically include: + +**[extract]** + +How raw data is sourced. Common patterns include: +- reading Parquet or CSV from blob storage +- applying schema checks on raw fields +- filtering out malformed input + +**[transform]** + +Defines transformation logic. Options include: +- SQL expressions (DuckDB or Polars SQL) +- Python functions +- multistage ETL pipelines (split into etl/modules) + +**[load]** + +Defines how the transformed dataset is written inot versioned storage. +Versions are timestampe-based and automatically assigned when new data is produced. + ```toml title="{your_catalog}/datasets/{dataset_name}.toml" [properties] From 8c9524ade7beeff7e45298a29a7260605e16fc40 Mon Sep 17 00:00:00 2001 From: Heather Patrick Date: Fri, 4 Sep 2026 22:08:05 +0000 Subject: [PATCH 13/13] added uv environment manager --- docs/catalog_creation.md | 27 +++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-) diff --git a/docs/catalog_creation.md b/docs/catalog_creation.md index 96ef458..9b80385 100644 --- a/docs/catalog_creation.md +++ b/docs/catalog_creation.md @@ -8,16 +8,25 @@ The `dataops_catalog_init` command creates a structured repository for managing ## Prerequisites -- Python 3.10 (required) +- Python 3.10 or newer (required) - `cfa.dataops` package installed - Access to create directories in your target location +- uv installed ## Installation +To verify uv installation + +`uv --version` + The `dataops_catalog_init` command is automatically available after installing the `cfa.dataops` package: ```bash pip install cfa.dataops + +or + +uv add cfa.dataops ``` ## Basic Usage @@ -191,9 +200,23 @@ After creating your catalog repository: 2. **Install in editable mode:** ```bash pip install -e .[dev] + + or + + uv pip install -e .[dev] ``` -3. **Start developing your datasets, workflows, and reports** +3. **Synchronize dependenceis** + ```bash + uv sync + ``` + +4. Run Python commands inside the environment +```bash +uv run python -c "from cfa.dataops import datacat; print(datacat._namespace_list_)" +``` + +5. **Start developing your datasets, workflows, and reports** ## Interactive Confirmation