Embed example notebooks in docs

README.md

@@ -20,7 +20,7 @@ In [partnership with NOAA](https://techpartnerships.noaa.gov/tpo_partnership/mak
 
 ## Data
 NNJA datasets are organized by sensor/source (e.g. all-sky radiances from the GOES ABI).
-The list of all NNJA datasets can be found on the [NNJA project page](https://psl.noaa.gov/data/nnja_obs/#data-sources), while the subset that is currently found in the NNJA-AI archive can be found [here](docs/datasets.md) or by exploring the data catalog (this will be be expanding rapidly).
+The list of all NNJA datasets can be found on the [NNJA project page](https://psl.noaa.gov/data/nnja_obs/#data-sources), while the subset that is currently found in the NNJA-AI archive can be found [here](datasets.md) or by exploring the data catalog (this will be be expanding rapidly).
 
 ## Getting Started
 

docs/datasets.md

@@ -114,7 +114,7 @@ dataset = dataset.sel(variables=primary_vars)
 dataset.variables
 ```
 
-Some fields have complex meanings; be sure to read [Understanding the Data](/docs/understanding-the-data.md).
+Some fields have complex meanings; be sure to read [Understanding the Data](understanding-the-data.md).
 
 #### ADPSFC - Synoptic Fixed Land (from WMO SYNOP bulletins)
 
@@ -123,7 +123,7 @@ Some fields have complex meanings; be sure to read [Understanding the Data](/doc
 - Message: `NC000001`
 - Dates processed: `1979-01-01 to 2025-03-31`
 
- Surface data from fixed land stations, received through WMO SYNOP bulletins. This dataset and [NC000101](/docs/datasets.md#adpsfc---synoptic-fixed-land-originally-in-bufr) come from the same sources; the WMO SYNOP bulletins are legacy reporting messages while the BUFR messages are from stations that have migrated to the newer report type. At the current time, most stations have migrated to BUFR.
+ Surface data from fixed land stations, received through WMO SYNOP bulletins. This dataset and [NC000101](#adpsfc-synoptic-fixed-land-originally-in-bufr) come from the same sources; the WMO SYNOP bulletins are legacy reporting messages while the BUFR messages are from stations that have migrated to the newer report type. At the current time, most stations have migrated to BUFR.
 
 
 #### ADPSFC - Synoptic Mobile Land (from WMO SYNOP MOBIL bulletins)
@@ -151,7 +151,7 @@ Surface data from aviation weather reports (METAR/SPECI). This dataset contains
 - Message: `NC000101`
 - Dates processed: `2020-10-22 to 2025-03-31`
 
-Surface data from fixed land stations, originally received in BUFR format. This dataset and [NC000001](/docs/datasets.md#adpsfc---synoptic-fixed-land-from-wmo-synop-bulletins) come from the same sources; the WMO SYNOP bulletins are legacy reporting messages while the BUFR messages are from stations that have migrated to the newer report type. At the current time, most stations have migrated to BUFR.
+Surface data from fixed land stations, originally received in BUFR format. This dataset and [NC000001](#adpsfc-synoptic-fixed-land-from-wmo-synop-bulletins) come from the same sources; the WMO SYNOP bulletins are legacy reporting messages while the BUFR messages are from stations that have migrated to the newer report type. At the current time, most stations have migrated to BUFR.
 
 #### Additional Resources
 
@@ -167,7 +167,7 @@ Surface data from fixed land stations, originally received in BUFR format. This
 - Message: `NC002001`
 - Dates processsed: `2009-12-31 to 2025-03-31`
 
-Upper-air rawinsonde (radiosonde) data. This is a sparse dataset with coverage over land only. The main data packet is temperature, dew point temperature, wind speed, and wind direction, and geopotential, on pressure levels (columns with prefixes, in order, `TMDB_`, `TMDP_`, `WSPD_`, `WDIR_`, `GP10_`). These data are originally packaged into the `UARLV` column, however their structure does not lend itself well to flattening. The UPA profile plotting [example notebook](/example_notebooks/adpupa_profile_example.ipynb) demonstrates some of the nuances of accessing this data.
+Upper-air rawinsonde (radiosonde) data. This is a sparse dataset with coverage over land only. The main data packet is temperature, dew point temperature, wind speed, and wind direction, and geopotential, on pressure levels (columns with prefixes, in order, `TMDB_`, `TMDP_`, `WSPD_`, `WDIR_`, `GP10_`). These data are originally packaged into the `UARLV` column, however their structure does not lend itself well to flattening. The UPA profile plotting [example notebook](example_notebooks/adpupa_profile_example.ipynb) demonstrates some of the nuances of accessing this data.
 
 #### Additional Resources
 - [BUFR source data on AWS](https://noaa-reanalyses-pds.s3.amazonaws.com/index.html#observations/reanalysis/conv/convbufr/adpupa/)

docs/example_notebooks

@@ -0,0 +1 @@
+../example_notebooks

docs/faq.md

@@ -1,49 +1,60 @@
 # Frequently Asked Questions
 
-- [What is the status of this archive?](#what-is-the-status-of-this-archive)
-- [What datasets do you have?](#what-datasets-do-you-have)
-- [What format is the data in and where is it stored?](#what-format-is-the-data-in-and-where-is-it-stored)
-- [Why is are some columns still a structured field?](#why-is-are-some-columns-still-a-structured-field)
-- [Why is this entire column missing data?](#why-is-this-entire-column-missing-data)
-- [Why didn't you use zarr instead of parquet?](#why-didn-t-you-use-zarr-instead-of-parquet)
-- [There are too many columns! How do I filter them?](#there-are-too-many-columns-how-do-i-filter-them)
-- [Will you keep the datasets current with the most recent data?](#will-you-keep-the-datasets-current-with-the-most-recent-data)
-- [How do I get in touch with you?](#how-do-i-get-in-touch-with-you)
+- [Frequently Asked Questions](#frequently-asked-questions)
+  - [What is the status of this archive?](#what-is-the-status-of-this-archive)
+  - [What datasets do you have?](#what-datasets-do-you-have)
+  - [What format is the data in and where is it stored?](#what-format-is-the-data-in-and-where-is-it-stored)
+  - [What have you done to process the NNJA BUFR files?](#what-have-you-done-to-process-the-nnja-bufr-files)
+  - [Why are some columns still a structured field?](#why-are-some-columns-still-a-structured-field)
+  - [Why didn't you use zarr instead of parquet?](#why-didnt-you-use-zarr-instead-of-parquet)
+  - [There are too many columns! How do I filter them?](#there-are-too-many-columns-how-do-i-filter-them)
+  - [What are code and flag table variables?](#what-are-code-and-flag-table-variables)
+  - [Why is this or that entire column missing data?](#why-is-this-or-that-entire-column-missing-data)
+  - [Will you keep the datasets current with the most recent data?](#will-you-keep-the-datasets-current-with-the-most-recent-data)
+  - [How do I get in touch with you?](#how-do-i-get-in-touch-with-you)
 
 ## What is the status of this archive?
+
 The NNJA-AI v1 release contains all the data currently processed to parquet form at time of writing (Aug 2025). Data processing from the NNJA archive is ongoing at Brightband. New datasets will be added periodically.
 
 ## What datasets do you have?
-All datasets currently available are listed in the [datasets documentation](/docs/datasets.md).
+
+All datasets currently available are listed in the [datasets documentation](datasets.md).
 We are working to add more datasets to the archive, and will update the documentation as we add more datasets.
 
 ## What format is the data in and where is it stored?
+
 The data is stored on GCS in parquets with partitions for each day.
-See the example notebook [here](/example_notebooks/basic_dataset_example.ipynb) for a guide on how to access the data.
+See the example notebook [here](example_notebooks/basic_dataset_example.ipynb) for a guide on how to access the data.
 If you prefer to bypass this SDK, you can currently find the v1 datasets here:
 `gs://nnja-ai/data/v1/` for direct access to the parquet files.
 
 ## What have you done to process the NNJA BUFR files?
+
 1) convert from BUFR to AVRO, preserving all the structure of the original BUFR messages (nested types, etc.).
 2) 'flatten' the complex columns (array, struct) into simple scalar columns.
 3) combined 6-hourly files into daily Parquet partitions based on the observation timestamp.
 
 ## Why are some columns still a structured field?
+
 The original BUFR data is highly structured, with multiple levels of nested data.
 While we have flattened the data as much as possible, there are some cases where the data is still structured.
 This is either because the original data could not be flattened
 (this is the case for fields which are variable-length lists, such as the upper air significant level data),
 or because the data is not worth flattening (e.g. because all subcolumns are null-valued).
 
 ## Why didn't you use zarr instead of parquet?
+
 We considered using zarr as the underlying storage format, but decided to use parquet for a few reasons:
+
 - The original BUFR data is inherently tabular, and parquet is a good format for tabular data.
 - Generally the data is one-dimensional (time), and zarr's strength is in multi-dimensional data.
 There are some datasets that have some additional dimensions (e.g. channel for satellite data),
 but we decided to stick with one format for simplicity.
 - There are many tools and libraries that support parquet (e.g. pandas, polars, dask, BigQuery, etc.)
 
 ## There are too many columns! How do I filter them?
+
 We have classified the columns into the following categories, based on our understanding of the data:
 `primary data`, `primary descriptors`, `secondary data`, `secondary descriptors`.
 You can quickly filter the columns with a snippet like the following:
@@ -54,6 +65,7 @@ dataset = dataset.sel(variables=primary_vars)
 ```
 
 ## What are code and flag table variables?
+
 Some variables are encoded as a code table or flag table.
 We're working on adding a helper utility to incorporate these into the archive and decode them,
 but in the meantime you can use the Variable.info() method to view the code table or flag table to link to the NOAA code table page, or look them up directly here: https://www.nco.ncep.noaa.gov/sib/jeff/CodeFlag_0_STDv31_LOC7.html
@@ -72,9 +84,10 @@ ds.variables['SIDENSEQ.SIDGRSEQ.SAID'].extra_metadata
 
 
 ## Why is this or that entire column missing data?
+
 We included all data fields from the original BUFR data.
 Some data fields were not populated in the original data (e.g. GOES warm channels are not present in the ABI data).
-If the [datasets documentation](/docs/datasets.md) does not mention a missing field that you think should be present,
+If the [datasets documentation](datasets.md) does not mention a missing field that you think should be present,
 please raise an issue on [GitHub](https://github.com/brightbandtech/nnja-ai/issues)
 and we will check with the data providers and update the documentation (or if you check, please let us know!).
 

example_notebooks/adpsfc_example.ipynb

@@ -4,24 +4,21 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/brightbandtech/nnja-ai/blob/main/example_notebooks/adpsfc_example.ipynb)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Uncomment the following line to install the package\n",
-    "#!pip install git+https://github.com/brightbandtech/nnja-ai.git"
+    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/brightbandtech/nnja-ai/blob/main/example_notebooks/adpsfc_example.ipynb)\n",
+    "\n",
+    "On Colab, run the following command in a Python cell.\n",
+    "\n",
+    "``` sh\n",
+    "!pip install git+https://github.com/brightbandtech/nnja-ai.git\n",
+    "```"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "## Navigating APDSFC data\n",
+    "\n",
     "The ADPSFC datasets, representing surface station observations, are very rich in data, but as a result, a bit difficult to dig through. Here we'll shows a few ways to explore the dataset and find the most useful variables"
    ]
   },
@@ -144,7 +141,8 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Subsetting to primary variables\n",
+    "## Subsetting to primary variables\n",
+    "\n",
     "A dataset like ADPSFC can have useful variables (such as measured air temperature, \"TMPSQ1.TMDB\"), along with less useful ones (such as the usually unreported estimated rate of ice accretion, \"ICESQ1.ROIA\"). Similarly, there are key descriptor fields (e.g. LAT and LON) and less valuable descriptor fields (e.g. \"TMPSQ1.MSST\", the method of water temperature and/or salinity measurement). All of these fields are included in the NNJA-AI dataset, but a quick way to subset the variables to explore is to use the 'category' field, which we have added to help filter out some of these less salient fields."
    ]
   },
@@ -390,19 +388,21 @@
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
+   "cell_type": "markdown",
    "metadata": {},
-   "outputs": [],
-   "source": []
+   "source": [
+    "Plot a map of temperatures for a day"
+   ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
-    "# plot map of temperatures for a day"
+    "import matplotlib.pyplot as plt\n",
+    "\n",
+    "plt.scatter(x=\"LON\", Y=\"LAT\", c=\"TMPSQ1.TMDB\", data=ds)"
    ]
   }
  ],

example_notebooks/adpupa_profile_example.ipynb

@@ -4,17 +4,21 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/brightbandtech/nnja-ai/blob/main/example_notebooks/adpupa_profile_example.ipynb)"
+    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/brightbandtech/nnja-ai/blob/main/example_notebooks/adpupa_profile_example.ipynb)\n",
+    "\n",
+    "On Colab, run the following command in a Python cell.\n",
+    "\n",
+    "``` sh\n",
+    "!pip install git+https://github.com/brightbandtech/nnja-ai.git\n",
+    "```"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Uncomment the following line to install the package\n",
-    "#!pip install git+https://github.com/brightbandtech/nnja-ai.git\n",
     "import pandas as pd\n",
     "import numpy as np\n",
     "import matplotlib.pyplot as plt"

example_notebooks/basic_dataset_example.ipynb

@@ -4,17 +4,13 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/brightbandtech/nnja-ai/blob/main/example_notebooks/basic_dataset_example.ipynb)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Uncomment the following line to install the package\n",
-    "#!pip install git+https://github.com/brightbandtech/nnja-ai.git"
+    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/brightbandtech/nnja-ai/blob/main/example_notebooks/basic_dataset_example.ipynb)\n",
+    "\n",
+    "On Colab, run the following command in a Python cell.\n",
+    "\n",
+    "``` sh\n",
+    "!pip install git+https://github.com/brightbandtech/nnja-ai.git\n",
+    "```"
    ]
   },
   {
@@ -402,13 +398,6 @@
     "df = amsu_ds.load_dataset(backend=\"pandas\")\n",
     "plot_df(df, \"BRITCSTC.TMBR_00001\")"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {

example_notebooks/dataset_overview.ipynb

@@ -1,5 +1,19 @@
 {
  "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "04f71471",
+   "metadata": {},
+   "source": [
+    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/brightbandtech/nnja-ai/blob/main/example_notebooks/dataset_overview.ipynb)\n",
+    "\n",
+    "On Colab, run the following command in a Python cell.\n",
+    "\n",
+    "``` sh\n",
+    "!pip install git+https://github.com/brightbandtech/nnja-ai.git\n",
+    "```"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "ea4cd315",
@@ -203,14 +217,6 @@
    "source": [
     "ds.manifest.index"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "b90391aa",
-   "metadata": {},
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {

mkdocs.yml

@@ -21,6 +21,7 @@ extra_css:
 
 plugins:
   - search
+  - mkdocs-jupyter
   - mkdocs-autoapi:
       autoapi_dir: src
   - mkdocstrings:
@@ -45,4 +46,10 @@ nav:
   - Data:
     - Datasets: datasets.md
     - Understanding the Data: understanding-the-data.md
+  - Example Notebooks:
+    - Basic Dataset Example: example_notebooks/basic_dataset_example.ipynb
+    - Dataset Overview: example_notebooks/dataset_overview.ipynb
+    - ADPSFC (weather station) Example: example_notebooks/adpsfc_example.ipynb
+    - ADPUPA (radiosonde) Profile Example: example_notebooks/adpupa_profile_example.ipynb
+
   - FAQ: faq.md

pyproject.toml

@@ -64,6 +64,7 @@ docs = [
     "mkdocstrings[python]>=0.29.1",
     "mkdocs-autoapi[python]>=0.4.1",
     "mkdocs-material>=9.6.14",
+    "mkdocs-jupyter>=0.25.1",
 ]
 interactive = [
     "ipykernel",