Phenobase Data Workflow

User Guide

Phenobase data loading prepares source observation releases for indexing in the Phenobase Elasticsearch datastore. The workflow begins by rebuilding the ontology-derived trait hierarchy, then uses that trait mapping during dataset normalization, validation, and loading so incoming observations can be indexed with consistent traitUrn, trait, mappedTraits, and mappedTraitsUrns values. For the current published trait outputs, use the traits viewer and the published traits.csv.

View Traits

Technical Details And Implementation Guide

What This Repo Does

This repository supports the Phenobase data pipeline around three core jobs:

Rebuild the ontology-derived trait hierarchy in data/traits.csv.
Load source CSV datasets into the Elasticsearch index used by Phenobase.
Maintain and inspect the live datastore with export and backfill helpers.

The trait reasoning step comes first. The loader depends on data/traits.csv to resolve each incoming traitUrn or source trait_urn into the current canonical trait label, derived mappedTraits, and derived mappedTraitsUrns used later for indexing and querying.

Reasoning

Reasoning is the first step because it generates the trait lookup consumed during ingestion.

Source of truth for the current workflow:

PPO GitHub main: https://raw.githubusercontent.com/PlantPhenoOntology/ppo/refs/heads/main/ppo.owl

For repeatability, this repo supports two ways to regenerate the trait mapping:

the default code method, which writes the canonical data/traits.csv
a separate ROBOT/SPARQL method, which writes reasoning/robot/traits.csv and a comparison report against the canonical file

Method 1: Default Code Method

Run the canonical rebuild from the repo root:

python3 reasoning/refresh_traits.py

Compatibility wrapper:

./reasoning/get_traits.sh

This method:

downloads the current PPO ontology from GitHub main
snapshots the exact ontology used under reasoning/<version>/ppo.owl
regenerates data/traits.csv
publishes docs/traits.csv for GitHub Pages
publishes docs/traits-data.json for the static viewer
writes reasoning/traits_build_metadata.json

Method 2: ROBOT/SPARQL Method

The SPARQL query is kept in its own file for repeatability:

reasoning/robot/traits_pairs.sparql

The direct ROBOT query command is:

../robot/robot query \
  --input reasoning/2026-05-06/ppo.owl \
  --query reasoning/robot/traits_pairs.sparql reasoning/robot/traits_pairs.csv

That query emits flat trait-to-mapped-trait pairs. To group those pairs back into the Phenobase traits.csv shape and compare them with the canonical file, run:

python3 reasoning/refresh_traits_robot.py --skip-query \
  --input-owl reasoning/2026-05-06/ppo.owl \
  --pairs-output reasoning/robot/traits_pairs.csv

If you want the helper to run both the ROBOT query and the post-processing for you, use:

python3 reasoning/refresh_traits_robot.py

The ROBOT/SPARQL path writes:

reasoning/robot/traits_pairs.csv: raw ROBOT query output
reasoning/robot/traits.csv: grouped alternative traits output
reasoning/robot/comparison.json: comparison to data/traits.csv

Current comparison summary for the local 2026-05-06 snapshot:

189/189 traits matched by mapped-ID set semantics
170/189 rows matched exactly
the remaining differences are ordering differences in mappedTraitIDs and mappedTraits

Current reasoning artifacts:

Quick verification after a rebuild:

python3 -m json.tool reasoning/traits_build_metadata.json
git diff -- data/traits.csv docs/traits.csv docs/traits-data.json
python3 -m json.tool reasoning/robot/comparison.json

Trait mapping rules used by the rebuild:

labels ending in present are included
labels ending in absent are included
present traits map to themselves plus transitive named PPO superclass traits that also end in present
absent traits map only to themselves

Ingest Procedure

Use this sequence for a new Phenobase data release:

Refresh the ontology-driven trait mapping with python3 reasoning/refresh_traits.py.
Review reasoning/traits_build_metadata.json and diff data/traits.csv.
Place source .csv files in the release directory you want to ingest.
Add dataset-local mappings.csv and/or transform.yaml if source-specific cleanup, case normalization, regex mapping, null handling, or trait remapping is needed. Use mappings.csv for source-to-PPO ID lookup and transform.yaml for row-level cleanup.
Confirm the loader inputs are in place:
- data/columns.csv defines field datatypes, requiredness, and system fields
- data/traits.csv provides the PPO ID-to-trait, mappedTraits, and mappedTraitsUrns expansion generated by the reasoning step
- transform.yaml is optional and only applies to the dataset directory being loaded
Before validation or loading, extend the reusable GBIF family cache for the source CSVs you are about to load. This keeps normal loader runs from blocking on live GBIF requests:

python3 prepare_gbif_family_cache.py \
  --names-from-csv <data_dir> \
  --gbif-workers 16

Run a dry ingestion pass in cache-only mode:

python3 loader.py --mode=<machine|in_situ|herbarium> --test --no-drop-existing --gbif-cache-only <data_dir>

Review loading_errors.csv. Common failures are:
- missing annotationID
- duplicate annotationID within an input file
- empty trait and traitUrn/source trait_urn
- traitUrn or source trait_urn values not found in data/traits.csv
- legacy trait values not found in data/traits.csv when no trait ID is provided
- strict-mode coercion failures for typed fields
Fix the source files or the dataset-local transform.yaml, then rerun test mode until the release is acceptable.
Run the production reload:

python3 loader.py --mode=<machine|in_situ|herbarium> --drop-source-records --gbif-cache-only <data_dir>

This is the default production pattern for a complete source refresh. It deletes existing records whose dataSource matches the input CSVs, then loads the regenerated records into the same index. It prevents stale records from surviving when regenerated annotationID values change.

If you are only appending new records or intentionally upserting by existing annotationID values without deleting stale source records, use:

python3 loader.py --mode=<machine|in_situ|herbarium> --no-drop-existing --gbif-cache-only <data_dir>

If you are rebuilding the index from scratch rather than appending or updating, use:

python3 loader.py --mode=<machine|in_situ|herbarium> --drop-existing --gbif-cache-only <data_dir>

If the live index needs decadeStart, run:

python3 update_decade_start.py --wait

Loading Data

The main loader is loader.py.

What the loader does during a run:

recursively finds .csv files under the provided dataset directory
applies optional row-level transforms from dataset-local transform.yaml
coerces values to the datatypes defined in data/columns.csv
computes system fields such as standardizedFamily, mappedTraits, mappedTraitsUrns, and taxonSearch
optionally deletes existing records for the input dataSource value(s) with --drop-source-records
rejects rows with missing annotationID, duplicate annotationID within a file, or unmapped traits
writes row-level failures to loading_errors.csv
indexes to phenobase2 using annotationID as the Elasticsearch document _id

Supported loading modes:

machine
in_situ
herbarium

Example commands:

python3 loader.py --mode=machine --drop-source-records --gbif-cache-only data/annotations.07.25.2025/ --batch-size 5000 --progress-every 50000
python3 loader.py --mode=in_situ --drop-source-records --gbif-cache-only data/npn.1956.01.01-2025.08.31/ --batch-size 5000 --progress-every 50000

GBIF Family Cache For Normal Loads

standardizedFamily is a system field derived from scientificName. The shared cache lives at:

downloads/taxonomy_family_compare/gbif_family_cache.csv

For normal source loads, pre-resolve names into that cache before running loader.py. This avoids slow per-row live GBIF calls during validation and indexing.

python3 prepare_gbif_family_cache.py \
  --names-from-csv downloads/npn/ingest \
  --names-from-csv downloads/phenoObs/ingest \
  --names-from-csv downloads/budburst/ingest \
  --names-from-csv downloads/seasonwatchindia/ingest \
  --gbif-workers 16 \
  --cache-write-every 500

Then run the loader with --gbif-cache-only:

python3 loader.py --mode=in_situ --test --no-drop-existing --gbif-cache-only downloads/npn/ingest
python3 loader.py --mode=in_situ --drop-source-records --gbif-cache-only downloads/npn/ingest

The loader uses cached family values even when live GBIF calls are disabled. Cache misses remain blank, so the cache-prep step should be run before production loads. --no-gbif-family is kept as a backward-compatible alias for the same cache-only behavior.

Full Reload Synopsis

For a source refresh in the current Elasticsearch index, generate each dataset-specific loader CSV first, dry-run each directory, then run the real loads with --drop-source-records. That option scans the input CSVs for dataSource values, deletes existing records in phenobase2 for those values, and then indexes the regenerated records. Use --drop-existing only when intentionally rebuilding the whole index from scratch.

Refresh the source CSVs:

# NPN / NEON, from downloads/npn
cd downloads/npn
python3 fetchAndTransformNPNData.py 1956-01-01 2026-05-17 ./mappings.csv
mkdir -p ingest
mv npn_observations_1956-01-01_to_2026-05-17.csv ingest/
cd ../..

# PhenoObs, from repo root
python3 downloads/phenoObs/prepare_phenoobs.py \
  --raw-root downloads/phenoObs \
  --mappings downloads/phenoObs/mappings.csv \
  --output downloads/phenoObs/ingest/phenoObs_observations.csv

# Budburst, from repo root
python3 downloads/budburst/fetch_budburst.py \
  --output downloads/budburst/ingest/budburst_observations.csv \
  --workers 3 \
  --timeout 300 \
  --retries 8

# SeasonWatch India, from repo root. Downloads the DwC-A only if missing.
# By default this resolves GBIF occurrence keys into direct GBIF record URLs
# and caches them in downloads/seasonwatchindia/seasonwatchindia_gbif_keys.csv.
python3 downloads/seasonwatchindia/fetch_seasonwatchindia.py \
  --output downloads/seasonwatchindia/ingest/seasonwatchindia_observations.csv

# iNaturalist
# Note that we have another set of procedures to generate the iNaturalist archive not covered here
unzip -j downloads/iNaturalist/ingest/inat.csv.gz

# Herbarium
# Note that we have another set of procedures to generate the herbarium archive not covered here
mkdir -p downloads/herbarium/ingest
unzip -j downloads/herbarium/herb_flower_inference_9.8.25.csv.zip 'flower_inference_formatted_edit_9.8.25.csv' -d downloads/herbarium/ingest

Refresh the GBIF family cache from the prepared ingest CSVs before validation or loading:

python3 prepare_gbif_family_cache.py \
  --names-from-csv downloads/npn/ingest \
  --names-from-csv downloads/phenoObs/ingest \
  --names-from-csv downloads/budburst/ingest \
  --names-from-csv downloads/seasonwatchindia/ingest \
  --names-from-csv downloads/iNaturalist/ingest \
  --gbif-workers 16 \
  --cache-write-every 500

Run validation first:

python3 loader.py --mode=in_situ --test --no-drop-existing --gbif-cache-only downloads/npn/ingest --batch-size 5000 --progress-every 50000
python3 loader.py --mode=in_situ --test --no-drop-existing --gbif-cache-only downloads/phenoObs/ingest --batch-size 5000 --progress-every 50000
python3 loader.py --mode=in_situ --test --no-drop-existing --gbif-cache-only downloads/budburst/ingest --batch-size 5000 --progress-every 50000
python3 loader.py --mode=in_situ --test --no-drop-existing --gbif-cache-only downloads/seasonwatchindia/ingest --batch-size 5000 --progress-every 50000
python3 loader.py --mode=in_situ --test --no-drop-existing --gbif-cache-only downloads/iNaturalist/ingest --batch-size 5000 --progress-every 50000

Run the real in-situ reload. Each command drops only the matching source records in the current index before loading:

python3 loader.py --mode=in_situ --drop-source-records --gbif-cache-only downloads/npn/ingest --batch-size 5000 --progress-every 50000
python3 loader.py --mode=in_situ --drop-source-records --gbif-cache-only downloads/phenoObs/ingest --batch-size 5000 --progress-every 50000
python3 loader.py --mode=in_situ --drop-source-records --gbif-cache-only downloads/budburst/ingest --batch-size 5000 --progress-every 50000
python3 loader.py --mode=in_situ --drop-source-records --gbif-cache-only downloads/seasonwatchindia/ingest --batch-size 5000 --progress-every 50000
python3 loader.py --mode=in_situ --drop-source-records --gbif-cache-only downloads/iNaturalist/ingest --batch-size 5000 --progress-every 50000

Load herbarium or machine-derived datasets after the in-situ sources. Use --drop-source-records for a full source refresh, or --no-drop-existing for append/update-only loads:

python3 prepare_gbif_family_cache.py \
  --names-from-csv downloads/herbarium/ingest \
  --gbif-workers 16 \
  --cache-write-every 500
python3 loader.py --mode=herbarium --test --no-drop-existing --gbif-cache-only downloads/herbarium/ingest --batch-size 5000 --progress-every 50000
python3 loader.py --mode=herbarium --drop-source-records --gbif-cache-only downloads/herbarium/ingest --batch-size 5000 --progress-every 50000

In herbarium mode, records with only year precision and no month-bearing date are rejected during loading.

PhenoObs source preparation writes one loader-ready CSV, similar to the NPN transformer. The script auto-detects raw rawdata_PhenObs_*.csv files under downloads/phenoObs first, then falls back to data/phenoObs. The output should go into a clean loader directory that also contains transform.yaml:

cd downloads/phenoObs
python3 prepare_phenoobs.py ./mappings.csv --output ingest/phenoObs_observations.csv
cd ../..
python3 prepare_gbif_family_cache.py --names-from-csv downloads/phenoObs/ingest --gbif-workers 16
python3 loader.py --mode=in_situ --test --no-drop-existing --gbif-cache-only downloads/phenoObs/ingest --batch-size 5000 --progress-every 50000
python3 loader.py --mode=in_situ --drop-source-records --gbif-cache-only downloads/phenoObs/ingest --batch-size 5000 --progress-every 50000

From the repo root, the equivalent explicit form is:

python3 downloads/phenoObs/prepare_phenoobs.py --raw-root downloads/phenoObs --mappings downloads/phenoObs/mappings.csv --output downloads/phenoObs/ingest/phenoObs_observations.csv

SeasonWatch India source preparation writes observedMetadataUrl, which the loader normalizes to sourceRecordUrl, as direct GBIF occurrence pages by default, for example https://www.gbif.org/occurrence/4943546900. The DwC-A does not include GBIF occurrence keys, so the fetcher resolves them from the GBIF API and writes a reusable cache at downloads/seasonwatchindia/seasonwatchindia_gbif_keys.csv. Use --no-gbif-direct-links only when you need to skip this lookup and accept GBIF occurrence search URLs instead.

Main options:

--test: validate and simulate without writing to Elasticsearch
--strict: reject rows with coercion or validation problems
--drop-existing: recreate the target index before loading
--drop-source-records: delete existing docs in the target index whose dataSource matches the input CSVs, then load the new records
--drop-poll-interval: seconds between internal Elasticsearch task checks for --drop-source-records
--gbif-cache: CSV cache used to populate standardizedFamily from scientificName
--gbif-cache-only: populate standardizedFamily from --gbif-cache only, with no live GBIF requests
--no-gbif-family: backward-compatible alias for cache-only GBIF family behavior
--batch-size: bulk size for indexing
--progress-every: progress logging interval
--no-drop-existing: keep the current index and upsert documents into it

What the loader relies on:

data/columns.csv for schema and required fields
data/traits.csv for traitUrn/source trait_urn to trait, mappedTraits, and mappedTraitsUrns expansion
optional dataset-local transform.yaml for normalization rules

`transform.yaml` Structure

Place transform.yaml inside the dataset directory being loaded. The loader checks for <data_dir>/transform.yaml automatically. If present, the file is applied row-by-row before datatype coercion, required-field checks, and mappedTraits derivation.

Use transform.yaml for:

trimming and case-normalizing raw source fields
regex-based cleanup of source strings
mapping raw trait labels to canonical Phenobase trait labels during transition
turning source-specific null tokens into empty values
adjusting parsing behavior for dates, booleans, and typed fields

Supported top-level keys:

fields: per-field transform steps and field-level overrides
trait_mappings: exact mapping from lowercase raw trait text to canonical trait labels. Prefer trait IDs in source prep, and treat this as a compatibility tool.
null_values: string tokens treated as null across all fields
coercions: global parsing and fallback rules for booleans, dates, integers, floats, and text

Minimal example:

fields:
  scientificName:
    transforms:
      - op: strip
      - op: case
        rule: scientific_name_standard

  trait:
    transforms:
      - op: strip
      - op: case
        rule: lower

  year:
    datatype: integer

trait_mappings:
  breaking leaf buds: breaking leaf bud present
  flowers: flower present

null_values:
  - ""
  - na
  - n/a
  - "-"

coercions:
  text:
    case: lower
  date:
    input_formats: ["%Y-%m-%d", "%m/%d/%Y"]
    output_format: "%Y-%m-%d"
    drop_invalid: true
  boolean:
    true_values: ["true", "t", "yes", "1"]
    false_values: ["false", "f", "no", "0"]
    drop_invalid: true
  integer:
    drop_invalid: true
  float:
    drop_invalid: true

Per-field Structure

Each entry under fields may define:

transforms: ordered transform steps applied to that field
datatype: override datatype for coercion
case: field-level case rule applied during text coercion
min: numeric minimum for integer or float fields
max: numeric maximum for integer or float fields
input_formats: date parse formats for that field
output_format: normalized output date format for that field

Supported transform steps under fields.<field>.transforms:

op: strip
op: case with rule: lower|upper|title|capitalize_first|scientific_name_standard
op: regex_sub with pattern, optional replacement, and optional flags
op: regex_map with pattern, to, and optional flags
op: null_if_in with values
op: map with exact values replacements

Notes on behavior:

transform steps run in the order written
trait_mappings is only applied to the trait field after field transforms run; it does not replace trait IDs
trait_mappings keys should be lowercase because the loader lowercases the incoming trait before lookup
regex_map uses regex match, so patterns should usually be anchored if you want full-value matching
supported regex flags are IGNORECASE, MULTILINE, and DOTALL
null_values is global, while null_if_in applies only to one field
text.case under coercions is a global default for text fields, but field-level case overrides it
invalid integers, floats, booleans, and dates are nulled by default unless --strict is used, in which case those coercion problems reject the row

Example with more complete field transforms:

fields:
  recordedBy:
    transforms:
      - op: strip
      - op: regex_sub
        pattern: "\\s+"
        replacement: " "

  collectionMethod:
    transforms:
      - op: strip
      - op: map
        values:
          specimen: herbarium specimen image
          photo: human observation

  date:
    input_formats: ["%Y-%m-%d", "%m/%d/%Y", "%Y/%m/%d"]
    output_format: "%Y-%m-%d"

  latitude:
    datatype: float
    min: -90
    max: 90

  longitude:
    datatype: float
    min: -180
    max: 180

  trait:
    transforms:
      - op: strip
      - op: regex_map
        pattern: "^open flowers?$"
        to: flower present
        flags: IGNORECASE

trait_mappings:
  flowering: flower present
  no flowers: flower absent

Export And Maintenance Utilities

Download A CSV Dump

Use download_csv_dump.py to scroll through the public Phenobase query API and write a local CSV.

python3 download_csv_dump.py

Useful variations:

python3 download_csv_dump.py --query 'genus:Quercus AND year:[2000 TO 2025]' --output downloads/quercus.csv
python3 download_csv_dump.py --limit 100000
python3 download_csv_dump.py --batch-size 10000 --scroll 1m
python3 download_csv_dump.py --request-timeout 60

Build A Zenodo Package

Use build_zenodo_package.py to export all current records and build a compressed package for public release or Zenodo submission.

python3 build_zenodo_package.py

The package is written under downloads/zenodo/<package-name>/ and, by default, also as downloads/zenodo/<package-name>.zip. It contains:

phenobase_observations.csv.gz: single compressed CSV export of all matching records
data_dictionary.csv: column metadata derived from data/columns.csv
column_metadata.json: JSON form of the same column metadata
source_summary.csv: record counts by dataSource
live_dataset_counts.csv: optional exact live and included counts by dataSource for --sample-per-datasource
trait_category_sample_counts.csv: optional exact live and included counts by dataSource and trait category for --sample-per-datasource-trait-category
record_summary.json: export parameters, row counts, date/year ranges, missing-link counts, and skipped year-only herbarium records
zenodo_metadata.json: editable Zenodo metadata template
manifest-sha256.txt: checksums and byte sizes
README.md: package documentation

By default, the CSV includes fields where data/columns.csv has visible_on_download=TRUE. Use --include-all-columns to export every field listed in data/columns.csv.

Production example:

python3 build_zenodo_package.py \
  --package-name phenobase-zenodo-2026-06-04 \
  --version 2026-06-04 \
  --publication-date 2026-06-04 \
  --creator "Phenobase Project" \
  --batch-size 10000 \
  --scroll 5m \
  --request-timeout 120

Smoke-test the package structure without exporting the full database:

python3 build_zenodo_package.py --limit 100 --package-name phenobase-zenodo-smoke-test

Build the all-datasets example package with 20 records per dataSource and exact live source counts:

python3 build_zenodo_package.py \
  --sample-per-datasource 20 \
  --package-name phenobase-zenodo-example-all-datasets-2026-07-06 \
  --version 2026-07-06 \
  --publication-date 2026-07-06 \
  --request-timeout 120

Build a broader trait-coverage example package with up to 20 flower, fruit, and leaf records per dataSource:

python3 build_zenodo_package.py \
  --sample-per-datasource-trait-category 20 \
  --sample-random-seed 20260821 \
  --package-name phenobase-zenodo-trait-coverage-example-2026-08-15 \
  --version 2026-08-15 \
  --publication-date 2026-08-15 \
  --request-timeout 120

This writes trait_category_sample_counts.csv with the live and included counts for each dataSource and trait category pair. --sample-random-seed makes the sample order random but reproducible; omit it to use the older _doc-order sample behavior.

The public API currently reports tens of millions of records, so run the full package build on the server in screen or tmux.

Backfill Version 2 Fields

Use backfill_version2_es.py to add version2 field mappings and backfill existing live records without reloading every source file. The script copies old field names into the version2 names, renames SeasonWatch India to SeasonWatch (India), derives mappedTraitsUrns from data/traits.csv, and fills standardizedFamily from a reusable GBIF cache.

For large GBIF family updates, use the two-phase process below. This avoids blocking Elasticsearch writes on live GBIF requests.

First build or extend the GBIF family cache from live ES records that have scientificName but are missing standardizedFamily:

python3 prepare_gbif_family_cache.py \
  --host 149.165.170.158 \
  --port 8081 \
  --scheme http \
  --name-source scroll \
  --gbif-workers 16 \
  --request-timeout 300 \
  --scroll 30m \
  --cache-write-every 500

If direct ES access is blocked, use the Phenobase query proxy instead:

python3 prepare_gbif_family_cache.py \
  --base-url https://biscicol.org/phenobase/api/v1/query \
  --name-source scroll \
  --gbif-workers 16 \
  --request-timeout 300 \
  --scroll 30m \
  --cache-write-every 500

Then run the ES backfill cache-only:

python3 backfill_version2_es.py \
  --host 149.165.170.158 \
  --port 8081 \
  --scheme http \
  --apply \
  --gbif-cache-only \
  --batch-size 2000 \
  --scroll 30m \
  --request-timeout 300

Do not use --overwrite for a restart or resume. Already-updated records should fall out of the default query. The default run is read-only; use --apply to write updates to Elasticsearch. If direct ES access is unavailable, run the same backfill command with --base-url https://biscicol.org/phenobase/api/v1/query instead of --host, --port, and --scheme.

Rename Legacy ES Fields

Use rename_es_fields.py for the one-time in-place ES rename:

gbifFamily -> standardizedFamily
basisOfRecord -> collectionMethod
accuracyExcludingUncertainFamily -> accuracyFamily
legacy accuracyFamily -> accuracyIncludingUncertainFamily

Dry-run through the proxy first:

python3 rename_es_fields.py \
  --base-url https://biscicol.org/phenobase/api/v1/query \
  --migrate-legacy-accuracy-family \
  --request-timeout 300

Apply the rename asynchronously:

python3 rename_es_fields.py \
  --base-url https://biscicol.org/phenobase/api/v1/query \
  --apply \
  --migrate-legacy-accuracy-family \
  --batch-size 5000 \
  --request-timeout 300

Use --migrate-legacy-accuracy-family only for the current legacy index migration. Future loads should write the new accuracyFamily meaning directly, so do not run that flag repeatedly after the legacy index has been converted.

Pages And Shared Outputs

The docs/ folder is intended for GitHub Pages publication and for quick sharing with collaborators.

Published outputs:

docs/index.html: GitHub Pages entry point redirecting to the traits viewer
docs/data-loading.md: source preparation notes and linkback URL behavior
docs/zenodo-package.md: public release and Zenodo package workflow
docs/traits.html: rendered trait explorer
docs/traits.csv: published CSV copy
docs/traits-data.json: viewer payload

Core Files

data/traits.csv: ontology-derived trait mapping
data/columns.csv: field definitions and schema metadata
data/datasource.csv: source-level citation, license, and data-use metadata for release exports
data/datasource_definitions.csv: definitions for source-level citation metadata fields
trait_lookup.py: local helper for resolving PPO IDs to current labels
loader.py: ingestion driver
reasoning/refresh_traits.py: reasoning rebuild driver
download_csv_dump.py: API export helper
update_decade_start.py: live index backfill helper

Requirements

Python 3.8+
Elasticsearch reachable for ingestion or maintenance commands

License

This repository is licensed under the MIT License. Bundled ontology snapshots and other third-party source data may remain subject to their own upstream terms.

Author

PhenoBase Project | Biocode, LLC

Navigation Menu

Uh oh!

Repository files navigation

Phenobase Data Workflow

User Guide

Technical Details And Implementation Guide

What This Repo Does

Reasoning

Method 1: Default Code Method

Method 2: ROBOT/SPARQL Method

Ingest Procedure

Loading Data

GBIF Family Cache For Normal Loads

Full Reload Synopsis

`transform.yaml` Structure

Per-field Structure

Export And Maintenance Utilities

Download A CSV Dump

Build A Zenodo Package

Backfill Version 2 Fields

Rename Legacy ES Fields

Pages And Shared Outputs

Core Files

Requirements

License

Author

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
data		data
docs		docs
downloads		downloads
reasoning		reasoning
tests		tests
.gitignore		.gitignore
.python-version		.python-version
LICENSE		LICENSE
README.md		README.md
STATUS.md		STATUS.md
backfill_npn_recorded_by.py		backfill_npn_recorded_by.py
backfill_version2_es.py		backfill_version2_es.py
build_zenodo_package.py		build_zenodo_package.py
build_zenodo_three_source_smoke_test.py		build_zenodo_three_source_smoke_test.py
download_csv_dump.py		download_csv_dump.py
export_schema.py		export_schema.py
gbif_family.py		gbif_family.py
loader.py		loader.py
normalize_npn_verbatim_traits_es.py		normalize_npn_verbatim_traits_es.py
prepare_gbif_family_cache.py		prepare_gbif_family_cache.py
rename_es_fields.py		rename_es_fields.py
run.sh		run.sh
source_record_url.py		source_record_url.py
trait_lookup.py		trait_lookup.py
update_decade_start.py		update_decade_start.py

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

Phenobase Data Workflow

User Guide

Technical Details And Implementation Guide

What This Repo Does

Reasoning

Method 1: Default Code Method

Method 2: ROBOT/SPARQL Method

Ingest Procedure

Loading Data

GBIF Family Cache For Normal Loads

Full Reload Synopsis

transform.yaml Structure

Per-field Structure

Export And Maintenance Utilities

Download A CSV Dump

Build A Zenodo Package

Backfill Version 2 Fields

Rename Legacy ES Fields

Pages And Shared Outputs

Core Files

Requirements

License

Author

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

`transform.yaml` Structure