Skip to content

STRUCTURE.md

Latest commit

 

History

History
70 lines (52 loc) · 2.84 KB

STRUCTURE.md

File metadata and controls

70 lines (52 loc) · 2.84 KB
Raw

DataCite Monthly Data File structure

Overview

The DataCite Monthly Data File is stored in an AWS S3 bucket.

  • Region: eu-west-1
  • Buckets:
    • stage: monthly-datafile.stage.datacite.org
    • production: monthly-datafile.datacite.org

The bucket contains a top-level MANIFEST file and data contained inside a dois folder.

For more detail on the exact contents of the data file, see the DataCite Public Data File support page.

Top-level files

The bucket contains several top-level files to assist with using the data file effectively.

The MANIFEST file

This file can be downloaded to discover which objects are currently available in the bucket without having to list the entire bucket contents via the S3 API. It can be used for workflows that wish to avoid calling the paginated list_objects_v2 endpoint.

Each line in the MANIFEST file is a whitespace separated record containing:

  • the object key (path within the bucket)
  • the object size (in bytes)
  • the SHA-256 hash of the object contents

E.g. dois/updated_2011-03/part_0000.jsonl.gz 41 2/YPZ3bPUHVfdgVK0DR/91/mZP9PzN9HrOXI9MxAzHQ=

The MANIFEST.json file

This file is similar to the MANIFEST file but with the records provided as JSON dictionaries.

E.g.

{
  "filename": "dois/updated_2011-03/part_0000.jsonl.gz",
  "size": 41,
  "sha256": "2/YPZ3bPUHVfdgVK0DR/91/mZP9PzN9HrOXI9MxAzHQ="
}

The STATUS.json file

This file contains a simple JSON dictionary expressing information about the current status of the data file generation. It can be used to determine if each monthly data file release is ready for consumption.

It contains three keys:

  • month, which contains the YYYY-MM value of the release for which the status applies
  • datetime, the timestamp at which the STATUS.json file was updated in ISO 8601-1:2019 format
  • status, one of In progress, Uploading or Complete, giving the current status of the data file generation

DOI Records

All DOI records are available under month-specific prefixes using the format YYYY-MM.

  • dois/updated_YYYY-MM/

Example:

  • dois/updated_2025-11/

Within a given month prefix you will find one or more objects (files). The exact filenames may vary over time, so it is recommended to use either the MANIFEST file or an S3 prefix listing to discover the available objects for a month.

Example object keys (illustrative):

  • dois/updated_2025-11/2025-11.csv.gz
  • dois/updated_2025-11/part_0000.json.gz

Practical notes

  • Credentials retrieved from the DataCite REST API (/credentials/datafile) are temporary AWS credentials and include a session token.
  • If you are using tooling that expects a bucket name without periods, note that this bucket name includes periods (e.g. monthly-datafile.stage.datacite.org).