Skip to content

Commit e01cd4c

Browse files
nuestclaude
andcommitted
Add Zenodo data deposition functionality
Implements functionality to deposit OPTIMAP data to Zenodo by creating/updating draft records. This feature enables automated archival and versioning of research data for long-term preservation and citation. Features: - Two Django management commands: - `render_zenodo`: Generates metadata files and data archives - `deposit_zenodo`: Uploads files and merges metadata to Zenodo drafts - Updates existing drafts only (requires deposition ID) - Never publishes automatically - manual approval required in Zenodo UI - Uploads: README.md, optimap-main.zip, latest GeoJSON and GeoPackage files - Merges metadata non-destructively without overwriting stable fields - Configurable via environment variables (ZENODO_API_TOKEN, etc.) - Comprehensive test coverage for rendering and deposition New files: - works/management/commands/deposit_zenodo.py - Upload to Zenodo - works/management/commands/render_zenodo.py - Generate metadata/archives - works/templates/README.md.j2 - Jinja2 template for README - data/README.md, data/last_version.txt, data/zenodo_dynamic.json - tests/test_deposit_zenodo.py - Deposition tests - tests/test_render_zenodo.py - Render tests Modified files: - .gitignore - Ignore Zenodo artifacts - optimap/settings.py - Add Zenodo configuration - requirements.txt - Add zenodo-client, markdown, jinja2 dependencies This implementation is adapted from PR #214 to work with the refactored codebase (publications/ → works/ directory structure). Closes #63 Co-authored-by: BharatVe <bharatveauli@live.com> Co-authored-by: BharatVe <150399011+BharatVe@users.noreply.github.com> 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 4d149b6 commit e01cd4c

11 files changed

Lines changed: 826 additions & 0 deletions

File tree

.gitignore

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -162,6 +162,12 @@ works/management/commands/goas_v01_simplified_0.1-90.geojson
162162

163163
works/management/commands/goas_v01_simplified-0.05-80.geojson
164164

165+
# Zenodo data artifacts
166+
data/optimap-main.zip
167+
data/*.gpkg
168+
data/*.geojson
169+
data/*.geojson.gz
170+
165171
works/management/commands/goas_v01_simplified.geojson
166172

167173
works/management/commands/goas_v01.gpkg

data/README.md

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# OPTIMAP FAIR Data Package
2+
3+
**Version:** v17
4+
5+
**Generated on:** 2025-09-24
6+
7+
8+
## Dataset Summary
9+
10+
- **Total articles:** 1
11+
- **Articles with spatial data:** 0
12+
- **Articles with temporal coverage:** 0
13+
- **Earliest publication date:** 2010-10-10
14+
- **Latest publication date:** 2010-10-10
15+
16+
17+
## Sources
18+
19+
- [OPTIMAP](http://optimap.science)
20+
21+
22+
## Codebook
23+
24+
| Field | Description |
25+
|------------------------|-------------------------------------------------------|
26+
| `id` | Primary key of the publication record |
27+
| `title` | Title of the article |
28+
| `abstract` | Abstract or summary |
29+
| `doi` | Digital Object Identifier (if available) |
30+
| `url` | URL to the article or preprint |
31+
| `publicationDate` | Date of publication (ISO format) |
32+
| `geometry` | Spatial geometry in GeoJSON/WKT |
33+
| `timeperiod_startdate` | Coverage start dates (ISO format) |
34+
| `timeperiod_enddate` | Coverage end dates (ISO format) |
35+
| `provenance` | Source/method by which the record was imported/added |
36+
37+
38+
## License
39+
40+
This record includes:
41+
42+
- **Data files** under **CC0-1.0** (<https://creativecommons.org/publicdomain/zero/1.0/>)
43+
- **optimap-main.zip** (code snapshot) under **GPL-3.0** (<https://opensource.org/licenses/GPL-3.0>)
44+
45+
**Note:** Data are CC0; the software snapshot is GPLv3.

data/last_version.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
v17

data/zenodo_dynamic.json

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
{
2+
"version": "v17",
3+
"related_identifiers": [
4+
{
5+
"scheme": "url",
6+
"identifier": "http://127.0.0.1:8000/data/optimap_data_dump_latest.geojson.gz",
7+
"relation": "isSupplementTo",
8+
"resource_type": "dataset"
9+
},
10+
{
11+
"scheme": "url",
12+
"identifier": "http://127.0.0.1:8000/data/optimap_data_dump_latest.gpkg",
13+
"relation": "isSupplementTo",
14+
"resource_type": "dataset"
15+
},
16+
{
17+
"scheme": "url",
18+
"identifier": "https://optimap.science",
19+
"relation": "describes",
20+
"resource_type": "publication"
21+
}
22+
]
23+
}

optimap/settings.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -349,6 +349,11 @@
349349
# Contact email for API user agents (OpenAlex, Wikidata, etc.)
350350
CONTACT_EMAIL = "login@optimap.science"
351351

352+
# Zenodo configuration
353+
ZENODO_API_TOKEN = env("ZENODO_API_TOKEN", default=None)
354+
ZENODO_SANDBOX_DEPOSITION_ID = env("ZENODO_SANDBOX_DEPOSITION_ID", default=None)
355+
ZENODO_API_BASE = env("ZENODO_API_BASE", default="https://sandbox.zenodo.org/api")
356+
352357
# Wikibase/Wikidata configuration
353358
WIKIBASE_API_URL = env("WIKIBASE_API_URL", default="")
354359

requirements.txt

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,11 @@ Pillow>=10.0
5454
# SVG → PNG for the OPTIMAP logo on the og:image preview
5555
cairosvg>=2.7
5656

57+
# Zenodo data deposition (issue #63)
58+
zenodo-client==0.3.6
59+
markdown>=3.7
60+
jinja2>=3.1.4
61+
5762

5863
# Geoextent library for spatial/temporal extent extraction
5964
git+https://github.com/nuest/geoextent.git@main#egg=geoextent

tests/test_deposit_zenodo.py

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
# tests/test_deposit_zenodo.py
2+
import json
3+
import tempfile
4+
from pathlib import Path
5+
from copy import deepcopy
6+
from unittest import TestCase
7+
from unittest.mock import patch
8+
9+
from django.core.management import call_command
10+
from django.test import override_settings
11+
from works.models import Publication, Source
12+
13+
14+
class DepositZenodoTest(TestCase):
15+
def setUp(self):
16+
self._tmpdir = tempfile.TemporaryDirectory()
17+
self.project_root = Path(self._tmpdir.name)
18+
self.templates_dir = self.project_root / "publications" / "templates"
19+
self.cmds_dir = self.project_root / "publications" / "management" / "commands"
20+
self.data_dir = self.project_root / "data"
21+
self.templates_dir.mkdir(parents=True, exist_ok=True)
22+
self.cmds_dir.mkdir(parents=True, exist_ok=True)
23+
self.data_dir.mkdir(parents=True, exist_ok=True)
24+
25+
# Minimal README so description→HTML works
26+
(self.data_dir / "README.md").write_text("# Title\n\nSome text.", encoding="utf-8")
27+
(self.data_dir / "optimap-main.zip").write_bytes(b"ZIP")
28+
# dynamic JSON with new related identifiers and version
29+
(self.data_dir / "zenodo_dynamic.json").write_text(json.dumps({
30+
"title": "OPTIMAP FAIR Data Package (test)",
31+
"version": "v999",
32+
"related_identifiers": [
33+
{"relation": "describes", "identifier": "https://optimap.science", "scheme": "url"}
34+
]
35+
}), encoding="utf-8")
36+
37+
# Fake dump files to upload
38+
(self.data_dir / "optimap_data_dump_20250101.geojson").write_text("{}", encoding="utf-8")
39+
(self.data_dir / "optimap_data_dump_20250101.gpkg").write_bytes(b"GPKG")
40+
41+
# Minimal DB so import paths work
42+
Publication.objects.create(title="A", publicationDate="2010-10-10")
43+
Source.objects.create(name="OPTIMAP", url_field="https://optimap.science")
44+
45+
# Command import – prefer deposit_zenodo; fallback to deploy_zenodo if needed
46+
import importlib
47+
try:
48+
self.deposit_mod = importlib.import_module(
49+
"works.management.commands.deposit_zenodo"
50+
)
51+
except ModuleNotFoundError:
52+
self.deposit_mod = importlib.import_module(
53+
"works.management.commands.deploy_zenodo"
54+
)
55+
56+
class FakePath(Path):
57+
_flavour = Path(".")._flavour
58+
def resolve(self):
59+
return self
60+
self.FakePath = FakePath
61+
self.deposit_file = str(self.cmds_dir / "deposit_zenodo.py")
62+
63+
def tearDown(self):
64+
self._tmpdir.cleanup()
65+
66+
def test_deposit_merges_metadata_and_uses_zenodo_client_for_uploads(self):
67+
# Fake Zenodo deposition (existing metadata)
68+
existing = {
69+
"submitted": False,
70+
"state": "unsubmitted",
71+
"links": {"edit": "http://edit", "bucket": "http://bucket"},
72+
"metadata": {
73+
"title": "Existing Title",
74+
"upload_type": "dataset",
75+
"publication_date": "2025-07-14",
76+
"creators": [{"name": "OPTIMAP"}],
77+
"keywords": ["Open Science"],
78+
"related_identifiers": [
79+
{"relation": "isSupplementTo", "identifier": "https://old.example", "scheme": "url"}
80+
],
81+
"language": "eng",
82+
"description": "<p>Old</p>",
83+
"version": "v1",
84+
},
85+
}
86+
87+
put_payload = {}
88+
89+
def _fake_get(url, params=None, **kwargs):
90+
class R:
91+
status_code = 200
92+
text = "ok"
93+
def json(self):
94+
# whatever object your test expects (e.g., deepcopy(existing))
95+
return deepcopy(existing)
96+
def raise_for_status(self):
97+
return None
98+
return R()
99+
100+
def _fake_post(url, params=None, json=None, **kwargs):
101+
class R:
102+
status_code = 200
103+
text = "ok"
104+
def json(self):
105+
# return what your code reads from POST responses, if anything
106+
return {"links": {"bucket": "https://example-bucket"}}
107+
def raise_for_status(self):
108+
return None
109+
return R()
110+
111+
def _fake_put(url, params=None, data=None, headers=None, **kwargs):
112+
class R:
113+
status_code = 200
114+
text = "ok"
115+
def raise_for_status(self):
116+
return None
117+
return R()
118+
119+
uploaded = {}
120+
121+
# zenodo-client upload shim: capture files that would be uploaded
122+
def _fake_update_zenodo(deposition_id, paths, sandbox=True, access_token=None, publish=False):
123+
self.assertEqual(deposition_id, "123456")
124+
self.assertTrue(sandbox)
125+
self.assertEqual(access_token, "tok")
126+
names = {Path(p).name for p in paths}
127+
self.assertIn("README.md", names)
128+
self.assertIn("optimap-main.zip", names)
129+
self.assertTrue(any(n.endswith(".geojson") for n in names))
130+
self.assertTrue(any(n.endswith(".gpkg") for n in names))
131+
uploaded["paths"] = [str(p) for p in paths]
132+
class R:
133+
def json(self): return {"links": {"html": f"https://sandbox.zenodo.org/deposit/{deposition_id}"}}
134+
return R()
135+
136+
with patch.object(self.deposit_mod, "__file__", new=self.deposit_file), \
137+
patch.object(self.deposit_mod, "Path", self.FakePath), \
138+
patch.object(self.deposit_mod.requests, "get", _fake_get), \
139+
patch.object(self.deposit_mod.requests, "put", _fake_put), \
140+
patch.object(self.deposit_mod, "update_zenodo", _fake_update_zenodo), \
141+
patch.object(self.deposit_mod, "_markdown_to_html", lambda s: "<p>HTML</p>"), \
142+
override_settings(ZENODO_UPLOADS_ENABLED=True):
143+
144+
call_command(
145+
"deposit_zenodo",
146+
"--deposition-id", "123456",
147+
)
148+
149+
# Merged metadata: required fields preserved, description/version updated, related merged
150+
merged = put_payload["metadata"]
151+
self.assertEqual(merged["title"], "Existing Title")
152+
self.assertEqual(merged["upload_type"], "dataset")
153+
self.assertEqual(merged["publication_date"], "2025-07-14")
154+
self.assertEqual(merged["creators"], [{"name": "OPTIMAP"}])
155+
156+
self.assertIn("description", merged)
157+
self.assertTrue(merged["description"].startswith("<p")) # from markdown->HTML
158+
159+
self.assertIsInstance(merged.get("version"), str)
160+
rel = {(d["identifier"], d["relation"]) for d in merged.get("related_identifiers", [])}
161+
self.assertIn(("https://old.example", "isSupplementTo"), rel)
162+
self.assertIn(("https://optimap.science", "describes"), rel)
163+
164+
# Uploader called with expected files
165+
self.assertIn("paths", uploaded)
166+
self.assertGreater(len(uploaded["paths"]), 0)

tests/test_render_zenodo.py

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# tests/test_render_zenodo.py
2+
import tempfile
3+
from pathlib import Path
4+
from unittest import TestCase
5+
from unittest.mock import patch
6+
7+
from django.core.management import call_command
8+
from works.models import Publication, Source
9+
10+
11+
class RenderZenodoTest(TestCase):
12+
def setUp(self):
13+
# Temp “project root”
14+
self._tmpdir = tempfile.TemporaryDirectory()
15+
self.project_root = Path(self._tmpdir.name)
16+
self.templates_dir = self.project_root / "publications" / "templates"
17+
self.cmds_dir = self.project_root / "publications" / "management" / "commands"
18+
self.data_dir = self.project_root / "data"
19+
self.templates_dir.mkdir(parents=True, exist_ok=True)
20+
self.cmds_dir.mkdir(parents=True, exist_ok=True)
21+
self.data_dir.mkdir(parents=True, exist_ok=True)
22+
23+
# Minimal README template with Sources
24+
(self.templates_dir / "README.md.j2").write_text(
25+
"# OPTIMAP FAIR Data Package\n"
26+
"**Version:** {{ version }}\n\n"
27+
"## Sources\n\n"
28+
"{% for src in sources %}- [{{ src.name }}]({{ src.url }})\n{% endfor %}\n"
29+
"\n## Codebook\n\n"
30+
"| Field | Description |\n|---|---|\n| id | pk |\n",
31+
encoding="utf-8",
32+
)
33+
34+
# DB fixtures
35+
Publication.objects.create(title="A", publicationDate="2010-10-10")
36+
37+
# Bad labels to clean
38+
Source.objects.create(name="2000", url_field="https://optimap.science") # numeric-only -> OPTIMAP
39+
Source.objects.create(name="", url_field="https://example.org") # blank -> domain label
40+
Source.objects.create(name=" ", url_field="https://example.org") # duplicate -> dedupe
41+
42+
# Good label
43+
Source.objects.create(
44+
name="AGILE: GIScience Series",
45+
url_field="https://agile-giss.copernicus.org"
46+
)
47+
48+
# Import after DB is ready
49+
import importlib
50+
self.render_mod = importlib.import_module(
51+
"works.management.commands.render_zenodo"
52+
)
53+
54+
# Fake Path so parents[3] stays inside tmp root
55+
class FakePath(Path):
56+
_flavour = Path(".")._flavour
57+
def resolve(self):
58+
return self
59+
self.FakePath = FakePath
60+
self.render_file = str(self.cmds_dir / "render_zenodo.py")
61+
62+
def tearDown(self):
63+
self._tmpdir.cleanup()
64+
65+
def test_render_produces_clean_readme_and_assets(self):
66+
# Don’t actually run `git archive`
67+
def _noop(*a, **k): return None
68+
69+
with patch.object(self.render_mod, "__file__", new=self.render_file), \
70+
patch.object(self.render_mod, "Path", self.FakePath), \
71+
patch("subprocess.run", _noop):
72+
call_command("render_zenodo")
73+
74+
readme_path = self.data_dir / "README.md"
75+
zip_path = self.data_dir / "optimap-main.zip"
76+
dyn_path = self.data_dir / "zenodo_dynamic.json"
77+
78+
self.assertTrue(readme_path.exists(), "README.md not generated")
79+
self.assertTrue(zip_path.exists(), "optimap-main.zip not generated")
80+
self.assertTrue(dyn_path.exists(), "zenodo_dynamic.json not generated")
81+
82+
md = readme_path.read_text(encoding="utf-8")
83+
# Sources cleanup assertions
84+
self.assertNotIn("- [2000](", md, "Numeric-only label leaked into Sources")
85+
self.assertIn("- [OPTIMAP](https://optimap.science)", md, "OPTIMAP override missing")
86+
self.assertIn("AGILE: GIScience Series", md, "Named source missing")
87+
# example.org should appear only once after dedupe
88+
self.assertEqual(md.count("example.org"), 1, "Duplicate source/domain not deduped")

0 commit comments

Comments
 (0)