CEP 47: Index timestamp in package record metadata - #154
Conversation
Propose adding an optional `upload_timestamp` field to per-record metadata in `repodata.json`. Unlike the existing `timestamp` field (build time, builder-controlled), `upload_timestamp` is set by the channel server when an artifact is indexed, providing a trustworthy publication time for supply chain security features (dependency cooldowns) and reproducible environment resolution. Requires: CEP 36
jaimergp
left a comment
There was a problem hiding this comment.
Thank you for bringing this up, I don't think we've discussed the exploitability of timestamp before, and this CEP brings up a few good points. Let's use this opportunity to also secure the acceptable values for timestamp.
| [CEP 36](./cep-0036.md) defines the schema for entries in `packages` and `packages.conda` within `repodata.json`. This CEP adds one optional field to the package record metadata: | ||
|
|
||
| - `upload_timestamp: int`. Optional. Unix time in milliseconds when the artifact was added to the channel index. It MUST be set by the channel server or indexing tool, not by the build tool. If absent, clients MAY fall back to `timestamp` for time-based operations. | ||
|
|
There was a problem hiding this comment.
Seeing timestamp with this angle of security, I think we could add more restrictions in this CEP:
- Limit
timestampto dates in the past, never in the future. - Rule:
timestamp <= upload_timestamp - Force indexing tools to reject packages where
timestampdoes not obey these rules.
There was a problem hiding this comment.
In conda-index we don't know when a package was uploaded. We could keep track of when it was first seen or first indexed. But there are cases where we might lose that information, like if we decide to re-index a channel. A common workflow is to index a package, and then upload everything to the CDN.
What about the last-modified date from HTTP or the filesystem timestamp?
There was a problem hiding this comment.
@jaimergp I have added a recommendation that indexing tools SHOULD validate timestamp <= indexed_timestamp and reject packages with timestamp values in the future. Kept it as a SHOULD rather than MUST since this tightens rules on an existing field and probably deserves a gradual rollout.
There was a problem hiding this comment.
@dholth I have revised the approach based on your feedback. The field is now called indexed_timestamp since that is what conda-index actually knows. For existing artifacts, the CEP allows seeding from timestamp as a pragmatic migration path. For new artifacts, the value should be set to the current time when first indexed.
There was a problem hiding this comment.
Kept this as SHOULD in the validation section rather than MUST: indexed_timestamp itself is server-set as MUST, but rejecting existing artifacts based on the older builder-controlled timestamp field is a compatibility tightening. Current text says indexers SHOULD validate no future timestamp and timestamp <= indexed_timestamp, and reject or flag violations.
|
|
||
| Channel servers and indexing tools (such as `conda-index`, `quetz`, or Anaconda's infrastructure) SHOULD set `upload_timestamp` on each package record when the artifact is first indexed into the channel. The value MUST reflect the actual time the artifact became available in the channel, not the build time or any value from the artifact itself. | ||
|
|
||
| If a channel re-indexes existing artifacts without a prior `upload_timestamp`, it MAY use the file modification time or another server-side signal as a best-effort approximation. It MUST NOT copy the value from the artifact's `timestamp` field. |
There was a problem hiding this comment.
This is going to be tricky to implement robustly, I fear. Some channels may not have the upload timestamp in their database, and their filesystem metadata may not be reliable (files were copied around, recovered from a backup, etc).
We could be a bit more permissive with using timestamp as long as the value is prior to the approval of this CEP, so we know they are not exploiting that weakness.
There was a problem hiding this comment.
Im also not sure about this. I think it makes more sense to initially set the field to the same value as timestamp. This would allow switching existing exclude-newer implementation to the new upload_timestamp without any interruption.
I was also considering leaving it empty for those records but that still poses a security risk when using exclude-newer.
There was a problem hiding this comment.
I have revised the re-indexing paragraph to allow seeding indexed_timestamp from timestamp for artifacts that predate this CEP. The security rationale: cooldowns protect against the initial publication window, and most supply chain attacks have windows under one week (per Woodruff's analysis). That window has long closed for existing artifacts. For newly indexed artifacts going forward, the value must be set by the server at indexing time. This should also address @baszalmstra's concern about keeping exclude-newer working without interruption.
There was a problem hiding this comment.
Yes. The migration text now allows pre-CEP artifacts to seed indexed_timestamp from timestamp or another server-side signal. For artifacts first indexed after approval, indexed_timestamp MUST be set at indexing time. That preserves existing exclude-newer behavior without letting new packages choose their own index time.
There was a problem hiding this comment.
Maybe redundant, but the value of this field must not change regardless the source, right? Otherwise we get clunky solves if extreme exclude-newer configs are used. Thinking of filesystems changing mtime of some files by accident, and that value used for indexed_timestamp, for example. Shall we record that decision?
There was a problem hiding this comment.
I agree, this invariant is worth making explicit. Suggested wording:
| If a channel re-indexes existing artifacts without a prior `upload_timestamp`, it MAY use the file modification time or another server-side signal as a best-effort approximation. It MUST NOT copy the value from the artifact's `timestamp` field. | |
| For artifacts that predate this CEP and lack an `indexed_timestamp`, channel servers MAY seed the value from the artifact's `timestamp` field or from another server-side signal such as file modification time. This is acceptable because the security benefits of `indexed_timestamp` (dependency cooldowns, exclude-newer) protect against the initial publication window, which has long closed for existing artifacts. | |
| Once `indexed_timestamp` has been assigned for a package record, channel servers and indexing tools MUST preserve that value across subsequent indexing runs. They MUST NOT recompute it from file modification time, the artifact's `timestamp`, or any other fallback signal while an existing `indexed_timestamp` is present. Channel operators MAY correct demonstrably erroneous values as an intentional metadata correction. |
|
@jezdez How can we move this forward? Anything I can do to help? |
|
@jezdez Gentle ping! I'd like to put this to a vote ASAP but there are still a few open questions. |
|
@baszalmstra Apologies, this dropped off my todo list in the past weeks, I'll take a look! |
- Set Discussion field to PR URL - Make conda/conda#15759 a proper markdown link
- Rename upload_timestamp to indexed_timestamp (reflects indexation time, not upload time; aligns with conda-forge staging workflow and conda-index terminology) - Rename CEP title to "Index timestamp in package record metadata" - Remove inline JSON schema section (schemas are downstream from CEPs) - Add rationale for naming choice and non-patchability - Revise backfill strategy: allow seeding from timestamp for pre-CEP artifacts, require server-set values for new artifacts - Add timestamp validation recommendation for indexing tools - Add pixi exclude-newer to ecosystem precedent section - Add channel model risk profile note to motivation - Add Nesbitt references
|
Thanks everyone for the thorough feedback. I have pushed changes that address all the review threads. Here is a summary: Renamed to Revised backfill strategy: The previous "MUST NOT copy from Removed the schema section: Agreed with Jaime that schemas are downstream from CEPs. Replaced with a prose note that the Added timestamp validation: Indexing tools SHOULD validate that Added pixi and channel model context: pixi's Added references: Andrew Nesbitt's posts on dependency cooldowns and package security for AI agents. I have replied to all inline threads and resolved the ones that are fully addressed. Left the backfill, timestamp validation, and Requires threads open for confirmation. |
Per PEP 12 convention, Requires is only for dependencies on in-development proposals, not final ones.
Link to the tracking issue, conda PR, and mamba PR from the references section. Update Implementation field to point at the tracking issue.
|
I have thought a bit about this, and sigstore uses something interesting called a "timestamp authority", which we could theoretically also investigate. If we have proper sigstore support, we'd get the timestamp for free from the attestation btw. The following was written by Claude because it can explain it better than me anyways :)
RFC 3161 Trusted TimestampsRFC 3161 defines a protocol where a Trusted Third Party (a Timestamp Authority / TSA) issues signed timestamps:
The TSA never sees your data (only the hash), and the signed timestamp is cryptographically bound to that specific hash at that specific time. You can't backdate it or forge it without the TSA's private key. This is already used in the sigstore ecosystem — the sigstore bundle format has a How this could work for conda packagesThere are two complementary places where a TSA timestamp adds value: Build-time TSA timestamp (embedded in the package)At build time (e.g., in
Note: we can't hash the final archive itself because embedding the TSR changes the archive hash (chicken-and-egg). But a content hash works well — at verification time, a client unpacks the package, recomputes the content hash using the same deterministic scheme (skipping the TSR field), and verifies the TSA's signature. This is analogous to how git signed commits work. This gives a cryptographically verifiable lower bound: "this package content existed no later than time T." Index-time TSA timestamp (alongside repodata)When a channel server indexes a package, it could:
This makes Relationship to this CEPI'm not suggesting this CEP needs to specify all of this —
For context, the sigstore integration in Happy to discuss further or help spec this out if there's interest. |
Acknowledge wolfv's review point that indexed_timestamp as a plain server-controlled integer is not independently verifiable. Document RFC 3161 TSA at index time and build-time TSA timestamps as future directions, leaving field names and verification semantics to dedicated follow-up CEPs.
|
Thanks @wolfv, this is a great point. You're right that I added a "Future work" section to the CEP that documents two directions:
I deliberately did not reserve a specific field name (e.g. For this CEP I'd like to keep the scope on the simple integer field to unblock |
|
Dear @conda/steering-council, this CEP is officially entering its Request For Comments (RFC) period. This will last two weeks, until May 12th EOD AoE. |
|
I've updated the conda and conda plugin PRs and made another pass over the CEP. I think it's ready for a vote. |
Co-authored-by: Jannis Leidel <jannis@leidel.info>
|
Dear @conda/steering-council, The vote for this CEP has started. It will be open for two weeks, until June 24th, 2026, 23:59 Anywhere on Earth. This time period has been chosen to make it eligible for time-out rules. As an Enhancement Proposal vote, it requires 60% affirmative votes to pass. To vote, please mark the relevant checkbox under your username:
|
|
@mariusvniekerk, @chenghlee, @msarahan, @jakirkham: final reminder to vote, thanks! |
|
Vote has closed, results are in: the CEP has passed!
@conda/steering-council, this will be minted as CEP 47. |
|
Congrats, @jezdez! |
Checklist for submitter
cep-0000.mdnamedcep-XXXX.mdin the root level.Checklist for CEP approvals
${greatest-number-in-main} + 1.cep-XXXX.mdfile has been renamed accordingly.# CEP XXXX -header has been edited accordingly.pre-commitchecks are passing.