Skip to content

CEP 47: Index timestamp in package record metadata - #154

Merged
jaimergp merged 15 commits into
conda:mainfrom
jezdez:cep-upload-timestamp
Jun 30, 2026
Merged

CEP 47: Index timestamp in package record metadata#154
jaimergp merged 15 commits into
conda:mainfrom
jezdez:cep-upload-timestamp

Conversation

@jezdez

@jezdez jezdez commented Mar 5, 2026

Copy link
Copy Markdown
Member

Checklist for submitter

  • I am submitting a new CEP: Index timestamp in package record metadata.
    • I am using the CEP template by creating a copy cep-0000.md named cep-XXXX.md in the root level.
  • I am submitting modifications to CEP XX.
  • Something else: (add your description here).

Checklist for CEP approvals

  • The vote period has ended and the vote has passed the necessary quorum and approval thresholds.
  • A new CEP number has been minted. Usually, this is ${greatest-number-in-main} + 1.
  • The cep-XXXX.md file has been renamed accordingly.
  • The # CEP XXXX - header has been edited accordingly.
  • The CEP status in the table has been changed to approved.
  • The last modification date in the table has been updated accordingly.
  • The table in the README has been updated with the new CEP entry.
  • The pre-commit checks are passing.

Propose adding an optional `upload_timestamp` field to per-record
metadata in `repodata.json`. Unlike the existing `timestamp` field
(build time, builder-controlled), `upload_timestamp` is set by the
channel server when an artifact is indexed, providing a trustworthy
publication time for supply chain security features (dependency
cooldowns) and reproducible environment resolution.

Requires: CEP 36

@jaimergp jaimergp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for bringing this up, I don't think we've discussed the exploitability of timestamp before, and this CEP brings up a few good points. Let's use this opportunity to also secure the acceptable values for timestamp.

Comment thread cep-XXXX.md Outdated
Comment thread cep-XXXX.md Outdated
Comment thread cep-0047.md
[CEP 36](./cep-0036.md) defines the schema for entries in `packages` and `packages.conda` within `repodata.json`. This CEP adds one optional field to the package record metadata:

- `upload_timestamp: int`. Optional. Unix time in milliseconds when the artifact was added to the channel index. It MUST be set by the channel server or indexing tool, not by the build tool. If absent, clients MAY fall back to `timestamp` for time-based operations.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seeing timestamp with this angle of security, I think we could add more restrictions in this CEP:

  • Limit timestamp to dates in the past, never in the future.
  • Rule: timestamp <= upload_timestamp
  • Force indexing tools to reject packages where timestamp does not obey these rules.

@dholth dholth Mar 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In conda-index we don't know when a package was uploaded. We could keep track of when it was first seen or first indexed. But there are cases where we might lose that information, like if we decide to re-index a channel. A common workflow is to index a package, and then upload everything to the CDN.
What about the last-modified date from HTTP or the filesystem timestamp?

@jezdez jezdez Apr 10, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jaimergp I have added a recommendation that indexing tools SHOULD validate timestamp <= indexed_timestamp and reject packages with timestamp values in the future. Kept it as a SHOULD rather than MUST since this tightens rules on an existing field and probably deserves a gradual rollout.

@jezdez jezdez Apr 10, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dholth I have revised the approach based on your feedback. The field is now called indexed_timestamp since that is what conda-index actually knows. For existing artifacts, the CEP allows seeding from timestamp as a pragmatic migration path. For new artifacts, the value should be set to the current time when first indexed.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kept this as SHOULD in the validation section rather than MUST: indexed_timestamp itself is server-set as MUST, but rejecting existing artifacts based on the older builder-controlled timestamp field is a compatibility tightening. Current text says indexers SHOULD validate no future timestamp and timestamp <= indexed_timestamp, and reject or flag violations.

Comment thread cep-XXXX.md Outdated
Comment thread cep-XXXX.md Outdated

Channel servers and indexing tools (such as `conda-index`, `quetz`, or Anaconda's infrastructure) SHOULD set `upload_timestamp` on each package record when the artifact is first indexed into the channel. The value MUST reflect the actual time the artifact became available in the channel, not the build time or any value from the artifact itself.

If a channel re-indexes existing artifacts without a prior `upload_timestamp`, it MAY use the file modification time or another server-side signal as a best-effort approximation. It MUST NOT copy the value from the artifact's `timestamp` field.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is going to be tricky to implement robustly, I fear. Some channels may not have the upload timestamp in their database, and their filesystem metadata may not be reliable (files were copied around, recovered from a backup, etc).

We could be a bit more permissive with using timestamp as long as the value is prior to the approval of this CEP, so we know they are not exploiting that weakness.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Im also not sure about this. I think it makes more sense to initially set the field to the same value as timestamp. This would allow switching existing exclude-newer implementation to the new upload_timestamp without any interruption.

I was also considering leaving it empty for those records but that still poses a security risk when using exclude-newer.

@jezdez jezdez Apr 10, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have revised the re-indexing paragraph to allow seeding indexed_timestamp from timestamp for artifacts that predate this CEP. The security rationale: cooldowns protect against the initial publication window, and most supply chain attacks have windows under one week (per Woodruff's analysis). That window has long closed for existing artifacts. For newly indexed artifacts going forward, the value must be set by the server at indexing time. This should also address @baszalmstra's concern about keeping exclude-newer working without interruption.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. The migration text now allows pre-CEP artifacts to seed indexed_timestamp from timestamp or another server-side signal. For artifacts first indexed after approval, indexed_timestamp MUST be set at indexing time. That preserves existing exclude-newer behavior without letting new packages choose their own index time.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe redundant, but the value of this field must not change regardless the source, right? Otherwise we get clunky solves if extreme exclude-newer configs are used. Thinking of filesystems changing mtime of some files by accident, and that value used for indexed_timestamp, for example. Shall we record that decision?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, this invariant is worth making explicit. Suggested wording:

Suggested change
If a channel re-indexes existing artifacts without a prior `upload_timestamp`, it MAY use the file modification time or another server-side signal as a best-effort approximation. It MUST NOT copy the value from the artifact's `timestamp` field.
For artifacts that predate this CEP and lack an `indexed_timestamp`, channel servers MAY seed the value from the artifact's `timestamp` field or from another server-side signal such as file modification time. This is acceptable because the security benefits of `indexed_timestamp` (dependency cooldowns, exclude-newer) protect against the initial publication window, which has long closed for existing artifacts.
Once `indexed_timestamp` has been assigned for a package record, channel servers and indexing tools MUST preserve that value across subsequent indexing runs. They MUST NOT recompute it from file modification time, the artifact's `timestamp`, or any other fallback signal while an existing `indexed_timestamp` is present. Channel operators MAY correct demonstrably erroneous values as an intentional metadata correction.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect!

Comment thread cep-XXXX.md Outdated
Comment thread cep-XXXX.md Outdated

@pavelzw pavelzw left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool!

Comment thread cep-XXXX.md Outdated
Comment thread cep-XXXX.md Outdated
Comment thread cep-XXXX.md Outdated
@baszalmstra

Copy link
Copy Markdown
Contributor

@jezdez How can we move this forward? Anything I can do to help?

@baszalmstra

Copy link
Copy Markdown
Contributor

@jezdez Gentle ping! I'd like to put this to a vote ASAP but there are still a few open questions.

@jezdez

jezdez commented Apr 9, 2026

Copy link
Copy Markdown
Member Author

@baszalmstra Apologies, this dropped off my todo list in the past weeks, I'll take a look!

jezdez added 3 commits April 10, 2026 12:10
- Set Discussion field to PR URL
- Make conda/conda#15759 a proper markdown link
- Rename upload_timestamp to indexed_timestamp (reflects indexation
  time, not upload time; aligns with conda-forge staging workflow
  and conda-index terminology)
- Rename CEP title to "Index timestamp in package record metadata"
- Remove inline JSON schema section (schemas are downstream from CEPs)
- Add rationale for naming choice and non-patchability
- Revise backfill strategy: allow seeding from timestamp for pre-CEP
  artifacts, require server-set values for new artifacts
- Add timestamp validation recommendation for indexing tools
- Add pixi exclude-newer to ecosystem precedent section
- Add channel model risk profile note to motivation
- Add Nesbitt references
@jezdez

jezdez commented Apr 10, 2026

Copy link
Copy Markdown
Member Author

Thanks everyone for the thorough feedback. I have pushed changes that address all the review threads. Here is a summary:

Renamed to indexed_timestamp: The field now reflects what it actually measures: when the artifact appeared in the channel index, not when it was uploaded. In conda-forge's workflow, upload (to cf-staging) and indexation (appearing in repodata.json) are separate steps. Daniel's framing of "first indexed" from the conda-index perspective confirmed this was the right name. The CEP title is now "Index timestamp in package record metadata."

Revised backfill strategy: The previous "MUST NOT copy from timestamp" rule was too strict. The CEP now allows seeding indexed_timestamp from timestamp for pre-existing artifacts. The security rationale: cooldowns protect against the initial publication window, and that window has long closed for packages already on channels. New artifacts indexed after this CEP is approved must have the value set by the server.

Removed the schema section: Agreed with Jaime that schemas are downstream from CEPs. Replaced with a prose note that the repodata-record-1 schema at schemas.conda.org should be updated accordingly. The non-patchability point is now covered in the rationale instead.

Added timestamp validation: Indexing tools SHOULD validate that timestamp <= indexed_timestamp and reject future timestamp values. Kept as SHOULD rather than MUST since this tightens rules on an existing field.

Added pixi and channel model context: pixi's exclude-newer is now mentioned in the Ecosystem precedent section, and there is a new section on how different channel models (curated vs community forge) have different risk profiles.

Added references: Andrew Nesbitt's posts on dependency cooldowns and package security for AI agents.

I have replied to all inline threads and resolved the ones that are fully addressed. Left the backfill, timestamp validation, and Requires threads open for confirmation.

Per PEP 12 convention, Requires is only for dependencies on
in-development proposals, not final ones.
Link to the tracking issue, conda PR, and mamba PR from the
references section. Update Implementation field to point at
the tracking issue.
@jezdez jezdez changed the title CEP XXXX: Upload timestamp in package record metadata CEP XXXX: Index timestamp in package record metadata Apr 11, 2026
@wolfv

wolfv commented Apr 15, 2026

Copy link
Copy Markdown
Contributor

I have thought a bit about this, and sigstore uses something interesting called a "timestamp authority", which we could theoretically also investigate. If we have proper sigstore support, we'd get the timestamp for free from the attestation btw.

The following was written by Claude because it can explain it better than me anyways :)

indexed_timestamp as proposed is a plain integer set by the channel server. This is better than timestamp (builder-controlled), but it's still a single party's unverifiable claim. If the server is compromised, or data is restored from backup, or a channel re-indexes, the value can silently change. There's no way for a client to independently verify that the timestamp is authentic.

RFC 3161 Trusted Timestamps

RFC 3161 defines a protocol where a Trusted Third Party (a Timestamp Authority / TSA) issues signed timestamps:

  1. You compute a hash of the data you want to timestamp
  2. You send that hash to a TSA (e.g., Sigstore's timestamp.sigstore.dev, FreeTSA, DigiCert)
  3. The TSA returns a signed Timestamp Response (TSR) containing the hash, the current time, and the TSA's signature
  4. Anyone can later verify the timestamp by checking the TSA's signature against its certificate chain

The TSA never sees your data (only the hash), and the signed timestamp is cryptographically bound to that specific hash at that specific time. You can't backdate it or forge it without the TSA's private key.

This is already used in the sigstore ecosystem — the sigstore bundle format has a TimestampVerificationData field specifically for RFC 3161 timestamps, and tools like cosign support --timestamp-server-url.

How this could work for conda packages

There are two complementary places where a TSA timestamp adds value:

Build-time TSA timestamp (embedded in the package)

At build time (e.g., in rattler-build or conda-build), after assembling the package contents but before creating the final archive:

  1. Compute a deterministic content hash over the package contents — e.g. a hash of a canonicalized manifest of (path, sha256, size) tuples, excluding the TSR field itself
  2. Send that hash to a TSA
  3. Embed the returned TSR in info/index.json (e.g. as "build_tsr": "<base64>") or as a sidecar file info/build.tsr
  4. Pack the .conda archive

Note: we can't hash the final archive itself because embedding the TSR changes the archive hash (chicken-and-egg). But a content hash works well — at verification time, a client unpacks the package, recomputes the content hash using the same deterministic scheme (skipping the TSR field), and verifies the TSA's signature. This is analogous to how git signed commits work.

This gives a cryptographically verifiable lower bound: "this package content existed no later than time T."

Index-time TSA timestamp (alongside repodata)

When a channel server indexes a package, it could:

  1. Hash the artifact (the .conda file hash is known at this point)
  2. Request a signed timestamp from a TSA
  3. Store the TSR alongside indexed_timestamp in repodata, or as a sidecar file

This makes indexed_timestamp independently verifiable rather than being a trust-me claim from the server.

Relationship to this CEP

I'm not suggesting this CEP needs to specify all of this — indexed_timestamp as a simple queryable integer in repodata is useful on its own for exclude-newer and dependency cooldowns. But it might be worth:

  1. Mentioning RFC 3161 TSA as a future direction for making indexed_timestamp cryptographically verifiable
  2. Considering whether the CEP should reserve a field (e.g. indexed_tsr) for an optional base64-encoded TSR that proves the indexed_timestamp value
  3. Noting that build tools could independently embed TSA timestamps in packages to provide a verifiable build time, complementing the server-set index time

For context, the sigstore integration in rattler-build already touches this infrastructure — the sigstore bundle format has native support for RFC 3161 timestamps via TimestampVerificationData. Extending this to also provide a verifiable build timestamp in index.json would be a natural next step.

Happy to discuss further or help spec this out if there's interest.

jezdez added 2 commits April 22, 2026 11:15
Acknowledge wolfv's review point that indexed_timestamp as a
plain server-controlled integer is not independently verifiable.
Document RFC 3161 TSA at index time and build-time TSA timestamps
as future directions, leaving field names and verification semantics
to dedicated follow-up CEPs.
@jezdez

jezdez commented Apr 22, 2026

Copy link
Copy Markdown
Member Author

Thanks @wolfv, this is a great point. You're right that indexed_timestamp as proposed is a single-party claim and therefore not independently verifiable. RFC 3161 TSA is the standard tool to close that gap, and Sigstore's timestamp.sigstore.dev was added precisely because relying on Rekor's internal clock had the same trust problem.

I added a "Future work" section to the CEP that documents two directions:

  • RFC 3161 TSA at index time, including the existing Sigstore bundle carrier (TimestampVerificationData) as a natural integration point for an attestation-aware indexer per CEP 27.
  • Build-time TSA timestamps as separate work that would more naturally amend CEP 34 than this CEP.

I deliberately did not reserve a specific field name (e.g. indexed_tsr). There's no precedent for reserving fields in our CEPs, CEP 27 follows the same "future work without reservation" pattern, and CEP 36's additive convention means new fields can be added later without needing reservation. That also leaves room for a future CEP to pick the right design (sidecar file, sigstore bundle reference, in-toto attestation tied to indexing, ...) rather than baking in an RFC 3161 TSR encoding now.

For this CEP I'd like to keep the scope on the simple integer field to unblock --exclude-newer parity with uv, pip, pixi, and rattler, and dependency cooldowns. Happy to collaborate on a follow-up CEP for the verifiable-timestamp work whenever there's interest. Your sigstore expertise would be invaluable there.

@jezdez

jezdez commented Apr 28, 2026

Copy link
Copy Markdown
Member Author

Dear @conda/steering-council, this CEP is officially entering its Request For Comments (RFC) period. This will last two weeks, until May 12th EOD AoE.

@jezdez jezdez added the rfc Request for comments period to wrap up the discussion before starting a vote. label Apr 28, 2026
@jaimergp
jaimergp marked this pull request as ready for review May 6, 2026 20:09
@jezdez

jezdez commented Jun 8, 2026

Copy link
Copy Markdown
Member Author

I've updated the conda and conda plugin PRs and made another pass over the CEP. I think it's ready for a vote.

Comment thread cep-XXXX.md Outdated
Comment thread cep-XXXX.md Outdated
Comment thread cep-XXXX.md Outdated
Comment thread cep-XXXX.md Outdated
Comment thread cep-XXXX.md Outdated
Comment thread cep-XXXX.md Outdated
Comment thread cep-XXXX.md Outdated
Comment thread cep-XXXX.md Outdated
Comment thread cep-XXXX.md Outdated
jezdez added 2 commits June 9, 2026 09:57
@jaimergp

jaimergp commented Jun 10, 2026

Copy link
Copy Markdown
Member

Dear @conda/steering-council,

The vote for this CEP has started. It will be open for two weeks, until June 24th, 2026, 23:59 Anywhere on Earth. This time period has been chosen to make it eligible for time-out rules. As an Enhancement Proposal vote, it requires 60% affirmative votes to pass.

To vote, please mark the relevant checkbox under your username:

@jaimergp jaimergp added vote Voting following governance policy and removed rfc Request for comments period to wrap up the discussion before starting a vote. labels Jun 10, 2026
@jaimergp

Copy link
Copy Markdown
Member

@mariusvniekerk, @chenghlee, @msarahan, @jakirkham: final reminder to vote, thanks!

@jaimergp

Copy link
Copy Markdown
Member

Vote has closed, results are in: the CEP has passed!

  • SC size: 13
  • Votes: 10 (76.9% of SC)
    • Yes: 10 (100% of votes)
    • No: 0
    • Abstain: 0

@conda/steering-council, this will be minted as CEP 47.

@jaimergp jaimergp changed the title CEP XXXX: Index timestamp in package record metadata CEP 47: Index timestamp in package record metadata Jun 30, 2026
@jaimergp
jaimergp merged commit cc15cf1 into conda:main Jun 30, 2026
1 check passed
@jaimergp

Copy link
Copy Markdown
Member

Congrats, @jezdez!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

vote Voting following governance policy

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants