Skip to content

Design the Occluded-Object memory diagnostic RFC #7

Description

@myheart521

Goal

Specify a paired toy diagnostic for whether a predicted future retains a task-relevant object after the object becomes unobserved.

Acceptance criteria

  • Define the shared initial context, occlusion event, legal action branches, and restore contract.
  • Separate memory correctness from pixel similarity and task success.
  • Define oracle, copy-last, memoryless, and wrong-memory baselines.
  • State expected metric ordering and at least two anti-gaming cases.
  • Define deterministic generation, train/evaluation leakage controls, and artifact hashes.
  • Set a CPU-first runtime and fixture-size budget.
  • Produce an RFC and benchmark-card outline before implementation.

Open questions

Decide whether the smallest version should use typed states, rendered pixels, or both. Do not use a hidden simulator state metric as if it were directly available from a pixel-only model.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:benchmarkBenchmark data or protocolhelp wantedExtra attention is neededneeds-rfcRequires a design RFC before implementation

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions