Skip to content

#[spec_operation] serves dual purpose — confuses both humans and AI agents #10

Description

@schgoo

Problem

#[spec_operation("name")] is used for two distinct purposes:

  1. On the operation function — instruments it for harness tracing (emits .outcome, .result, .error)
  2. On test functions — marks which spec assertion the test covers (for coverage tracking)

This confusion led a SuperScalar AI coding agent to annotate test functions with assertion IDs instead of operation names:

// WRONG — agent thought spec_operation takes assertion IDs
#[test]
#[spec_operation("CSDL-XML-3.3-A2")]
fn resolve_binary() { ... }

#[test]
#[spec_operation("CSDL-XML-3.3-A3")]  
fn resolve_boolean() { ... }

The correct usage for the harness is:

// On the actual operation — one per operation name
#[spec_operation("resolve_primitive")]
pub fn resolve_primitive(type_name: &str) -> Result<PrimitiveType, String> { ... }

Impact

  • The harness scan (scan.rs) uses a BTreeMap<String, OpDecl> — last-writer-wins. Multiple test functions with different assertion ID strings just overwrote each other.
  • AI agents (and likely humans unfamiliar with the codebase) naturally assume the attribute's argument should be the assertion being tested, not the operation name.

Suggestion

Use separate attributes for separate concerns:

  • #[spec_operation("resolve_primitive")] — on the function, for harness instrumentation
  • #[spec_assertion("CSDL-XML-3.3-A2")] or #[spec_covers("CSDL-XML-3.3-A2")] — on tests, for coverage tracking

This makes intent unambiguous and prevents misuse.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions