POC - CCR analysis CLI / pipeline

[JennyPng]

To understand:

• Read README.md for how to use the CLI
• Read DESIGN.md and IMPLEMENTATION_PLAN.md for Copilot's plan input
• Read EXECUTION_LOG.md for exactly what Copilot implemented and the order in which it did so

Example analyzer report output:

analyzer report
Run 5  Azure/azure-sdk-for-python  state=merged  prs=49  status=completed
             Metrics             
┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃ Metric                ┃ Value ┃
┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
│ miss_rate             │ 0.933 │
│ copilot_overlap_rate  │ 0.048 │
│ copilot_acted_on_rate │ 0.847 │
│ human_burden_per_pr   │ 0.429 │
└───────────────────────┴───────┘
             Data quality             
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃ Counter                    ┃ Value ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
│ substantive_human_count    │    21 │
│ copilot_comment_count      │   151 │
│ gap_count                  │    14 │
│ judged_human_count         │    37 │
│ unjudged_human_count       │     5 │
│ low_confidence_human_count │     0 │
└────────────────────────────┴───────┘
       Top themes        
┏━━━━━━━━━━━━━━━━┳━━━━━━┓
┃ Theme          ┃ Gaps ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━┩
│ api-design     │    7 │
│ docs           │    3 │
│ security       │    2 │
│ error-handling │    1 │
│ perf           │    1 │
└────────────────┴──────┘
                                     What Copilot missed → prompt fixes                                      
┏━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃     PR ┃ Theme          ┃ Missed finding                         ┃ Prompt improvement                     ┃
┡━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ #47203 │ api-design     │ The workflow contains four separate    │ Identify and suggest consolidating     │
│        │                │ steps that could be consolidated into  │ multiple workflow steps into a single  │
│        │                │ a single step using the                │ step when a more efficient alternative │
│        │                │ 'github-script' action for efficiency. │ exists.                                │
│ #47203 │ api-design     │ The custom 'loadSharedModule' function │ Avoid implementing custom functions    │
│        │                │ duplicates functionality that is       │ that duplicate functionality already   │
│        │                │ already provided by the shared         │ provided by shared or standard         │
│        │                │ 'execFile' module.                     │ libraries.                             │
│ #47203 │ api-design     │ The 'execFile' function in 'runNode'   │ Avoid duplicating error handling logic │
│        │                │ appears to duplicate error handling    │ when it is already implemented in      │
│        │                │ logic that is already implemented in   │ shared or reusable components.         │
│        │                │ the shared 'execFile' function.        │                                        │
│ #47290 │ api-design     │ The dependency version constraint      │ Flag dependency version constraints    │
│        │                │ `opentelemetry-api==1.42` is overly    │ that use `==` for non-pre-release      │
│        │                │ restrictive and does not allow for     │ versions and suggest using `~=` to     │
│        │                │ bugfix releases (e.g., `1.42.1`),      │ allow for bugfix updates within the    │
│        │                │ which could lead to compatibility      │ same minor version.                    │
│        │                │ issues or missed updates.              │                                        │
│ #47428 │ api-design     │ The 'on_emit' method in                │ Avoid assigning duplicate or redundant │
│        │                │ '_BaggageLogRecordProcessor' assigns   │ attributes in data structures to       │
│        │                │ duplicate attributes for 'agent_name'  │ prevent confusion and potential        │
│        │                │ in the 'attrs' dictionary.             │ errors.                                │
│ #47428 │ api-design     │ The 'on_emit' method in                │ Avoid assigning duplicate or redundant │
│        │                │ '_BaggageLogRecordProcessor' assigns   │ attributes in data structures to       │
│        │                │ duplicate attributes for               │ prevent confusion and potential        │
│        │                │ 'agent_version' in the 'attrs'         │ errors.                                │
│        │                │ dictionary.                            │                                        │
│ #47428 │ api-design     │ The constructor of                     │ Flag cases where duplicate attributes  │
│        │                │ `_BaggageLogRecordProcessor`           │ with the same semantic meaning are     │
│        │                │ introduces duplicate attributes for    │ added to a data structure, as this can │
│        │                │ `session_id` in the log record         │ lead to ambiguity or unintended        │
│        │                │ attributes, which could lead to        │ behavior.                              │
│        │                │ ambiguity or unintended behavior.      │                                        │
│ #47203 │ docs           │ The use of 'require' instead of        │ Enforce the use of ES module syntax    │
│        │                │ 'import' in a JavaScript file, which   │ (e.g., 'import' instead of 'require')  │
│        │                │ deviates from the project's standard   │ in JavaScript files if the project     │
│        │                │ of using ES modules.                   │ standard specifies it.                 │
│ #47456 │ docs           │ The absence of a log message when the  │ Log a message when a file or resource  │
│        │                │ post-emitter script file is not found  │ is expected but not found to improve   │
│        │                │ at 'script_path' in the                │ debugging and observability.           │
│        │                │ 'run_post_emitter_script' function.    │                                        │
│ #47456 │ docs           │ The hardcoded value                    │ Ensure that docstrings and comments    │
│        │                │ '_post_emitter.ps1' in the docstring   │ accurately reflect the actual values   │
│        │                │ of 'run_post_emitter_script' does not  │ or constants used in the code.         │
│        │                │ match the actual constant              │                                        │
│        │                │ 'POST_EMITTER_SCRIPT_NAME'.            │                                        │
│ #47214 │ error-handling │ The `_pump` function in `_tee_stream`  │ Flag cases where exceptions are caught │
│        │                │ suppresses exceptions during           │ and suppressed without logging or      │
│        │                │ `sink.write` and `sink.flush` without  │ providing visibility into the error,   │
│        │                │ logging or displaying the errors,      │ as this can hinder debugging and error │
│        │                │ which could make debugging difficult.  │ resolution.                            │
│ #47203 │ perf           │ The 'setup-node' step is unnecessary   │ Avoid redundant setup steps in         │
│        │                │ as the 'github-script' action          │ workflows when the required tools or   │
│        │                │ automatically uses Node.js.            │ dependencies are already provided by   │
│        │                │                                        │ the actions being used.                │
│ #47368 │ security       │ The `_is_allowed_redirect_target`      │ Ensure that redirect validation        │
│        │                │ function does not validate the scheme  │ functions check the scheme, effective  │
│        │                │ or effective port of the target URL,   │ port, and host/subdomain to prevent    │
│        │                │ which could allow insecure redirects   │ insecure or unintended redirects.      │
│        │                │ or redirects to unintended ports,      │                                        │
│        │                │ potentially exposing sensitive         │                                        │
│        │                │ information.                           │                                        │
│ #47459 │ security       │ The client name in the                 │ Verify that sensitive or confidential  │
│        │                │ 'pyproject.toml' file does not match   │ information is not inadvertently       │
│        │                │ the real client name, potentially      │ included in configuration files or     │
│        │                │ exposing sensitive information.        │ code.                                  │
└────────┴────────────────┴────────────────────────────────────────┴────────────────────────────────────────┘
Paste-ready prompt additions below (also in --format markdown/json):
## Suggested review-prompt additions

Paste the rules below into your Copilot review prompt. Each was inferred from a substantive issue a human 
reviewer caught that the Copilot reviewer missed.

### api-design
- Avoid assigning duplicate or redundant attributes in data structures to prevent confusion and potential 
errors.  _(from #47428)_
- Avoid duplicating error handling logic when it is already implemented in shared or reusable components.  
_(from #47203)_
- Avoid implementing custom functions that duplicate functionality already provided by shared or standard 
libraries.  _(from #47203)_
- Flag cases where duplicate attributes with the same semantic meaning are added to a data structure, as this
can lead to ambiguity or unintended behavior.  _(from #47428)_
- Flag dependency version constraints that use `==` for non-pre-release versions and suggest using `~=` to 
allow for bugfix updates within the same minor version.  _(from #47290)_
- Identify and suggest consolidating multiple workflow steps into a single step when a more efficient 
alternative exists.  _(from #47203)_

### docs
- Enforce the use of ES module syntax (e.g., 'import' instead of 'require') in JavaScript files if the 
project standard specifies it.  _(from #47203)_
- Ensure that docstrings and comments accurately reflect the actual values or constants used in the code.  
_(from #47456)_
- Log a message when a file or resource is expected but not found to improve debugging and observability.  
_(from #47456)_

### error-handling
- Flag cases where exceptions are caught and suppressed without logging or providing visibility into the 
error, as this can hinder debugging and error resolution.  _(from #47214)_

### perf
- Avoid redundant setup steps in workflows when the required tools or dependencies are already provided by 
the actions being used.  _(from #47203)_

### security
- Ensure that redirect validation functions check the scheme, effective port, and host/subdomain to prevent 
insecure or unintended redirects.  _(from #47368)_
- Verify that sensitive or confidential information is not inadvertently included in configuration files or 
code.  _(from #47459)_

Caveat: miss_rate is RELATIVE to issues humans actually commented on, not ground truth. Issues nobody flagged
are invisible. Track precision and recall separately; never read miss_rate as an absolute.