feat: Provider Sensitivity demo (LLM comparison) - #14
Conversation
Adds a new demo that sends identical prompts to all ready LLM providers and compares responses side-by-side. Includes four prompt categories (factual, creative, reasoning, decision), llm:choose comparison, custom prompt input, and per-provider length statistics. - demos/provider-sensitivity/provider-sensitivity.nlogo: main model - demos/provider-sensitivity/tests.nlogo: test suite (config, switching, comparison, history isolation) - demos/provider-sensitivity/config: multi-provider config template - demos/provider-sensitivity/README.md: usage documentation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a3b3abd2e5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| openai_api_key=YOUR_OPENAI_API_KEY_HERE | ||
| anthropic_api_key=YOUR_ANTHROPIC_API_KEY_HERE | ||
| gemini_api_key=YOUR_GEMINI_API_KEY_HERE |
There was a problem hiding this comment.
These placeholder values are non-empty strings, and llm:providers currently treats any non-empty key as "ready" (it only checks trim.nonEmpty), so a fresh checkout will incorrectly include OpenAI/Anthropic/Gemini as ready and then fail live calls with auth errors instead of skipping them. That breaks the demo/test harness flow and contradicts the README claim that providers without valid keys are skipped.
Useful? React with 👍 / 👎.
|
|
||
| to test-metric-capture | ||
| print "-- test-metric-capture --" | ||
| let provider-name first ready-providers |
There was a problem hiding this comment.
Using first ready-providers makes this test brittle with partial credentials: if users configure only one provider but other providers still appear in llm:providers (for example due placeholder keys), the alphabetically first provider is exercised and the test fails even though another configured provider works. Iterating providers with skip-on-error (or selecting one that successfully chats) would prevent false failures; the same pattern appears again in test-choose-valid-option.
Useful? React with 👍 / 👎.
Code Review: Provider Sensitivity Demo ✅Strengths:
Logic Review:
Minor Recommendations:
Status: APPROVED FOR MERGE ✅ |
Convert provider-sensitivity.nlogo to XML-based .nlogox format with CDATA-wrapped code, structured widgets, and proper shape definitions. Remove legacy .nlogo files (tests.nlogo, test-harness.nlogo, and provider-comparison-tests.nlogo) replaced by Python test suite. Update README to reference .nlogox and NetLogo 7.0.3.
…ssion checks 26 Python tests across 4 classes: artifact validation, XML structure parsing, NetLogo 7.0.3 format compliance, and behavior regression guards covering deprecated primitives, procedure closure, prompt categories, provider switching, quality scoring, and cost estimation.
…into feature/ccl-03-provider-sensitivity
Replaces the flat sequential text collector with a visual ABM where parallel chains of turtles paraphrase a seed message through different LLM providers. Semantic drift is measured via Jaccard similarity and visualized as a green-to-red color gradient. Includes optional thinking mode using llm:chat-with-thinking, drift/length plots, and six seed message categories.
Replaces previous provider-comparison docs with Telephone Game documentation covering setup, thinking mode, drift visualization, and the test suite.
38 tests covering XML structure, widget types, core procedures, seed message categories, thinking mode primitives, config validation, and regression guards against deprecated primitives.
NetLogo 7 uses lower-case (hyphenated) not lowercase. This caused "Nothing named LOWERCASE has been defined" on setup.
NetLogo has no built-in lower-case primitive. Added lower-case-string and lower-case-char reporters following the standard pattern from NetLogo's Case Conversion Example model.
- Larger world (53x33 patches) with colored lane backgrounds per chain - Bigger turtles (2.5) with drift score labels instead of truncated text - Full message text printed in output on each step and in Show Results - Thicker colored links that reflect drift gradient - "Inspect Turtle" button to click any turtle and see its full message - Provider name headers in UPPERCASE in results - Summary section with avg/final drift per provider - Added upper-case-string for formatted output
Each provider gets a unique hue (green, blue, orange, violet, red, yellow). Drift fades saturation and brightness rather than switching to a flat green-red gradient, so providers remain visually distinct even at high drift. Links also carry the provider hue.
Uses NetLogo color numbers (green=55, blue=105, orange=25, violet=115, red=15, yellow=45) so each provider's line is visually distinct in both the drift and message length charts.
Non-empty placeholder strings like YOUR_OPENAI_API_KEY_HERE caused llm:providers to treat unconfigured providers as ready, leading to auth errors during chain processing. Empty values ensure only providers with real keys (or Ollama) appear in ready-providers-list.
1 participant
Compares LLM providers (OpenAI, Anthropic, Gemini, Ollama) in real-time switching scenarios.
What it does:
Why it matters:
Production systems need cost/quality tradeoffs; this demo shows real-world impact
Files:
Demonstrates: Provider agility in production LLM systems