Add Glyphh Ada 1.1 (HDC+FC) to BFCL V4 Leaderboard - #1308
Open
timmetim wants to merge 2 commits into
Open
Conversation
Adds GlyphhHDCHandler for decoding pre-computed HDC model results. Handles JSON-stringified parameters in decode_execute. Registers glyphh-hdc-v1 and glyphh-ada-1.1 model configs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add Glyphh Ada 1.1, a hybrid Hyperdimensional Computing (HDC) + Function Calling model. Uses HDC for deterministic function routing (cosine similarity in 10,000-dimensional space) and Claude Haiku 4.5 for argument extraction. Overall accuracy: 74.50% - Non-Live: 88.71% - Live: 74.32% - Irrelevance: 87.56% - Multi-Turn: 53.75% - Web Search: 79.50% - Memory: 87.10% Total cost: $2.08 | Mean latency: 8.52s Changes: - Add GlyphhHandler (extends ClaudeHandler) in model_handler/api_inference/glyphh.py - Add model config entry with Haiku pricing ($0.05/$0.40 per 1M tokens) - Add to SUPPORTED_MODELS.md and supported_models.py - Include pre-computed result files for all categories
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Glyphh Ada 1.1, a hybrid Hyperdimensional Computing (HDC) + Function Calling model to the BFCL V4 leaderboard.
Architecture
Glyphh Ada uses a two-stage pipeline:
tool_useThis means function selection is deterministic and instant, while the LLM is only used for argument extraction on a filtered candidate set.
Results
Total cost: $2.08 | Mean latency: 8.52s
Changes
bfcl_eval/model_handler/api_inference/glyphh.py—GlyphhHandlerextendingClaudeHandlerbfcl_eval/constants/model_config.py— Model config with Haiku pricing ($0.05/$0.40 per 1M tokens)bfcl_eval/constants/supported_models.py— Addedglyphh-ada-1.1SUPPORTED_MODELS.md— Added to supported models tableresult/glyphh-ada-1.1/— Pre-computed result files for all categoriesscore/— Evaluation scores generated byeval_runner.py --test-category allAll results verified locally using
eval_runner.py.