Repository navigation
Added AoT for Qwen, Phi, Gemma & Whisper - #4
Closed
neuropilot-captain wants to merge 726 commits into
Closed
neuropilot-captain wants to merge 726 commits into
neuropilot-captain wants to merge 726 commits into
Conversation
Move check_platform_support() to utils.sh file since it will be used by other scripts in coming commits. Add check_os_support() in utils.sh and call it from main. Signed-off-by: per.held@arm.com
…ch#13757) ### Summary Add pre-processing pass on the aten dialect level, which splits `gru` nodes with `num_layers > 1` into an equivalent sequence of single layer `gru` nodes. ### Test plan Unit tests provided in `backends/nxp/tests/test_gru_splitting.py`.
Summary: This diff adds extra logging in CoreML to indicate whether the model in the pte file was pre-compiled. Differential Revision: D81256207
This reverts commit 185c96a. ### Summary [PLEASE REMOVE] See [CONTRIBUTING.md's Pull Requests](https://github.com/pytorch/executorch/blob/main/CONTRIBUTING.md#pull-requests) for ExecuTorch PR guidelines. [PLEASE REMOVE] If this PR closes an issue, please add a `Fixes #<issue-id>` line. [PLEASE REMOVE] If this PR introduces a fix or feature that should be the upcoming release notes, please add a "Release notes: <area>" label. For a list of available release notes labels, check out [CONTRIBUTING.md's Pull Requests](https://github.com/pytorch/executorch/blob/main/CONTRIBUTING.md#pull-requests). ### Test plan [PLEASE REMOVE] How did you test this PR? Please write down any manual commands you used and note down tests that you have written if applicable.
### Summary Update the tokenizers dependency to 4ed91cc to pick up the Windows changes. Note that this also involves updating the submodule URL, as it's been recently migrated from pytorch-labs to meta-pytorch. ### Test plan CI
Differential Revision: D81199585 Pull Request resolved: pytorch#13765
Differential Revision: D81537565
Now for QNN users, they can use the dependency `implementation "com.qualcomm.qti:qnn-runtime:$qnnVersion"` to add the runtime deps Define `qnnVersion` in gradle.properties
### Summary - Support Flip - Support Index_Select ### Test plan - `python backends/qualcomm/tests/test_qnn_delegate.py -k TestQNNQuantizedOperator.test_qnn_backend_flip --model SM8750 --device $DEVICE --build_folder build-android` - `python backends/qualcomm/tests/test_qnn_delegate.py -k TestQNNQuantizedOperator.test_qnn_backend_index_select --model SM8750 --device $DEVICE --build_folder build-android` - `python backends/qualcomm/tests/test_qnn_delegate.py -k TestQNNQuantizedModel.test_qnn_backend_conv2d_flip --model SM8750 --device $DEVICE --build_folder build-android` - `python backends/qualcomm/tests/test_qnn_delegate.py -k TestQNNQuantizedModel.test_qnn_backend_conv2d_slice_copy --model SM8750 --device $DEVICE --build_folder build-android`
Run unit test CI on Windows. I've disabled a few jobs (mainly pending tokenizers being available on Windows), but most everything is working out of box. I did also make a few additional fixes: * Shorten names for a few external project targets in CMake, in order to work around max path length limitations. * Update load code for kernel libraries in python. The binary names do not start with lib by default on Windows. This could be solved in the build, but it seems easier to just extend the glob. * Fix an issue blocking iterative builds for XNNPACK - we need to cause the schema move command to overwrite the existing artifact in the build directory. * Disable tokenizer install on Windows for now. Pending enablement in the tokenizers repo. * Fix paths in setup.py for Windows builds.
…ation (pytorch#13908) * Add _serialize_operator() function to NodeVisitor * Enable passing of DebugHook to NodeVisitor if enabled * Add option dump_debug_info to ArmCompileSpecBuilder
Differential Revision: D81142086 Pull Request resolved: pytorch#13731
Add Windows CI jobs to test models. From looking at the existing CI, it looks like our model test jobs are pretty minimal and don't verify much other than it runs without crashing. I'd like to re-visit this in the near future and integrate with the new test harness, but for the purpose this PR, I'm just matching the Windows jobs to what the Linux jobs do.
Differential Revision: D81359053 Pull Request resolved: pytorch#13829
Exporting the model continues to fail now.
This PR enables per-row grouping in CoreML LUT ops.
ghstack-source-id: 9f1fca5 Pull Request resolved: pytorch#13662
Differential Revision: D77232760 Pull Request resolved: pytorch#12597
Differential Revision: D77232764 Pull Request resolved: pytorch#13918
### Summary Utilize `multimodal_runner.h` to run [Voxtral exported from Optimum Executorch](huggingface/optimum-executorch#126). The runner takes in a `.pt` file of a preprocessed audio recording and feeds it a C++ multimodal runner. Example output: ``` This audio is a casual and somewhat silly conversation between two speakers who seem to be discussing their tattoos. The speakers are engaging in a game where they ask each other what their tattoos say, but both repeatedly say "sweet" instead of the actual words. The speakers are aware of their mistake and try to correct it by asking the other what their tattoo says, but they still end up saying "sweet" again. The conversation ends with a speaker telling the other that their tattoo says " PyTorchObserver {"prompt_tokens":1138,"generated_tokens":99,"model_load_start_ms":0,"model_load_end_ms":0,"inference_start_ms":1756159197436,"inference_end_ms":1756159222710,"prompt_eval_end_ms":1756159209605,"first_token_ms":1756159209605,"aggregate_sampling_time_ms":96,"SCALING_FACTOR_UNITS_PER_SECOND":1000} I 00:00:33.116291 executorch:stats.h:104] Prompt Tokens: 1138 Generated Tokens: 99 I 00:00:33.116304 executorch:stats.h:110] Model Load Time: 0.000000 (seconds) I 00:00:33.116312 executorch:stats.h:117] Total inference time: 25.274000 (seconds) Rate: 3.917069 (tokens/second) I 00:00:33.116320 executorch:stats.h:127] Prompt evaluation: 12.169000 (seconds) Rate: 93.516312 (tokens/second) I 00:00:33.116327 executorch:stats.h:136] Generated 99 tokens: 13.105000 (seconds) Rate: 7.554369 (tokens/second) I 00:00:33.116338 executorch:stats.h:147] Time to first generated token: 12.169000 (seconds) I 00:00:33.116344 executorch:stats.h:153] Sampling time over 1237 tokens: 0.096000 (seconds) ``` ### Test plan Build and run: ``` # Build and install ExecuTorch cmake --preset llm -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=cmake-out -DEXECUTORCH_ENABLE_LOGGING=ON && cmake --build cmake-out -j16 --target install --config Release # Build and install Voxtral runner cmake -DCMAKE_INSTALL_PREFIX=cmake-out -DBUILD_TESTING=OFF -DCMAKE_BUILD_TYPE=Release -Bcmake-out/examples/models/voxtral examples/models/voxtral && cmake --build cmake-out/examples/models/voxtral -j16 --config Release # Run Voxtral runner ./cmake-out/examples/models/voxtral/voxtral_runner --model_path ~/models/voxtral/voxtral_q8da4w_edm_qe4w_d_split_metadata_unsqueeze.pte --tokenizer_path ~/hf/models--mistralai--Voxtral-Mini-3B-2507/snapshots/3060fe34b35ba5d44202ce9ff3c097642914f8f3/tekken.json --prompt "What can you tell me about this audio?" --audio_path ~/models/voxtral/input_features.bin ``` Pull Request resolved: pytorch#13663
Differential Revision: D81351792 Pull Request resolved: pytorch#13823
## Summary Runs audio preprocessing (mel spectrogram conversion) on raw audio tensor .bin file, using an exported `.pte` from https://github.com/pytorch/executorch/blob/main/extension/audio/mel_spectrogram.py Current limitations - no batching of output in the spectrogram processing module, so can only support audio of <30 seconds. ``` The speaker in this audio seems to be talking about their concerns about a device called the model or maybe they're just talking about the model in general. They mention that the model was trained with the speaker for inference, which suggests that the model was trained based on the speaker's data or instructions. They also mention that the volume is quite small, which could imply that the speaker is trying to control the volume of the model's output, likely because they are concerned about how loud the model's responses might PyTorchObserver {"prompt_tokens":388,"generated_tokens":99,"model_load_start_ms":0,"model_load_end_ms":0,"inference_start_ms":1756351346381,"inference_end_ms":1756351362602,"prompt_eval_end_ms":1756351351435,"first_token_ms":1756351351435,"aggregate_sampling_time_ms":99,"SCALING_FACTOR_UNITS_PER_SECOND":1000} I 00:00:24.036773 executorch:stats.h:104] Prompt Tokens: 388 Generated Tokens: 99 I 00:00:24.036800 executorch:stats.h:110] Model Load Time: 0.000000 (seconds) I 00:00:24.036805 executorch:stats.h:117] Total inference time: 16.221000 (seconds) Rate: 6.103200 (tokens/second) I 00:00:24.036815 executorch:stats.h:127] Prompt evaluation: 5.054000 (seconds) Rate: 76.770875 (tokens/second) I 00:00:24.036819 executorch:stats.h:136] Generated 99 tokens: 11.167000 (seconds) Rate: 8.865407 (tokens/second) I 00:00:24.036822 executorch:stats.h:147] Time to first generated token: 5.054000 (seconds) I 00:00:24.036828 executorch:stats.h:153] Sampling time over 487 tokens: 0.099000 (seconds) ```
This PR was created by the merge bot to help merge the original PR into the main branch. ghstack PR number: pytorch#13899 by @Ninja91 ^ Please use this as the source of truth for the PR details, comments, and reviews ghstack PR base: https://github.com/pytorch/executorch/tree/gh/Ninja91/18/base ghstack PR head: https://github.com/pytorch/executorch/tree/gh/Ninja91/18/head Merge bot PR base: https://github.com/pytorch/executorch/tree/main Merge bot PR head: https://github.com/pytorch/executorch/tree/gh/Ninja91/18/orig @diff-train-skip-merge Co-authored-by: Nitin Jain <jainnitin@meta.com>
Differential Revision: D81245500 Pull Request resolved: pytorch#13919
Differential Revision: D81634155 Pull Request resolved: pytorch#13921
Differential Revision: D80980948 Pull Request resolved: pytorch#13660
…h#13909) Summary: In another PR, pytorch#13722, for whatever reason, this test was failing. Adjusting the margin here since I have seen this fail before on trunk but somehow it got resolved. So there is some level of flakiness particularly around quantized kv cache + ring attention Test Plan: CI Reviewers: Subscribers: Tasks: Tags: ### Summary [PLEASE REMOVE] See [CONTRIBUTING.md's Pull Requests](https://github.com/pytorch/executorch/blob/main/CONTRIBUTING.md#pull-requests) for ExecuTorch PR guidelines. [PLEASE REMOVE] If this PR closes an issue, please add a `Fixes #<issue-id>` line. [PLEASE REMOVE] If this PR introduces a fix or feature that should be the upcoming release notes, please add a "Release notes: <area>" label. For a list of available release notes labels, check out [CONTRIBUTING.md's Pull Requests](https://github.com/pytorch/executorch/blob/main/CONTRIBUTING.md#pull-requests). ### Test plan [PLEASE REMOVE] How did you test this PR? Please write down any manual commands you used and note down tests that you have written if applicable.
Not intended for landing
…torch#14154) We should be going through the runtime_wrapper for everything IIUC. Differential Revision: [D82127664](https://our.internmc.facebook.com/intern/diff/D82127664)
…torch#13946) ### Summary Fixes pytorch#13044 Release notes: build
Differential Revision: D81948451 Pull Request resolved: pytorch#14160
### Summary Build and run C++ tests in the unit test CI job on Windows. Fix a few lingering issues on Windows that surfaced. Changes: * Add CMake build + CTest execution the Windows unit test job. * Gate Core ML export recipe tests (and coremltools imports) to non-Windows platforms. * Add some explicit cases for size_t 0 literals to make MSVC overload resolution happy. * Remove 2> /dev/null from some CMake PTE export custom commands. I'm not entirely sure why this is there in the first place (keep the log a little cleaner?), but it doesn't work on Windows. Easiest to just remove it. * Update the temp file utility code (in extension/testing_util/temp_file.h) to use standard library file IO (as opposed to posix). * **(Critical)**: Replace long with int64_t in a few portable kernels that deal with indices. It should be int64_t, as this is the correct type for long tensors. On Windows, for example, long is actually 32-bit for historical reasons, and it causes memory corruption / crashes when trying to read/write ScalarType::Long tensors with long* pointers. * Update a bunch of custom commands to use platform-specific syntax or replace with a cross-platform approach. * Use non-strict export for test models. We can support strict export on Windows, but it take a bit of env setup. This is covered in CI in places, but non-strict is the default so it makes it a bit easier to build tests this way. I've also excluded method_test and tensor_parser_test for now, due to the env issues with some of the PTE test models. The tests pass locally - but I need to debug the build-time PTE creation for these tests. I'm disabling to get the remaining 98% of the tests up first, and will fix these shortly. Tracking in pytorch#13883. ### Test plan CI - the tests are covered by the unittest jobs (debug, release, and editable).
Summary: It's moved to executorch-examples repo Differential Revision: D82183720
* Rewrite to make it more concise Change-Id: I57c5dbc8f87a4e8f527597ec721d9697f41e5d3b cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 Signed-off-by: Tom Allsop <tom.allsop@arm.com>
…sion (pytorch#14154)" (pytorch#14233) This reverts commit 6d8583d. Something went wrong with the automated merge of it + several PRs stacked on it initially, and my attempt to recover merged an old version.
Something went wrong with the automated merge here. This is an exact export of the corresponding diff. Differential Revision: D82127664
Something went wrong with the automated merge here. This is an exact export of the corresponding diff. Differential Revision: D82127663
Something went wrong with the automated merge here. This is an exact export of the corresponding diff. Differential Revision: D82127665
Something went wrong with the automated merge here. This is an exact export of the corresponding diff. Differential Revision: D82127666
Differential Revision: D82129824 Pull Request resolved: pytorch#14165
### Summary Add export scripts for supporting bert distilbert --------- Co-authored-by: neuropilot-captain <Kevin.Goo@mediatek.com>
…g buffer (pytorch#14244) ## Context During export, Vulkan sometimes converts certain tensor dtypes. The most common case of this is that int64 and float64 are internally represented as int32 and float32 tensors. The primary reason for this is to reduce the number of dtype variants that need to be generated for each shader, and also due to the fact that 64-bit types are not guaranteed to be supported. However, this raises an issue if an int64 or float64 tensor is marked as an input/output tensor of the model. The source/destination ETensor will have a different dtype than the internal representation, meaning that the input/output bytes will be interpreted incorrectly. ## Changes This diff fixes this behaviour by introducing the concept of a "staging dtype". This allows the staging buffer of a tensor to have a different dtype than the underlying GPU buffer or texture. When copying to/from the GPU resource, the dtype can then be converted to the correct dtype expected by the client code. As a bonus, also add an optional setting to force fp16 to be used internally for fp32 tensors. This allows models to access half precision inference without needing to incur the cost of dtype conversion ops being inserted into the graph, or needing to manually convert inputs/outputs to half type. Differential Revision: [D82234180](https://our.internmc.facebook.com/intern/diff/D82234180/)
Differential Revision: D82168471 Pull Request resolved: pytorch#14188
…lkan export with force_fp16 flag (pytorch#14245) Title says it all! Differential Revision: [D82234179](https://our.internmc.facebook.com/intern/diff/D82234179/)
try fix openvino test
…ance cores (pytorch#14090) ### Summary Allow build-time configuration of the thread pool size and default to a performance heuristic. There are 2 modes that we want to support: * Heuristic-based. Choose the number of threads according to a performance heuristic. Use threads equal to the number of detected performance cores, but we can continue to iterate on this by adding fine-grained heuristics for specific chipsets in the future. * All cores (threads=cores). This is the current behavior. We need to maintain this as an option for some use cases. With this PR, the default (for OSS) is to use performance cores. From testing with CV models on ~10 representative devices across the performance spectrum, this gives anywhere from parity with the existing perf to up to a 13x speedup (measured on Pixel 6). Many common devices (S20, S22, iPhone 15 Pro) show a 2-4x speedup. #### Specifying Threadpool Size To specify the threadpool size, I've added two preprocessor options (and corresponding CMake options): * `EXECUTORCH_THREADPOOL_USE_PERFORMANCE_CORES`- Use threads = detected perf cores. * `EXECUTORCH_THREADPOOL_USE_ALL_LOGICAL_CORES` - Use threads = logical cores. ### Test plan I've verified that logic functions correctly in OSS by building the executor_runner on M1 Mac and observing the existing logging in cpuinfo_utils. Measuring MobileNet V3 (exported from examples) on XNNPACK, time to run 100 iterations drops from ~450ms to ~230ms on M1 Pro with this change.
neuropilot-captain
had a problem deploying
to
upload-benchmark-results
September 15, 2025 07:46 — with 
GitHub Actions
Error
GitHub Actions
Error
neuropilot-captain
had a problem deploying
to
upload-benchmark-results
September 15, 2025 07:46 — with
GitHub Actions
Error
GitHub Actions
Error
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Added AoT for Qwen2, Qwen3, Phi3, Phi4, gemma2, gemma3 and whisper