アップストリームを取り込み - #3
Open
kyakuno wants to merge 2054 commits into
Open
Conversation
…27424) * server: (router) lazy-load startup_models after main setup * only allow is_first_load to populate it * nits * nits 2
…#26678) * CI: Use LLVM's OpenMP over MSFT_DEBUG_non_redist on Windows Currently, we ship the non-redist debug version of microsoft's libomp. This PR changes this to official LLVM's release, also packaging the license as needed. * Remove LLVM SHA from job name to increase legibility * Add temp validations to CI * Revert "Add temp validations to CI" This reverts commit eef97c8. * Build OpenMP in CI * Make OpenMP fetch self-contained in cmake and cache in CI * Robustify Licens-packaging 1. Ship OpenMP license, not LLVM's. 2. Invalidate cache also on checksum of the license * Remove stale reference in docs/build.md * No longer package base license in release This was scope-creep * Add explanatory comment to OpenMP license * Remove arm64 smoke Forgot this during conflict resolution during rebase of c54c0e9 * Remove GGML_OPENMP_FETCH_CACHE_DIR as requested by @CISC * whitespace changes
…org#27413) Codex found that qd could be a denorm and 1/qd would overflow.
* feat: add --mmproj-device arg & backwards compatible MTMD_BACKEND_DEVICE env var * feat: load mmproj device backend immediately, add -mmdev shortflag * fix: its a pointer now get the name * clean up * gen docs * nits --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* ui: Extract server stream lifecycle from chatStore into ChatStreamManager
Discovery, attach/replay, resume retry and the remote-running snapshot
formed a cohesive cluster inside chatStore. It now lives in
chat-streams.svelte.ts as ChatStreamManager, owned by chatStore, which
keeps the public entry points as delegates so components are
unchanged. chatStore: 2877 -> 2418 lines.
* ui: Extract user interaction gates from agenticStore into AgenticGates
Tool permission requests, turn-limit continue prompts and queued
steering messages are the state the loop waits on between turns. They
had no coupling to session state, so they now live in
agentic-gates.svelte.ts; agenticStore keeps delegates so components
are unchanged. agenticStore: 1196 -> 1073 lines.
* ui: Compose MCP resources under mcpStore.resources
Resource state was a second import scope next to mcpStore. Consumers
now go through mcpStore.resources, so the MCP surface is one store;
mcp-resources.svelte.ts stays a separate file owned by mcpStore.
* ui: Reorganize stores into domain namespaces
* fix: Update stale doc comments
* ui: Consolidate conv running-state into a chat activity ledger
Running-state was split across chatStore.chatLoadingStates (local
pipes), ChatStreamManager.remoteRunningConvs (backend sessions) and
attachingConvs (attach lifecycle), unioned by hand in
getAllLoadingChats and cross-cleaned by setChatLoading calling
streams.clearRemoteRunning - the 'spinner ghosts until tab toggle'
workaround.
chatActivityStore now owns both sets with one transition per event:
markLocal / localEnded (local pipe end also drops the stale remote
hint, no cross-owner call) / applyRemoteSnapshot (diffed). The
sidebar reads chatStore.activity.loadingConvs through the unchanged
getAllLoadingChats entry point.
Consequences:
- isStreamingActive and its five manual writers are gone; isStreaming()
now reports whether the active conversation has a live streaming
pipe, which is what all four consumers (assistant row, stop action,
context gauge, chat screen) actually check
- isLoading/isReasoning become derived from the per-conv maps plus
the active conversation, dropping the manual resync in
syncLoadingStateForChat and clearUIState
- attachingConvs and the last-attach coordination disappear from
ChatStreamManager
- getAllStreamingChats (no consumers) is removed
* ui: Give store collaborators narrow host interfaces
Collaborators took 'host: typeof <store>', i.e. the store's entire
public surface, which is how chatStore's streamChatCompletion,
createAssistantMessage, getApiOptions and setStreamingActive got
widened to public. Replace with per-collaborator interfaces carrying
only the members each one drives:
- ChatStreamHost (chat/streams) - activity, processing, streaming
states, abort controller, loading/streaming setters
- ChatFlowsHost (chat/flows) - streaming core, message creation,
per-conv state setters
- McpHealthHost (mcp/health) - connection registry + reconnection
- ModelPropsHost / ModelStatusHost (models) - model rows, feed
updates; the managers write modalities/status back onto the host's
rows, so those members stay writable
- ConversationsPreferencesHost (conversations) - the active row and
the conversation list
The store classes now declare 'implements <Host>' so the contract is
visible at the class level, and the 'import type { <store> }' back
references in the collaborators disappear entirely - the host
contract is local to each collaborator file, and collaborators can
no longer reach around their slice. Members stay public (structural
typing), but the collaborator side is now compiler-enforced.
* test: Chat Activity store test
* refactor: Cleanup
* chore: Remove legacy architecture docs
* ui: Memoize findMessageIndex for the streaming hot path
Streaming looks up the same message index on every chunk, a linear
scan of activeMessages each time. Cache the last lookup and reuse it
after validating the id still sits at the same position (O(1)); any
structural change to the array fails validation and falls back to a
full scan.
* ui: Throttle per-chunk stream state writes to localStorage
saveStreamState ran JSON.stringify + a synchronous localStorage.setItem
on every decoded chunk of the stream. The read loop now goes through a
new saveStreamStateThrottled (one write per conversation per 500ms,
latest value held pending); the public saveStreamState keeps its
immediate-write contract for stream start and pre-fetch, and also
resets the throttle window.
A pending offset is force-flushed at resume boundaries (resumeStream
reads the offset back from localStorage), on visibilitychange->hidden
and on pagehide, so a reload always finds a usable offset. The resume
offset only needs to be roughly current since the server retransmits
from a line boundary and the client discards its partial line.
Adds unit tests for the throttled/flush/clear interplay.
* ui: Compute context gauge timing stats in one pass
currentRead/Fresh/Cache/Output were separate deriveds, each running a
full reverse scan of activeMessages for the last assistant timings,
and cumulative ran its own forward scan plus an agentic filter - 4-5
O(n) passes per chunk while streaming. Replace with a single
summarizeAssistantTimings() pass (last assistant timings, last
agentic llm totals and the cumulative sums) feeding a shared derived
snapshot. Semantics unchanged, including the live-stats overrides and
the agentic llm-totals branch.
* agentic : clear session state when a conversation is deleted
Every conversation that ran an agentic flow left an AgenticSession in the
store forever; clearSession was never called. conversationsStore now
notifies deletion listeners and agenticStore drops the matching sessions,
avoiding a circular import back into conversationsStore.
* chat : extract ChatService.normalizeMessagesForApi
The DB->API message normalization (convert + drop empty system messages)
was duplicated in sendMessage, preEncode and the agentic flow. Extract it
into one shared method and call it from all three.
* sse : share record splitting and data extraction
splitSseRecords and extractSseDataPayload centralize the record-boundary
splitting and data: line extraction used by parseSseJsonStream and the
models status feed. chat.service keeps its own line-based parser for
resume support.
* api : delegate apiFetchWithParams to apiFetch
apiFetchWithParams duplicated apiFetch's headers/fetch/error handling
body-for-body; it only differs in URL construction. Build the URL and
delegate.
* chat flows : dedupe title, timings and cleanup handling
- conversationsStore.applyTitleFromContent centralizes the title-from-first-
message logic duplicated in 5 places
- ChatProcessingStore.applyStreamTimings centralizes the onTimings handler
shared by the chat and continue flows
- host.cleanupStreaming centralizes the loading/streaming/processing reset
repeated across the continue flow's exit paths
* conversations : centralize conversation update mirroring
rename, pin, mcp override, reasoning effort and cwd all repeated the same
write-DB-then-mirror-into-list-and-active dance. A single
applyConversationUpdate(id, updates) on the host collapses all five and
removes the forgot-to-mirror-one-field bug class. Drops the redundant
array reassignment in setCwd (deep field assignment is reactive).
* mcp : dedupe tool execution, server parsing and tool indexing
- executeTool delegates to executeToolByName (only diff was argument parsing)
- drop the private #parseServerSettings copy; use parseMcpServerSettings
- cache getServers() keyed on the raw config value (hot path)
- indexServerTools() unifies the three identical toolsIndex rebuild loops
Assisted-by: Claude
* mcp : share cursor pagination and tool indexing
- MCPService.paginate() collapses the identical do-while loops in
listAllResources and listAllResourceTemplates
- promoteHealthCheckToConnection now uses indexServerTools like the other
connect paths
Assisted-by: Claude
* database : share message parent-child bookkeeping
- addChildToParent() dedups the append-to-children update in createMessageBranch
and createSystemMessage
- removeChildFromParent() dedups the remove-from-children cleanup in deleteMessage
and deleteMessageCascading
- bulkAdd the cloned messages when forking a conversation instead of one add
per message
Assisted-by: Claude
* chore: Lint/format
* fix: `pagehide` event from `window`
* refactor: Api Fetch util
* docs : rewrite architecture sections in README
Update the high-level diagram, routes, hooks, stores, services and data
flow tables to match the current UI structure (mcp/settings/search
routes, agentic/tools/mcp stores, MCPService/ToolsService/SandboxService,
/tools API). Fix stale architectural patterns for per-conversation state
and modality validation.
* chore : add ESLint rule for blank lines between accessors
Enforce a blank line between consecutive class accessors. The core
padding-line-between-statements rule does not cover class members, so a
local rule is needed.
* refactor : reorder store members and unify naming
Order store class members as public fields, private fields, constructor,
getters, public methods, then private methods. Normalize private naming
to the `private` keyword (drop `#` and the `_` prefix where there is no
matching public getter). Rename conversationsStore.init() to
initialize() to match the other stores.
* refactor : prefix lookup methods with get in agentic and chat stores
Unify bare-name lookup methods with the get* prefix used across the
other stores (mcp, models, tools, settings). Renames currentTurn,
totalToolCalls, lastError, streamingToolCall, executingToolCallId,
pendingPermissionRequest, pendingContinueRequest,
pendingSteeringMessageContent, pendingSteeringMessageExtras in the
agentic store and pendingMessageContent, pendingMessageExtras in the
chat store. Updates the two consuming components and a doc comment.
* refactor: Clean up comments in stores' and services' code
* chore : add ESLint rule for class member ordering
Enforce structural order (public fields -> private fields -> constructor ->
getters -> setters -> public methods -> private methods) with alphabetical
sorting within each group via perfectionist/sort-classes. Dependency
detection keeps Svelte $derived fields in a valid dependency order instead
of alphabetizing them, since Svelte rejects forward references.
Assisted-by: Claude
* refactor : reorder class members to match new ESLint rule
Apply the sort-classes rule across stores, services, hooks and utils.
Pure reordering - verified no logic changes by comparing sorted line
multisets before/after. All tests and svelte-check pass.
… compilers (ggml-org#26476) * opencl: decline KV-convert flash_attn variants on Adreno A7X (compiler SIGSEGV) The Adreno 740 (A7X) compiler E031.41 crashes inside clBuildProgram when building the flash_attn programs whose KV path is mixed-type or dequantized: flash_attn_f32_f16, flash_attn_f32_q8_0, flash_attn_f32_q4_0. It is a driver crash rather than a compile-error return, so build_program_from_source_ex() cannot catch it. The uniform f32 and f16 programs build correctly. Decline the three KV-convert variants on the A7X in supports_op so they never lazy-compile; those attention layers run on the CPU backend instead. Same idiom as the existing Intel DK=512 and X1E carve-outs. test-backend-ops FLASH_ATTN_EXT on the 740: 226 OK / 0 FAIL, previously exit 139. Other parts are unaffected - the gate is dead code there. * opencl: fix q6_K flat mul_mat on older Adreno E031 compilers, gated kernel_mul_mv_q6_K_f32_flat produces ~10x-wrong output on the older Adreno E031 compilers while q4_K and q5_K are correct. Four codegen defects, each confirmed on-device against the CPU reference: 1. 64-bit ulong arithmetic is miscompiled, so every weight and scale read hit the wrong address - the primary cause, and why q5_K (int offsets) was unaffected. The block index is computed in int and widened only inside the pointer expression. 2. The vectorized dequant (int4/float4 bit-ops, convert_*4, dot()) is miscompiled; the 6-bit weights are reconstructed and the dot done scalar. 3. vload4 of the f32 activations is miscompiled; replaced by a scalar-indexed load. 4. The accumulation is miscompiled unless a side effect forces the partial sums to materialize. A printf under a guard the compiler cannot prove false acts as a zero-cost optimizer barrier; its placement is load-bearing. The defect tracks the compiler, not the GPU generation: it reproduces on E031.38 (Adreno 642L) and E031.41 (Adreno 740) and is fixed by E031.45 (Adreno 619), so the workarounds are gated on the compiler version. Where they are not needed they cost real throughput - 42.4 -> 35.1 GFLOPS on an Adreno 840 q6_K GEMV. The explicit compiler-type check is required, not redundant: newer_than_or_same() is false for every non-E031 compiler, so negating it alone would enable the workarounds on E17 and DX. test-backend-ops MUL_MAT is 919/919 on the Adreno 740, 642L, 619, 840 and 850; the 740 and 642L were 909/919 before. The 642L additionally needs the A6X per-kernel-program support to reach these tests at all.
…ple of 32 (ggml-org#27450) The Tensor API mat-mat path of kernel_mul_mm (GGML_METAL_HAS_TENSOR) fed a static K=32 tile to the matmul2d op on every iteration. On the last, partial K tile (ne00 % 32 != 0) the src1 slice extends past the K extent of the tensor, and the op reads those out-of-bounds elements (undefined behavior per the MSL specification, section 2.22.2). Depending on stale memory contents, this corrupted the result or produced NaN. Make the matmul2d op use dynamic_extent for K, and clamp the K extent of both operand tensor views to the remaining valid K range (min(32, K - loop_k)) per iteration, so the op reads exactly the valid K range on every iteration (mirroring the tail handling of the MPP matmul2d examples). On K-aligned inputs the clamp degenerates to the full 32-wide tile: the only difference from the static-K op is that the dynamic-K op derives K from the operand extents and edge-checks the tile against the tensor extents (a handful of integer ops per iteration). Add test-backend-ops MUL_MAT cases with K not a multiple of 32 to exercise the unaligned K path. Assisted-by: pi:llama.cpp/Qwen3.8-27B
* use regular script to build cmake pkg * use old grep without perl
…ml-org#27345) * ggml: support ggml_rope_set_offset on opencl, sycl, wgpu, hexagon * rm inplace optimization
Assisted-by: deepseek-v4-flash
* Update norm.cpp * Update helper.hpp * Update im2col.cpp * Update fattn-mkl.cpp * Update element_wise.cpp * Update fattn-mkl.cpp * Update set_rows.cpp * Update element_wise.cpp * Update ggml-sycl.cpp * Update ggml-sycl.cpp * Update ggml-sycl.cpp * Update ggml-sycl.cpp * Update ggml-sycl.cpp * Update norm.cpp * Update CMakeLists.txt * Update CMakeLists.txt * Update CMakeLists.txt * Update ggml-sycl.cpp
…ggml-org#26635) * feat: updated gating logic of fattn-onednn.cpp * verified device types * Update ggml/src/ggml-sycl/fattn-onednn.cpp Accepted recommendations to add bmg_g31 arch. Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com> * Improved SPDA gate, added documentation. * Added arch var to reworked gate, fixing build errors. * Fix trailing whitespaces. --------- Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com>
…X (compiler issue workaround) (ggml-org#26440) * opencl: keep the vocab-scale K-quant lm_head on the CPU on the Adreno A7X * opencl: revise comments --------- Co-authored-by: Li He <lih@qti.qualcomm.com>
* Add DMMV Q4_K and Q6_K ESIMD kernels Configure cmake build with -DGGML_SYCL_ESIMD=ON to enable. Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * Refactor ESIMD kernels to share common code Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * Move control of ESIMD from compile to runtime Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * Use ESIMD by default when available Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * Fix possible error when using ESIMD by default While not an issue in the current version, this will become an issue when additional QK ESIMD kernels are added (such as Q2_K). Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * Add explicit unroll to ESIMD kernels Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * Tidy up ESIMD kernels a bit Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * Add DMMV Q5_K ESIMD kernel Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * Remove redundant copyright notice Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> --------- Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
…g#27394) Resolve the TODO in test_flash_attn_ext: the branch that creates V as a sub-view of K (MLA-based models) was hardcoded for the 576/512 head shapes. Add a v_is_view_of_k test case parameter (default false) and select the sub-view branch on it; the existing 576/512 (DeepSeek MLA) cases now pass it explicitly, so the test coverage is unchanged. Also add more V-is-sub-view-of-K cases: the 320/256 (Mistral4 MLA) and 192/128 head shapes, and full views with equal head sizes (128/128 F16, 64/64 q8_0). Assisted-by: pi:llama.cpp/Qwen3.8-27B
* Add DMMV Q4_K and Q6_K ESIMD kernels Configure cmake build with -DGGML_SYCL_ESIMD=ON to enable. Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * Refactor ESIMD kernels to share common code Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * Move control of ESIMD from compile to runtime Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * Use ESIMD by default when available Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * Fix possible error when using ESIMD by default While not an issue in the current version, this will become an issue when additional QK ESIMD kernels are added (such as Q2_K). Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * Add explicit unroll to ESIMD kernels Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * Tidy up ESIMD kernels a bit Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * Add a reordered Q2_K MMVQ kernel Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * Add DMMV Q2_K ESIMD kernel Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> --------- Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
As agreed in ggml discussion ggml-org#1579, the official semver releases now include a nightly-tag.txt asset containing the tag of the corresponding nightly release (e.g. b10485). The Web UI assets are published to the HF bucket under the nightly tag, so this makes them discoverable for each official release. - make-release-desc.sh: expose the resolved nightly tag as a nightly_tag output - make-release.yml: create nightly-tag.txt from that tag, upload it as a release asset (skipped on dry-run), mention it in the release body and in the dry-run summary Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ui : rework the settings registry into ordered raw-data sections SETTINGS_REGISTRY becomes an ordered SettingsSectionEntry[] array; the array order is the sidebar display order. Section titles, color mode options and title radio options are declared inline in their section or entry. Entries gain showInUi; MCP servers, the system-message toggle and the title LLM flag become hidden entries of their own section. Derived values (config defaults, help info, chat sections, numeric field lists, syncable parameters) are still derived here; they move to their actual consumers in follow-up commits. * ui : extract settings localStorage persistence into SettingsService Stateless load/save of the settings config and user-override keys, plus the legacy theme key migration. Business logic (default merging, mobile sendOnEnter default, applying the migrated theme) stays in the store. * ui : move the settings exit route into ROUTES SETTINGS_FALLBACK_EXIT_ROUTE is just a route, so it lives with the other routes as ROUTES.SETTINGS_EXIT. * ui : derive the syncable parameter list in the parameter sync service The syncable parameter mapping is only consumed by the sync service, so derive it there from the registry instead of exporting it from the constants file. * ui : restore isPrivate for API key masking * ui : clean up settings registry and router fetch guard Drop the per-entry section field (duplicates the parent slug and is never read) and guard the router model fetch on fields?.length so the Tools/Import-Export pages with empty fields are excluded again. Assisted-by: pi * ui : merge sampling and penalties settings into one section Assisted-by: pi
)" (ggml-org#27486) This reverts commit ff14356.
Skip the nb[3] check when ne[3] == 1, the shader never reads it for a single stream. Cache views carry the full-buffer stride there, so the old check reduced to n_kv == kv_size and the path only engaged with the cache full.
* CUDA: Allow CUDA optimization per split for multi-GPU. Previous guard caused multi-GPU to skip the graph optimization. The graph is already split per device and the optimization doesnt run over the whole model but once per split, and thus should be allowed. However, the CUDA event ggml_cuda_concurrent_event belongs to whichever GPU was "current" when created. If the pass ran while GPU 0 was current, it would stick and during event creation for the second GPU it would land on GPU 0. The fix: set the device explicitly ggml_cuda_set_device(cuda_ctx->device); Default behaviour remains unchanged, only active for GGML_CUDA_GRAPH_OPT=1. Explicit device setting pattern re-used from ggml_backend_cuda_graph_compute. * Update ggml/src/ggml-cuda/ggml-cuda.cu Co-authored-by: Aman Gupta <amangupta052@gmail.com> --------- Co-authored-by: tannerbruhn <tannerbruhn@users.noreply.github.com> Co-authored-by: Aman Gupta <amangupta052@gmail.com>
* mtmd : mark context as const in more methods Mark `mtmd_context` as `const` in: - mtmd_bitmap_init_lazy - mtmd_tokenize - mtmd_tokenize_from_parts - mtmd_helper_support_video - mtmd_helper_bitmap_init_from_file - mtmd_helper_bitmap_init_from_buf - mtmd_helper_video_init - mtmd_helper_video_init_from_buf - mtmd_helper_model_can_chat The tokenization functions in particular are useful to have marked `const`, as that allows more easily telling the compiler that we can safely tokenize from multiple threads (`mtmd_tokenize` is already documented as thread-safe, this just reifies that in the signature). * mtmd : mark tokenization input pointer as const Mark the `bitmaps` and `parts` pointers in `mtmd_tokenize` and `mtmd_tokenize_from_parts` as `const`. This allows more easily calling these with immutable arrays / vectors. * mtmd : mark llama_context as const in mtmd_helper_model_can_chat
Conflict resolution: - common/arg.cpp, tools/mtmd/clip.cpp, tools/mtmd/mtmd-helper.cpp: upstream now contains equivalent Windows non-ASCII (UTF-8 argv and file path) handling, so the upstream version is used as-is. - ggml/src/ggml-metal/ggml-metal-device.m: upstream moved the language version selection into ggml_metal_compile_options_set_lang(). The fork's fix that forces MSL 3.1 (so BF16 kernels stay available when the host process, e.g. Unity, defaults to an older MSL) is kept there for devices without the tensor API. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
github-actions
Bot
added
documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
b10783まで取り込み。MTMD対応は公式に入ったのでForkでは対応不要になった。Metalのバージョン指定はまだ必要。
Merge upstream llama.cpp b10783 into patched
Conflict resolution: