Riley dixon/hipfile shim layer#1
Draft
riley-dixon wants to merge 74 commits into
Draft
Conversation
riley-dixon
force-pushed
the
riley-dixon/hipfile-shim-layer
branch
from
April 21, 2026 23:31
1db49fd to
72764ee
Compare
| hipFile is alpha software and has been tested on limited hardware. | ||
| For full installation details, see the | ||
| `hipFile install guide <https://github.com/ROCm/hipFile/blob/develop/INSTALL.md>`__. | ||
| `hipFile install guide <https://github.com/ROCm/rocm-systems/blob/develop/projects/hipfile/INSTALL.md>`__. |
There was a problem hiding this comment.
@riley-dixon should we point this at rocm documentation and not the repo?
Owner
Author
There was a problem hiding this comment.
If that is possible, I would happily make the change. I was unable to locate where the hipFile documentation is hosted on rocm.docs.amd.com however.
| # (e.g., without GPUDirect), so it's temporarily put here. | ||
| # Third Party | ||
| import hipfile | ||
| from lmcache.v1.storage_backend import hipfile_shim |
There was a problem hiding this comment.
@riley-dixon is there a reason we are using _shim here? What does the cufile equivalent line do? I wonder if the _shim term is redundent?
Owner
Author
There was a problem hiding this comment.
The import cufile line above refers to a 3rd-party package that does provide direct cufile support. The hipfile_shim module masquerades as this cufile package to align with the existing LMCache implementation of hipFile support, rather than working directly with our hipfile package.
I am just trying to make clear that the module we are importing here is not our hipfile package.
|
|
||
| This module wraps ``hipfile.Driver``, ``hipfile.FileHandle``, and | ||
| ``hipfile.Buffer`` so that callers written against the cufile-python API | ||
| (``CuFileDriver`` / ``CuFile``) can use hipfile without any branching. |
| import ctypes | ||
| import os | ||
|
|
||
| from hipfile import Buffer, Driver, FileHandle |
There was a problem hiding this comment.
OK so it is a _shim. You can ignore my comment above on that topic.
| from hipfile import Buffer, Driver, FileHandle | ||
|
|
||
|
|
||
| def _os_flags(mode: str) -> int: |
There was a problem hiding this comment.
Is there really no python lib function for this already? I am sure someone has had to do this before?
Owner
Author
There was a problem hiding this comment.
This code is copied over from the cufile package interestingly. We could probably just import that function instead to reuse it.
Owner
Author
There was a problem hiding this comment.
Actually, just kidding we can't. I forgot the cufile Python package does try to load the cufile shared library. So it is best if we just leave it copied over if someone is running a ROCm only environment.
riley-dixon
force-pushed
the
riley-dixon/hipfile-shim-layer
branch
3 times, most recently
from
April 22, 2026 22:09
e846040 to
f1121e4
Compare
This shim layer matches the cufile-python API that is currently in use by LMCache. This allows hipFile to maintain its own API while not requiring additional code changes in LMCache. Signed-off-by: Riley Dixon <riley.dixon@amd.com>
Signed-off-by: Riley Dixon <riley.dixon@amd.com>
riley-dixon
force-pushed
the
riley-dixon/hipfile-shim-layer
branch
from
April 22, 2026 22:33
f1121e4 to
15f8afe
Compare
Signed-off-by: Riley Dixon <riley.dixon@amd.com>
…cies `uv pip install -e . --no-build-isolation` will not download and install missing build dependencies. hipFile uses `scikit_build_core` as the build backend. Since hipFile is currently under active development, the hipFile Python bindings are not tied to a particular release and are only published in an sdist format. As such, the hipFile bindings need to be built at install time. Signed-off-by: Riley Dixon <riley.dixon@amd.com>
Signed-off-by: Riley Dixon <riley.dixon@amd.com>
…/hipfile-shim-layer
The hipFile Python bindings require that the hipFile C library & header is available on the system. Unfortunately at this time the hipFile C library is still in early access and does not have a package published on the ROCm package repo. This presents a challenge for how to fetch & install the hipFile C library in the LMCache CI. For now, lets leave hipFile as a soft-dependency but I will follow-up and add it back once I solve this early-release distribution problem. Signed-off-by: Riley Dixon <riley.dixon@amd.com>
riley-dixon
force-pushed
the
riley-dixon/hipfile-shim-layer
branch
from
May 5, 2026 17:49
d7a6153 to
bb6a8dc
Compare
Signed-off-by: Riley Dixon <riley.dixon@amd.com>
…/hipfile-shim-layer
…/hipfile-shim-layer Signed-off-by: Riley Dixon <riley.dixon@amd.com>
Signed-off-by: deng451e <838677410@qq.com>
Signed-off-by: deng451e <838677410@qq.com>
…LMCache#3276) Signed-off-by: deng451e <838677410@qq.com>
…2989) When nixl_async_put is enabled, the async PUT cleanup thread (deregister_memory) and the sync GET worker thread (register_memory + prepXfer) both use device_id = 0, 1, 2, ... for every call. NIXL's OBJ backend tracks registrations in a flat unordered_map<uint64, string> keyed by devId with no reference counting. The PUT's deregister erases devId=0, which deletes the GET's registration, causing postXfer to fail with NIXL_ERR_INVALID_PARAM and crashing the EngineCore. Fix: replace the per-call sequential device_id (idx) with a monotonically increasing counter protected by a threading.Lock. Each register/deregister cycle now gets globally unique IDs, so concurrent operations never collide in devIdToObjKey_. Fixes LMCache#2983 Signed-off-by: Jason Goldschmidt <jason.goldschmidt@dell.com>
…onnector (LMCache#3287) * fix(hpu): implement device-specific initialize_kvcaches_ptr for HPU connector Signed-off-by: Tony Lin <tony.lin@intel.com>
…distribution (LMCache#2861) * Add nixl_endpoint_list for per-worker object-storage endpoint distribution Allow TP workers to target different object-storage endpoints when using the NIXL OBJ backend. When nixl_endpoint_list is set in extra_config, each worker selects an endpoint round-robin by worker_id, overriding any nixl_backend_params.endpoint_override (with a logged warning). Also copies nixl_backend_params into a new dict before mutating it to avoid modifying the original config. Signed-off-by: Jason Goldschmidt <jason.goldschmidt@dell.com> * Use local_worker_id instead of worker_id is correct for multi-host deployments because: • worker_id is the global worker ID across all hosts (e.g., host 0 has workers 0-7, host 1 has workers 8-15) • local_worker_id is the worker ID within the local host (e.g., workers 0-7 on each host) • For endpoint distribution, each host should distribute its local workers across endpoints independently • This ensures proper load balancing in multi-host scenarios where each host may have its own set of endpoints Signed-off-by: Jason Goldschmidt <jason.goldschmidt@dell.com> --------- Signed-off-by: Jason Goldschmidt <jason.goldschmidt@dell.com>
…ache#3295) [CI] allow PyTorch/NVIDIA egress endpoints in release workflows Signed-off-by: deng451e <838677410@qq.com> Co-authored-by: Roy Huang <roy.y.huang@gmail.com>
* Add SYCL/XPU port of mem_kernels with Intel optimizations and build scripts Performance optimization for flash_infer on Intel XPU: 1. Added page_buffer_base_offset() and key_value_base_offset() helpers that compute loop-invariant base offsets (incl. integer division for flash_infer block mapping) once before the inner loop. 2. Added submit_multi_layer_kernel_fused_kv() that processes both K and V in a single work-group for non-MLA formats, halving work-group count and avoiding redundant slot lookups, pointer reads, and division. 3. Updated dispatch logic to use fused kernel for non-MLA formats (flash_infer, flash_attention, cross-layer) and original kernel for MLA formats. 4. Applied base-offset hoisting to unilateral kernel as well. Signed-off-by: Feng Tian <feng.tian@intel.com> * Add build instructions for Intel XPU Signed-off-by: Feng Tian <feng.tian@intel.com> * Update csrc/sycl/mem_kernels_sycl.cpp Fix this typo in code Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Tian, Feng <feng.tian@intel.com> * Fix typo in setup.py and __init__.py Signed-off-by: Feng Tian <feng.tian@intel.com> * Refine the logic of importing GPU/XPU/HPU connector symbols Signed-off-by: Feng Tian <feng.tian@intel.com> * fix issues raised by cursor 1. add releasing gil lock for single layer transfer 2. fix typo when printing error message 3. add .spv file to ignore file list Signed-off-by: Feng Tian <feng.tian@intel.com> * fix issues raised by cusor 1. change dtype from uint64 to int64 for old_position calculation 2. change potential out-of-bound access in slot_mapping Signed-off-by: Feng Tian <feng.tian@intel.com> * Fix bugs by merging conflicts Signed-off-by: Feng Tian <feng.tian@intel.com> * Update backend loading logic to support XPU backend The non_cuda_equivalents is the base module to provide default kernels, if backend implments optimized or new ops, use them. Signed-off-by: Feng Tian <feng.tian@intel.com> * skip c ops signature check as XPU compiler doesn't generate signature Signed-off-by: Feng Tian <feng.tian@intel.com> * fix issue found in test_non_cuda_equivalents.py The original code override the non_cuda_equivalents import lib. The change uses a merged module to avoid that. Signed-off-by: Feng Tian <feng.tian@intel.com> * fix bugs introduced by merging dev to this PR 1. adopt the name change from `discover_gpu_kv_format` to `normalize_kv_and_discover_format` in (LMCache#3162) 2. adopt the KVLayerGroupsManager change in (LMCache#3078) Signed-off-by: Feng Tian <feng.tian@intel.com> * Not install cuda or hip related libs when build sycl version Signed-off-by: Feng Tian <feng.tian@intel.com> * merge the Intel XPU related contents to the latest docs Signed-off-by: Feng Tian <feng.tian@intel.com> * sync GPUKVFormat enum to latest one Signed-off-by: Feng Tian <feng.tian@intel.com> * Remove SGLang related mem SYCL kernels Signed-off-by: Feng Tian <feng.tian@intel.com> * Fix cpu backend importing c_ops issue Signed-off-by: Feng Tian <feng.tian@intel.com> * fix pre-commit check error on test_xpu_connector_benchmark.py Signed-off-by: Feng Tian <feng.tian@intel.com> --------- Signed-off-by: Feng Tian <feng.tian@intel.com> Signed-off-by: Tian, Feng <feng.tian@intel.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: deng451e <838677410@qq.com>
…e#3293) Signed-off-by: royyhuang <roy.y.huang@gmail.com>
…e#3338) Signed-off-by: idellzheng <idellzheng@tencent.com>
* Refactor: abstract discover_subclasses util method Signed-off-by: baoloongmao <baoloongmao@tencent.com> * Fix Signed-off-by: baoloongmao <baoloongmao@tencent.com> * Fix Signed-off-by: baoloongmao <baoloongmao@tencent.com> * Fix Signed-off-by: baoloongmao <baoloongmao@tencent.com> * Fix Signed-off-by: baoloongmao <baoloongmao@tencent.com> * Fix Signed-off-by: baoloongmao <baoloongmao@tencent.com> * Raise exception on_import_error Signed-off-by: baoloongmao <baoloongmao@tencent.com> * Move test_cache CLI module under bench sub-package TestCacheCommand is the impl behind 'bench kvcache', not a top-level CLI verb. Putting test_cache.py under lmcache/cli/commands/ caused discover_subclasses to surface it as a stand-alone command. Move the module (and its tests) into lmcache/cli/commands/bench/ so it sits next to the engine_bench peer and is naturally invisible to the top-level scan. Signed-off-by: baoloongmao <baoloongmao@tencent.com> --------- Signed-off-by: baoloongmao <baoloongmao@tencent.com>
Signed-off-by: deng451e <838677410@qq.com>
* feat(mp): CPU Context by pickle Signed-off-by: Tony Lin <tony.lin@intel.com> * renaming bounce keyword to cpu context Signed-off-by: Tony Lin <tony.lin@intel.com> * fix unit test failures Signed-off-by: Tony Lin <tony.lin@intel.com> * address bot review comment Signed-off-by: Tony Lin <tony.lin@intel.com> * refactor: standardize cpu context naming conventions Signed-off-by: Tony Lin <tony.lin@intel.com> * small fix on error handling Signed-off-by: Tony Lin <tony.lin@intel.com> * refactor: polymorphic TransferContext for MP adapter transport layer Replace scattered if/else CPU/CUDA branches in vllm_multi_process_adapter with a TransferContext abstraction (ABC + CudaTransferContext + CPUTransferContext). - Add transfer_context.py with unified register/store/retrieve/poll interface - Device-type dispatch centralized in create_transfer_context() factory - Adapter delegates all transport logic via polymorphism, no branching - Future transports (e.g. SHM) only need a new subclass, zero adapter changes Signed-off-by: Tony Lin <tony.lin@intel.com> * restore unnecessary changes Signed-off-by: Tony Lin <tony.lin@intel.com> * Revert CPU registration payload from pickle bytes to scalar fields Signed-off-by: Tony Lin <tony.lin@intel.com> * update design doc Signed-off-by: Tony Lin <tony.lin@intel.com> * rebase dsv4: propagate vllm_logical_block_size through TransferContext.register() to restore DeepSeek V4 compress_ratio Signed-off-by: Tony Lin <tony.lin@intel.com> * rename to more general names: non_gpu_context & non_cuda_transfer_context Signed-off-by: Tony Lin <tony.lin@intel.com> * add todo note for deepseek v4 on non-cuda path Signed-off-by: Tony Lin <tony.lin@intel.com> * Consolidate MPCacheEngine context state into unified registry Signed-off-by: Tony Lin <tony.lin@intel.com> * use dataclass for payload Signed-off-by: Tony Lin <tony.lin@intel.com> * Auto-disable l1-use-lazy on non-CUDA backends Signed-off-by: Tony Lin <tony.lin@intel.com> * Lift layout hints to the caller layer to avoid redundant computation in transfer-context registration Signed-off-by: Tony Lin <tony.lin@intel.com> * [refactor] reserve zero-copy buffer allocation interface to NonGpuContext for shm solution Signed-off-by: Tony Lin <tony.lin@intel.com> * fix: update test to use new unified protocol methods Signed-off-by: Tony Lin <tony.lin@intel.com> * refactor: rename transfer context classes to handle/data semantics Signed-off-by: Tony Lin <tony.lin@intel.com> * update test Signed-off-by: Tony Lin <tony.lin@intel.com> * update docs Signed-off-by: Tony Lin <tony.lin@intel.com> * rename test file Signed-off-by: Tony Lin <tony.lin@intel.com> --------- Signed-off-by: Tony Lin <tony.lin@intel.com>
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: deng451e <57919305+deng451e@users.noreply.github.com>
Signed-off-by: deng451e <838677410@qq.com>
…3354) rename NO_CUDA_EXT Signed-off-by: deng451e <838677410@qq.com>
… server` boots (fixes LMCache#3353) (LMCache#3355) fix(docker): install CLI requirements in standalone image so `lmcache server` boots `lmcache/standalone` (v0.4.5, latest, nightly) crashes on `lmcache server` with: ModuleNotFoundError: No module named 'openai' Root cause: `setup.py` only puts `requirements/common.txt` and `requirements/cuda{12,13}_core.txt` into `install_requires`. `requirements/cli.txt` -- which contains `openai` (used by the `bench` engine client) -- is not installed. `lmcache server` is registered via `lmcache.cli.main:main`, which does `from lmcache.cli.commands import ALL_COMMANDS`. That triggers `lmcache.cli.commands._discover_commands()`, which uses `discover_subclasses(...)` to **eagerly importlib.import_module** every submodule under `cli/commands/`. That includes `bench/__init__.py`, whose top-level imports chain into `engine_bench/request_sender.py:11 -> from openai import AsyncOpenAI`, and the standalone image has no `openai`. `cli/commands/__init__.py:18-30` deliberately re-raises import errors ("a broken CLI command module should fail loudly rather than silently disappear from the CLI"), so the discovery cannot be softened on the slim install path -- the install path itself has to be fixed. The other Docker images (`Dockerfile`, `Dockerfile.rocm`, `Dockerfile.lightweight`, `Dockerfile.rocm-lightweight`) all install `vllm[runai,tensorizer,...]` or are `FROM vllm/vllm-openai*`, and `openai` arrives transitively through vLLM there. Only the standalone image is vLLM-free and hits this crash. Minimal fix: install `requirements/cli.txt` alongside the existing `ray nvidia-ml-py` step in the base stage, using a `--mount=type=bind` so the file does not persist into the image (matching the `build.txt` style used later by the lmcache-build stage). Out of scope (intentional, can be a follow-up): the deeper question of whether `cli.txt` should be in `install_requires`, an `extras_require["cli"]` block in `setup.py`, or whether the fail-loudly contract in `cli/commands/__init__.py` should be loosened, is an architectural call I'd defer to maintainers. Either of those would also fix LMCache#3353 (and would fix the same crash for users who `pip install lmcache` outside of Docker), but keeping this PR scoped to the reported failure mode -- the standalone Docker image -- keeps the review surface tight and unblocks the broken images today. Fixes LMCache#3353 Signed-off-by: abinggo <107740309+abinggo@users.noreply.github.com>
new cpu only ci Signed-off-by: deng451e <838677410@qq.com>
observability: add l2_usage_bytes gauge + dashboard panel Register an observable gauge that reports each L2 adapter's current usage via adapter.get_usage(), tagged by l2_name, plus a Grafana panel to visualize it. Lets operators query how much each L2 tier currently holds. Signed-off-by: Samuel Shen <slshen@tensormesh.ai>
* ci: add macOS compatibility check Bring over the macOS CI workflow and smoke test from LMCache#3148. Signed-off-by: baoloongmao <baoloongmao@tencent.com> * Address comment Signed-off-by: baoloongmao <baoloongmao@tencent.com> * introduce c++ event_notifier abstraction Signed-off-by: baoloongmao <baoloongmao@tencent.com> * Address comment from liqian Signed-off-by: baoloongmao <baoloongmao@tencent.com> * ci(macos): use NO_GPU_EXT=1 to align with build_cpu_artifacts.yml NO_CUDA_EXT was deprecated by LMCache#3357 in favor of two distinct knobs: - NO_NATIVE_EXT=1: skip all C++ extensions (pure-Python build) - NO_GPU_EXT=1: build common C++ ext, skip CUDA/ROCm/SYCL Switch the macOS workflow to NO_GPU_EXT=1 so we actually compile native_storage_ops / lmcache_redis / lmcache_fs on macOS and exercise the new PipeNotifier fallback in csrc/storage_backends/event_notifier.h. Also extend the smoke script with an import check that mirrors the verify step in build_cpu_artifacts.yml. Signed-off-by: baoloongmao <baoloongmao@tencent.com> --------- Signed-off-by: baoloongmao <baoloongmao@tencent.com>
…Cache#3366) Signed-off-by: baoloongmao <baoloongmao@tencent.com>
…ver (LMCache#3365) Signed-off-by: baoloongmao <baoloongmao@tencent.com>
…namo compilation (LMCache#3390) [CI][Hotfix] Add pytest to runtime deps to fix cupy lazy import crash Signed-off-by: ApostaC <yihua98@uchicago.edu>
) * docs: add docstrings to public helpers in lmcache/utils.py Signed-off-by: zhengchenyu <zhengchenyu16@163.com> * fix style Signed-off-by: zhengchenyu <zhengchenyu16@163.com> * trigger ci Signed-off-by: zhengchenyu <zhengchenyu16@163.com> --------- Signed-off-by: zhengchenyu <zhengchenyu16@163.com>
cleanup: remove duplicate time import Signed-off-by: Nachiket Torwekar <nachiket.torwekar@gmail.com>
…cks through C++ host callback (LMCache#3336) Signed-off-by: elliotz <elliot@character.ai>
LMCache#2999) * [BugFix] fix sha256_cbor + async_loading type mismatch in _hash_tokens When LMCACHE_PRE_CACHING_HASH_ALGORITHM=sha256_cbor is used together with LMCACHE_ENABLE_ASYNC_LOADING=True, the async lookup path fails because sha256_cbor (vLLM PR#23673) returns a 32-byte bytes digest. This digest is placed into LookupRequestMsg.hashes which msgpack cannot encode as int, raising OverflowError. Fix: after calling hash_func in _hash_tokens, if the result is bytes, fold the first 8 bytes into a uint64 int. This keeps all downstream code (CacheEngineKey, msgpack serialisation) working with plain ints regardless of the configured hash algorithm. Both the store path and the async lookup path go through _hash_tokens, so the keys produced are always consistent. Fixes LMCache#2997 Signed-off-by: GentleCold <gentlecold@qq.com> * [BugFix] address review: fix NONE_HASH type and use public test interface - token_database.py: convert NONE_HASH from bytes to int immediately after it is set from vLLM. This ensures the entire prefix-hash chain uses a consistent int type as canon_prefix input to hash_func, not bytes on the first chunk and int on subsequent ones. - test_token_database.py: replace direct call to private _hash_tokens with the public process_tokens interface (make_key=False). Drop cbor2 import in favour of a simpler hashlib stub. Signed-off-by: GentleCold <gentlecold@qq.com> * [BugFix] normalize byte hash outputs in token database - token_database.py: add a module-level helper to normalize hash outputs from vLLM hash functions into LMCache's int chunk-hash representation. - Reuse the helper for both vLLM NONE_HASH initialization and token chunk hashing so sha256_cbor byte digests are folded consistently before entering the prefix hash chain. Signed-off-by: GentleCold <gentlecold@qq.com> * [CI] Fix isort in token database test Signed-off-by: GentleCold <gentlecold@qq.com> --------- Signed-off-by: GentleCold <gentlecold@qq.com>
…sferred bytes (LMCache#3363) Signed-off-by: ApostaC <yihua98@uchicago.edu>
Signed-off-by: idellzheng <idellzheng@tencent.com>
* docs: add recipes for phi, mistral, llama Signed-off-by: yeoshuheng <100367948+yeoshuheng@users.noreply.github.com> * docs: update tool calling link Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: sh <100367948+yeoshuheng@users.noreply.github.com> * docs: update jinja template fp Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: sh <100367948+yeoshuheng@users.noreply.github.com> * docs: update jinja template fp Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: sh <100367948+yeoshuheng@users.noreply.github.com> --------- Signed-off-by: yeoshuheng <100367948+yeoshuheng@users.noreply.github.com> Signed-off-by: sh <100367948+yeoshuheng@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
… thread (LMCache#3271) * Replace Condvar polling with eventfd+epoll for io_uring worker Signed-off-by: zhengfeihe <hezhengfei1999@gmail.com> * Fix rust format check Signed-off-by: zhengfeihe <hezhengfei1999@gmail.com> * Add RAII guard for file descriptor in rust raw block Signed-off-by: zhengfeihe <hezhengfei1999@gmail.com> --------- Signed-off-by: zhengfeihe <hezhengfei1999@gmail.com>
* nixl_storage: naive support for files + dynamic
static is not actually very usable with files: we bump into the OS
limits on the open files very quickly, limiting the size of the cache.
With tons of VRAM, having G3 of comparable size is not helping much, we
really need it to be much bigger.
This commit adds support for files in dynamic mode of nixl storage.
There are some naive things done:
1. No support for sharing the cache storage. We assume the worker has
exclusive access to the files and it is always safe to overwrite
existing files. Note that with TP!=1 we will have several workers
with the same target directory, that's why it's important to have a
worker id as part of the key, so they don't affect each other.
2. No eviction support. As before, dynamic mode has no evicting support.
This kinda made sense when it was only working with OBJ storages, but
now that is something to be aware of. It's not hard to add eviction
though.
3. Flat directory with all the cache files. There are going to be a lot
of them, especially provided we don't have eviction. Most filesystems
don't optimize for the case of a directory with millions of files,
and we constantly open/close files there, that might be a source
of additional latency.
- There is an existing PR to support sharding the directory, we can
merge it as a band aid.
- As a more advanced solution, we can do multi-layer subdirectory
structure, like gds backend does.
4. We switched directly from having _all_ files open at once, to
open/close _every_ single file on access. We might explore having an
LRU cache of open files instead.
Signed-off-by: Ilya Yanok <iyanok@nvidia.com>
* Fix formatting
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
* Fix PR Comments
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
* Extract _build_descs
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
* Fix test
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
* Refactor to increase readability
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
* Fix another leak issue
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
* Fix release order
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
* Treat file not found as a miss
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
* Add docstring about file create mode 0o644
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
* Unlink files on failure
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
* Add missing docs
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
* Fix pre-commit issues
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
* Fix mypy error
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
* Fix test
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
* Make tests pass
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
* Fix formatting
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
* Fix nixl tests and stop ignoring them in CI
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
* Fix path to support GDS
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
* Revert path creation in NixlDynamicStorageBackend
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
* Fix code quality issue
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
---------
Signed-off-by: Ilya Yanok <iyanok@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Co-authored-by: Ilya Yanok <iyanok@nvidia.com>
* memory: huge page support (C bits) Add 3 pairs of alloc/free functions for huge pages. The 4th option is for SHM and it's missing, since normal SHM use cases don't support huge pages. Old API is left untouched, except for the small refactor: common mmap code is factored out. This doesn't affect the behavior. Signed-off-by: Ilya Yanok <iyanok@nvidia.com> * memory_management: use new functions for huge pages Change two allocators to support us_huge_pages flag. Signed-off-by: Ilya Yanok <iyanok@nvidia.com> * memory_management: log if allocating huge pages fail Signed-off-by: Ilya Yanok <iyanok@nvidia.com> * non_cuda_equivalents: change memory functions to match cuda Adding use_hugepages to alloc and size to free. There is no actual huge page support though. It should be pretty easy to add to non-shm cases (mmap and give torch a buffer). For shm we need to look into shared_memory module implementation. Signed-off-by: Ilya Yanok <iyanok@nvidia.com> * local_cpu: support for huge pages Signed-off-by: Ilya Yanok <iyanok@nvidia.com> * nixl_storage: add config option to alloc huge pages We want to be able to alloc huge pages for the NIXL buffer in DRAM. Signed-off-by: Ilya Yanok <iyanok@nvidia.com> * PR comments fixes, Code cleanup Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com> * Skip test if not enough free huge pages Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com> * Fix non-cuda fallbacks Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com> * Add docs Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com> * Remove unused func Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com> * Fix hugepage info to check for 2 MiB only Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com> * Fix pre-commit issues Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com> * Merge branch 'dev' into iyanok/huge-pages Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com> --------- Signed-off-by: Ilya Yanok <iyanok@nvidia.com> Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com> Co-authored-by: Guy Ealey Morag <gealeymorag@nvidia.com> Co-authored-by: Yihua Cheng <yihua98@uchicago.edu>
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
…ifferent metadata (LMCache#3147) Signed-off-by: cr7258 <chengzw258@163.com>
docs(mp): note non-CUDA auto-disable of --l1-use-lazy PR LMCache#3259 (feat(mp): Non-GPU Context by pickle) added a post-init guard in lmcache/v1/distributed/config.py:L1MemoryManagerConfig that auto-disables use_lazy on backends where lmcache.torch_dev has no cudart attribute, logging a warning ("LazyMemoryAllocator requires cudart which is not available on the current backend. Disabling l1-use-lazy."). The user-facing reference in docs/source/mp/configuration.rst still listed the --l1-use-lazy default as plain "True" without mentioning that non-CUDA backends silently downgrade to eager allocation. Extend the description so the documented behavior matches the actual code path. Doc-only change; no code is modified. Signed-off-by: ApostaC <yihua98@uchicago.edu>
… page-aligned for O_DIRECT (LMCache#3191) * [Fix] non_cuda_equivalents: page-align pinned allocations for O_DIRECT Signed-off-by: zhengfei.he <hezhengfei1999@gmail.com> Co-authored-by: Yihua Cheng <yihua98@uchicago.edu>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Introduces a shim layer
hipfile_shim.pyto wrap AMD'shipfilepackage behind thecufile-pythonAPI. This allows hipfile to masquerade ascufile-pythonand lets LMCache interact with both cufile and hipfile through the same interface.Special notes for your reviewers:
If applicable:
Tests passing: https://gist.github.com/riley-dixon/b7ff1902155073342c54ba8751ae2714
Note: This required adding the kwarg
gds_backend="hipfile"in theLMCacheEngineConfig.from_defaults(...)call intest_gds_backend.py.