[FEATURE] Prefetch a Model to cache without an InferenceService (eager/standalone download)

[Defilan]

Feature Description

Allow a Model to be prefetched (downloaded into a cache) on declaration, independent of any InferenceService. Today a Model with a remote source (https://…, hf://…) is only a declaration: nothing is fetched until an InferenceService serving it spins up a model-downloader init container. A Model with no serving workload sits Ready having downloaded nothing.

Problem Statement

As an operator, I want to declare a Model and have its artifact pulled to the node cache ahead of time, so that the first InferenceService serves immediately and I can stage/validate large GGUFs without standing up a serving pod.

Two concrete pain points:

• Surprise: kubectl apply a Model with a HuggingFace URL, see it go Ready, assume it's cached — but it isn't. The download only happens lazily at first serve.
• No first-class staging: prefetching a 20–30 GB GGUF today requires hand-rolling a download Job + PVC and switching the source to pvc://… (this is how every Strix/AMD model is staged). That's exactly the manual loader-Job dance the operator could own.

Proposed Solution

A prefetch/refresh switch on Model that makes the operator run a download Job (the same init-container logic, hoisted to a standalone Job) into the model cache when the Model is created/updated, and reflect progress in status.

Example YAML:

apiVersion: inference.llmkube.dev/v1alpha1
kind: Model
metadata:
  name: qwopus36-mtp-q6k
spec:
  format: gguf
  source: https://huggingface.co/.../Qwopus3.6-35B-A3B-Coder-MTP-Q6_K.gguf
  quantization: Q6_K
  prefetch: true            # operator pulls to cache now, no InferenceService required
  hardware:
    gpu: { enabled: true, vendor: amd, runtime: vulkan }
status:
  phase: Downloading        # -> Cached
  cache: { pvc: qwopus36-mtp-q6k-model-cache, progressBytes: 18253611008, totalBytes: 31300000000 }

Then status.phase: Cached is a real readiness signal, and an InferenceService referencing it serves with a cache hit.

Alternatives Considered

• Status quo (manual loader Job + pvc://): works but is undocumented tribal knowledge and easy to get wrong (PVC sizing, file layout, node placement, taint tolerations).
• Eager-by-default: make every remote Model prefetch automatically. Simpler API but pulls artifacts you may never serve; an opt-in prefetch field is safer.

Additional Context

• Download logic today lives in the InferenceService init container (internal/controller/model_storage.go, model-downloader), keyed off the serving pod — there is no Model-level download path.
• Related issues: [BUG] hostpath PVC provisioning blocked on tainted GPU nodes (breaks AMD/Vulkan model staging) #850 — on AMD/Vulkan nodes the serve-time download is broken anyway (the cache PVC can't provision on the devic.es/dri-render-tainted node), so prefetch + a properly-tolerated download Job would also be the natural place to fix AMD staging.
• Workaround in use: a standalone curlimages/curl Job (with the GPU-taint toleration) downloading into a pre-bound PVC, then a pvc:// source.

Priority

• Medium - Nice to have

Willingness to Contribute

• Yes, I can submit a PR
• Yes, I can help test

[Defilan]

Scope correction after verifying on the live fleet (2026-06-30):

The serve-time auto-download path works fine for URL/hf sources on the metal tier — it is not limited to local files or manual staging. Confirmed on the Macs:

Model	Source	Loader Job?
devstral-24b	https://huggingface.co/.../Devstral-Small-2-24B-...-Q6_K.gguf	none
gemma4-26b-q4	https://huggingface.co/.../gemma-4-26B-...-Q4_K_M.gguf	none
mistral-small-32-24b	https://huggingface.co/.../Mistral-Small-3.2-24B-...-Q6_K.gguf	none

Each is accelerator: metal, served from a remote URL, with no hand-rolled download Job and no pvc:// source. The metal-agent pulls the GGUF itself when the InferenceService spins up. The only metal models on local-path sources (ornith-35b-m5, qwopus-coder-mtp) are deliberate local stages, not a requirement.

So the "no first-class staging" pain in this issue is not a metal problem. It splits cleanly:

1. bare-Model prefetch (this issue, all tiers): still valid — a Model with a remote source and no InferenceService downloads nothing. prefetch: true would give a real Cached readiness signal and warm the first serve. This is the feature ask and it stands.
2. AMD/Vulkan serve-time download is genuinely broken ([BUG] hostpath PVC provisioning blocked on tainted GPU nodes (breaks AMD/Vulkan model staging) #850): the hostpath cache PVC cannot provision on the devic.es/dri-render-tainted Strix node, so even a URL-source InferenceService cannot pull there. That is why every Strix model uses a hand-rolled loader Job + pvc://. The manual-loader dance is an AMD-tier workaround for [BUG] hostpath PVC provisioning blocked on tainted GPU nodes (breaks AMD/Vulkan model staging) #850, not the general staging story.

Net: the metal tier already does lazy URL auto-download correctly; #904 remains scoped to eager/standalone prefetch, and the manual loader Job is AMD-only until #850 is fixed.