Skip to content

CONVERTER_API.md

Latest commit

 

History

History
419 lines (329 loc) · 13.5 KB

CONVERTER_API.md

File metadata and controls

419 lines (329 loc) · 13.5 KB
Raw

Converter Module - API Reference

Module: npu_proxy.models.converter

Quick conversion and validation of HuggingFace models to OpenVINO format with smart caching.

Conversion safety model

Conversion runs optimum-cli export openvino in a child process (subprocess.run() for convert_to_openvino() and subprocess.Popen() for get_conversion_progress()). Both paths use the same 1-hour timeout (CONVERSION_TIMEOUT_SECONDS = 3600).

The converter writes to a sibling temporary directory named .<output-name>.partial first. On success, it validates that the temporary directory contains non-empty openvino_model.xml and openvino_model.bin, then publishes the result by renaming the temporary directory to the requested output directory. On failure, timeout, missing files, or process errors, the partial directory is removed. This avoids leaving a partially-written model directory at the final output path.

Return values do not include a SHA-256 digest. Earlier documentation claimed a digest existed, but the current API returns only status/path/model/source fields and error messages.

Functions

is_openvino_model(path: Path) -> bool

Check if a directory contains a valid OpenVINO model.

Validates that a directory contains both required OpenVINO files as regular, non-empty files.

Parameters

  • path (Path | str): Directory path to validate
    • Can be pathlib.Path or string path
    • Non-existent paths return False
    • File paths return False (directory check fails)

Returns

  • bool: True if both openvino_model.xml and openvino_model.bin exist and are non-empty, False otherwise

Examples

from npu_proxy.models.converter import is_openvino_model

# Check a model directory
if is_openvino_model(r"C:\models\gpt2"):
    print("Ready for inference!")

# String paths work too
is_openvino_model(r"C:\models\gpt2")  # True or False

# Non-existent paths
is_openvino_model(r"C:\does\not\exist")  # False

convert_to_openvino(hf_repo: str, output_dir: Path, task: str = "text-generation-with-past", progress_callback: Callable[[str], None] | None = None) -> dict

Convert a HuggingFace model to OpenVINO format.

Uses optimum-cli export openvino to convert models. Validates the temporary output directory and returns status.

Parameters

  • hf_repo (str): HuggingFace model repository ID
    • Examples: "gpt2", "meta-llama/Llama-2-7b", "sentence-transformers/all-MiniLM-L6-v2"
  • output_dir (Path | str): Final directory where the converted model will be published
    • Parent directory is created if it doesn't exist
    • Conversion first writes to .<output-name>.partial next to this directory
    • Must be writable
  • task (str, optional): Task type for conversion (default: "text-generation-with-past")
    • Valid values: "text-generation-with-past", "text-generation", "feature-extraction"
    • "text-generation-with-past" is the current default for LLM models
    • "feature-extraction" is used for embedding models
  • progress_callback (Callable, optional): Function called with progress messages
    • Called with string messages during conversion
    • Useful for logging or UI updates

Returns

  • Success (dict): 
    {
    "status": "success",
    "path": "C:\\path\\to\\converted\\model",
    "model": "gpt2"
}
  • Error (dict): 
    {
    "error": "Error message describing what went wrong"
}

No success response includes a digest field.

Possible Errors

  • "optimum-cli not found..." - Install with pip install optimum-intel
  • "Invalid task type..." - Use only "text-generation-with-past", "text-generation", or "feature-extraction"
  • "Conversion failed: ..." - Model not found, network error, process error, etc.
  • "Conversion timed out..." - Took longer than 1 hour
  • "output files not found" - Conversion succeeded but required files were missing

Examples

from pathlib import Path
from npu_proxy.models.converter import convert_to_openvino

# Basic conversion using the default task: text-generation-with-past
result = convert_to_openvino("gpt2", Path(r"C:\models\gpt2"))
if "error" not in result:
    print(f"Converted to: {result['path']}")

# With progress callback
def log_progress(msg):
    print(f"[Conversion] {msg}")

result = convert_to_openvino(
    "meta-llama/Llama-2-7b",
    Path(r"C:\models\llama2"),
    progress_callback=log_progress,
)

# Feature extraction task
result = convert_to_openvino(
    "sentence-transformers/all-MiniLM-L6-v2",
    Path(r"C:\models\embedding"),
    task="feature-extraction",
)

# Using string path
result = convert_to_openvino("gpt2", r"C:\models\gpt2")

If you want the exported embedding model to be auto-detected by the running service, write it into that model's runtime cache path (for example ~/.cache/npu-proxy/models/embeddings/all-minilm-l6-v2 for sentence-transformers/all-MiniLM-L6-v2).

auto_download_and_convert(model_name: str, task: str = "text-generation-with-past", cache_dir: Path | None = None) -> dict

Download and convert a model if needed, with smart caching.

Checks cache first, skips conversion if already cached, converts if needed.

Parameters

  • model_name (str): Model name or HuggingFace repo ID
    • Ollama-style names: "tinyllama", "llama2"
    • HuggingFace repos: "gpt2", "meta-llama/Llama-2-7b"
    • Names are resolved via npu_proxy.models.mapper.resolve_model_repo()
  • task (str, optional): Task type (default: "text-generation-with-past")
    • Valid values: "text-generation-with-past", "text-generation", "feature-extraction"
  • cache_dir (Path | str, optional): Directory for cached models
    • Defaults to ~/.cache/npu-proxy/models/
    • Allows custom caching location

Returns

  • Success (dict): 
    {
    "status": "success",
    "path": "C:\\path\\to\\model",
    "model": "tinyllama-1.1b-chat-int4-ov",
    "source": "cache"  # or "converted"
}
  • Error (dict): 
    {
    "error": "Error message"
}

Key Features

  • Caching: If model is already in cache, returns immediately without conversion
  • Source tracking: Returns "source" to indicate if from cache or newly converted
  • Automatic resolution: Converts registered Ollama names to HuggingFace repos and runtime storage keys
  • Safe publish: New conversions use the same partial-directory-and-rename flow as convert_to_openvino()

Examples

from npu_proxy.models.converter import auto_download_and_convert

# Auto-convert with default cache
result = auto_download_and_convert("tinyllama")
if "error" not in result:
    print(f"Model path: {result['path']}")
    print(f"From: {result['source']}")  # "cache" or "converted"

# Custom cache directory
result = auto_download_and_convert(
    "tinyllama",
    cache_dir=r"C:\custom\cache",
)

# Feature extraction model
result = auto_download_and_convert(
    "sentence-transformers/all-MiniLM-L6-v2",
    task="feature-extraction",
)

# Caching demo
result1 = auto_download_and_convert("gpt2")  # Converts if not cached
result2 = auto_download_and_convert("gpt2")  # Uses cache when valid

get_conversion_progress(hf_repo: str, output_dir: Path, task: str = "text-generation-with-past") -> Generator[dict, None, None]

Yield progress updates during model conversion for streaming.

Generator that yields progress updates as the conversion child process runs.

Parameters

  • hf_repo (str): HuggingFace model repository ID
  • output_dir (Path | str): Final directory where the converted model will be published
  • task (str, optional): Task type (default: "text-generation-with-past")
    • Valid values: "text-generation-with-past", "text-generation", "feature-extraction"

Yields

Progress dictionaries with these fields:

  • "status" (str): Current status
    • "starting" - Conversion initialization
    • "running" - Subprocess started
    • "converting" - Captured conversion output
    • "success" - Conversion completed and was renamed into place
    • "error" - An error occurred
  • "message" (str): Status message or output line
  • "path" (str, optional): Path to converted model (only on success)

Examples

from pathlib import Path
from npu_proxy.models.converter import get_conversion_progress

# Stream conversion progress
for progress in get_conversion_progress("gpt2", Path(r"C:\models\gpt2")):
    print(f"{progress['status']}: {progress['message']}")
    if progress["status"] == "success":
        print(f"Converted to: {progress['path']}")

# For UI updates
for progress in get_conversion_progress("llama2", r"C:\models\llama2"):
    if progress["status"] == "error":
        display_error(progress["message"])
    else:
        update_progress_bar(progress["message"])

# Feature extraction conversion with progress
for p in get_conversion_progress(
    "sentence-transformers/all-MiniLM-L6-v2",
    r"C:\models\embedding",
    task="feature-extraction",
):
    if p["status"] == "converting":
        logger.info(p["message"])

Constants

DEFAULT_CONVERSION_DIR

Path.home() / ".cache" / "npu-proxy" / "models"

Default cache directory for converted models. Typically: ~/.cache/npu-proxy/models/

REQUIRED_OPENVINO_FILES

["openvino_model.xml", "openvino_model.bin"]

Files required for a valid OpenVINO model.

VALID_EXPORT_TASKS

("text-generation-with-past", "text-generation", "feature-extraction")

Task values accepted by conversion functions.

CONVERSION_TIMEOUT_SECONDS

3600

One-hour timeout used for conversion child processes.

Error Handling

Common Error Scenarios

Missing optimum-cli

result = convert_to_openvino("gpt2", r"C:\models\gpt2")
# Returns: {"error": "optimum-cli not found. Install it with: pip install optimum-intel"}

Solution: pip install optimum-intel

Invalid Task Type

result = convert_to_openvino("gpt2", r"C:\models\gpt2", task="invalid-task")
# Returns: {"error": "Invalid task type. Must be one of ['text-generation-with-past', 'text-generation', 'feature-extraction']"}

Solution: Use only "text-generation-with-past", "text-generation", or "feature-extraction"

Model Not Found

result = convert_to_openvino("nonexistent-model-xyz", r"C:\models\xyz")
# Returns: {"error": "Conversion failed: ..."}

Solution: Verify the model name/repo exists on HuggingFace

Timeout

result = convert_to_openvino("large-model", r"C:\models\large")
# If conversion takes >1 hour: {"error": "Conversion timed out after 1 hour"}

Solution: Models requiring >1 hour may need custom infrastructure

Unknown Model Name

result = auto_download_and_convert("unknown-xyz")
# Returns: {"error": "Model 'unknown-xyz' not found"}

Solution: Use HuggingFace repo ID or a registered Ollama-style name

Return Value Examples

Successful is_openvino_model

is_openvino_model(r"C:\models\gpt2")  # True

Successful convert_to_openvino

{
    "status": "success",
    "path": "C:\\Users\\you\\.cache\\npu-proxy\\models\\gpt2",
    "model": "gpt2"
}

Successful auto_download_and_convert (from cache)

{
    "status": "success",
    "path": "C:\\Users\\you\\.cache\\npu-proxy\\models\\tinyllama-1.1b-chat-int4-ov",
    "model": "tinyllama-1.1b-chat-int4-ov",
    "source": "cache"
}

Successful auto_download_and_convert (newly converted)

{
    "status": "success",
    "path": "C:\\Users\\you\\.cache\\npu-proxy\\models\\tinyllama-1.1b-chat-int4-ov",
    "model": "tinyllama-1.1b-chat-int4-ov",
    "source": "converted"
}

get_conversion_progress yields

{"status": "starting", "message": "Starting conversion of gpt2"}
{"status": "running", "message": "Running: optimum-cli export openvino ..."}
{"status": "converting", "message": "Exporting GPT2 model to OpenVINO format..."}
{"status": "success", "message": "Conversion complete: C:\\models\\gpt2", "path": "C:\\models\\gpt2"}

Integration with Existing Code

With npu_proxy.models.mapper

from npu_proxy.models.mapper import resolve_model_repo
from npu_proxy.models.converter import auto_download_and_convert

# auto_download_and_convert uses resolve_model_repo internally
result = auto_download_and_convert("tinyllama")
# Internally resolves "tinyllama" to the registered HuggingFace repo and runtime storage key

With Error Patterns

# Consistent error handling with rest of codebase
result = auto_download_and_convert("gpt2")
if "error" in result:
    logger.error(result["error"])
    return None
else:
    return result["path"]

Performance Notes

  • Caching: Second call to same model returns instantly when a valid cache exists
  • Timeout: 1-hour limit prevents unbounded conversions
  • Streaming: get_conversion_progress() yields captured output after the child process completes or times out
  • Validation: Models are validated before the temporary directory is renamed into place
  • No digest: Current converter return values do not include SHA-256 or other content digests

Dependencies

  • Required: Python 3.9+
  • For actual conversion: optimum-intel package providing optimum-cli
  • Internal: npu_proxy.models.mapper.resolve_model_repo()

See Also