Skip to content

Design Doc

Jump to bottom
Phil Winder edited this page Jun 24, 2021 · 6 revisions

TL;DR; Easy to build, universal ML containers, that are fast and production ready.

Value Proposition

Incredibly Simple to Use

Problem: Building deployable models is more difficult than it needs to be. There’s a lot of unnecessary boilerplate code. Data scientists don’t care about the supporting code.

If you’ve got a model to hand, it’s really easy to create a deployable container.

Create deployable artifacts with one command. Inspired by Seldon’s use of s2i. Use the CLI to securely build model. Supports all common libraries. For example:

  • Sklearn
  • Pytorch
  • Tensorflow
  • XGBoost
  • ...

Choose between building locally, on your production cluster, or use an external service. We leverage Kaniko to securely build containers. This means you can build your model container locally, by passing the model artifact, remotely on your production cluster, or even using an externally hosted service.

Deploy your model to ML SaaS platforms. Deploy your model directly to Modzy, Sagemaker, etc. (inspired by Serverless deploy)

Python SDK. Do all of the above from the CLI or the Python SDK.

Defacto ML Containers

Problem: Data Scientists don’t care about Docker. They don’t care about anything other than a model artifact. But they want something that can be deployed.

Our containers work with any orchestrator. KFServing promoted the idea of having pre-made containers that work with various trained models. We want to take this further and expose a unified interface that can be used with any serving platform. For example, this artifact can work with:

  • KFServing
  • Modzy
  • Sagemaker…
  • Seldon-Core
  • HTTP
  • etc.

Our containers are the fastest and work with all use cases. By creating our own containers, we will not only be able to own THE ML deployment part of the stack, we can add extra features like:

  • Faster processing (no (multiple) HTTP requests, no Python, no Flask, no unnecessary calls, etc.)
  • Batch workloads (only one-shot inference available at the moment)
  • Streaming workloads

Our containers are production ready and secure. We improve the security posture of the containers by unifying:

  • Container security best practices
  • Dependency analysis
  • Automated pipelines, etc.
  • ...

Requirements

Installation

  • Helm

DS Interfaces

  • Python SDK
  • CLI

Targets

  • modzy
  • Docker-registry
    • Needs devops user to configure registry credentials in a k8s secret
  • download-tar
    • Passes url of object storage back to user
    • Demo: docker load < downloaded.tar; docker run ...

Demos

Simple HTTP Demo

  1. Devops setup
  2. containerizer.publish(“model.foo”, sklearn, target=”docker-registry”) [default?] 3a. !kubectl apply -f ‘apiVersion: v1/deployment… yaml which references registry repo for the model image’ 3b. Or! kubectl run equivalent which is less typing

Key Interfaces

Model Interface

All models are expected to be packaged with MLflow. The MLFlow bundle is then submitted to the builder service, along with some metadata.

The benefit of this is that we can take advantage of all the work that currently exists that wrap a consistent interface around many different ML libraries. You can read more about the models it supports in the MLFlow documentation.

Serving API

Once a model package (and metadata) is sent to the builder service, the builder service creates a container. The container contains code that exposes a number of servers which are programmed to satisfy the APIs of any serving framework.

Initially it will support:

In the future it will support:

Selecting an API

When the user submits their model to the builder service, the metadata tells the builder which API to implement. The container will only implement one API at a time because:

  • You don't want to waste resources serving an API that isn't used
  • Hosting multiple unused APIs increases the attack surface
  • Hard to make servers work nicely together (e.g. GRPC and HTTP servers would have to be run in separate processes, which complicates the code)

Data Conversion

There is an impedance mismatch between what is expected by the MLFlow interface, and that provided by the interface inputs.

Therefore, the API implementations will convert the data from the incoming API type, to that expected by MLFlow.

Related Work

Artifact Output

The built containers produce a tar file. This is the default return type, and is returned to the user.

The container tar can optionally be pushed to docker hub and the Modzy model marketplace.

Clone this wiki locally