mlops-kubernetes

This repo is an experiment into creating a fully self-contained, local kubernetes deployment of a MLops cycle. It showcases various ML tools and, more importantly, how they are managed. Run this end to end to get a glimpse into the true complexity of setting up a MLOps platform.

🚀 Features 🚀

1. Setting up a local k8s cluster via minikube
2. Deploying postgres DB as a datastore via terraform
3. Metaflow used job orchestration
4. Feast for feature store implementation
5. Mlflow for model registry
6. Seldon for model serving

See a high level overview here:

Note that Kafka, Prometheus, Grafana are not shown (yet) in this example.

🔨 Versions 🔨

• Minikube for k8s cluster setup

  • minikube v1.31.2 on Ubuntu 22.04
  • Kubernetes v1.27.4 on Docker 24.0.4
  • Helm version 3.10

• Terraform for IaC

  • 1.6.3

• K9s for ease of k8s management

  • 0.27.4

💻 Commands 💻

1. Launching local minikube cluster:

Start docker if necessary:

sudo dockerd

Then start minikube:

minikube start

2. Launching terraform

Make sure to set kube config variable first:

export KUBE_CONFIG_PATH=~/.kube/config

Now you can run terraform :

terraform init
terraform apply

This will install everything into the cluster via the provided terraform files. Note this will take some time to startup.

3. Building / testing out a job

Because this is built with local python code, begin by making sure the local docker image can be built and accessible by minikube:

eval $(minikube docker-env)

Note Currently the values for postgres DB are hardcoded, as they need to be accessible to pods within the cluster. Thus, all references to the host need to be replaced first. These are hardcoded in also to make it easier to understand the steps, whereas in practice they should be encrypted and pulled as secrets.

E.g:

• Go to the feature_store.yaml file within app/feast
• Find the current set host
• Find the IP of postgressql within the cluster
• Replace all instances with the new IP

Once that is done, do the same with the password. The password can be found via:

kubectl get secret --namespace default psql-postgresql -o jsonpath="{.data.postgres-password}" | base64 -d

With an example value: 1ki6EsXo4s.

Replace all instances within the code with the new value.

Then build the docker image:

docker build -t pipeline .

Test out the dataflow pipeline:

kubectl apply -f metaflow_dataflow_pod.yaml

Within the kubernetes logs, you should see that the job executes successfully. Next, try the modelflow job:

kubectl apply -f metaflow_modelflow_pod.yaml

This job writes a model to a fake S3 bucket hosted by minio.

4. Setup fake minio bucket for model deployment

Normally seldon accesses s3/gcs buckets with granted permission using secrets. However, since this repo example is based purely on a local, self-contained cluster, the rclone step needs to be configured specifically to read from our internal minio bucket.

Unfortunately this involves a small hack. We need to shell into the seldon modelserver pod and set the rclone config:

1. Edit the mlserver serverconfig to run as root (otherwise the config won't save)
2. restart mlserver statefulset deployment for changes to apply
3. Go onto the rclone container and update the config to allow for this new way of syncing:

Update seldon's rclone to pull from our fake minio bucket:

echo "[minio]
type = s3
provider = Minio
access_key_id = test-access-key
secret_access_key = test-secret-key
endpoint = http://10.244.0.21:9000" > /rclone/rclone.conf

5. Deploy model artifacts to cluster!

Port-forward the service account to be discoverable locally:

kubectl port-forward svc/seldon-mesh -n seldon-test 8080:80

Deploy the model to seldon via a kubectl command:

kubectl apply -f model.yaml

Now you can make a sample request to a model!

curl http://localhost:8080/v2/models/seldon-model/infer \
   -H "Content-Type: application/json" \
   -d '{"inputs": [{"name": "predict", "shape": [1, 4], "datatype": "FP32", "data": [[1, 2, 3, 4]]}]}'

Suppose you wanted to deploy a challenger model after a retraining; you can deploy it now via an experiment.

kubectl apply -f experiment.yaml

This example now shows how you can obtain an A/B split:

curl http://localhost:8080/v2/models/experiment-example/infer \
   -H "Content-Type: application/json" \
   -H "seldon-model: experiment-example.experiment" \
   -d '{"inputs": [{"name": "predict", "shape": [1, 4], "datatype": "FP32", "data": [[1, 2, 3, 4]]}]}'

Now we can run a sample batch of requests using

python app/sample_requests.py

Note the output looks something like this:

Counter({'seldon-model-challenger_1': 40, 'seldon-model_1': 20}) example simulation of 60 requests

This shows that 40/60 requests went to the challenger model, 20/60 requests to the base model. We defined a 50/50 split, but obviously we need some more samples!

Video Demo

If the video doesn't play automaically, please see the example_workflow.webm file within the repo.

Manual installations

This is only necessary if testing this out manually. The following notes are only kept from the development cycle:

Installing postgres

Helpful tips for postgres:

psql -h 10.244.0.11 -U postgres -d postgres -p 5432  # for connecting to the pod containing postgres
DROP DATABASE mydb; # for deleting db

In order to view password for postgres:

kubectl get secret --namespace default psql-postgresql -o jsonpath="{.data.postgres-password}" | base64 -d

Other, helpful commands:

kubectl port-forward --namespace default svc/psql-postgresql 5432:5432 &
    PGPASSWORD="$POSTGRES_PASSWORD" psql --host 127.0.0.1 -U postgres -d postgres -p 5432

Testing Jobs

For testing purposes an on iterations:

docker build -t pipeline . && kubectl delete -f metaflow_modelflow_pod.yaml && kubectl apply -f metaflow_modelflow_pod.yaml

Setting Rclone

For setting up rclone on the mlserver. Requires going into the rclone container and updating config

rclone config

Once running rclone config, here are some settings to try out:

minio
5 / Amazon compliant
16 / Minio

test-access-key
test-secret-key
http://10.244.0.11:9000

Once you have the config, you can test to make sure rclone is working correctly via:

rclone copy --config=/rclone/rclone.conf minio:demo-test-bucket/ temp/