Building a Data Science Platform with Kubernetes | by Avinash Kanumuru | Jul, 2024

How Kubernetes — the back-end device — powers the information science workforce with end-to-end ML life-cycle from mannequin improvement to deployment

After I began in my new position as Supervisor of Knowledge Science, little did I find out about establishing an information science platform for the workforce. In all my earlier roles, I had labored on constructing fashions and to some extent deploying fashions (or at the very least supporting the workforce that was deploying fashions), however I by no means wanted to arrange one thing from scratch (infra, I imply). The information science workforce didn’t exist then.

So first of my goal was to arrange a platform, not only for the information science workforce in a silo, however that may be built-in with knowledge engineering and software program groups. That is once I was launched to Kubernetes (k8s) immediately. I had heard of it earlier however hadn’t labored past creating docker pictures and another person would deploy in some infra.

Now, why is Kubernetes required for the information science workforce? What are among the challenges confronted by knowledge science groups?

• A scalable laptop based mostly on requirement — as an information scientist we work on totally different issues day-after-day and every has totally different useful resource necessities. There isn’t a one-size-fits-all laptop. Even when it exists, it might probably’t be given to everybody on the information science workforce
• Model points — Python and package deal model points when working in a workforce or once we deploy to manufacturing
• Completely different applied sciences and platforms — some pre-processing and mannequin constructing require spark, and a few could be executed in pandas. So once more, there isn’t a one-size-fits-all in native laptop
• Sharing work inside the workforce — Sharing and monitoring of mannequin outcomes executed in an Excel spreadsheet and circulated after every iteration
• And most significantly, Manufacturing deployment — how do I get the completed mannequin to manufacturing? Fashions don’t get to manufacturing for real-time use instances, as we as knowledge scientists aren’t conscious of constructing API/system round a mannequin. Finally, we find yourself operating the mannequin rating in batch

I’ve explored options, together with Cloud Platform options (AWS SageMaker, GCP AI Platform, Azure Machine Studying), however our principal issue is price and subsequent is cloud-agnostic. If price shouldn’t be an element, then one can use the above-mentioned cloud platform providers.

We recognized that Kubernetes is a perfect platform that satisfies most of those necessities — to scale and serve containerized pictures. Additionally this fashion, we’re cloud-agnostic. If now we have to maneuver to a unique vendor, we simply carry and shift the whole lot with minimal modifications.

Many instruments present full/related options like KubeFlow, Weights & Biases, Kedro, …, however I ended up deploying the under 3 providers as the primary model of the information science platform. Although these don’t present the entire MLOps framework, this will get us began to construct the information science platform and workforce.

1. JupyterHub — Containerized person environments for creating fashions in interactive Jupyter Notebooks
2. MLflow — Experiment monitoring and storing mannequin artifacts
3. Seldon Core — Simplified strategy to deploy fashions in Kubernetes

With these 3 providers, I get my workforce to construct fashions together with massive knowledge processing in JupyterHub, monitor totally different fine-tuned parameters, and metrics, and retailer artifacts utilizing MLflow and serve the mannequin for manufacturing utilizing Seldon-Core.

JupyterHub

Deploying this was the trickiest of all. JupyterHub in a standalone setup is straightforward in comparison with Kubernetes set up. However many of the required configuration was out there right here —

Since we need to use Spark for a few of our knowledge processing, we created 2 docker pictures —

1. Fundamental Pocket book — prolonged from jupyter/minimal-notebook:python-3.9
2. Spark Pocket book — prolonged from above with extra spark setup.

Code for these pocket book docker pictures and helm values for putting in JupyterHub utilizing these docker pictures can be found here.

There are a whole lot of tweaks executed to allow Google Oauth, beginning Pocket book as a root person, however operating them as a person person, retrieving the username, user-level permissions, persistent quantity claims, and repair accounts, … which took me days to get it working, particularly with the Auth. However this code within the repo, can provide you a skeleton to get began.

MLflow

Establishing MLFlow was straightforward.

MLflow provides mannequin monitoring, mannequin registry, and mannequin serving capabilities. However for mannequin serving, we use the subsequent device (Seldon-Core).

Construct a Docker picture with the required Python packages.

FROM python:3.11-slim
RUN pip set up mlflow==2.0.1 boto3==1.26.12 awscli==1.27.22 psycopg2-binary==2.9.5
EXPOSE 5000

As soon as the docker picture is created and pushed to the container registry of your selection, we create a deployment and repair file for Kubernetes (much like some other docker picture deployment). A snippet of the deployment yaml is given under.

containers:
- picture: avinashknmr/mlflow:2.0.1
imagePullPolicy: IfNotPresent
identify: mlflow-server
command: ["mlflow", "server"]
args:
- --host=0.0.0.0
- --port=5000
- --artifacts-destination=$(MLFLOW_ARTIFACTS_LOCATION)
- --backend-store-uri=postgresql+psycopg2://$(MLFLOW_DB_USER):$(MLFLOW_DB_PWD)@$(MLFLOW_DB_HOST):$(MLFLOW_DB_PORT)/$(MLFLOW_DB_NAME)
- --workers=2

There are 2 principal configurations right here that took time for me to grasp and configure —

1. artifact’s location
2. backend retailer

The artifact location can be a blob storage the place your mannequin file can be saved and can be utilized for model-serving functions. However in our case, that is AWS S3 the place all fashions are saved, and is a mannequin registry for us. There are a few different choices to retailer the mannequin domestically within the server, however at any time when the pod restarts the information is completed, and PersistentVolume is accessible solely through the server. By utilizing Cloud Storage, we will combine with different providers — for instance, Seldon-Core can decide from this location to serve the mannequin. The backend retailer shops all metadata required to run the appliance together with mannequin monitoring — parameters and metrics of every experiment/run.

Seldon-Core

The second most trickiest of the three is Seldon-Core.

Seldon-Core is sort of a wrapper to your mannequin that may package deal, deploy, and monitor ML fashions. This removes the dependency on ML engineers to make the deployment pipelines.

We did the set up utilizing a Helm chart and Istio for ingress. There are 2 choices for ingress — Istio & Ambassador. I’m not stepping into establishing Istio, because the DevOps workforce did this setup. Seldon is put in with the under Helm and Kubectl instructions.

kubectl create namespace seldon-system
kubectl label namespace seldon-system istio-injection=enabled
helm repo add seldonio https://storage.googleapis.com/seldon-charts
helm repo replace
helm set up seldon-core seldon-core-operator 
--repo https://storage.googleapis.com/seldon-charts 
--set usageMetrics.enabled=true 
--set istio.enabled=true 
--set istio.gateway=seldon-system/seldon-gateway 
--namespace seldon-system

However assuming you might have Istio set, under is the Yaml to arrange Gateway and VirtualService for our Seldon.

apiVersion: networking.istio.io/v1alpha3
variety: Gateway
metadata:
identify: seldon-gateway
namespace: seldon-system
spec:
selector:
istio: ingressgateway
servers:
- port:
quantity: 80
identify: http
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
variety: VirtualService
metadata:
identify: seldon-vs
namespace: seldon-system
spec:
hosts:
- "*"
gateways:
- seldon-gateway
http:
- match:
- uri:
prefix: /seldon
route:
- vacation spot:
host: seldon-webhook-service.seldon-system.svc.cluster.native
port:
quantity: 8000

Under is a pattern k8s deployment file to serve the iris mannequin from GCS. If utilizing scikit-learn package deal for mannequin improvement, the mannequin ought to be exported utilizing joblib and named as mannequin.joblib .

apiVersion: machinelearning.seldon.io/v1
variety: SeldonDeployment
metadata:
identify: iris-model
namespace: prod-data-science
spec:
identify: iris
predictors:
- graph:
implementation: SKLEARN_SERVER
modelUri: gs://seldon-models/v1.16.0-dev/sklearn/iris
identify: classifier
identify: default
replicas: 1

On this instance, we use SKLEARN_SERVER, however it has integrations for MLFLOW_SERVER, and TF_SERVER for MLflow and TensorFlow respectively.

Seldon-Core not solely helps REST API but additionally gRPC for seamless server-server calls.

Conclusion

These instruments are open supply and deployable in Kubernetes, so they’re cost-effective for small groups and in addition cloud-agnostic. They cowl most challenges of an information science workforce like a centralized Jupyter Pocket book for collaboration with out model points and serving fashions with out devoted ML engineers.

JupyterHub and Seldon-Core leverage the Kubernetes capabilities. JupyterHub spins up a pod for customers after they log in and kills it when idle. Seldon-Core wraps the mannequin and serves it as an API in a couple of minutes. MLflow is the one standalone set up that connects mannequin improvement and mannequin deployment. MLflow acts as a mannequin registry to trace fashions and retailer artifacts for later use.