Example Seldon Core Deployments using Helm with Istio — seldon-core documentation (original) (raw)

Prequisites

Setup Cluster and Ingress

Use the setup notebook to Setup Cluster with Istio Ingress. Instructions also online.

!kubectl create namespace seldon

!kubectl config set-context $(kubectl config current-context) --namespace=seldon

Context "kind-kind" modified.

Configure Istio

For this example we will create the default istio gateway for seldon which needs to be called seldon-gateway. You can supply your own gateway by adding to your SeldonDeployments resources the annotation seldon.io/istio-gateway with values the name of your istio gateway.

Create a gateway for our istio-ingress

%%writefile resources/seldon-gateway.yaml apiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: seldon-gateway namespace: istio-system spec: selector: istio: ingressgateway # use istio default controller servers:

Overwriting resources/seldon-gateway.yaml

!kubectl create -f resources/seldon-gateway.yaml -n istio-system

gateway.networking.istio.io/seldon-gateway created

Ensure the istio ingress gatewaty is port-forwarded to localhost:8004

ISTIO_GATEWAY = "localhost:8004" VERSION = !cat ../version.txt VERSION = VERSION[0] VERSION

from IPython.core.magic import register_line_cell_magic

@register_line_cell_magic def writetemplate(line, cell): with open(line, "w") as f: f.write(cell.format(**globals()))

Start Seldon Core

Use the setup notebook to Install Seldon Core with Istio Ingress. Instructions also online.

Serve Single Model

!helm install mymodel ../helm-charts/seldon-single-model --set model.image=seldonio/mock_classifier:$VERSION

NAME: mymodel LAST DEPLOYED: Wed Mar 10 16:37:01 2021 NAMESPACE: seldon STATUS: deployed REVISION: 1 TEST SUITE: None

!helm template mymodel ../helm-charts/seldon-single-model --set model.image=seldonio/mock_classifier:$VERSION | pygmentize -l json

Source: seldon-single-model/templates/seldondeployment.json

{ "kind": "SeldonDeployment", "apiVersion": "machinelearning.seldon.io/v1", "metadata": { "name": "mymodel", "namespace": "seldon", "labels": {} }, "spec": { "name": "mymodel", "protocol": "seldon", "annotations": {}, "predictors": [ { "name": "default", "graph": { "name": "model", "type": "MODEL", }, "componentSpecs": [ { "spec": { "containers": [ { "name": "model", "image": "seldonio/mock_classifier:1.7.0-dev", "env": [ { "name": "LOG_LEVEL", "value": "INFO" }, ], "resources": {"requests":{"memory":"1Mi"}}, } ] }, } ], "replicas": 1 } ] } }

!kubectl rollout status deploy/mymodel-default-0-model

Waiting for deployment "mymodel-default-0-model" rollout to finish: 0 of 1 updated replicas are available... deployment "mymodel-default-0-model" successfully rolled out

Get predictions

from seldon_core.seldon_client import SeldonClient

sc = SeldonClient( deployment_name="mymodel", namespace="seldon", gateway_endpoint=ISTIO_GATEWAY )

REST Request

r = sc.predict(gateway="istio", transport="rest") assert r.success == True print(r)

Success:True message: Request: meta { } data { tensor { shape: 1 shape: 1 values: 0.721679221744617 } }

Response: {'data': {'names': ['proba'], 'tensor': {'shape': [1, 1], 'values': [0.1002015221659356]}}, 'meta': {'requestPath': {'model': 'seldonio/mock_classifier:1.7.0-dev'}}}

gRPC Request

r = sc.predict(gateway="istio", transport="grpc") assert r.success == True print(r)

Success:True message: Request: {'meta': {}, 'data': {'tensor': {'shape': [1, 1], 'values': [0.17825624441824628]}}} Response: {'meta': {'requestPath': {'model': 'seldonio/mock_classifier:1.7.0-dev'}}, 'data': {'names': ['proba'], 'tensor': {'shape': [1, 1], 'values': [0.06074453279395597]}}}

release "mymodel" uninstalled

Host Restriction

In this example we will restriction request to those with the Host header “seldon.io”

%%writetemplate resources/model_seldon.yaml apiVersion: machinelearning.seldon.io/v1 kind: SeldonDeployment metadata: name: example-seldon annotations: "seldon.io/istio-host": "seldon.io" spec: protocol: seldon predictors:

!kubectl apply -f resources/model_seldon.yaml

seldondeployment.machinelearning.seldon.io/example-seldon created

!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=example-seldon -o jsonpath='{.items[0].metadata.name}')

Waiting for deployment "example-seldon-model-0-classifier" rollout to finish: 0 of 1 updated replicas are available... deployment "example-seldon-model-0-classifier" successfully rolled out

for i in range(60): state = !kubectl get sdep example-seldon -o jsonpath='{.status.state}' state = state[0] print(state) if state == "Available": break time.sleep(1) assert state == "Available"

X=!curl -s -d '{"data": {"ndarray":[[1.0, 2.0, 5.0]]}}'
-X POST http://localhost:8003/seldon/seldon/example-seldon/api/v1.0/predictions
-H "Content-Type: application/json"
assert X == []

import json X=!curl -s -d '{"data": {"ndarray":[[1.0, 2.0, 5.0]]}}'
-X POST http://localhost:8003/seldon/seldon/example-seldon/api/v1.0/predictions
-H "Content-Type: application/json"
-H "Host: seldon.io" d=json.loads(X[0]) print(d) assert(d["data"]["ndarray"][0][0] > 0.4)

{'data': {'names': ['proba'], 'ndarray': [[0.43782349911420193]]}, 'meta': {'requestPath': {'classifier': 'seldonio/mock_classifier:1.9.0-dev'}}}

!kubectl delete -f resources/model_seldon.yaml

seldondeployment.machinelearning.seldon.io "example-seldon" deleted