ML Pipelines
A machine learning system is shaped by more than its code. Its behaviour comes from the data it was trained on and the weights that training produced, and both keep changing after the first release. Shipping a model is therefore not one event but a loop: prepare data, check it, train, evaluate, decide whether the result is good enough, release it carefully, watch it in production, and start again when the world moves. ML Pipelines gives that loop a single home. You describe how your model is trained and evaluated and what "good enough" means; Tantilium Cloud Runner (TCR) runs the training on Argo Workflows, on GPUs when you ask for them, records every model it produces as an immutable version with its full lineage, holds back the versions that don't meet your bar, and releases the ones that do through a canary that rolls itself back if production disagrees with your evaluation.
Platform administrators and tenants create ML pipelines from the same interface, with the same wizard, as data pipelines, and every run appears live in Workflows.
The shape of an ML pipeline
An ML pipeline has two halves. The first half is yours: the compute stages that prepare and validate a dataset, train or fine-tune a model, and evaluate it. They are ordinary pipeline steps running your images, connected by dependencies and handing data to each other as artifacts, exactly as in a data pipeline. The second half is TCR's: the control stages that register the model, apply the promotion gate, deploy to staging, wait for approval, run the canary, analyse it and promote the winner. You configure them, but you never write or run them.
ML Pipeline — from data to production
You own the compute stages; TCR runs the privileged control stages that register, gate and release the model.
That split is deliberate. Registering a model, judging it and putting it in front of users are privileged operations: they write to the model registry and change what production serves. In TCR they are carried out by the platform itself, which runs them as steps of the same workflow so that you watch them in the same graph as your training. Your training code never holds the rights to overwrite a registered model or to touch a serving endpoint, and a pipeline cannot talk its way past its own gate.
Training
Each compute stage is a container step, and the wizard's training example shows the whole flow on a CPU: a step that prepares a dataset, one that validates it, one that trains a scikit-learn model and one that evaluates it and writes its scorecard. Real pipelines swap in their own images, and steps that need more than a CPU ask for it.
A step that asks for GPUs is scheduled onto the platform's GPU node pool. That pool is made of spot instances and scales from zero: when a GPU step is waiting, the cluster autoscaler adds a GPU node; when the pool has been idle for a few minutes, it removes it again, so GPUs cost nothing while no one is training. The NVIDIA GPU Operator prepares each node with its driver and device plugin as it joins, and every tenant has a GPU quota of its own.
Spot capacity can be reclaimed mid-run, so GPU steps are retried once by default, and any step can be marked to checkpoint. A checkpointing step receives KUBEOPERA_CHECKPOINT_URI, a location in your tenant's storage that stays the same across that step's retries within a run, together with credentials that can read and write only your tenant's training storage. A training loop that saves its state there and resumes from it on start loses minutes rather than hours when a node disappears. Retries stay deliberately conservative, because a GPU step that keeps failing should fail visibly rather than burn hours of GPU time.
When one machine isn't enough, a step can be distributed training instead: a Kubeflow PyTorchJob with a master and as many workers as you choose, each with its own GPUs, all running the same image so that torchrun can coordinate them. The step waits for the job to finish just as it would wait for a container. Distributed replicas read their data from and save their model to your tenant's training storage, KUBEOPERA_ML_STORAGE_URI, and a following container step exports the finished model.
Hyperparameter search uses the fan-out every pipeline step supports. A training step that fans out over a list of configurations runs one trial per configuration in parallel, each logging to the same experiment, and an evaluation step picks the best trial before the model moves on.
Experiment tracking
Every ML pipeline can log to MLflow. TCR runs one MLflow server for the whole platform with authentication switched on: each tenant has its own MLflow account, and each pipeline its own experiment, named after the tenant and the pipeline, that only that tenant's account can read or change. Steps receive MLFLOW_TRACKING_URI, MLFLOW_EXPERIMENT_NAME and their tenant's login automatically, so calling mlflow.start_run() is all a training script needs to do. Artifacts logged to MLflow travel through the tracking server itself, so steps never hold storage credentials for it.
The pipeline page shows the experiment's runs, with the metrics your gate cares about as columns, their parameters, and a chart of any metric's history for the run you select, so you can compare trials without leaving TCR.
The model registry
When the compute stages finish, TCR registers the result. It reads the step output you named as the trained model, which can be a single file or a directory, and copies it into the registry as the next version of the model: models/<tenant>/<model>/v<N>/model/. The evaluation scorecard is stored beside it as eval_scorecard.json, along with a lineage.json that records where the version came from: the pipeline and the pipeline version that produced it, the workflow run, the Git commit when the pipeline is defined in Git, the parameters of the run, the image every step ran, the artifacts the run produced and the MLflow experiment it logged to.
Versions are immutable. Only TCR writes to the registry; your tenant's credentials can read its models but never change them, so the files behind v7 are the files that v7 was evaluated on, for as long as the version exists. Each run registers exactly one version. If the platform restarts while a run is registering, it finds the version it already created rather than making another, so a version number always corresponds to a run.
The scorecard is the contract between your evaluation step and the gate. It is a JSON object of metric names and numbers, such as {"accuracy": 0.91, "f1": 0.88, "p95_latency_ms": 42}; nested objects are flattened into dotted names, so per-slice metrics like {"slices": {"eu": 0.9}} become slices.eu.
The Promotion Gate
The gate decides whether a version may be deployed. You give it thresholds on scorecard metrics, each either "at least" or "at most", which is how latency budgets are gated alongside accuracy: accuracy >= 0.85 and p95_latency_ms <= 50 sit side by side. You can also turn on the champion and challenger comparison, which additionally requires the new version to be no worse than the version currently in production on every gated metric, within a tolerance you set as a percentage of the production value. A version is measured against the best you already have, not only against a fixed bar.
A failed gate never loses work. The version stays in the registry marked rejected, with its scorecard and the gate's verdict check by check, so you can compare it with the versions that passed. Deployment is simply skipped for that run. The gate's result is part of the run's graph, and its reasoning is part of the version's record on the Models page.
Deployment
A version that passes the gate is deployed by TCR on KServe, running on Knative. Each tenant's models are served from a namespace of their own, kubeopera-serve-<tenant>, kept separate from the namespace their training runs in, so that training workloads can never reach production endpoints and model servers can never reach training data. TCR picks the serving runtime from the model's framework: scikit-learn, XGBoost, PyTorch, ONNX, NVIDIA Triton, or the Hugging Face runtime with its vLLM backend for large language models.
The path to production can be as cautious as you want it to be. With a staging environment, the version is first deployed to a staging endpoint of its own, and a smoke test, which is any container step of your pipeline you choose, runs against it with KUBEOPERA_STAGING_URL and KUBEOPERA_STAGING_TOKEN in its environment. Only if the smoke test passes does the version move on. With approval switched on, the run then waits until someone with the right to promote models approves it on the model's page, and a rejection ends the run without touching production.
Production releases are canaries. The new version is deployed next to the one already serving, and it receives the share of traffic you chose, ten percent by default, for the length of the analysis window. When the window closes, TCR reads the canary's own request metrics from Prometheus, its share of server errors and its 95th-percentile latency, and compares them with the limits you set. If the canary stayed within them, it is promoted to all of the traffic, it becomes the model's production version, and the version it replaced is retired. If it didn't, TCR rolls back on its own: all traffic returns to the version that was already serving, the new version is marked rolled back, and the run fails with the reason, so your notifications tell you what happened. A model's first deployment has nothing to be compared with and goes straight to all of the traffic. If a run is stopped or fails while its canary is still taking traffic, the run's exit handler sends that traffic back to the stable version, so an abandoned release never lingers in production.
Each model serves at https://<model>.<tenant>.<models-domain>, behind a bearer token that belongs to that deployment. The platform's ingress checks the token with TCR before a request reaches the model, and only a hash of it is stored; tenant administrators can reveal the token, together with a ready-made example request, on the model's page. Models can scale to zero when idle, so a model that is rarely called costs nothing between calls and wakes on the first request; replica counts, CPU and memory, GPUs for the server, and the runtime version are all part of the deployment settings.
Retraining when the data moves
Models drift. The inputs they see in production slowly stop looking like the data they were trained on, and their predictions degrade without a single error being thrown. An ML pipeline can be started by a drift webhook: point your drift detector, an Evidently report, an Alertmanager rule on prediction drift or anything else that can make an HTTPS request, at the pipeline's webhook, and a retraining run starts whenever drift is detected, with the detector's reason and score passed into the run's parameters. Combined with the gate's champion and challenger comparison, this closes the loop: the world changes, the model retrains on fresh data, and the new version reaches production only if it is genuinely better. Schedules, S3 and SQS triggers, Kafka triggers, Slack and PagerDuty notifications and Git-defined pipelines all work exactly as they do for data pipelines; a Git-defined ML pipeline is a kubeopera.io/v1alpha1 MLPipeline document, the wizard's model with the ML settings under spec.ml.
The Models page
Every model your pipelines register has a page under Workflows → Models. It shows where the model is deployed, with each environment's version, its readiness and a live view of any canary in progress and its share of traffic. It lists the runs waiting for approval, with Approve and Reject. A chart plots the metrics of every version's scorecard over time, so you can watch the model's quality across releases, and the version list shows each version's status: registered, candidate, rejected, staging, canary, production, rolled back, retired or archived. Selecting a version shows the gate's verdict check by check, against which production version it was compared, its full scorecard and its lineage, with links back to the pipeline and the exact run that produced it.
The same page is where people take over from the automation when they need to. Deploy puts any version into production at once, which is how you override a gate you disagree with or bring back an older version, Roll back returns production to the previous version, and Archive retires a version so that it can no longer be deployed.
The pipeline's own page shows the ML picture from the other side: the model it trains, its gate and deployment settings, its experiment runs, and for every run the CPU-hours and GPU-hours it consumed. When the GPU pool is running, the Performance dashboard shows its utilisation, memory and power from the NVIDIA DCGM exporter.
Who can do what
Everyone in a tenant can see its ML pipelines, its models and their history. Changing them needs permissions a platform administrator grants to roles in Settings → Permissions: ml-pipelines:write, ml-pipelines:execute and ml-pipelines:delete for the pipelines themselves, and models:promote to deploy, roll back and archive versions, approve releases and reveal endpoint tokens. Tenant administrators hold all of them by default. Platform administrators see every tenant's pipelines and models, act on any model, and create or change a tenant's ML pipelines after choosing that tenant in the selector at the top of the page, in which case the pipeline lives in the tenant's namespaces and under its quotas.
Next steps
- Data Pipelines: the foundation ML pipelines are built on: steps, artifacts, secrets, versions, schedules and triggers.
- Workflows: watch training and release runs live, read logs and act on runs.
- workflow-manager: the service that compiles pipelines and runs the control stages.