Workflow Manager
Service: workflow-manager · Port: 8120
Workflow Manager is Tantilium Cloud Runner's workflow management module. It has two responsibilities that share one foundation. It is the platform's eyes on Argo Workflows, keeping a live, cross-cluster record of every workflow run on every cluster Tantilium Cloud Runner (TCR) manages, which is what the Workflows area shows. And it is the engine behind data pipelines and ML pipelines: it turns the pipelines tenants describe into Argo workflow templates, provisions the isolated namespaces they run in, starts their runs, and, for ML pipelines, carries out the registration, gating and deployment stages that tenant code is never allowed to perform.
workflow-manager is reached only through kubeopera-api. kubeopera-api authenticates the caller, derives their tenant, whether they are a platform administrator and the permissions of their roles from their token, and passes those to workflow-manager with a shared key. workflow-manager trusts nothing else: there is no request it serves on behalf of an unidentified tenant, and a request without a tenant is refused unless it comes from a platform administrator.
Watching every cluster
workflow-manager reads the list of clusters from kubeopera-api and runs one informer per cluster over Argo's Workflow resources in every namespace. It uses the in-cluster identity on the host cluster, and each registered cluster's own kubeconfig everywhere else. It recognises the host cluster the same way Register this cluster does, from the identity of its kube-system namespace, so it needs no configuration to know where it runs; when the host is registered in the interface, kubeopera-api tells it immediately and it adopts the registered identity, carrying the host's history with it. Clusters connected or removed later take effect the same way.
Each run it sees is normalised into a single record: the cluster with its region and provider, the namespace, the run's identity, the tenant it belongs to, where it came from, its phase and progress, its timings, its parameters, the resources it consumed and a snapshot of its execution graph, including every node's inputs, outputs and artifact locations. These records live in PostgreSQL, and every change is published through PostgreSQL notifications to a live stream that every replica serves. That is why the run list updates in every browser at once however many replicas are running, and why a run's history remains after Argo has removed the run from its cluster. Only one replica watches the clusters at a time, chosen through a Kubernetes lease; all of them serve the API and the stream. Retention of finished runs is configurable, ninety days by default.
Actions on runs go to the cluster that owns them. Stopping and terminating set Argo's own shutdown switch on the workflow. Retrying asks that cluster's Argo server, which workflow-manager reaches through the cluster's Kubernetes API service proxy, so no Argo server is ever exposed; when a run can't be retried in place, it is resubmitted from its template with the same parameters. Logs come from the step's pod while it exists and from Argo's archived log afterwards, and artifacts stream from object storage through workflow-manager after the caller's access has been checked. Every action is recorded with its actor and outcome.
Running pipelines
Tenants' pipelines run on a dedicated Argo Workflows controller on the host cluster, separate from the controller that runs CI/CD builds, configured so that it only runs workflows that reference a template TCR has applied. Each tenant gets a namespace of its own, kubeopera-data-<tenant>, which workflow-manager provisions on first use and reconciles whenever its provisioning specification changes. The namespace enforces the baseline Pod Security standard and carries a quota, including a GPU quota, and default limits. It has an executor identity that Argo uses and that user containers never receive, the Spark and training identities that Spark and PyTorch jobs run as, a network policy confining traffic to DNS, the namespace itself, the API server, the shared tracking server and the external ports the platform allows, and the namespace's default artifact repository.
Object storage is a single bucket with a prefix per tenant. Because the host cluster is self-managed, each tenant has its own IAM user rather than a role, created by workflow-manager under a fixed path and with a permissions boundary that limits it to that bucket; workflow-manager's own permissions only let it create users on that path with that boundary, so not even workflow-manager can mint broader credentials. A tenant's user can read and write its pipeline artifacts and its ML training storage, and can only read its registered models.
Every save of a pipeline renders a WorkflowTemplate. Pipelines built in the wizard are compiled from their specification; pipelines written in YAML are the tenant's own template. In both cases workflow-manager stamps the template's identity, namespace, labels and service account itself and then checks the whole document against an allowlist that refuses anything able to escape the namespace, with Argo's linter as a second opinion. The template is applied before the version is recorded, so a version never points at a template that was never applied, and the newest templates are kept so that runs of recent versions can still be retried. Schedules become CronWorkflow resources pointing at the current version, and event triggers become Argo Events sources and sensors in a namespace of their own, from which the only thing that can happen is the creation of a workflow from an applied template. Git-defined pipelines are checked for new commits every two minutes. When a run finishes, workflow-manager sends its Slack and PagerDuty notifications itself, exactly once per run and channel.
The ML control stages
An ML pipeline's template ends with stages that belong to the platform: register, gate, deploy to staging, approval, canary, canary analysis, promotion, and an exit handler that cleans up. They are compiled as Argo suspend nodes. While a run waits on one of them, workflow-manager's stage controller does that stage's work with its own rights and then completes the node through the Argo server, setting the node's outputs, such as the registered version number or the gate's verdict, and marking it succeeded or failed so that the workflow continues. The deployment stages only run when the gate's output says the version passed. The controller acts only on stages that the pipeline version stored in its own database actually contains, so a template that merely looks like a control stage gets nothing done. Before completing a node it records the stage's outcome, which makes every stage safe to resume after a restart or a change of leader: a run registers exactly one model version however many times its register stage is observed.
Registration copies the run's model artifact and scorecard into the registry under a newly allocated, immutable version number and writes the version's lineage beside it. The gate evaluates the scorecard against the thresholds and, when asked, against the production version's scorecard. Deployment manages each tenant's serving namespace, kubeopera-serve-<tenant>, its quota and the storage credentials KServe downloads models with, applies the InferenceService for the model and environment, and publishes it behind an ingress that asks workflow-manager to check the deployment's bearer token before forwarding each request. Canary analysis reads the canary revision's request metrics from Prometheus and either promotes it or sends all traffic back to the stable version. Experiment tracking is configured the same way: workflow-manager is the administrator of the shared MLflow server, creates each tenant's MLflow account and each pipeline's experiment, and reads runs and metric histories back for the interface.
REST API
All routes below are served under /api/v1 and called by kubeopera-api, which forwards the caller's identity. A platform administrator acting for one tenant does so through kubeopera-api's acting_tenant parameter.
Workflow runs
| Method | Path | Description |
|---|---|---|
GET | /workflows | Runs across clusters, filtered by cluster, region, provider, phase, source, template, pipeline, text and time. |
GET | /workflows/summary | Counts by phase and failures in the last 24 hours. |
GET | /workflows/stream | Live run changes (Server-Sent Events). |
GET | /workflows/{cluster}/{ns}/{name} | A run with its graph and action history. |
GET | /workflows/{cluster}/{ns}/{name}/logs/{node} | A step's log; follow=true streams it. |
GET | /workflows/{cluster}/{ns}/{name}/artifacts/{node}/{artifact} | Download a step's output artifact. |
POST | /workflows/{cluster}/{ns}/{name}/retry · /stop · /terminate | Act on a run. |
POST | /workflow-templates/{cluster}/{ns}/{template}/submit | Start a run from a template. |
GET | /workflow-clusters | Watched clusters and their Argo status. |
GET · POST · DELETE | /access-grants | Grant tenants access to namespaces on other clusters (administrators). |
Data and ML pipelines
Data pipelines live under /data-pipelines and ML pipelines under /ml-pipelines; both offer the same routes.
| Method | Path | Description |
|---|---|---|
GET · POST | /data-pipelines | List or create pipelines. |
POST | /data-pipelines/validate | Render, check and lint a pipeline without saving it. |
GET · PUT · DELETE | /data-pipelines/{id} | Read, update (a new version) or delete a pipeline. |
GET | /data-pipelines/{id}/versions | Version history. |
POST | /data-pipelines/{id}/rollback | Make an earlier version current again. |
POST | /data-pipelines/{id}/run | Run now, with parameter overrides. |
POST | /data-pipelines/{id}/sync | Pull the latest commit of a Git-defined pipeline. |
GET | /data-pipelines/{id}/triggers/{trigger}/webhook | A webhook trigger's URL and token. |
POST | /data-pipelines/{id}/schedule/pause · /resume | Pause or resume the schedule. |
GET · PUT · DELETE | /data-secrets[/{name}] | The tenant's write-only secrets. |
Models and tracking
| Method | Path | Description |
|---|---|---|
GET | /models | Registered models. |
GET | /models/{id} | A model with its versions, deployments and pending approvals. |
GET | /models/{id}/versions/{v}/files/{file} | A version's eval_scorecard.json or lineage.json. |
POST | /models/{id}/versions/{v}/promote · /archive | Deploy a version to production at once, or archive it. |
POST | /models/{id}/rollback | Return production to the previous (or a given) version. |
GET | /models/{id}/deployments/{env}/endpoint | A deployment's URL, token and example request. |
POST | /models/{id}/approvals/{run}/approve · /reject | Decide a release waiting for approval. |
GET | /ml-tracking/{pipeline}/runs | The pipeline's MLflow runs. |
GET | /ml-tracking/{pipeline}/runs/{run}/metrics/{key} | A metric's history for one run. |
Two routes are served outside /api/v1: POST /internal/v1/clusters/refresh, which kubeopera-api calls when clusters change, and GET /internal/v1/inference-auth, which the ingress in front of model endpoints calls to check a request's bearer token.
Configuration
| Variable | Default | Description |
|---|---|---|
DATABASE_URL | — | PostgreSQL connection (schema workflow_manager). |
API_KEY | — | Shared key kubeopera-api presents. Without it every request is refused. |
KUBEOPERA_API_BASE_URL · KUBEOPERA_API_INTERNAL_KEY | — | Where the cluster registry is read from. |
ARGO_DATA_SERVER_URL | in-cluster data controller | The pipeline controller's Argo server. |
ARTIFACT_BUCKET · AWS_REGION | — · eu-north-1 | Object storage for artifacts, logs and models. |
ARTIFACT_IAM_USER_PATH · ARTIFACT_IAM_USER_PREFIX · ARTIFACT_IAM_BOUNDARY_ARN | — | Conventions for per-tenant storage users. |
DATA_NS_QUOTA_CPU · DATA_NS_QUOTA_MEMORY · DATA_NS_QUOTA_PODS | 8 · 16Gi · 50 | Quota of each tenant's pipeline namespace. |
DATA_MAX_CPU · DATA_MAX_MEMORY | 4 · 8Gi | Largest single step. |
DATA_EGRESS_PORTS · DATA_EGRESS_EXCEPT_CIDRS | 443 · private ranges | Where pipeline steps may connect. |
EVENTS_NAMESPACE · EVENTS_PUBLIC_HOST | kubeopera-pipeline-events | Event triggers and their public webhook host. |
GIT_SYNC_INTERVAL | 2m | How often Git-defined pipelines are checked. |
PUBLIC_UI_URL | — | Links in notifications. |
ML_MAX_GPU · ML_NS_QUOTA_GPU | 0 · 0 | GPUs per step and per tenant. |
MLFLOW_URL · MLFLOW_ADMIN_USERNAME · MLFLOW_ADMIN_PASSWORD | — | The shared experiment-tracking server. |
ML_SERVING_DOMAIN | — | Domain model endpoints are published under. |
KNATIVE_GATEWAY | Kourier's internal gateway | Where model endpoints forward to. |
INFERENCE_AUTH_URL | in-cluster | The token-check URL given to the ingress. |
ML_SERVE_QUOTA_CPU · ML_SERVE_QUOTA_MEMORY · ML_SERVE_QUOTA_GPU | 8 · 16Gi · 0 | Quota of each tenant's serving namespace. |
PROMETHEUS_URL | — | Source of canary request metrics. |
RUN_RETENTION_DAYS | 90 | How long finished runs are kept. |
HTTP_PORT | 8120 | HTTP port. |