Data Pipelines

A data pipeline moves and shapes data: it extracts from a source system, checks what it extracted, transforms it, and loads the result somewhere useful. In Tantilium Cloud Runner (TCR) you describe that work as a graph of steps, and the platform runs it on Argo Workflows for you. You never install Argo, never write Kubernetes manifests, and never hand anyone credentials to a cluster. You describe what each step runs and what it depends on; TCR turns that into a workflow, runs it in a namespace that belongs to your tenant alone, stores what each step produces, and shows you every run as it happens in Workflows.

Describing a pipeline

A pipeline is a set of steps. Each step runs a container image you choose, with the command, arguments and environment you give it, and each step can depend on others. The dependencies form a graph: steps that don't depend on each other run in parallel, and a step starts as soon as everything it waits for has finished. A typical pipeline looks like the one the wizard can start you with, an extract step that pulls orders from a source system, a validate-schema step that checks them, a transform step that aggregates them, and a load-warehouse step that writes the result, but the shape is yours to choose.

Steps hand data to each other through artifacts. A step declares the files or directories it produces as named outputs; a later step that depends on it asks for one of those outputs and says where it wants it mounted. TCR stores every artifact in your tenant's area of the platform's object storage and fetches it into the next step's container, so steps never share a disk and never need to know where the data lives. Pipelines take parameters too. You declare them with default values, refer to them in a step's arguments or environment as {{workflow.parameters.<name>}}, and override them for a single run when you start it by hand, when a schedule fires, or from the event that triggered the run.

Each step can be made as robust as it needs to be. You can give it a retry policy, with how many times to retry, on which kinds of failure, and with what backoff between attempts. You can give it a timeout, and the CPU and memory it requests and may use. A step can fan out, running once for each item in a fixed list, or once for each item in a list that an earlier step printed, with the current item available as {{item}}. At the pipeline level you decide whether one failure stops new steps from starting, how many steps may run at once, and how long the whole run may take.

Credentials never appear in a pipeline's definition. You store them as secrets on the Data Pipelines page, where each key of a secret becomes an environment variable in the steps you attach it to. Secret values are write-only: you can replace them, but no screen and no API ever shows them again.

Three ways to define it

Most pipelines are built in the wizard, which walks you through the basics and parameters, the steps and how they connect, the triggers and alerts, and a final review. A live graph of the pipeline redraws as you add steps and dependencies, so you can see the shape you are building.

When you need an Argo feature the wizard doesn't offer, you can switch a pipeline to YAML and edit the Argo WorkflowTemplate it compiles to directly. Switching is one-way, because the YAML can express things the form can't. Even in YAML, TCR decides the template's identity: it sets the name, namespace, labels and service account itself, and it checks the template against an allowlist before accepting it. Container, script, DAG, steps and suspend templates, artifacts, retries and exit handlers are all welcome. Anything that would let a pipeline escape its namespace is refused, including privileged containers, host access, pod spec patches, node selection, persistent volumes and references to secrets other than your own. After that check, Argo's own linter gets a second look, so a template that saves is a template that runs.

Teams that keep everything in Git can keep their pipelines there as well. A pipeline can take its definition from a file in a GitHub or GitLab repository, either a kubeopera.io/v1alpha1 DataPipeline document, which uses the same model as the wizard, or an Argo WorkflowTemplate. TCR watches the file on the branch you name, and every commit that changes it becomes a new version of the pipeline, checked by exactly the same rules as an edit in the interface. A commit that fails validation never replaces a working pipeline: the current version keeps running and the pipeline page shows what was wrong with the commit. Sync now pulls the latest commit immediately, and a secret holding a GIT_TOKEN gives TCR read access to a private repository.

Every change is a version

A pipeline never changes in place. Each save, whether from the wizard, the YAML editor or a Git commit, creates a new immutable version, and new runs and the schedule use the newest one. The pipeline page lists every version with who made it, when and why, and shows the exact WorkflowTemplate each version applied. Rolling back makes an old version's definition current again by saving it as the next version, so the history only ever grows and a run always knows precisely which definition it executed. Runs already in progress are never affected by a new version.

When pipelines run

You can run a pipeline on demand with Run now, overriding any of its parameters for that run. You can schedule it with a cron expression in the timezone of your choice, decide what happens when a run is still going as the next one is due (skip the new run, replace the old one, or let both run), and pause and resume the schedule without touching anything else.

Pipelines can also start themselves when something happens. A webhook trigger gives the pipeline a URL and a bearer token; anything that can make an HTTPS request can start a run and pass fields from its JSON body into the run's parameters. An S3 or SQS trigger watches a queue, typically the destination of a bucket's "object created" notifications, and can filter on the object key, so a pipeline runs every time a new file lands and receives that file's key as a parameter. A Kafka trigger consumes a topic, with SASL and TLS when your brokers need them. These triggers run on Argo Events in a namespace that tenants can't reach, and the component that turns an event into a run can do nothing but create workflows from templates TCR has applied.

Spark

Some transformations belong on Spark. A step can be a Spark application, run by the Spark Operator: you give the Spark image, the main application file, the arguments, the Spark version and the driver and executor sizes, and the step waits for the application to finish just as it would wait for a container. When your code already calls spark-submit itself, a container step can instead be marked as running spark-submit --deploy-mode cluster; it then receives an identity that is allowed to create the Spark driver in your namespace, and the namespace and driver account to pass to Spark. Either way the driver runs in its own pod, so Spark jobs read and write their data directly, for example over s3a://, rather than through step artifacts. Spark settings that would change who the driver runs as, or where, are refused.

Hearing how a run went

A pipeline can notify Slack when a run fails, when it succeeds, or both, and it can page PagerDuty when a run fails, resolving the incident automatically when the next run succeeds. Notifications are sent by TCR itself when it sees the run finish, so webhook URLs and routing keys stay in your secrets and never enter a pipeline's pods, and every run is announced exactly once per channel however it was started. Failed runs also count towards the failed-workflow figures on the dashboards and in the sidebar.

Where your pipelines run

Every tenant's pipelines run on a dedicated Argo Workflows controller on the platform's host cluster, separate from the one that runs CI/CD builds, in a namespace of their own named kubeopera-data-<tenant>. TCR creates that namespace the first time you save a pipeline and keeps it reconciled from then on. It enforces the baseline Pod Security standard, carries a quota and default limits so that one tenant's pipelines can't starve another's, and runs every step without a Kubernetes token in its containers. A network policy lets steps reach DNS, each other and the internet on the ports your platform administrator allows, so they can talk to external databases, warehouses and APIs, but never to other tenants or to the platform's own services.

Artifacts and archived logs live under your tenant's own prefix in the platform's object storage, written with credentials that can reach that prefix and nothing else. Those credentials are used by Argo's own executor and never by your code. Everything a run produces reaches the interface by streaming through TCR after it has checked that you may see it.

Who can do what

Everyone in a tenant can see its pipelines, their versions and their runs. Changing them needs permissions that a platform administrator grants to roles in Settings → Permissions: data-pipelines:write to create and edit pipelines, schedules, triggers, notifications and secrets, data-pipelines:execute to run a pipeline, and data-pipelines:delete to delete one. Tenant administrators hold all three by default, while other members see a read-only page.

Platform administrators see every tenant's pipelines. To create or change one on a tenant's behalf, they choose that tenant in the selector at the top of the page. The pipeline is then created in the tenant's namespace, under its quota, and the version history records the administrator who made it.

Next steps

• Workflows: follow runs live, read logs, download artifacts, retry and stop.
• ML Pipelines: the same foundation, extended to training, evaluating and deploying models.
• workflow-manager: the service that compiles, validates and runs your pipelines.