Skip to main content
Version: 2.0

Monitoring Guide

This guide covers two things:

  1. Monitoring your applications: how to instrument an app so Tantilium Cloud Runner's dashboards, SLOs, APM, tracing and AI agents can see inside it.
  2. Monitoring Tantilium Cloud Runner (TCR) itself: how the platform's own services are scraped, visualized and alerted on.

Monitoring your applications​

TCR automatically collects cluster-level signals such as CPU, memory, restarts and pod status for every workload. To see inside your app (request rate, latency, errors, traces), give TCR three things.

1. Expose Prometheus metrics​

Serve metrics on a /metrics endpoint using a Prometheus client library for your language (Go, Python, Java, Node.js). At a minimum, record HTTP requests with the standard labels:

http_requests_total{service, route, method, status}
http_request_duration_seconds_bucket{service, route, method, le}

Then tell TCR to scrape it. In the Deploy Application wizard, turn on Metrics and enter the port and path, or add the annotations yourself:

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9898"
    prometheus.io/path: "/metrics"

Your app's latency, throughput and error rate now appear in APM, on the Dashboard's observability panels and in the AI agents' view of your app.

2. Send traces​

Instrument your app with OpenTelemetry and export traces with OTLP to TCR's collector:

OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.monitoring:4317
OTEL_SERVICE_NAME=payments-api

Traces appear in Monitoring → Traces, and the service map shows how your services call each other.

3. Log to standard output​

Write logs to stdout/stderr, ideally as JSON with level, msg and a trace_id. TCR collects them with Loki automatically; they appear on your app's Logs tab, and log patterns feed root-cause analysis.

Define SLOs​

Once your app exposes metrics, define service-level objectives in Monitoring → SLOs, for example 99.9% of checkout requests succeed over 30 days. TCR tracks the error budget and burn rate and can alert you before the budget runs out. See slo-manager.

Monitoring Tantilium Cloud Runner itself​

TCR's services are monitored with the same tools as any workload: Prometheus, Grafana and Alertmanager, installed as part of every environment.

Metrics​

Every TCR service exposes Prometheus metrics on /metrics, on the same port as its API, and ships a ServiceMonitor so the Prometheus Operator scrapes it:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kubeopera-api
spec:
  selector:
    matchLabels:
      app: kubeopera-api
  endpoints:
    - port: http
      interval: 30s
      path: /metrics

Each service reports request rate, latency and errors per route, plus metrics specific to the service, such as messages consumed and processing time for event-driven services.

tip

If a service's panels go flat, check that its ServiceMonitor port matches a port the container actually listens on. Prometheus's Targets page shows any target it can't scrape.

Dashboards​

Grafana ships with dashboards for the platform, provisioned from Git alongside the services they cover:

  • Platform overview: health of every TCR service at a glance.
  • Per-service: request rate, latency, errors and saturation.
  • Reactive pipeline: telemetry throughput, analysis latency, actions taken and feedback signals.
  • Message bus: RabbitMQ queue depth, publish and consume rates, and unroutable messages.
  • Data stores: PostgreSQL and Redis.

Tenant-facing views of apps (cost, performance, security, SLOs and traces) are in the TCR dashboard itself, under Monitoring.

Alerting​

Alert rules are PrometheusRule resources in Git, evaluated by Prometheus and routed by Alertmanager. Out of the box they cover:

  • elevated HTTP error rate or latency on any service;
  • high CPU or memory per pod;
  • growing RabbitMQ queues or unroutable messages;
  • failed Flux reconciliations;
  • certificates close to expiry.

Route alerts by severity to the channels your team uses, whether Slack, PagerDuty, Opsgenie or email, by editing the Alertmanager configuration in your environment:

route:
  receiver: slack-alerts
  routes:
    - matchers: [ severity="critical" ]
      receiver: pagerduty
receivers:
  - name: slack-alerts
    slack_configs:
      - channel: "#alerts"
  - name: pagerduty
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pagerduty-key

Next steps​