Skip to content
ArtificialFlow

Operations

Process health is not a functional check. Confirm identity, the connector, and a write that shows up in the query API before calling a deployment ready.

Health

  • Command and query /health report that the HTTP process is up. They do not probe IAM, Postgres, search, or the bus.
  • Sync /health can return HTTP 200 with status starting or degraded while healthFailOnStale is false.
  • The gateway and frontend are ready when the public page loads.

The command service also exposes compact counters for outbox relay activity, idempotency records, publish lag, and retry exhaustion. The sync worker reports freshness on its health endpoint.

A deployment is ready when

  1. The browser login succeeds.
  2. /api/identity/me returns the expected role.
  3. A workflow can be deployed and started.
  4. That instance appears through /api/query/.

The Debezium connector and its tasks report RUNNING. The status URL still uses the connector name artificialflow-postgres-connector, which is the legacy wire identifier kept for rolling upgrades. Pin image tags or digests and review signatures, SBOM, and provenance before a production rollout. Tags 1.1.0 and v1.1.0 are the same digest. New installs pull artificialflow/*.

Backups

Back up Postgres, the search indexes or the data required to rebuild them, IAM configuration, secrets, Helm values, and rendered manifests. Test a restore.

Recovery

  • Rebuild search projections from source events or CDC topics when you can.
  • make init-connector is a manual recovery path for local Compose profiles, used when the connector is missing after the command service has initialized the database.
  • Keep the Debezium connector configuration versioned. Do not place a database password in a ConfigMap.

Replicas

The supported production shape is two or more gateway, frontend, command, and query pods. The sync worker scales with Kafka partitions under a stable consumer group. The runtime default is one replica. Extra runtime pods are standbys: a Postgres lease (runtime_scheduler_lease) lets one holder run timer, SLA, and scheduler ticks. The holder renews for RUNTIME_LEASE_TTL (default 15s). Another pod takes the lease only after it expires. Helm disruption budgets cover command, query, frontend, and gateway when those components have at least two replicas.

Command pod loss fails the in-flight request. Clients retry with Idempotency-Key. Sync worker loss grows projection lag while writes stay in Postgres. Search loss degrades the query API and leaves the command path up. Postgres loss stops writes and runtime ticks.

Upgrade

  1. Back up Postgres and record the image digests you can roll back to.
  2. Drain the outbox. artificialflow_outbox_pending should be near zero.
  3. Apply the Helm or Compose upgrade. Command, query, and frontend can roll. Runtime ticks pause until the lease is held again.
  4. Check health, an OIDC login, a deployed process, a started instance, a completed job, an inbox task, and that instance in the query API.

Roll back by redeploying the previous image digests. Restore Postgres only when the schema change cannot run on the old image, and rebuild search afterward. Admins can read engine counters at GET /api/internal/metrics. The runtime process also exposes Prometheus on port 9091.

Published latency figures in the product repository were measured on a single-node Compose stack at v0.3.0, not on a production cluster and not on v1.1.0. Re-measure with make test-perf before using them for capacity. The aspirational targets are outbox age and CQRS lag under 15–30 seconds at p99, and runtime failover inside the lease TTL plus one tick.

Production checklist

  • TLS on public endpoints, and network policy around any internal plaintext route.
  • Example hostnames and Compose credentials replaced.
  • Audience validation left on, with short-lived SDK tokens.
  • Secrets in a secret manager, including SDK profiles. Those profiles never belong in a browser.
  • Resource requests, limits, and a tested backup.
  • Alerts on the gateway, command, runtime, query, sync, IAM, database, bus, connector, and search.

Schema changes need care: command and runtime run GORM AutoMigrate during startup. Pair this list with Troubleshooting when an install does not become ready.