Operations

Clustering and operations

Clustering is off by default (cluster: false). Turn it on only when distribution is already a private network with a cookie and, where you need it, TLS distribution. Peers are trusted. Do not publish EPMD or the distribution port.

What clustering does

Wiregrid.start_instance(:chat,
  profile: :balanced,
  cluster: true,
  cluster_shards: 16,
  cluster_namespace: :chat,
  cluster_dedupe_ttl_ms: 60_000,
  cluster_pending_per_shard: 10_000
)

Enabled clustering uses a fixed shard count and fixed :pg groups. It does not create a process group per topic. Local ETS remains authoritative for local sessions and the local hot fanout path. Cluster messages carry event ids and are dropped when the dedupe table has already seen them. That table is capped by max_cluster_dedupe and entries expire after cluster_dedupe_ttl_ms (60,000).

cluster_shards defaults to 16 and must be from 1 to 256. Each shard mailbox is capped by cluster_pending_per_shard (10,000). Remote presence rows and remote room edges have their own profile caps, max_remote_presence and max_remote_room_edges. The namespace keeps two instances on one distributed node from sharing traffic. It defaults to :instance and may be an atom, a non-negative integer, a short binary, or a tuple of up to four such parts.

Node join starts a bounded resync. A periodic pass repairs state if the first resync was incomplete. Presence and room indexes are keyed so a lost node can be cleaned in batches. During a partition, presence and room views can disagree. Durable events that matter should be in the storage adapter, which is not a cluster consensus log. Publish forwards to the cluster when the call’s :cluster option and the instance flag both say so. The publish result includes a cluster field describing that forward.

Health

Wiregrid.health(:chat)
Wiregrid.readiness(:chat)          # :ready | :not_ready
Wiregrid.readiness_report(:chat)
Wiregrid.liveness(:chat)
Wiregrid.stats(:chat)
Wiregrid.pressure(:chat)
Wiregrid.prometheus_metrics(:chat)

health/1 returns a map with status (:ok, :degraded, or :down when the instance is missing), plus runtime, expiry, storage, cache, webhook, cluster, draining, and sessions. Status is :ok only when every present component is healthy.

Readiness is :not_ready when health is degraded (reason: :unhealthy), when the instance is draining (reason: :draining), or when any resource is at or above readiness_pressure_threshold (default 1.0, so the default trips only at the cap). The report lists saturated resources under reason: :resource_pressure. A missing instance is reason: :instance_unavailable. Liveness checks that the instance tables exist and the runtime process answers.

Metrics

Wiregrid.Observability.prometheus/1 prints a fixed text payload. The application exposes it on whatever HTTP endpoint it already has. Wiregrid does not open a port for metrics. Gauges include sessions, subscriptions, presence_watches, room_memberships, and delivery_reservations. Counters include:

• connections_total, disconnects_total, deliveries_total
• dropped_ephemeral, slow_consumers, admission_rejections, control_rejections
• callback_rejections, rate_limit_rejections
• adapter_calls_total, adapter_failures, adapter_timeouts, adapter_rejections
• cluster_messages_sent, cluster_messages_received, cluster_messages_dropped, cluster_resyncs
• webhook_enqueued, webhook_delivered, webhook_retries, webhook_failures
• topology_syncs, topology_sync_failures
• dispatch_plans_compiled, dispatch_plan_executions, dispatch_plan_failures

telemetry_metric_events defaults to false. When true, the same numbers can be emitted as telemetry. There are still no per-user labels.

Drain

:ok = Wiregrid.drain(:chat)
:ok = Wiregrid.await_idle(:chat, 10_000)
:ok = Wiregrid.undrain(:chat)
:ok = Wiregrid.graceful_shutdown(:chat, 10_000)

drain/1 stops new admission and still accepts acknowledgements and cleanup. await_idle/2 waits until delivery reservations, webhook work, and cluster pending work are zero. The default timeout is 10,000 ms. A negative timeout is {:error, :invalid_timeout}. graceful_shutdown/2 drains, waits, and stops the instance. Use it during a rolling replacement. A healthy Wiregrid process does not make an exhausted Postgres pool healthy; watch the caller-owned pools beside these gauges.

A release checklist

• ./scripts/verify.sh on the OTP and Elixir pair you deploy.
• Adapter integration tests against services that match production.
• scripts/load.sh with realistic sizes, fanout, ack delay, and slow clients.
• Limits chosen on purpose. :large is a ceiling, not a tuning shortcut.
• A real authorizer on every internet-facing transport.
• Distribution kept private.
• Backups and restore tested for the durable adapter.
• Drain during replacement.

The Linux release asset is built by .github/workflows/release.yml when a v* tag is pushed. The asset name is wiregrid-<version>-linux-x86_64.tar.gz, with SHA256SUMS beside it.