Failure Modes

Read

A gateway outage returns a connection error to the client. SDKs do not retry directly against the backing store. Check gateway health and connectivity; store failures remain visible through the gateway response.

The document cache is stateless and can scale to zero with no disruption: document fetches fall through to origin (turbopuffer, or S3 for snapshots) on a miss or cache outage, so a cache failure degrades latency, not availability.

Write

Successful writes require success from the backing store. If the gateway is unreachable, the client returns the connection error. Cache warming is best-effort and does not replace the durable store write.

Pipeline stop-writes

The primary failure mode for writes through a healthy gateway is Aerospike stop-writes during a multi-stage pipeline job: staged documents stay warm in the cache but carry no vector data yet, and once that data exceeds the Aerospike drive allocation the cache rejects further writes.

The pipeline does not stall. Each stage persists its chunk bodies to S3 before it touches the cache, and pipeline state lives in PostgreSQL, so the Aerospike write is best-effort: on stop-writes the gateway logs the skipped write and the stage still completes. Downstream chunk reads degrade to the S3 backing for as long as the cache is rejecting writes.

Recovery is automatic. The Helm document cache restarts on stop-writes by default (documentCache.autoRestartOnStopWrites: true) and clears its Aerospike backing file on pod start (documentCache.storage.resetOnStart: true); the gateway reconnects in the background and refills the cache from S3 on demand. No pipeline work is lost — S3 and PostgreSQL are the durable recovery boundary and must stay healthy.

Operator signals:

• layer_aerospike_op_duration_seconds{status="aerospike_stop_writes"} — the stop-writes condition itself, the same series the dashboard charts.
• hevlayer_cache_cold_responses_total — reads being served from S3 backing instead of the cache while it recovers.
• hevlayer_document_cache_cold_starts_total and hevlayer_document_cache_cold_start_seconds — the demand-triggered reconnect-and-refill cycle after the cache restarts.
• Gateway warn logs Aerospike chunk write failed (best-effort) and Aerospike chunk read failed; falling back to S3 backing.

Planned gateway rollout

A gateway rollout is a short, real outage, not a zero-downtime handover. The gateway runs as a single pod because its embedded Postgres (the Pipeline and Function queues) is pod-local; the chart rejects gateway.replicas above 1 with gateway.queue.embedded, and a second replica or a PodDisruptionBudget would not add availability (a budget of one blocks the roll or does nothing). During a roll the old pod stops answering at the start of its pre-stop drain (gateway.drain.preStopSleepSeconds, 20s), the new pod starts after it exits, and readiness follows its startup. The gap is bounded by that drain plus the pod’s startup, not by traffic.

What survives it:

• Queue state: Postgres data stays on the pod’s volume, so Pipeline and Function queues, claims and leases are intact when the pod returns on the same node. Losing the node is a different event: node-local storage goes with it.
• Function and Pipeline workers built on the Layer extract image: they wait the gap out and carry on without a relaunch. See Gateway rollouts.
• Setup calls and reads that SDKs can safely repeat.

What does not: requests made during the gap fail with a connection error or 503 (gateway_draining from the pod itself, or a proxy 503). Retry reads and idempotent setup; do not blindly resend a write or Function completion whose outcome you did not see.

Serving through a rollout needs the queue state off the pod (external Postgres) and more than one gateway replica. That is a deployment and spend decision, not a default.

Client failures

When the gateway is unreachable, the SDKs return the gateway connection error. They do not retry directly against a backing store. That keeps the client surface backend-neutral: a namespace may be backed by turbopuffer or another store selected by VectorStore, and callers should not need a store-specific credential or fallback path.