Operations

Dashboard

The Layer dashboard is the operator UI that ships in-cluster alongside the gateway, as the layer-dashboard Deployment and Service. This page covers running it: the access it needs, how to reach it, how to gate it, and how to turn it off.

Access it needs

The dashboard is read-mostly and backed by three sources, each with its own grant:

  • The gateway API — the same endpoints customers use, plus the Prometheus-compatible metrics proxy at /v2/metrics. Authenticated with a gateway bearer (LAYER_GATEWAY_API_KEY). In deriveFromStore mode this is the default VectorStore credential; in keys mode it is the configured inbound worker key. It does not touch PostgreSQL, Aerospike, or VictoriaMetrics directly — metrics and cost arrive through the gateway proxy and /v2/cost.
  • The Kubernetes API — reads hevlayer.com CRDs (VectorStores, Indexes, InfraRules, Pipelines) and the workload objects behind them (pods, deployments/statefulsets, HPAs, KEDA ScaledObjects, nodes) through RBAC bound to its ServiceAccount. dashboard.kubeAccess.enabled grants the read role; with it off the dashboard still runs but the cluster/scaling views show a “kube access not configured” banner. dashboard.writeAccess.enabled adds the operator write role: Index spec patches, Karpenter NodePool disruption, VectorStore and Warehouse create/edit, and namespace-scoped Secret writes for credentials. Set it false for a read-only install.
  • Cost data — the cost view reads the gateway cost API. turbopuffer spend is metered from upstream billing counters; AWS spend is read by the gateway from Cost Explorer using the gateway IRSA role. The dashboard does not need direct AWS cost API permissions for the standard cost view. In a Terraform install, still pass dashboard.serviceAccount.roleArn from the layer_dashboard_role_arn output so the dashboard ServiceAccount is annotated with its cluster-specific IRSA role at first boot.

Who spent the queried bytes

The cost tab has a queried bytes · top callers panel. Pick a namespace and a window (last hour by default, or 6 or 24 hours) to see which Function, pipeline or API key accounts for the most Turbopuffer billable queried bytes, with each caller’s operation breakdown (query, ranked_query, scan, fetch, passthrough, upsert, patch, delete, snapshot).

  • Bytes come from hevlayer_tpuf_billable_bytes_queried_total, summed across gateway instances after a per-series increase, so a restarted gateway does not hide or double count its traffic. A cache hit adds none.
  • Callers are the bounded labels the gateway emits: a registered Function name, a pipeline id or a verified API key name. unknown, the other overflow bucket and series written before the caller labels existed are shown, marked as unattributed, and never named as the top spender.
  • The panel lists the top 20 callers and rolls the rest into one line.
  • The namespace is checked against the gateway with the dashboard’s own key first; a namespace that key cannot read returns no data. Namespaces that no longer exist are not listed.
  • Counters restart with the gateway, so the first minutes after a deploy can undercount.
  • Historical coverage is partial. Before the instrumentation rollout the gateway did not meter scan_page, fetch, fetch_many, fetch_vector or Function discovery bytes, and it recorded no caller identity. Older totals are partial metered-path figures, not full namespace billing, so do not compare them with these expanded counters to claim a cost reduction. Expanded coverage begins at the rollout, and the panel always shows a coverage note saying so. Earlier caller attribution is unknown, not zero: unknown, other and unlabeled series are shown as they are, never reconstructed or zero-filled. Full historical namespace billing needs the provider’s historical namespace export.
  • Figures come from the metrics store, so they lag the gateway by at least one scrape interval; the last seconds of traffic may not appear yet.

The same data is at GET /api/cost/attribution?namespace=<name>&window=1h. The underlying query is:

sum by (caller_kind, caller, operation) (
  increase(hevlayer_tpuf_billable_bytes_queried_total{namespace="<name>"}[1h])
)

Networking

The dashboard is an operator tool. Reach it over a port-forward rather than exposing it publicly:

kubectl port-forward -n <release-namespace> svc/layer-dashboard 8081:8081

Then open http://localhost:8081. Customer workloads only ever receive the gateway base URL and credentials — never the dashboard.

Basic auth

HTTP Basic auth sits in front of every dashboard route and is required — the dashboard refuses to start without it. Set credentials through the chart:

dashboard:
  basicAuth:
    user: ops
    password: <strong-password>

The chart render fails if either field is blank while the dashboard is enabled.

Disabling the dashboard

The dashboard is optional. Disable it and the Deployment, Service, RBAC, and ingress all skip rendering:

dashboard:
  enabled: false

The gateway and transform runtime run unchanged without it; you lose only the operator UI.

Operational notes

Pipeline queue states

The pipeline overview combines gateway queue counters with the matching Pipeline resources and their operator-managed Deployments and KEDA ScaledObjects. A queue with pending work is presented as:

  • PAUSED when every matching Pipeline has spec.paused: true. Its workers are intentionally at zero replicas; pending documents remain queued.
  • CRON-GATED when its cron ScaledObject is inactive. The card shows the next UTC window, scaler activity, and managed worker replica counts.
  • STALLED when work is pending with no processing or indexing activity and neither an intentional pause nor a closed cron window explains the stop.

Failed-document counts, operator condition messages, and missing or inactive managed workers appear as reason hints on the same card. A growing queue that is still processing remains BACKLOG rather than STALLED.

The data tab is split into indexes, vector store, and warehouse views. Indexes come from /v2/namespaces; vector stores and warehouses come from the gateway’s /v2/vectorstores and /v2/warehouses projections, so the dashboard shows the same credential-safe shape as the CLI and SDKs.

When write access is enabled, the vector store and warehouse panes can create and edit those CRDs. Raw credentials are accepted only in the form submission; the dashboard writes them into Kubernetes Secrets first, then creates or patches the CR. The read views keep showing only Secret references.

esc
↵ ask AI · ↑↓ open hev/ask