Namespace metadata

Request

metadata = await client.get_namespace_metadata("products")

metadata, err := client.GetNamespaceMetadata(ctx, "products")

const metadata = await client.getNamespaceMetadata("products");

curl "$LAYER_GATEWAY_URL/v2/namespaces/products/metadata" \
  -H "Authorization: Bearer $LAYER_GATEWAY_API_KEY"

{
  // Proxied from turbopuffer verbatim
  "schema": { },
  "approx_row_count": 12500,
  "approx_logical_bytes": 48800000,
  "created_at": "2026-03-15T10:30:45Z",
  "updated_at": "2026-05-12T18:49:00Z",
  "last_write_at": "2026-05-12T18:48:30Z",
  "index": { "status": "up-to-date" },

  // Layer enhancement
  "layer": {
    "stable_as_of": 1715600400000,
    "is_stable": true,
    "indexed": true,
    "index_lag_rows": 0
  }
}

Source sync health

layer.sync records the most recent completed source sweep. A key with read access to the namespace can read it through this endpoint. REST and clouddrive workers persist the signal in the snapshot bucket, including sweeps that change zero rows. Scheduled REST sweeps publish the same signal as continuous sync.

{
  "last_success_at": "2026-11-03T18:00:53Z",
  "last_attempt_at": "2026-11-03T18:00:53Z",
  "last_status": "succeeded",
  "rows_seen": 42,
  "source_kind": "rest",
  "pipeline": "ki-projects"
}

Read the last successful sync time at layer.sync.last_success_at. Times are UTC completion times. A failed or partial sweep advances last_attempt_at and preserves last_success_at; before the first success, last_success_at is null. rows_seen counts rows observed in that sweep, not upserts. The object is omitted when no source has reported or the gateway cannot read or parse its health record. Other namespace metadata remains available. Treat an absent object or null success time as unknown, and reject stale, malformed or future times when enforcing freshness. last_write_at still describes data writes. Pipeline Ready means reconciled and does not prove a successful sweep.

The signal survives worker and gateway restarts. Reading it does not require Pipeline, Kubernetes or bucket access. If several Pipelines target one namespace, this object describes the latest published sweep, not the health of every source. A change of Pipeline or source kind starts a new success history.

The layer block

A read for a namespace that does not exist returns upstream’s 404, matching turbopuffer’s own metadata endpoint.

is_stable is the current signal — it drives the per-query filter-skip decision on the query path. stable_as_of is the historical watermark — the cut a filtered query would apply. After a namespace is observed stable, the watcher refreshes this watermark on the stable-tier cadence (CONSISTENCY_STABLE_POLL_INTERVAL_MS, default 60000 ms). Writes re-arm the fast tier, so active namespaces are polled on CONSISTENCY_POLL_INTERVAL_MS.

indexed answers a different question than is_stable. is_stable reports whether the upstream index has caught up on the rows it has seen, which is what read-after-write depends on. indexed reports whether every row that should be present is present and queryable, which is what a bulk load or a pipeline needs to know it has finished: rows can be staged and counted before their vectors are indexed, so a namespace can read is_stable: true while indexed: false with a non-zero index_lag_rows. The reconciliation runs against the latest snapshot, so indexed advances on the snapshot cadence rather than per write.

For snapshot history derived from these freshness signals, see Snapshots.

Field stats

layer.field_stats describes what is in each attribute of schema: how many distinct values it has, how many rows lack it, its range, and its most common values. It sits next to the schema it describes, keyed by the same attribute names, so one metadata read tells a client both what a field is and what is in it. The gateway keeps the stats current as rows are written, so the read is a few cache lookups, not a scan and not a query against your store.

Field stats are opt-in. Ask for them with ?field_stats=true. Without it the metadata response is exactly what it is today, so existing clients see no change.

metadata = await client.get_namespace_metadata("bcc-pages", field_stats=True)

metadata, err := client.GetNamespaceMetadata(ctx, "bcc-pages", &hevlayer.GetNamespaceMetadataParams{
    FieldStats: true,
})

const metadata = await client.getNamespaceMetadata("bcc-pages", { fieldStats: true });

curl "$LAYER_GATEWAY_URL/v2/namespaces/bcc-pages/metadata?field_stats=true" \
  -H "Authorization: Bearer $LAYER_GATEWAY_API_KEY"

A search UI uses them to build the right filter for each field without being told. A field with one value has nothing to filter on. A field with five values wants checkboxes, one with 3,000 wants a search box, and a numeric field wants a slider over its real range. Layer UI reads field_stats and makes those choices itself.

{
  // Proxied from turbopuffer verbatim
  "schema": {
    "doc_type": { "type": "string", "filterable": true },
    "company": { "type": "string", "filterable": true },
    "file_year": { "type": "uint", "filterable": true },
    "path": { "type": "string", "filterable": true }
  },
  "approx_row_count": 48211,

  // Layer enhancement
  "layer": {
    "stable_as_of": 1791035112000,
    "is_stable": true,
    // Only with ?field_stats=true
    "field_stats": {
      "as_of": 1791035112000,
      "rows": 48211,
      "fields": {
        "doc_type": {
          "missing": 231,
          "distinct": { "value": 14, "quality": "exact" },
          "state": "exact"
        },
        "company": {
          "missing": 0,
          "distinct": { "value": 1, "quality": "exact" },
          "state": "exact"
        },
        "file_year": {
          "missing": 1209,
          "distinct": { "value": 19, "quality": "exact" },
          "min": 2007,
          "max": 2026,
          "state": "approximate",
          "approximate_since": 1791034990000
        },
        "path": {
          "missing": 0,
          "distinct": { "value": 48102, "quality": "estimate" },
          "state": "approximate"
        }
      }
    }
  }
}

Metadata reads use cached field statistics; they do not trigger a namespace scan. Reconciliation refreshes the cache separately. The performance acceptance fixture checks that both field_stats=true and field_stats_top=20 add at most 20 ms at p95 over plain metadata on a quiet host. This is a fixture target, not a staging latency SLA; network and store latency still affect your requests.

Add field_stats_top=N (up to 1,000) to include each field’s N most common values with their counts. It implies field_stats=true:

curl "$LAYER_GATEWAY_URL/v2/namespaces/bcc-pages/metadata?field_stats_top=20" \
  -H "Authorization: Bearer $LAYER_GATEWAY_API_KEY"

"doc_type": {
  "missing": 231,
  "distinct": { "value": 14, "quality": "exact" },
  "top": [
    { "v": "invoice", "n": 18233 },
    { "v": "drawing", "n": 9120 },
    { "v": "change_order", "n": 2210 }
  ],
  "state": "exact"
}

To page through every value of a field, or count values within a filter, use a values scan.

How the gateway keeps them current

Every write through the gateway updates the stats as it lands.

• Writes add counts and widen ranges. Overwrites, patches and deletes subtract cached old values where available; missing old values, filtered writes and invalidated extrema leave approximate aggregates.
• Freshness. With reviewed capture installed, metadata reads validate the complete persisted bundle under retained source guards before reporting exact counts. A stored receipt alone is insufficient. Missing coverage or unavailable guards leave historical counts approximate. Reads never scan, admit a scan attempt, schedule reconciliation or register a namespace.
• Reconcile. A gated source integration uses an opaque capture receipt validated under the original retained guards. Complete persisted artifacts with matching coverage may be reused while guarded. A false validation requires one strong, exhaustive scan projecting the union of snapshot facets and declared stats fields; an error never permits skipping. Concurrent triggers share one flight and the larger configured interval sets the floor. Pro startup can install this integration from an explicit LAYER_FIELD_STATS_CAPTURE_CONFIG file (the Helm chart mounts it from an existing ConfigMap named by gateway.fieldStatsCapture.configMapName) containing reviewed client/account, pricing/calibration, schema/projection and validity inputs, plus an existing witness identity and fence quotas. No policy defaults or witness bootstrap are supplied. All configured Turbopuffer aliases must share the reviewed writer identity. Missing configuration preserves legacy snapshots. Source installation requires independent all-writer and terminal-drain evidence; it is not evidence that a live namespace has accepted current stats.
• Legacy snapshots continue their periodic facet scans at their snapshot interval. Declared stats do not broaden those scans. Repeated unchanged refreshes still scan and reaggregate; new stats-only coverage is unavailable without the capture contract. Turbopuffer timestamps and row counts do not certify content identity or silent external writes.

The live stats sit in Aerospike, one record per field, updated with compare-and-set so concurrent writers never lose a delta. Each reconcile also writes them to S3. A gateway that starts cold, or loses Aerospike, reloads from S3 rather than rescanning.

A field passes the configured value cap (default 10,000 distinct values) when it has more distinct values than that. Past it the gateway stops counting values one by one: top is dropped and distinct becomes an estimate from a HyperLogLog sketch that the same writes and reconcile scan feed. No extra scan, accounting or scan bytes are involved.

Estimate accuracy

The sketch has 16,384 one-byte registers (16 KiB per field, about 22 KiB stored) and Ertl’s improved estimator. Its relative standard error is about 0.8%: typically the estimate is within 1%, about 95% of estimates are within 1.6%, and about 98.7% are within 2%. Layer documents about 2%, not a hard bound. Its tests measure 10,001 to 2,000,000 distinct values over repeated independent key sets and require the mean error under 1.2%, at least 95% of estimates within 2% and none beyond 3%. The estimate never drops below the exact count it had at the cap. Sketches only add values: a delete or an overwrite of a value in an overflowed field leaves the estimate possibly too high, so the field’s state becomes approximate until the next reconcile rebuilds the sketch from the scan. Use a values scan when you need the exact distinct count of a high-cardinality field.

What as_of and state promise

quality and state answer different questions. quality says how a count was computed; state says whether it is current.

exact is never reported for stale or incomplete data. Without reviewed capture installed (the default), nothing can certify a cut, so every returned field is approximate, even on an append-only namespace. Counts then only err high and ranges only wide. After deletes, the next reconcile within one interval replaces them with exact scan results, and the field returns to exact state wherever the capture contract certifies it.

Which fields are tracked

Every filterable, non-vector, non-reserved attribute of an Index’s namespace, by default. Configure coverage on the Index:

spec:
  backend:
    namespace: bcc-pages
  fieldStats:
    fields: auto          # default; or [doc_type, year]
    exclude: [blocks]
    valueCap: 10000       # default
    interval: 5m          # default; positive duration

An explicit list selects only those filterable fields. fields: [] disables stats, even when snapshot facets remain enabled. Exclusions apply to either mode. Invalid field names, a zero value cap or a nonpositive interval are rejected. Auto follows schema changes at the next scheduled reconciliation.

Enabling a new field requires reading that attribute for every row once at the next reconcile. Wider projections increase upstream scan bytes; stats add no second scan. Until initial coverage completes, absent stats remain in field_stats.untracked; a client must keep their controls visible rather than treat them as exact zero. Excluded, nonselected and unavailable filterable fields are also listed there. Layer without Aerospike returns empty fields. A namespace without a declarative Index keeps the legacy snapshot-facet coverage.

Scan cost of enabling fields

A reconcile that has to scan reads the selected attributes of every row, in pages of 1,000 rows, and the provider bills each page as a query. Each page query is billed over the namespace’s queried bytes, not just the page’s rows, so the cost of one traversal grows with rows times namespace size, not with the number of fields alone. One measured traversal of a 208,000-row, 1.9 GB namespace projecting 14 attributes was 208 pages, 62 MB of response bytes and about 4.4e11 billed queried bytes.

Three things keep that cost down:

• List fields instead of auto. auto adds every filterable attribute, including high-cardinality ones such as paths and content hashes, and widens every row the scan returns. List the filters people actually use.
• Enabling a field is a scan. A new field is untracked until the next reconcile reads it. Expanding coverage consumes the same scan allocation as any other reconcile, so enable fields in one change, not one at a time.
• Unchanged namespaces cost nothing, where capture certifies it. With reviewed capture installed, a reconcile whose guards certify no change reuses the stored bundle and reads no rows. Without it, every reconcile at the interval rescans, so choose the interval for the cost you accept.

Metadata reads never scan, whatever the configuration.

Capping the snapshot scan

Without guarded capture, the snapshot scan (snapshot.facetFields) is the only scan that fills field stats, and its stats are always approximate. A namespace can be put under the same spend ledger as guarded capture by listing it in the file named by LAYER_FIELD_STATS_SNAPSHOT_BUDGET_CONFIG: a reviewed store, and per namespace the account and pricing review, the page calibration and the static metadata estimate. The gateway refuses to start if that file is invalid or the shared financial store is missing. A listed namespace’s scan then:

• reserves one attempt, and every page at its reviewed ceiling, before the page is sent, against 144 attempts and 963,000 micro-USD of reserved spend per rolling 24 hours, keyed by provider and namespace (capture and snapshot scans share one allowance);
• settles each page’s billed bytes before the next page, and never refunds a reservation, a failed attempt or a refused page;
• is refused, with no provider request, when the window is exhausted, another replica has an unsettled page, the listed projection differs from the namespace’s, the size metadata is missing, a projected attribute’s schema entry is missing or changed (the review pins those entries only, so an unrelated new attribute does not stop the scan), or the review has expired. A lapsed review does not stop the gateway from starting; the scan simply refuses until the file is re-rendered and the gateway restarts. The previous stats and as_of stay, approximate.

A listed namespace the provider reports as written in the last 15 seconds is not scanned that interval at all: no attempt, no reservation, no provider page. This only avoids billing a scan that would be discarded; as_of and the cadence are unchanged, and a write that lands during a scan still discards its result.

A namespace that is not listed scans as before, uncapped. At the reviewed 2x page ceiling the window admits fewer than 144 scans a day for a namespace of about 209 pages: see the scan cost estimate above. The spend is an estimate at public rates, not an invoice.

The value cap is configured per Index; LAYER_FIELD_STATS_VALUE_CAP remains the fallback for legacy facet coverage. Above the cap this increment retains the sketch described above.

How Layer UI uses them

Layer UI’s live runtime reads metadata with field_stats=true at startup and on each refresh, and picks each filter’s control from the field’s stats:

An approximate field keeps its control. Only an exact count of 0 or 1 hides it, so a filter never disappears because of a stale count.

Field stats and snapshots

Snapshots are history: a body per watermark, kept for retention, for comparing a namespace over time. Field stats are the current state, for building a UI now. They share the reconcile scan. Snapshot facets stay independent of stats coverage. Their projection is included in the same reconcile, even with field stats disabled.

Limits

• Field stats describe the whole namespace. They take no filter; for counts within a filter use a values scan.
• GET /v1/namespaces/{ns}/schema stays a verbatim proxy. Stats are on metadata only.
• Field stats need Aerospike and S3, so they are a Pro feature.

List namespaces

GET /v2/namespaces is a Layer-only augmented listing. It pages the upstream namespace list and enriches each row with stability and cache signals. It is the endpoint the dashboard’s inventory view reads.

namespaces = await client.list_namespaces(prefix="prod", page_size=100)

namespaces, err := client.ListNamespaces(ctx, &hevlayer.ListNamespacesParams{
    Prefix:   "prod",
    PageSize: 100,
})

const namespaces = await client.listNamespaces({
  prefix: "prod",
  pageSize: 100,
});

curl "$LAYER_GATEWAY_URL/v2/namespaces?prefix=prod&page_size=100" \
  -H "Authorization: Bearer $LAYER_GATEWAY_API_KEY"

{
  "namespaces": [
    {
      "name": "products",
      "row_count": 12500,
      "size_bytes": 48800000,
      "stable_as_of_ms": 1715600400000,
      "is_stable": true,
      "index": { "status": "up-to-date" },
      "cache_state": {"state": "warm", "warm_inflight": false},
      "last_write_ms": 1715600399000,
      "shadow": false,
      "labels": {},
      "title": "Products",
      "description": "Shop catalog, one row per product."
    }
  ],
  "next_cursor": "..."
}

title and description come from the namespace’s Index and are omitted when unset, so a client shows title ?? name. They are present even when metadata_error is.

Each row carries freshness signals derived from that row’s metadata fetch. is_stable is true when index.status is "up-to-date", false when it is "updating", and omitted when metadata has no index signal or the fetch failed. stable_as_of_ms is set to the metadata observation time for rows reported up to date.

indexed and index_lag_rows live on GET /v2/namespaces/{namespace}/metadata, where the gateway can do the snapshot lookup for one namespace without adding object-store reads to the high-fanout list path.

index is turbopuffer’s indexing state, passed through verbatim:

Listing is read-only and does not register namespaces with the consistency watcher. Write traffic and snapshot facet configuration register the namespaces that need durable watermarks.

A per-row metadata failure degrades to a row with metadata_error set rather than dropping the namespace, so the list stays complete even when a single namespace’s metadata call fails. Responses are served from a short-TTL cache (NAMESPACE_LIST_CACHE_TTL_MS, default 10000) so dashboard polling does not fan out a metadata call per namespace per refresh.