Functions

A Function is a User Defined Function (UDF) that runs over rows that already exist in an Index. It is the right shape for classifiers, enrichment, backfills, fan-out from an existing row, and deterministic re-upserts. UDFs are best defined in YAML and invoked by the layer CLI. The operator creates worker resources; the gateway owns discovery, queueing, retries, leases, and completion markers. Workers own their data writes. Discovery-triggered Functions scan existing rows after registration, then repeat on schedule.discoveryIntervalSeconds. Changing spec.version makes discovery due again so rows with older completion markers are queued.

POST /v2/udfs/{id}/discover processes at most max_pages source pages across all requested namespaces (default 10, range 1–100). page_size limits each page (default 1,000, range 1–1,000). Unfinished namespaces take turns so a large namespace cannot keep smaller ones out of discovery. The response includes pages_scanned, complete, and an opaque next_cursor. Pass next_cursor as cursor to continue. A stale cursor returns HTTP 409; omit it to resume the latest committed progress. Gateway replacement preserves progress in Postgres. Changing the Function version starts discovery from the beginning.

A checkpoint advances after canonical source observation and queue insertion commit. Interrupted pages can be replayed; queue and receipt fences protect against unsafe acknowledgment. A completed sweep records traversal, not a snapshot census of a changing source. Source-write triggers and subsequent sweeps recover changes behind the cursor. Claims rotate among pending namespaces, including when workers request one item at a time. Claim limit is 1–1,000; selection considers at most 64 pending namespaces per batch.

Paused Functions do not scan or claim work. The operator copies gateway active_namespaces and pending_count into Function status activeNamespaces and queueDepth.

Use a Pipeline when external data becomes rows in Layer. Use a Function when compute starts from rows that are already in Layer.

apiVersion: hevlayer.com/v1alpha1
kind: Function
metadata:
  name: tag-products
  namespace: layer
spec:
  targetNamespaces:
    - products
  inputs:
    - id
    - title
  version: v1
  filter:
    - category
    - Eq
    - outdoor
  worker:
    image: <acct>.dkr.ecr.us-east-1.amazonaws.com/hev-tag-products:latest
    dispatch: pull
    computeClass: cpu
    batchSize: 32
    timeoutSeconds: 30
  schedule:
    discoveryIntervalSeconds: 300
    leaseSeconds: 120
    maxInFlightBatches: 8
    maxConcurrentScans: 1
  retry:
    maxAttempts: 8
    initialBackoffSeconds: 5
    maxBackoffSeconds: 300
  triggers:
    - discovery
  scaling:
    pool: cpu
    mode: autoscale
    replicas:
      min: 0
      max: 6

Selection

Use targetNamespaces for explicit namespaces. Use indexSelector when labels on Index resources should choose the namespaces.

Workers claim only from the namespaces the Function currently targets. When you narrow targetNamespaces, queue rows already pending in a namespace you removed stay queued and are not claimed (nothing is deleted); adding the namespace back makes them claimable again. A Function that uses indexSelector has no fixed list and claims from the namespaces the selector chose.

filter preserves arbitrary JSON, including array-form turbopuffer filters. The operator stores the shape as-is; the gateway evaluates it during discovery after AND-ing it with the generated completion-marker predicate. Do not include a version-marker predicate in filter; the gateway creates that from spec.version.

Worker

To apply the CR, register the gateway UDF, trigger discovery, and watch the queue with one command, use layer run -f.

The worker pod receives HEVLAYER_UDF_ID, HEVLAYER_BASE_URL, HEVLAYER_UDF_BATCH_SIZE, HEVLAYER_UDF_TIMEOUT_SECONDS, HEVLAYER_UDF_LEASE_SECONDS, and LAYER_GATEWAY_API_KEY. The gateway bearer is sourced from the default VectorStore credential in deriveFromStore mode, or from the configured inbound worker key in keys mode.

Simple classifier

The Python client turns a normal function into the claim/process/complete loop. output="tags" is client-side metadata: the resource does not declare an output attribute. run_udf_worker sends the returned value as a completion attributes.tags patch, and the gateway stamps the reserved completion marker in the same patch. The Go client drives the same worker protocol directly, as does the TypeScript client — claim a batch, process rows, report completions and failures.

import asyncio
from hevlayer.udf import PermanentError, TransientError, run_udf_worker, udf

@udf(inputs=["id", "title", "description"], output="tags", kind="tags")
def tag_product(*, id: str, title: str | None, description: str | None) -> list[str]:
    if not title:
        raise PermanentError(f"{id}: missing title")
    try:
        text = f"{title} {description or ''}".lower()
    except TypeError as exc:
        raise TransientError(str(exc)) from exc

    tags: list[str] = []
    if "wireless" in text:
        tags.append("wireless")
    if "waterproof" in text:
        tags.append("waterproof")
    return tags or ["uncategorized"]

if __name__ == "__main__":
    asyncio.run(run_udf_worker(tag_product, udf_id="product-tags"))

package main

import (
	"context"
	"os"
	"strings"

	hevlayer "github.com/hev/layer-go"
)

func tags(title, description string) []string {
	text := strings.ToLower(title + " " + description)
	var out []string
	if strings.Contains(text, "wireless") {
		out = append(out, "wireless")
	}
	if strings.Contains(text, "waterproof") {
		out = append(out, "waterproof")
	}
	if len(out) == 0 {
		out = []string{"uncategorized"}
	}
	return out
}

func main() {
	ctx := context.Background()
	udfID := os.Getenv("HEVLAYER_UDF_ID")
	layer := hevlayer.NewClient(
		hevlayer.WithBaseURL(os.Getenv("HEVLAYER_BASE_URL")),
		hevlayer.WithAPIKey(os.Getenv("LAYER_GATEWAY_API_KEY")),
	)

	for {
		claimed, err := layer.ClaimUdfItems(ctx, udfID, &hevlayer.UdfClaimRequest{
			WorkerID: "tag-products-0",
			Limit:    32,
		})
		if err != nil {
			continue
		}
		var done []hevlayer.UdfCompleteItem
		var failed []hevlayer.UdfFailItem
		for _, item := range claimed.Items {
			title, _ := item.Input["title"].(string)
			description, _ := item.Input["description"].(string)
			if title == "" {
				failed = append(failed, hevlayer.UdfFailItem{
					Namespace: item.Namespace, ID: item.ID,
					Kind: "permanent", Message: "missing title",
				})
				continue
			}
			done = append(done, hevlayer.UdfCompleteItem{
				Namespace:  item.Namespace,
				ID:         item.ID,
				Attributes: map[string]interface{}{"tags": tags(title, description)},
			})
		}
		if len(done) > 0 {
			layer.CompleteUdfItems(ctx, udfID, &hevlayer.UdfCompleteRequest{
				WorkerID: "tag-products-0", Items: done,
			})
		}
		if len(failed) > 0 {
			layer.FailUdfItems(ctx, udfID, &hevlayer.UdfFailRequest{
				WorkerID: "tag-products-0", Items: failed,
			})
		}
	}
}

import { Hevlayer } from "hevlayer";

function tags(title: string, description: string): string[] {
  const text = `${title} ${description}`.toLowerCase();
  const out: string[] = [];
  if (text.includes("wireless")) out.push("wireless");
  if (text.includes("waterproof")) out.push("waterproof");
  return out.length ? out : ["uncategorized"];
}

const udfId = process.env.HEVLAYER_UDF_ID!;
const layer = new Hevlayer({
  baseUrl: process.env.HEVLAYER_BASE_URL,
  apiKey: process.env.LAYER_GATEWAY_API_KEY,
});

while (true) {
  const claimed = await layer.claimUdfItems(udfId, {
    worker_id: "tag-products-0",
    limit: 32,
  });
  const done = [];
  const failed = [];
  for (const item of claimed.items) {
    const title = typeof item.input.title === "string" ? item.input.title : "";
    const description = typeof item.input.description === "string" ? item.input.description : "";
    if (!title) {
      failed.push({
        namespace: item.namespace,
        id: item.id,
        kind: "permanent",
        message: "missing title",
      });
      continue;
    }
    done.push({
      namespace: item.namespace,
      id: item.id,
      attributes: { tags: tags(title, description) },
    });
  }
  if (done.length > 0) {
    await layer.completeUdfItems(udfId, { worker_id: "tag-products-0", items: done });
  }
  if (failed.length > 0) {
    await layer.failUdfItems(udfId, { worker_id: "tag-products-0", items: failed });
  }
}

In Python, function parameters are keyword-only and named to match inputs; raise TransientError for retryable work and PermanentError for unrecoverable input. In Go and TypeScript, report the same split through FailUdfItems / failUdfItems with kind: "transient" or kind: "permanent".

Backfill original-file references

layer-extract originals is a stock Function worker. With groupByParent: true it claims whole documents whose page rows have an original_sha256 and no original_blob, stores the cached original in the namespace’s private blob storage, and completes every page with just original_blob and original_bytes. It is row-preserving and idempotent. It does not check the blob set before storing: that check is a query billed at the size of every original already stored. It downloads, hash-checks and stores by hash instead, so a repeated document is stored again with the same bytes and the same reference. Only a row with no URL at all is looked up by hash, and never during a backfill. Bytes come from a sealed manifest of cached originals, or through the clouddrive connector using the Warehouse you mount on the pod (podSpec, container udf). Rows indexed before original_* existed are matched by their webUrl and attached only if the downloaded bytes hash to the row’s own content_hash. A document whose only copy came from a rest Warehouse is fetched through that Warehouse’s connector instead (declare LAYER_ORIGINALS_REST and mount its credential on the pod); select legacy rows with content_hash NotEq null, which every page row satisfies. Pipelines retain originals by default (extract.retainOriginals: false opts out) and write the same attributes at ingest, so the Function is only for rows that predate it. See Original files.

GPU classifier

More complicated classifiers (e.g. a vision-language classifier) may require a model to run on a GPU. The shop source provides a product-color classifier example.

apiVersion: hevlayer.com/v1alpha1
kind: Function
metadata:
  name: product-color
  namespace: layer
spec:
  targetNamespaces:
    - amazon-products
  inputs:
    - id
    - image_url
  version: v1
  worker:
    image: <acct>.dkr.ecr.us-east-1.amazonaws.com/hev-shop-udf-product-color:latest
    dispatch: pull
    computeClass: gpu
    batchSize: 8
    timeoutSeconds: 120
  schedule:
    leaseSeconds: 300
    maxInFlightBatches: 2
  triggers:
    - discovery
  scaling:
    pool: gpu
    mode: autoscale
    replicas:
      min: 0
      max: 2

worker.computeClass: gpu defaults omitted scaling.pool to the built-in gpu pool. The stock pool selects layer.hev.dev/node-role=worker-gpu, requests one NVIDIA GPU, and carries the worker and NVIDIA tolerations:

computePools:
  - name: gpu
    kind: gpu
    maxReplicasPerWorkload: 4
    nodeSelector:
      layer.hev.dev/node-role: worker-gpu
      layer.hev.dev/compute: gpu
    tolerations:
      - key: layer.hev.dev/node-role
        operator: Equal
        value: worker-gpu
        effect: NoSchedule
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
    resources:
      requests: { memory: 4Gi, nvidia.com/gpu: "1" }
      limits: { memory: 10Gi, nvidia.com/gpu: "1" }

The worker loads the model once at startup and classifies per row. CLIP zero-shot classification labels each product image with its dominant color:

import asyncio
import io

import httpx
import torch
from PIL import Image
from transformers import pipeline

from hevlayer.udf import PermanentError, TransientError, run_udf_worker, udf

COLORS = ["black", "white", "gray", "red", "blue", "green", "brown", "multicolor"]

classifier = pipeline(
    "zero-shot-image-classification",
    model="openai/clip-vit-large-patch14",
    device="cuda" if torch.cuda.is_available() else "cpu",
)

@udf(inputs=["id", "image_url"], output="color", kind="classification")
def classify_color(*, id: str, image_url: str | None) -> str:
    if not image_url:
        raise PermanentError(f"{id}: missing image_url")
    try:
        resp = httpx.get(image_url, timeout=10.0, follow_redirects=True)
        resp.raise_for_status()
        image = Image.open(io.BytesIO(resp.content)).convert("RGB")
    except httpx.HTTPError as exc:
        raise TransientError(f"{id}: image fetch failed: {exc}") from exc
    except OSError as exc:
        raise PermanentError(f"{id}: undecodable image: {exc}") from exc

    scores = classifier(image, candidate_labels=COLORS)
    return scores[0]["label"]

if __name__ == "__main__":
    asyncio.run(run_udf_worker(classify_color, udf_id="product-color"))

The worker image needs torch, transformers, pillow, and httpx alongside the hevlayer Python client. Bake the model weights into the image so autoscaled pods do not re-download them on every cold start.

Sizing for inference: keep worker.batchSize low and worker.timeoutSeconds high enough for one batch of forward passes, and make schedule.leaseSeconds outlast a full batch so claims do not reissue mid-inference. replicas.min: 1 keeps a warm worker when model cold-start dominates; min: 0 scales to zero between sweeps.

Document grouping

Rows written by PDF extraction are one row per page, with ids like report#p3 and a shared _hevlayer_parent_id. A classifier over the whole document should run once, not once per page. Set spec.groupByParent: true and the gateway claims by document:

spec:
  targetNamespaces:
    - pages
  inputs:
    - id
    - text
  groupByParent: true

• A claim returns one item per document. The item’s id and input are the first page by id, and pages lists every page of the document, including that one, each with its own id, input_revision and input.
• The gateway claims the sibling pages for the same worker, whether or not discovery has queued them yet, so no other worker can take a page of a document that is in flight. The claim limit counts the rows picked from the queue, not the pages added with them.
• Echo pages (id and input_revision per page) on the completion item. The gateway writes the same attributes (and vector, if any) plus the version marker to every page row. Each page is fenced by its own revision, so a page that changed during the run is requeued like any stale claim.
• Echo pages on a failure item and on a heartbeat item to fail or renew every page of the document together.
• Once every page carries the marker, discovery does not queue the document again. Bumping spec.version, or changing a page’s declared inputs, runs the whole document again.
• Rows without _hevlayer_parent_id behave as they do without the option: one item per row, no pages.

A document groups at most 10,000 pages.

In Python, run_udf_worker echoes pages for you. Declare a pages parameter to receive the input of every page, in id order:

@udf(inputs=["id", "text"], output="doc_type", kind="classification")
def classify_document(*, id: str, text: str | None, pages: list[dict]) -> str:
    return classify("\n".join(page["text"] or "" for page in pages))

In Go, TypeScript and REST, copy the claimed item’s pages entries into the completion, failure or heartbeat item.

Scaling

spec.scaling is the same scaling config Pipelines use: a pool from the built-in set or InfraRules/default, a mode, and replica bounds. For Functions, mode: autoscale emits a KEDA ScaledObject triggered by layer_udf_queue_depth. Replica maxima above the pool’s maxReplicasPerWorkload are rejected in status.

For GPU Functions on a scale-to-zero pool, set spec.scaling.warmWindowSeconds to hold the worker — and its node — warm for a cooldown after the queue drains, so adjacent batches skip the cold start (fresh node, image pull, model load) before the pool returns to zero. See Workload scaling.

Writeback

Workers own data writes. The common single-attribute case uses the Python client’s sugar: @udf(output="tags") makes run_udf_worker send returned values as attributes.tags in the completion call — in Go (or over REST) the same thing is attributes on each completion item. The gateway applies those attributes and the reserved completion marker in one patch_columns write. Completion attributes must not use the reserved _hevlayer_* prefix.

Embedding Functions can include vector on each completion item. The Python helper emits vector for @udf(kind="embedding") return values. The gateway fetches the existing row, merges returned attributes and reserved markers, then re-upserts the full row with the replacement vector.

Python workers that need more control can declare the tpuf parameter, write through the client, and return None; completion then stamps only the marker. Use deterministic IDs when a Function creates rows so at-least-once retries remain idempotent.

Deleting a Function garbage-collects operator-managed Kubernetes resources. It does not delete already-written attributes.

Lifecycle

kubectl get function product-tags
kubectl describe function product-tags

layer udf get product-tags

kubectl patch function product-tags --type=merge -p '{"spec":{"paused":true}}'
kubectl patch function product-tags --type=merge -p '{"spec":{"paused":false}}'

curl -X POST -H "authorization: Bearer $LAYER_GATEWAY_API_KEY" \
  $LAYER_GATEWAY_URL/v2/udfs/product-tags/reset-failed

kubectl delete function product-tags

Registration in the gateway’s UDF registry happens at reconciliation, not at first discovery run — a Function created with spec.paused: true (or paused later) is registered immediately with paused: true, so it is observable from creation onward. API callers can pass paused: true alongside id and spec to POST /v2/udfs to register atomically without starting discovery:

When the Function spec changes, reconciliation upserts the registered UDF definition in place. Pending, processing, failed, and indexed queue state remain attached to the same UDF id.

GET /v2/udfs
  → 200 {"udfs": [{"id": "product-tags", "paused": true, ...}]}

GET /v2/udfs/product-tags/status
  → 200 {"udf_id": "product-tags", "paused": true, ...}

A 404 from /v2/udfs/{id}/status means the Function was never registered — a real failure, not an intentional pause. paused on the Udf and UdfStatus resources is the single source of truth for “is this installed and paused” versus “does this exist at all.”

Version markers

spec.version is the re-run safety rail and defaults to v1. On completion, the gateway stamps _hevlayer_udf_<function>_v with that version, normalizing hyphens in the Function name to underscores. For metadata.name: product-color, the marker is _hevlayer_udf_product_color_v.

Discovery automatically looks for rows whose marker is missing, differs from spec.version, or has an expired _hevlayer_udf_<function>_stale_after marker. Bump spec.version when a model, taxonomy, or prompt changes. To check the change on live data before it overwrites outputs, run it as a candidate.

Candidate runs

Test a changed prompt, taxonomy or model against live data before it replaces the live output. A candidate run is derived from the live Function, so you do not write a second Function. Candidate runs use the gateway API; the Function resource has no candidate field.

1. Register the candidate. Name the output attributes it produces and the new version. Anything else (worker, inputs, filter, target_namespaces) overrides the live value only if you set it.

   curl -X POST -H "authorization: Bearer $LAYER_GATEWAY_API_KEY" \
     -H "content-type: application/json" \
     $LAYER_GATEWAY_URL/v2/udfs/doc-type/candidate \
     -d '{"version":"v2","outputs":["doc_type"],
          "worker":{"image":"<acct>.dkr.ecr.us-east-1.amazonaws.com/doc-type:v2"}}'

   This registers doc-type-candidate. Discovery, claims, retries and status work as for any Function (GET /v2/udfs/doc-type-candidate/status). Run a worker for it with HEVLAYER_UDF_ID=doc-type-candidate. The worker returns the same attribute names as the live Function. The gateway writes them to _hevlayer_cand_<function>_<output> and stamps a separate completion marker. The live output and live marker never change. A candidate completion may set only attributes listed in outputs and cannot write vectors.

2. Compare. The report covers rows the candidate has completed.

   curl -H "authorization: Bearer $LAYER_GATEWAY_API_KEY" \
     "$LAYER_GATEWAY_URL/v2/udfs/doc-type/candidate/compare?sample=5"

   {
     "rows_compared": 3, "rows_agreed": 2, "agreement_rate": 0.667,
     "outputs": [{
       "output": "doc_type", "compared": 3, "agreed": 2, "agreement_rate": 0.667,
       "confusion": {"invoice": {"invoice": 2, "receipt": 1}}
     }],
     "disagreements": [{"namespace": "docs", "id": "c", "output": "doc_type",
                        "live": "invoice", "candidate": "receipt"}]
   }

   confusion counts rows by live value, then candidate value, for outputs whose values are strings, numbers or booleans. A missing value counts as null. Array outputs, such as tags, compare without regard to order and have no confusion table. max_rows (default 10,000, maximum 100,000) bounds the scan; truncated is true when it stopped early. The report also carries the candidate’s pending_count, processing_count and failed_count so you can tell a finished run from a partial one.

3. Promote, or discard.

   curl -X POST -H "authorization: Bearer $LAYER_GATEWAY_API_KEY" \
     -H "content-type: application/json" \
     $LAYER_GATEWAY_URL/v2/udfs/doc-type/candidate/promote -d '{}'
   
   curl -X DELETE -H "authorization: Bearer $LAYER_GATEWAY_API_KEY" \
     $LAYER_GATEWAY_URL/v2/udfs/doc-type/candidate

   Promote copies each shadow value into the live output, stamps the live marker with the candidate version, clears the shadow attributes, and updates the live Function’s version, worker, inputs, filter and target_namespaces to the candidate’s. It then deletes the candidate. The live Function’s invalidates fire as for any completion. Promote returns HTTP 409 while the candidate has pending or processing rows; pass "allow_partial": true to promote what is finished. Rows the candidate did not finish keep the old marker, so the live Function reprocesses them at the new version. Discard clears the shadow attributes and deletes the candidate without touching live output.

For a Function managed by a Function CR, update the CR’s spec.version and spec.worker to the candidate’s values after promoting. Otherwise the next reconcile upserts the old definition, and discovery reprocesses every row.

Tuning knobs

Gateway rollouts

The gateway is one pod (embedded Postgres holds the Function queues, so the chart refuses gateway.replicas above 1). Rolling it out takes the gateway away for tens of seconds. A Function worker waits that out instead of exiting: the work it holds and the work still queued both carry on once the new pod answers, with no relaunch.

The worker image’s gateway client treats these as “not here yet” and polls with backoff (1s doubling to 10s) for up to HEVLAYER_GATEWAY_OUTAGE_SECONDS (default 240, kept under the 300s default lease; 0 never waits):

Credential denial, throttling and spend guards are answers from a running gateway, so they still stop the worker’s run immediately; only silence is waited out. A wait that outlasts the window ends the call with an error, as before, and the worker restarts with its durable claim ledger and hold marker intact.

Completions stay fenced by the echoed input_revision and the worker’s lease, so a completion that arrives after the lease expired is a conflict (see below), not a duplicate write.

Completion conflicts

Workers can set report_dispositions: true on completeUdfItems to receive HTTP 200 with updated and an items array in request order. Each outcome has an index (the submitted item position), a disposition, and a bounded reason for conflicts. Check every outcome; HTTP 200 alone is not full success.

Safe stale recovery is part of completion: it fences the echoed input revision and worker owner and never requeues a claim with an unacknowledged prepared receipt. It does not require resetting failures or retrying a pipeline. Expired claims cannot prepare new output. Acquisition fences earlier receipts, even when a worker name is reused.

Without report_dispositions, completion retains batch conflict HTTP 409 and adds bounded reason and disposition fields. Store failures remain non-200 and can occur after earlier items in a batch succeeded. Retry the identical request after a transient transport/store failure; prepared receipts prevent replacement output and recognize already acknowledged completions. Do not convert an unknown store error into success or replay changed output.

Claims with a durable unacknowledged completion include prepared_receipt: its original input_revision, intent_digest and output_digest. These are protected digest-only evidence; the gateway does not retain output values. A worker recovering a durable prepared completion submits the current claim’s input_revision and the original receipt’s prepared_input_revision, with exactly the original attributes and vectors. This explicitly requests adoption. The gateway checks the strong current source, retained source-change fence, original output intent, current owner and live lease before transferring the receipt to the new generation. Identical repeated calls are safe; ACK counters advance once. Changed source, changed output, absent proof or conflicting ownership preserves the receipt and stops recovery. Never recompute a timestamp or inference result to replace an uncertain prepared output.

Source epochs fence changes and absence observed through the gateway. They do not prove an unobserved delete-and-recreate performed directly in the store. Recovery requires authoritative current input reads; uncertain source history must remain stopped. Legacy receipts lacking retained source proof are preserved and rejected, even if their current output looks plausible.

Discovery cost bounds

Scheduled discovery spends at most ten pages per chunk and waits discoveryIntervalSeconds before continuing its durable cursor. A chunk that reaches the page budget is incomplete; it is not a failed sweep. Gateway restarts resume committed progress. Failed chunks use exponential delays capped at one hour. A pause stops a running chunk at the next page boundary. That is not a failure: the chunk ends without an error (a manual POST …/discover returns 200 with complete: false), and the failure count is unchanged. Eight consecutive failures pause the Function. Its Ready condition reports DiscoveryAutoPaused with recovery guidance. Check target namespaces, filter schema and upstream availability, then correct the Function definition or explicitly resume it. Manually paused Functions remain paused.

Discovery sends the stale version marker predicate together with the user filter to the store, including rows with missing markers. Gateway-observed changed inputs are recovered from durable input revisions. For stores exposing valid namespace creation and modification timestamps, completed unchanged targets need one metadata request and no row scan. Approximate row counts and byte counts are not change detectors. Stores without reliable timestamps continue bounded filtered scans. Namespace writes bypassing the gateway must clear the affected Function marker when they change a Function input.

Canonical document read cache

The runtime Function read cache is disabled by default. Pro startup can compose the shared cache from an existing reviewed authority configuration, as described below. LAYER_FUNCTION_READ_CACHE=exclusive-writer does not enable it: process-local invalidation cannot fence an abandoned provider request that commits later, even with one gateway. Without configured authority, canonical reads use the store. Multi-replica cache activation requires durable mutation fencing and an enforced upstream writer credential boundary; neither a topology assumption nor this old environment setting establishes that boundary.

The following certificate behavior describes the source cache implementation; it is not an enabled production consistency guarantee.

Canonical ID reads use a separate cache of complete, committed attribute rows. Projected scans and the ordinary document cache do not certify a canonical row. A cache miss makes one strong ID query for the missing IDs in the batch; confirmed missing IDs are cached as absent. Cache errors fall back to the store. When the retained shared fence certifies namespace ID type and generated columns, the strong miss query reuses that identity without another metadata request. Missing identity or schema/delete epoch invalidation requires fresh strong metadata; a process-local metadata TTL does not certify it. Shared certificates are scoped by the canonical provider/writer domain and full namespace identity. A row’s cache key also includes its durable namespace epoch, document revision and ID; sibling membership includes the namespace revision. The original shared fence is retained and validated around provider and cache work. Mutations record durable pending state and advance affected versions before provider dispatch; successful certificate publication precedes exact-token completion. An abandoned or uncertain write cannot become readable merely because a later write succeeds.

The Aerospike adapter gives rebuildable certificates a one-hour TTL. Expiry or cache loss causes a miss; it does not remove durable pending or poisoned state. A new gateway process can reuse shared certificates only after validating the same durable identity, independent witness and retained fence. Restart does not itself create a fresh cache scope or certify rows. Missing or mismatched authority prevents certificate use and selects strong uncached reads; loss of an original fence fails the protected operation. Pending or poison prevents cache reuse, and shared-mode mutations are rejected when their required authority is unavailable. If native embedding produces attributes absent from the write body, the client requires a complete upstream read rather than certifying an incomplete row. Conditional and filtered mutations invalidate their certificate identity. A cache miss never becomes a partial row.

Claim and completion read each namespace’s IDs together. A write containing both upserts and patches performs one post-write canonical read for their combined IDs. Cache hits retain all Function input columns and completion markers, even when a worker later projects the returned attributes.

Parent and sibling lookup for workers

Replace the worker’s separate page-two and parent-filter queries with POST /v2/udfs/{function_id}/items/lookup. Use the same authenticated gateway and the current claim’s worker_id, namespace, id and input_revision:

lookup = await client.lookup_udf_siblings(
    function_id,
    {
        "worker_id": worker_id,
        "namespace": item["namespace"],
        "id": item["id"],
        "input_revision": item["input_revision"],
        "include_attributes": ["page", "text"],
    },
).model_dump()
siblings = lookup["documents"]
page_two = next(
    (row for row in siblings if row["id"] == lookup["page_two_id"]), None
)

client here is the worker’s authenticated HTTP client, with the gateway base URL already configured. The response contains udf_id, namespace, parent_id, page_two_id (null when absent), and documents, each with id and attributes. IDs and page_two_id are returned independently of the attribute projection. The parent comes from the claimed source’s reserved _hevlayer_parent_id; the caller cannot substitute another parent. The lookup requires a live owned claim and rejects a stale revision with HTTP 409. page_two_id identifies a sibling whose numeric page attribute is 2.

The first lookup of an uncached parent performs one strong provider query and certifies complete membership and full rows. Repeated lookups use the cache. Source mutations invalidate membership before dispatch. Cache eviction or failure causes a fresh strong lookup. Groups with 10,000 or more siblings fail instead of returning incomplete membership. This endpoint does not silently fall back to a full namespace scan for another store kind.

For a backfill Function, the lookup makes no provider read. It answers from the sibling groups the backfill scan holds, and returns only the Function’s inputs.

Per-item query budget

Set spec.queries_per_item in the gateway API or spec.queriesPerItem on the Function resource to limit provider query attempts. Omit it for accounting without a cap; zero allows only cache hits. Status exposes query_budget with queries_per_item, provider_queries, rejected_queries, exhausted_items, and discovery_queries.

The ledger is keyed by Function ID, Function version, namespace and source ID. A batch query charges one attempt to every participating item, once per unique ID, so provider_queries sums item charges, not physical queries. Reads that a gateway write makes for write-triggered Functions are charged to those Functions, never to a backfill. hevlayer_function_provider_queries_total and hevlayer_function_billable_bytes_queried_total (labels function, stage) count each physical read and its billed bytes once per Function. A multi-query charges its number of query legs. Reservations commit before dispatch: errors, 429s, cancellation and transport failures retain the charge. Cache hits and empty ID lists cost zero. Retries, lease recovery, completion replay, failed-item resets and gateway restart retain the ledger. A version change starts a new budget; increasing the cap allows work to continue. Source edits alone do not reset the budget for an existing ID.

Claim hydration, canonical write-trigger reads, completion reads and the claim-bound lookup all reserve queries. Discovery scans appear as discovery_queries and in provider_queries as namespace overhead; they are not subject to an individual item’s cap. The provider’s fetch_many and scan_page billing remains visible in the billing metrics. provider_queries is the sum of item charges and discovery overhead, so shared batch queries can contribute more than one item charge. Metadata calls and writes retain their existing billing attribution and are outside this query-attempt cap. Workers must use the claim-bound lookup for document reads: arbitrary queries made outside these Function operations cannot be associated with an item and are outside its budget.

An exhausted reservation returns HTTP 409 before contacting Turbopuffer. Completion keeps its existing revision, source and receipt fences; a budget error does not acknowledge work or discard a prepared receipt.

Real-namespace measurement and certified-cache activation remain held. Before activation, an operator must enforce one canonical provider/writer identity across every registered dispatcher, prohibit endpoint aliases and bypass credentials, and verify the shared namespace ledger under its original PostgreSQL session guards against the independent object-store witness. Registration alone does not establish this writer boundary. Restoring or replacing authority requires fencing all old dispatchers and bypass writers, proving every accepted provider request terminal, and rotating the writer binding and witness incarnation while retaining old history. Production drain verification, initial provisioning and gateway configuration remain unavailable; multiple gateway processes alone do not establish safe cache coherence.

Only after these prerequisites and separate activation authorization, verify document-cache configuration and Aerospike readiness. Warm certified canonical rows through claim hydration, a parent lookup, or confirmed unconditional writes under the retained shared fences. Ordinary cache scans and public document reads can use projected/eventual cache rows and do not establish this prerequisite. Use the lookup API in the worker and pin gateway and worker revisions. Record provider query/billing deltas for namespace metadata, the write trigger, claim, sibling lookup, fan-out patches and completion, including misses and retries. A synthetic provider-call regression is not a measurement of the BCC namespace. Keep all Function pauses until the specific cheap-read acceptance and an explicit runtime handoff authorize a bounded measurement.

Shared Function-cache startup

Pro gateways accept LAYER_FUNCTION_READ_CACHE_CONFIG, a path to a reviewed JSON configuration. The Helm value gateway.functionReadCache.configMapName mounts an existing reviewed ConfigMap with key cache.json read-only and sets that path; its default is empty. The chart does not create authority or review inputs. The obsolete LAYER_FUNCTION_READ_CACHE flag does not enable caching. This composition is independent of field-stats capture; configure one writer authority, not both. Every configured Turbopuffer alias must use the exact same endpoint and static credential generation, and all aliases receive the same fenced HTTP client. Unreviewed aliases refuse startup.

Required fields are store_ref, writer_domain, writer_boundary_evidence, terminal_drain_evidence, witness_prefix, witness_incarnation, witness_writer_binding, max_namespaces, max_generations, versions_per_namespace, and key_bytes. Quotas have no defaults. Startup validates the existing independent S3 witness and PG ledger policy; it never creates or repairs authority. Aerospike, S3 and the transform control pool must be configured. Witness binding must match the actual endpoint/key-bound client. Missing configuration leaves reads strongly uncached. Lost authority, pending writes, poison, or failed guard validation cannot certify cache rows.

Evidence references identify operator-reviewed records; strings alone do not enforce credentials or prove a terminal drain. Before adoption, the runtime coordinator must establish the canonical all-writer boundary and provision the existing PG policy and independent witness using a proved drain. No production reset, credential change, or activation follows from this source configuration. The measurement owner runs the bounded warm-document acceptance only after that handoff.

Certified-cache recovery boundary

Certified reads remain disabled without an enforced canonical writer boundary and independent witness configuration. The object-store witness uses conditional versions independent of PostgreSQL restore. Missing authority, mismatched versions, pending writes or poison cannot be repaired by replaying a database snapshot: reads fall back to the strong provider path and shared-mode mutations are rejected.

A recovery controller must fence every old dispatcher and bypass writer, prove all accepted provider requests terminal, then rotate to a fresh writer binding and witness incarnation while retaining old history. Cancellation, key revocation and elapsed time alone are insufficient. The source adapter supplies checkpoint and frozen-root rotation primitives; production drain verification and initial provisioning are not available yet; gateway configuration only validates existing authority. No automatic seed/reset/reseed or runtime activation is provided. Analytics caches are replaceable projections and cannot supply this proof.

The source-only certified-capture primitive can hold a bounded set of namespace fences around strong, separately metered reads. It groups physical advisory-lock collisions without merging logical namespace certificates. Any pending write, poison, witness mismatch or session loss invalidates the whole capture. This is not an immutable analytics snapshot API; consumers must hold all Function locks first and publish nothing after a failed capture. Service validation and production recovery/boundary configuration remain prerequisites.

A source-only opaque capture receipt can bind completed strong-scan coverage to its canonical provider and namespaces, independent witness incarnation and readable generation. Comparison requires the original validated capture guards; missing proof or mismatched coverage cannot authorize skipping a scan. Receipts belong beside trusted, complete derived results and cannot be supplied as client proof. This API does not activate production capture or replace the required writer/drain enforcement.

Administrative authority preparation

The layer-function-cache-authority tool prepares a create-only, inactive witness root and an explicit empty PG policy. Preparation does not establish terminal provider drain or enable cached reads. Activation requires a real verifier bound to the persisted request; an existing root or correctness state is never reset. Turbopuffer read-only metadata is not a transactional write barrier, so it cannot resolve unknown old writes. The platform execution owner must obtain that evidence and enforce the all-writer boundary before adopting the optional configuration.

The tool’s request command retrieves the original inactive bootstrap manifest for a provider capability inquiry. The reviewed public Turbopuffer API does not establish a terminal legacy-write drain operation. Keep existing namespaces strongly uncached without that proof. An independent namespace branch can isolate later source commits, but migration still requires an enforced destination writer boundary, an approved data cut and spend; it is not an automatic bootstrap.

Abandoned scan reservations

The financial scan ledger has a separate operator-only recovery tool: layer-scan-budget-recovery settle-dead PRIVATE_PLAN.json. It requires retained attempt-to-owner linkage, operator-reviewed terminal runtime evidence, and a fresh Kubernetes inventory proving that the old pod UID is absent. An API object being absent alone is not proof of process death.

Recovery charges every unknown page at its full reservation, without inventing observed billing or refunding spend. The audit is retained and a late higher provider receipt reopens debt. Completion starts a fresh 24-hour retention window; it does not certify scan completeness or publish field stats. Existing legacy attempts lack durable owner bindings, so startup cannot automatically recover them. New scan attempts bind to an immutable original PostgreSQL-session incarnation. On session loss, canonical ledger transactions revoke that incarnation before conservative settlement; live sessions remain protected. A revoked client cannot reserve or complete again, while late billing can still raise debt. This fences financial admission without proving process death or provider drain. See docs/runbooks/scan-budget-recovery.md for the private plan and direct-session requirements.

An ownerless legacy scan can also be financially fenced with the administrative layer-scan-budget-recovery fence-legacy tool. This closes that exact attempt to new page admission without claiming that its old process died. Existing reservations remain fully charged, missing billing remains explicitly estimated, and late excess remains debt. It does not authorize cache activation or certify a source snapshot. See the scan-budget recovery runbook for the exact-match procedure.

Bulk backfill

spec.backfill: true runs the Function once over rows that already exist, reading inputs in paged scans of 9,999 rows and making no per-document provider read at claim, completion or write. It requires triggers: [discovery], writes attributes only, and reports backfill progress and billed bytes in Function status. The Function CRD and the layer-operator must both be from layer-pro 3dcdeee6 or later: an older CRD prunes the field and an older operator drops it when registering the Function, which leaves the per-document path running. Declare the Function paused, confirm that spec.backfill reads back true from both Kubernetes and GET /v2/udfs/{id}, then unpause it. See Backfill a Function over existing rows.

Bounded backfill runs

For an expensive namespace, set spec.discoveryOnce: true and include both discovery and write in spec.triggers. The gateway traverses existing work once per Function definition using durable page cursors. Completed checkpoints stay complete when Function completion markers change. Subsequent gateway writes maintain the queue without another sweep. A definition or source-store change starts a new traversal. Unresolved gateway write intents or dirty revisions also reopen discovery, so failed write-trigger queue updates can recover. Writes outside the gateway are not covered; use periodic discovery for those workloads. This setting does not create a provider snapshot.

Use spec.attemptRun: story-probe-1 and spec.attemptLimit: 40 to bound a measurement to forty representative claims. The gateway reserves attempts in Postgres before returning claims. Failures, lost responses and expired leases consume attempts; success does not refund them. Parent-group expansion belongs to the representative attempt. Concurrent workers share the ceiling. Zero blocks new claims. Already processing work can finish, so pause and drain it before starting a measurement. Reusing the run name retains its count even after a version change; use a new name for a new measurement. Raising the cap extends the same run. Removing the cap explicitly restores unlimited acquisition.

targetNamespaces restricts already queued work even when indexSelector is also present. Narrow it before a probe; excluded queue entries remain pending. These controls bound work acquisition, not provider retries within one attempt; combine them with queriesPerItem and spend admission.

Gateway API spellings are discovery_once, attempt_run, and attempt_limit. These are Function fields; no Pipeline field or application ingest change is needed for this backfill.

Layer-only Function reads

Set gateway LAYER_FUNCTION_LAYER_ONLY_NAMESPACES to a comma-separated list of physical namespaces whose Function work must dispatch no provider reads. For example, LAYER_FUNCTION_LAYER_ONLY_NAMESPACES=pov-bcc-story-pages. The gateway forces a zero provider-query budget for every Function input, claim-bound parent/group lookup, completion verification and write-trigger read in those namespaces, even when a Function declares a higher queriesPerItem. Metadata reads are covered too. Cache misses or unavailable certificates fail closed; they never silently hydrate from turbopuffer. Denied reads increment the Function’s rejected_queries; provider_queries remains zero.

Discovery in this mode seeds existing work from the complete Layer canonical copy once per Function definition and materialization incarnation, then consumes durable gateway write intents and dirty input revisions. It scans no provider namespace and reports pages_scanned: 0. Its local cursor survives gateway replacement. New Functions can discover previously hydrated rows without a source rewrite. Mixed-target Functions use local discovery when a protected namespace participates; hydrate all participating targets for complete backfills.

Cold namespaces require a separate admitted bulk reconciliation. Configure canonical_storage: "postgres" in the gateway’s existing LAYER_FUNCTION_READ_CACHE_CONFIG JSON. The Pro image includes layer-function-bulk-hydrate AUTHORITY.json PLAN.json: it uses the same original PG ledger, static writer identity and independent witness, scans strong ordered pages of 10,000 rows, and stores complete Function attributes plus durable _hevlayer_parent_id membership. It does not scan from an ordinary Function request. Vectors reserved by the wire are not part of Function canonical inputs.

The hydration plan contains namespace, max_pages (1–1000 pages per invocation) and admission: run_id, namespace_bytes (reviewed billed bytes per read ceiling), max_queries, max_queried_bytes, max_rows and max_storage_bytes (JSON payload bytes; provision PG disk/index overhead separately). Every physical attempt, including the initial metadata request and failures, reserves its ceiling before dispatch. Unknown or over-ceiling successful row billing blocks that run. Row receipt totals and conservative reservations are distinct; rejected publication and failed/uncertain attempts still consume reservations. Retry the identical plan to resume the durable cursor; changing budgets requires a fresh run and a new full traversal. A completed identical run performs no provider reads.

Pause source writers and Functions through initial hydration. A namespace revision/incarnation change between pages refuses resumption; it cannot certify an inconsistent traversal. Rows, membership and the page cursor commit together; only completion grants whole-namespace membership. Typed writes then maintain rows and membership under the shared fence. Repeating unchanged schema declarations preserves the copy. Unknown writes, changed schema/native generated attributes, failed publication or lost authority revoke eligibility; strict mode continues to refuse provider hydration. Retain the original ledger and witness through gateway replacement and follow the cache authority runbook for initial authority, sizing, hydration and bounded acceptance.

Write-trigger preparation records durable uncertainty before dispatch and does not fetch a provider pre-image. Successful complete writes populate certified cache rows; row-only Functions then consume that written state. Patches need an existing complete certified row. A post-write cache/read failure can surface after the provider accepted the write; retain its intent and resolve it through Layer’s guarded recovery. Do not infer that an HTTP failure rolled the write back.

This policy is a financial admission boundary. It does not manufacture cache authority: activation still requires the independently verified writer boundary and original ledger/witness protocol. Bulk hydration removes per-document seeding cost and global group invalidation for the durable copy. Count its admitted provider costs separately, then require successful bounded Function completion with zero billed reads per attempted document. Refused misses alone are not successful backfill acceptance.