Operations

Backfill a Function over existing rows

A Function normally treats each row on its own: discovery finds it, a claim reads it, completion reads it again, and a write trigger can read it a third time. turbopuffer bills every query at the full size of the namespace being queried, so on a large namespace those per-document reads cost far more than the work. A backfill declares the same Function with backfill: true. Layer then reads the inputs in bulk, 9,999 rows per query, hands them to your worker in batches, and patches the results back with no further reads from the store.

Use it to add an attribute to rows that already exist: classify, tag, derive a file reference, extract a field. Rows written after the backfill are not covered; see What a backfill does not do.

Before you start

backfill is a newer Function field. Two cluster components must know it, or the field is dropped and the Function silently runs on the per-document path that a backfill exists to avoid:

The Function CRD. The Kubernetes API server prunes fields its stored schema does not declare. Apply the CRDs from a chart that includes spec.backfill (infra/helm/layer/crds/hevlayer.com_crds.yaml, layer-pro 3dcdeee6 or later).
The layer-operator. The operator re-serializes the Function spec before registering it with the gateway, so an older operator drops the field even when the CRD keeps it. Run an operator image built from layer-pro 3dcdeee6 or later.

If you start backfills with layer run, use a layer CLI from the same release or later for the same reason.

Check both before the first run. The CRD must list the field:

kubectl get crd functions.hevlayer.com -o json \
  | jq '.spec.versions[] | select(.name=="v1alpha1")
        | .schema.openAPIV3Schema.properties.spec.properties.backfill'

A null result means the CRD is too old. Then confirm the operator image is the one you expect:

kubectl -n layer get deploy layer-operator \
  -o jsonpath='{.spec.template.spec.containers[?(@.name=="operator")].image}'
kubectl -n layer get pods -l app.kubernetes.io/component=operator \
  -o jsonpath='{range .items[*]}{.status.containerStatuses[0].imageID}{"\n"}{end}'

Compare the running image digest with the digest published for an operator revision that includes backfill.

Declare it

Start from an ordinary Function. Add backfill: true and keep discovery as the only trigger. Declare it paused the first time, so nothing runs until you have checked that the field survived.

apiVersion: hevlayer.com/v1alpha1
kind: Function
metadata:
  name: tag-products
  namespace: layer
spec:
  paused: true
  targetNamespaces:
    - products
  inputs:
    - id
    - title
  version: v1
  backfill: true
  triggers:
    - discovery
  filter:
    - category
    - Eq
    - outdoor
  worker:
    image: <acct>.dkr.ecr.us-east-1.amazonaws.com/hev-tag-products:latest
    dispatch: pull
    batchSize: 200
    timeoutSeconds: 120
  schedule:
    discoveryIntervalSeconds: 60
    maxInFlightBatches: 8

Your worker is the same pull worker you would write for any Function: it claims items, computes, and completes them with attributes. The claim already contains the inputs you declared, so the worker needs no lookup of its own. With groupByParent: true a claim carries every page row of a document, taken from the rows the backfill scanned. Backfill claims carry input_snapshot: true.

Keep the worker’s own provider reads off the per-document path. The bulk scan is cheap because it reads many rows per query. Every other read your worker makes through Layer is billed at the full size of the namespace it reads: a row read or a blob read from the namespace’s blob set, which holds every stored file. One such read per document undoes the backfill’s savings. Use the claim’s input instead of re-reading the row. Write blobs with PUT, which is an idempotent upsert keyed by hash and billed by bytes written, rather than checking first whether they exist. The stock layer-extract originals worker works this way.

Send x-hevlayer-function: <function id> on the worker’s gateway calls (the operator sets HEVLAYER_UDF_ID on worker pods). The gateway then labels that traffic’s provider billing caller_kind="function_worker" with the Function id, apart from anything else using the same key. The stock workers send it.

Writes can cost reads too. A backfill’s completion writes its outputs without a provider read. A worker that writes other rows itself through POST /v2/namespaces/{ns} (for example, the label on every page of a document) can make that write read. A write reads the written rows before and after for each Function that observes it: a write-triggered or discovery Function, not a backfill, that targets the namespace. Each such read is billed at the namespace’s size. A patch-only write (patch_columns or patch_rows) skips two kinds of Function, so it reads nothing for them:

A Function the patch cannot concern: none of the patched attributes is one of its inputs or appears in its filter.
A paused Function with a discovery trigger. It claims nothing while paused, so the write records a durable intent instead, and the Function’s discovery re-reads those rows in its batched scan when it resumes.

So patch only the attributes your Function produces. Rewriting an unchanged attribute that another Function reads (for example, page) makes every such write read for that Function. Every write also looks up the namespace’s shard marker; with the Function read cache enabled, that lookup is served from cache.

These write-path reads are charged to the write-triggered Functions that need them, never to a backfill: a backfill neither observes nor is queued by writes, and its queriesPerItem never blocks another writer. Two metrics attribute provider reads to Functions:

Metric	Labels	Counts
`hevlayer_function_provider_queries_total`	`function`, `stage`	Physical provider queries made on behalf of the Function’s items
`hevlayer_function_billable_bytes_queried_total`	`function`, `stage`	Their billed queried bytes

stage is lookup, claim, completion, discovery, write_trigger or other. A read that serves several Functions counts once for each, so don’t sum these metrics across Functions. For a backfill Function, lookup, claim and completion should stay at 0. The backfill’s own scan pages are reported in the status backfill block, not here.

Look up a document’s other pages

A worker that classifies a document from its first page often needs the other pages too. The parent lookup (POST /v2/udfs/{id}/items/lookup) returns them. In a backfill it makes no provider read: it answers from rows the scan already holds.

For each scan page, the backfill also reads the complete sibling group of every parent document on that page (_hevlayer_parent_id In the page’s parents, 9,999 rows per query). It holds those rows next to the claimed inputs. A filter that selects only page 1 still gets every page of the document. For a groupByParent Function the scan skips this step: its claims already carry the scanned pages, and the lookup returns those. A namespace whose schema has no _hevlayer_parent_id (rows written without pages) is scanned without it: the scan asks only for the Function’s inputs, and group_queries stays 0.

spec:
  backfill: true
  triggers: [discovery]
  filter: [page, Eq, 1]
  inputs: [page, name, text, page_count, _hevlayer_parent_id]

lookup = await client.lookup_udf_siblings(
    function_id,
    {
        "worker_id": worker_id,
        "namespace": item["namespace"],
        "id": item["id"],
        "input_revision": item["input_revision"],
        "include_attributes": ["page", "text"],
    },
).model_dump()
pages = lookup["documents"]  # every page of the document, ordered by id

The response has the same shape as on the per-document path, with three differences:

Inputs only. The held rows carry the Function’s inputs. Every name in include_attributes must be one of them, or the lookup returns 422. page_two_id needs page among the inputs.
Scan-time values. Like claims, the lookup returns the values the scan read, not later writes.
Rows scanned by an older gateway. Rows scanned before the gateway held sibling groups have none. Their lookup returns 409 instead of reading the store. Change any field of the definition to scan again. Rows that already carry the version marker are not selected again, so the new scan covers only the remaining work.

Confirm the field round-trips

Apply the paused Function and read it back twice: once from Kubernetes (the CRD kept the field) and once from the gateway (the operator passed it on).

kubectl apply -f tag-products.yaml
kubectl -n layer get function tag-products -o jsonpath='{.spec.backfill}'
# true

curl -s -H "Authorization: Bearer $LAYER_GATEWAY_API_KEY" \
  "$HEVLAYER_BASE_URL/v2/udfs/tag-products" | jq '.udf.spec.backfill, .udf.paused'
# true
# true

Wait for the operator to reconcile before the second read; the Function’s Ready condition shows when it has. If either read is not true, stop: leave the Function paused (or delete it) and update the CRD or the operator first. A Function registered without backfill scans and reads per document.

Run it

Unpause the Function to start the scan:

kubectl -n layer patch function tag-products --type merge -p '{"spec":{"paused":false}}'

The gateway continues the scan on its own every schedule.discoveryIntervalSeconds. Call POST /v2/udfs/{id}/discover to advance it immediately.

Once a cluster has passed these checks, layer run -f tag-products.yaml (with paused removed from the manifest) does the same in one command: it applies the Function, starts the scan and waits until the scan has finished and the queue is empty.

Follow progress

GET /v2/udfs/{id}/status (or getUdfStatus in the SDKs) reports a backfill block once the first page is scanned. layer udf get tag-products --watch shows the queue counts alongside it:

{
  "backfill": {
    "version": "v1",
    "scan_complete": false,
    "pages": 12,
    "rows_scanned": 119988,
    "queries": 12,
    "group_queries": 120,
    "queried_bytes": 92400000000,
    "returned_bytes": 31000000,
    "rows_done": 97310,
    "billed_queried_bytes_per_row": 770077.0
  }
}

Field	Meaning
`pages`, `queries`	Scan pages read, and the row queries made for them, each billed at the namespace’s size. A page read again after an interruption counts again.
`metadata_requests`	Namespace metadata requests made alongside the pages (the id-type lookup for a cursor page, cached for 60 s). They are not counted in `queries`.
`group_queries`	Row queries that read the sibling groups of each page’s parents, so the parent lookup needs no read. Each is billed at the namespace’s size. Each 1,000 parents take one query per 9,999 of their rows. A full scan page makes 10 when documents average under 10 rows, and 20 under 20. Zero for `groupByParent` and for rows without a parent.
`rows_scanned`	Rows the scans returned, counted when the page’s rows are queued.
`rows_done`	Scanned rows that are no longer pending, processing or failed.
`queried_bytes`	Billed logical bytes queried, from the provider’s own response billing. Includes the sibling-group queries.
`scan_complete`	Every target namespace has been traversed once.

A page’s billed read is recorded as soon as the provider answers, and its rows are counted in the same commit that queues them. A page interrupted in between, for example by a pause, shows in pages, queries and queried_bytes but adds nothing to rows_scanned. Resuming reads that page again, so it is billed twice and its rows are counted once. While no worker runs, rows_done stays 0.

The cost of the reads is (queries + group_queries) × namespace size. One page covers 9,999 rows, so a namespace of 387,826 rows needs 39 pages. A full scan page adds 10 group queries when documents average under 10 rows, and 20 when they average under 20. Compare that with a query or more per document, and per lookup, on the per-document path. After a run, check that queries equals pages, and that queried_bytes / rows_scanned is far below the size of the namespace.

scan_complete is true only when every current target has a completed checkpoint for the current Function definition. A definition change, including changing the worker image, makes it false before discovery resumes. Historical scan totals and held queue rows remain available; they do not prove that the current definition has finished scanning. A change to the source store can also reopen a checkpoint when discovery resumes.

Resume and re-run

Resume. Scan progress is stored in Layer’s Postgres after each page is queued. A gateway restart, a lost lease or a paused-then-resumed Function continues from the last page. Workers can stop and start; unfinished claims are recovered when their lease expires.
Pause. Pausing stops the scan at a page boundary, without an error and without counting a discovery failure. Pages already queued stay queued and are not read again. A page that was still being read when the pause landed is dropped, so unpausing reads that one page again: pages and queries each gain 1 and its billed bytes count twice.
```
kubectl -n layer patch function tag-products --type merge -p '{"spec":{"paused":true}}'
# later: continue from the last queued page
kubectl -n layer patch function tag-products --type merge -p '{"spec":{"paused":false}}'
```
Safe to repeat. Each completion stamps a version marker on the row, and the scan only selects rows without the current marker. A row finished twice gets the same patch. Re-applying the manifest does not scan again once the scan is complete.
Run again. Change spec.version (or any other part of the definition) to scan from the start. Use this after changing the worker’s output or to catch rows that were written during the first run.
Failures. Failed items stay in the queue with their error; fix the worker and run reset-failed. The scan itself fails and retries per page.

Switching an existing Function to backfill

A Function that ran on the per-document path can still have queue rows from that run: pending, or failed. A backfill leaves them as they are. It claims only rows whose inputs its own scans hold, so it never claims, fails, retries or rewrites an older row:

An older pending row that the scan selects again is processed as ordinary backfill work.
An older pending row outside every scan stays pending.
An older failed row stays failed, even when a scan selects its document again. Retry it separately if you choose to.

The backfill block in status counts the two kinds apart:

Field	Meaning
`held_pending`, `held_processing`, `held_failed`	Queue rows whose inputs a scan holds. `held_failed` includes older failures a scan selected again.
`legacy_pending`, `legacy_processing`, `legacy_failed`	Older queue rows without held inputs.

Each status response reads the whole backfill block and the top-level pending_count, processing_count and failed_count from one snapshot. These hold in every response, even in the middle of a scan:

held_X + legacy_X equals the matching X_count.
rows_done equals rows_scanned less held_pending, held_processing and held_failed.

Compare values from the same response, not from two reads. If you check these in a script and a check fails, save the whole response before you act on it. It holds only counts, and it shows which values disagreed.

A backfill is finished when scan_complete is true and held_pending and held_processing are both 0. pending_count and failed_count keep the older rows. rows_done counts held rows only.

What a backfill does not do

Rows written later. Gateway writes never queue a backfill Function, and the backfill makes no read for them. Run a backfill again when you need to cover new rows, or use a regular Function with a write trigger for ongoing work.
Vector output. A backfill writes attributes only. Embedding and other vector output reads the whole row to rewrite it; use a regular Function.
A consistent snapshot. Scans use eventual reads. A row changed while the backfill runs may be processed with the older inputs. The next run corrects it.
Large inputs. The scanned inputs, and the sibling groups held for the lookup, wait in Layer’s Postgres until the queue drains, then are deleted. Declare only the attributes you need.

A backfill needs no cache or finality configuration. It does not use the certified Function read cache, queriesPerItem or the layer-only namespace setting. Its paged scans are labeled with the Function in billing metrics like any other Function read.

Example: attach original files to existing Story rows

Backfill original-file references describes layer-extract originals, the stock worker that fills original_blob on rows indexed before originals were kept. Add backfill: true and triggers: [discovery] to its Function and the backfill reads the page rows in 10,000-row scans instead of once per document.