Scan

Complete projected source rows

POST /v2/namespaces/{ns}/scans/rows returns a bounded complete observed source as projected rows. It requires read authorization for the same namespace and store as query. No job, cache warm, cursor continuation, or snapshot is created. The request is:

{"include_attributes":["session_id","end","cost"],"scan":{"max_rows":1000000,"max_bytes":268435456,"consistency":"observed"}}

The response contains rows and source counters (rows, pages, bytes, complete: true, consistency: "observed", threads). Physical IDs are always included. Rows are returned in physical ID order after complete acquisition. The gateway uses the shared namespace shard fanout and scan width resolver: ordinary default 8, capped by active shard count and namespace limits. An unready or unsharded namespace uses one complete cursor including tagged and untagged rows. With active sharding a disjoint residual cursor also reads null, missing and out-of-range shard tags; cached readiness alone cannot prove source coverage. It shares the same fanout width and budgets. A foreign-namespace manifest is rejected. Topology auth, throttle, transport and malformed-data errors fail the request. Only absent metadata or an unconfigured object store allows compatibility resolution; no readiness drain scan or cache promotion is needed for this route. The reserved namespace marker is excluded locally without string-ID predicates, including on uint stores. Zero or more than 4,096 advertised shards use one complete unpartitioned cursor to bound topology metadata expansion.

Only explicitly supported projected source adapters may serve this route; others return structured 422 UnsupportedByStore / projected_source_rows before source reads. Projection errors never retry with full attributes. Rows and decoded source response bytes share one request budget across shards; all provider calls retain query admission. Any source, budget, identity or cursor failure rejects the whole response. Disconnect or deadline drops pending work; already accepted provider requests can finish remotely. No partial result is returned. Source counters exclude readiness/metadata calls and HTTP framing.

This is parallel projected source acquisition, not backend-native aggregates or compact source reduction. Every revision is transferred to the gateway and the projected result to the caller. Consumers must choose latest before mutable filters and retain their existing complete folds and joins. Concurrent writes can cross observed pages; completeness does not imply a transactional snapshot.

Live scans select matching IDs, count rows, or aggregate distinct field values against the configured store. Choose source: "origin" for standalone scans. Backend support is listed in the store matrix.

Routes

Create a job with POST /v2/namespaces/{ns}/scans. Poll it through GET /v2/namespaces/{ns}/scans/{id} and read its result pages through GET /v2/namespaces/{ns}/scans/{id}/results. Count mode responds synchronously. ID and values job state is in-memory and resets on gateway restart.

ID mode

{"mode":"ids","source":"origin","filters":["category","Eq","audio"],"page_size":1000}

Count mode

{"mode":"count","source":"origin","filters":["category","Eq","audio"]}

Values mode

{"mode":"values","source":"origin","field":"category","page_size":1000}

Values scans return distinct v values and their n counts. The result is bounded to 1,000,000 distinct values; truncated: true indicates that only the most frequent values were retained, each with its exact count.

Full-text count

Count rows matching a BM25 query with the fts selector. Full-text counts are exact and always run origin scatter/gather, so source must be omitted, auto, or origin. A filters array, when present, is ANDed on as an extra constraint.

count = await client.create_scan("products", {
    "mode": "count",
    "fts": {"field": "title", "query": "wireless headphones"},
    "filters": ["category", "Eq", "Electronics"],
    "exhaustive": True,
})

count, err := client.CreateScan(ctx, "products", &hevlayer.CreateScanRequest{
    Mode:       "count",
    Fts:        hevlayer.FtsScan{Field: "title", Query: "wireless headphones"},
    Filters:    []interface{}{"category", "Eq", "Electronics"},
    Exhaustive: true,
})

const count = await client.createScan("products", {
  mode: "count",
  fts: { field: "title", query: "wireless headphones" },
  filters: ["category", "Eq", "Electronics"],
  exhaustive: true,
});

curl -X POST "$LAYER_GATEWAY_URL/v2/namespaces/products/scans" \
  -H "Authorization: Bearer $LAYER_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "mode": "count",
    "fts": {"field": "title", "query": "wireless headphones"},
    "filters": ["category", "Eq", "Electronics"],
    "exhaustive": true
  }'

Hybrid text count

Count rows in the keyword/fuzzy neighborhood of a HybridText query with the hybrid_text selector. The scan tokenizes query with the HybridText policy, then evaluates the BM25 leg, one fuzzy leg per token, and one surfacing leg per token (the RFC 0057 empty-result fallback’s legs), and counts the de-duplicated union of returned row ids.

This count is a superset of the hybrid_text query route’s deduped rows: the scan always includes the surfacing legs, whereas the query route only adds them when its primary legs (BM25 + fuzzy) return nothing. On a partial-typo query whose primary legs do match, the scan can therefore count more rows than the route returns. Use this selector for a generous live count next to hybrid_text or auto results that routed to hybrid_text; plain fts counts exact BM25 only.

count = await client.create_scan("products", {
    "mode": "count",
    "hybrid_text": {"field": "title", "query": "wireles headphones"},
    "filters": ["category", "Eq", "Electronics"],
})

count, err := client.CreateScan(ctx, "products", &hevlayer.CreateScanRequest{
    Mode:       "count",
    HybridText: hevlayer.HybridTextScan{Field: "title", Query: "wireles headphones"},
    Filters:    []interface{}{"category", "Eq", "Electronics"},
})

const count = await client.createScan("products", {
  mode: "count",
  hybrid_text: { field: "title", query: "wireles headphones" },
  filters: ["category", "Eq", "Electronics"],
});

curl -X POST "$LAYER_GATEWAY_URL/v2/namespaces/products/scans" \
  -H "Authorization: Bearer $LAYER_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "mode": "count",
    "hybrid_text": {"field": "title", "query": "wireles headphones"},
    "filters": ["category", "Eq", "Electronics"]
  }'

Radius count

Count rows within radius of a query vector with the ann selector — a distance-ball scan. radius is required and finite (without an upper bound every row is in the ball); field defaults to vector. Like fts, radius counts always run origin scatter/gather.

The count is approximate: ANN recall means the index’s membership of the ball may differ from the true set, independent of saturation, so the response carries approximate: true.

The radius bound is applied by the gateway to the $dist returned by the ranked query. It is not sent upstream as a filter.

count = await client.create_scan("products", {
    "mode": "count",
    "ann": {"field": "vector", "vector": [0.12, -0.3, 0.88], "radius": 0.25},
})

count, err := client.CreateScan(ctx, "products", &hevlayer.CreateScanRequest{
    Mode: "count",
    Ann:  hevlayer.AnnScan{Field: "vector", Vector: []float64{0.12, -0.3, 0.88}, Radius: 0.25},
})

const count = await client.createScan("products", {
  mode: "count",
  ann: { field: "vector", vector: [0.12, -0.3, 0.88], radius: 0.25 },
});

curl -X POST "$LAYER_GATEWAY_URL/v2/namespaces/products/scans" \
  -H "Authorization: Bearer $LAYER_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "mode": "count",
    "ann": {"field": "vector", "vector": [0.12, -0.3, 0.88], "radius": 0.25}
  }'

{
  "count": 980,
  "served_by": "origin",
  "approximate": true,
  "bounded": false,
  "timed_out": false,
  "shards_saturated": 0,
  "shards_total": 1,
  "threads": 1,
  "elapsed_ms": 51
}

Bounding ranked scans

Ranked selectors fan out one turbopuffer query per shard, each capped at top_k = 10_000. threads bounds fan-out width: how many shard requests can run at once. exhaustive and timeout_seconds bound depth: what happens when a shard hits that cap and how long recursion can run.

• exhaustive: false (default) — one scatter/gather. A saturated shard contributes its cap as a lower bound; the response carries bounded: true with shards_saturated > 0.
• exhaustive: true — for BM25, recurse on each saturated shard via score-band pagination ($score < last with an id tiebreak) until every page is short or timeout_seconds elapses. ANN radius scans do not push $dist filters upstream; the gateway counts returned rows whose $dist <= radius and marks the shard exhausted when the first over-radius row appears. If the full page is still inside the radius, the shard remains bounded.

The same threads value applies to the initial round and every exhaustive round over the remaining saturated shards.

bounded and approximate are independent. bounded means a shard saturated and the count is a >= lower bound for the rows the index returned; approximate means the distance ball’s membership is itself fuzzy. An ann count can be bounded: false yet still approximate: true.

Sources

Use origin for live store reads. Managed-cache and precomputed snapshot serving are documented in the full view. Origin filters use the selected store’s supported filter grammar.

Pinning and fan-out

For Turbopuffer origin scans, namespaces observed with pinning.status.ready_replicas > 0 default to 32 concurrent upstream requests, capped at 512. Other namespaces default to 8, capped at 32. Missing, failed, or older-than-two-minute observations use the unpinned limits. A request’s threads remains bounded by the applicable maximum and active shard count. These controls do not enable pinning.