Embed

Add an embed block to a string attribute in your schema and Layer computes that attribute’s vector on every write. Put ["Embed", "..."] in rank_by and Layer computes the query vector with the same model. Your application never loads a model or sends a vector of its own.

Example

This write creates the articles namespace with an embedded text attribute and upserts two rows. Layer embeds each row’s text with MiniLM, one of the CPU models, before storing it.

await client.write_namespace("articles", {
    "upsert_rows": [
        {"id": "planet-1", "title": "Planet",
         "text": "Jupiter is the biggest planet in the Solar System."},
        {"id": "photo-1", "title": "Photosynthesis",
         "text": "Plants turn sunlight, water, and carbon dioxide into food."},
    ],
    "distance_metric": "cosine_distance",
    "schema": {
        "text": {
            "type": "string",
            "embed": {
                "model": "sentence-transformers/all-MiniLM-L6-v2",
                "dims": 384,
                "serving": {"prefer": "local"},
            },
        },
    },
})

_, err := client.WriteNamespace(ctx, "articles", hevlayer.TurbopufferWriteRequest{
    "upsert_rows": []map[string]any{
        {"id": "planet-1", "title": "Planet",
            "text": "Jupiter is the biggest planet in the Solar System."},
        {"id": "photo-1", "title": "Photosynthesis",
            "text": "Plants turn sunlight, water, and carbon dioxide into food."},
    },
    "distance_metric": "cosine_distance",
    "schema": map[string]any{
        "text": map[string]any{
            "type": "string",
            "embed": map[string]any{
                "model":   "sentence-transformers/all-MiniLM-L6-v2",
                "dims":    384,
                "serving": map[string]any{"prefer": "local"},
            },
        },
    },
})

await client.writeNamespace("articles", {
  upsert_rows: [
    { id: "planet-1", title: "Planet",
      text: "Jupiter is the biggest planet in the Solar System." },
    { id: "photo-1", title: "Photosynthesis",
      text: "Plants turn sunlight, water, and carbon dioxide into food." },
  ],
  distance_metric: "cosine_distance",
  schema: {
    text: {
      type: "string",
      embed: {
        model: "sentence-transformers/all-MiniLM-L6-v2",
        dims: 384,
        serving: { prefer: "local" },
      },
    },
  },
});

curl -X POST "$LAYER_GATEWAY_URL/v2/namespaces/articles" \
  -H "Authorization: Bearer $LAYER_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "upsert_rows": [
      {"id": "planet-1", "title": "Planet",
       "text": "Jupiter is the biggest planet in the Solar System."},
      {"id": "photo-1", "title": "Photosynthesis",
       "text": "Plants turn sunlight, water, and carbon dioxide into food."}
    ],
    "distance_metric": "cosine_distance",
    "schema": {
      "text": {
        "type": "string",
        "embed": {
          "model": "sentence-transformers/all-MiniLM-L6-v2",
          "dims": 384,
          "serving": { "prefer": "local" }
        }
      }
    }
  }'

Query by meaning with Embed. Layer reads the model from the text schema and embeds the query string with it:

response = await client.query_namespace("articles", {
    "rank_by": ["text", "ANN", ["Embed", "largest planet in the solar system"]],
    "top_k": 3,
    "include_attributes": ["title", "text"],
})
print(response.rows, response.performance)

response, err := client.QueryNamespace(ctx, "articles", &hevlayer.QueryRequest{
    RankBy:            []any{"text", "ANN", []any{"Embed", "largest planet in the solar system"}},
    TopK:              3,
    IncludeAttributes: []string{"title", "text"},
})

const response = await client.queryNamespace("articles", {
  rank_by: ["text", "ANN", ["Embed", "largest planet in the solar system"]],
  top_k: 3,
  include_attributes: ["title", "text"],
});

curl -X POST "$LAYER_GATEWAY_URL/v2/namespaces/articles/query" \
  -H "Authorization: Bearer $LAYER_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "rank_by": ["text", "ANN", ["Embed", "largest planet in the solar system"]],
    "top_k": 3,
    "include_attributes": ["title", "text"]
  }'

{
  "rows": [
    { "id": "planet-1", "$dist": 0.137, "title": "Planet",
      "text": "Jupiter is the biggest planet in the Solar System." }
  ],
  "performance": {
    "embedding_tokens": 7,
    "embedding_ms": 9
  }
}

Where the vector comes from

embed.serving.prefer picks the provider that runs the model:

The provider is independent of the store the vector is written to, so any store that accepts embed can use any configured provider.

Layer never substitutes one provider for another. A profile that selects a provider the deployment has not configured fails validation at write time, and a provider that fails at request time returns an error instead of falling back to a different model.

autoscaler is accepted as an alias for turbopuffer, and lattice for local on a Lattice profile. native is its own value, not an alias.

Per-store resolution

Omitting serving selects native. Only Turbopuffer can compute that vector itself. On a store that cannot embed, Layer computes the vector and the store receives an ordinary vector column. The rule depends on the store, never on the request:

Layer resolves a gateway-resolved profile once, on the write that declares the attribute, and the namespace keeps that provider for good. If neither provider serves the model, the write returns 503 and nothing is stored.

CPU models

Layer serves three CPU embedding models, small enough to run on a laptop under Docker. They use the same schema as the larger GPU and hosted models, with lower retrieval quality. Community Edition Compose starts the text service next to the gateway, and the Helm chart runs it as a gateway sidecar; neither needs a key, GPU or download after the image pull, and the service has no route out of the machine.

The two text models are served by layer-embed, a separate CPU process whose image contains both sets of weights. The gateway reaches it over a private network at LAYER_EMBED_URL (for example http://embed:8081); nothing is downloaded at startup, and the gateway image carries no model weights. Set the variable and prefer: local resolves any model the service advertises. Leave it unset and a prefer: local write that names a text model returns 422. CLIP runs inside the gateway process from the directory named by LAYER_LOCAL_CLIP_MODEL_PATH; see Images.

There is no default model: a prefer: local profile always names one. Omit dims and Layer uses the dimension the service advertises for that model. Every model here pins a fixed artifact; revision, instructions, and chunk are not accepted on a local profile.

Query embedding has a 10 second budget and each write 60 seconds, shared by every batch of that write. Set LAYER_EMBED_QUERY_TIMEOUT_MS and LAYER_EMBED_WRITE_TIMEOUT_MS to change them. A service that is still loading returns 503 for local profiles only; native and GPU profiles are unaffected.

Text

The two text models are embedded differently, and Layer handles both; the example at the top of this page uses MiniLM. bge expects an instruction prefix on queries and MiniLM expects none, so the gateway adds the prefix exactly once, only when it embeds a query, and never on the text it stores. bge uses CLS pooling and MiniLM uses mean pooling. Both outputs are L2-normalized, so cosine_distance scores are comparable across rows. Switching a namespace between them requires a full re-index.

Images

With modality: image on a CLIP model, Layer embeds each attribute value with the image tower at write time and embeds Embed query text with the text tower, against the same vector column. Both towers run on CPU beside the gateway.

await client.write_namespace("photos", {
    "upsert_rows": [
        {"id": "commons-131", "title": "Sunset at Porto Covo",
         "image_url": "https://upload.wikimedia.org/…/640px-Porto_Covo.jpg"},
    ],
    "distance_metric": "cosine_distance",
    "schema": {
        "image_url": {
            "type": "string",
            "embed": {
                "model": "openai/clip-vit-base-patch32",
                "dims": 512,
                "modality": "image",
                "serving": {"prefer": "local"},
            },
        },
    },
})

_, err := client.WriteNamespace(ctx, "photos", hevlayer.TurbopufferWriteRequest{
    "upsert_rows": []map[string]any{
        {"id": "commons-131", "title": "Sunset at Porto Covo",
            "image_url": "https://upload.wikimedia.org/…/640px-Porto_Covo.jpg"},
    },
    "distance_metric": "cosine_distance",
    "schema": map[string]any{
        "image_url": map[string]any{
            "type": "string",
            "embed": map[string]any{
                "model":    "openai/clip-vit-base-patch32",
                "dims":     512,
                "modality": "image",
                "serving":  map[string]any{"prefer": "local"},
            },
        },
    },
})

await client.writeNamespace("photos", {
  upsert_rows: [
    { id: "commons-131", title: "Sunset at Porto Covo",
      image_url: "https://upload.wikimedia.org/…/640px-Porto_Covo.jpg" },
  ],
  distance_metric: "cosine_distance",
  schema: {
    image_url: {
      type: "string",
      embed: {
        model: "openai/clip-vit-base-patch32",
        dims: 512,
        modality: "image",
        serving: { prefer: "local" },
      },
    },
  },
});

curl -X POST "$LAYER_GATEWAY_URL/v2/namespaces/photos" \
  -H "Authorization: Bearer $LAYER_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "upsert_rows": [
      {"id": "commons-131", "title": "Sunset at Porto Covo",
       "image_url": "https://upload.wikimedia.org/…/640px-Porto_Covo.jpg"}
    ],
    "distance_metric": "cosine_distance",
    "schema": {
      "image_url": {
        "type": "string",
        "embed": {
          "model": "openai/clip-vit-base-patch32",
          "dims": 512,
          "modality": "image",
          "serving": { "prefer": "local" }
        }
      }
    }
  }'

Query the image column with text:

response = await client.query_namespace("photos", {
    "rank_by": ["image_url", "ANN", ["Embed", "sunset over water"]],
    "top_k": 16,
    "include_attributes": ["title", "image_url"],
})

response, err := client.QueryNamespace(ctx, "photos", &hevlayer.QueryRequest{
    RankBy:            []any{"image_url", "ANN", []any{"Embed", "sunset over water"}},
    TopK:              16,
    IncludeAttributes: []string{"title", "image_url"},
})

const response = await client.queryNamespace("photos", {
  rank_by: ["image_url", "ANN", ["Embed", "sunset over water"]],
  top_k: 16,
  include_attributes: ["title", "image_url"],
});

curl -X POST "$LAYER_GATEWAY_URL/v2/namespaces/photos/query" \
  -H "Authorization: Bearer $LAYER_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "rank_by": ["image_url", "ANN", ["Embed", "sunset over water"]],
    "top_k": 16,
    "include_attributes": ["title", "image_url"]
  }'

{
  "rows": [
    { "id": "commons-131", "$dist": 0.412, "title": "Sunset at Porto Covo",
      "image_url": "https://upload.wikimedia.org/…/640px-Porto_Covo.jpg" }
  ],
  "performance": {
    "embedding_tokens": 5,
    "embedding_ms": 134
  }
}

Write responses report embedding_images instead of embedding_tokens. CPU CLIP handles query-time text embeds and write-time image embeds for small and medium corpora. A bulk image backfill needs a GPU. The provider is set per namespace, so a backfill can run on a GPU while queries embed on CPU.

Declare the checkpoint for that backfill under open-weights models on your own GPU.

The lens demo runs this setup: Wikimedia Commons Quality images embedded and searched on CPU, with the gateway’s performance echo shown beside each result. Source at github.com/hev/lens.

Image input

An image profile accepts an HTTP(S) URL or a base64 image string and fetches at most 20 MiB per URL. A rate-limited image host returns HTTP 429 with error upstream_error; a server error from the image host returns HTTP 503 with error service_unavailable. Layer preserves the host’s Retry-After response header for both. Malformed URLs or base64, non-retryable 4xx responses, oversize content, and undecodable images return 422 validation_error.

Changing models requires a re-index

The two text models both emit 384 dimensions, so switching between them leaves the namespace schema valid. It still requires re-embedding every row.

Different models place vectors in unrelated spaces. If documents were embedded with MiniLM and queries are embedded with bge, the distances compare incompatible vectors and the ranking is meaningless. The same applies when moving from a CPU model to a hosted or GPU model, and when the weights, tokenizer, pooling, or prefixes behind a model id change: equal dimensions never make two vector spaces compatible.

Layer records the artifact fingerprint a local profile was declared with. A write that names a different model for an attribute that already has rows, or a write or Embed query after the service starts serving a different artifact for the pinned model, returns 422 and names the pinned model. Re-index from your source data into a fresh namespace so every vector comes from the new model, verify it, then switch your application to it.

Bringing your own weights

You can mount a model directory into the layer-embed service to add or override text checkpoints. Set LAYER_EMBED_MODELS_DIR on that service to a read-only bundle in the same manifest format as its baked models: a directory with a manifest.json listing each model’s files with their sizes and SHA-256 hashes. Mounted entries override baked entries by model id. The service verifies every listed file, loads and warms the whole menu, and only then reports ready; the gateway discovers the merged menu at startup. Only BERT text models with 384-dimensional CLS or mean pooling are accepted. The mount is optional and the bundled models need none. Overriding a model id with different weights changes its artifact fingerprint, so existing namespaces on that id need a full re-index after the restart.

Under Compose, mount the directory into the embed service read-only and set LAYER_EMBED_MODELS_DIR to its path in a Compose override file. The stock Compose file mounts nothing.

On Kubernetes, the Helm chart provisions a mounted bundle with the opt-in gateway.localModels values. An init container copies the bundle from a private S3 prefix, verifies manifest.json against a pinned SHA-256 before the pod starts, and mounts the directory read-only into the embedding sidecar; the sidecar then verifies each file against the manifest. A checksum mismatch fails the pod instead of serving unverified weights. The gateway container itself never downloads a checkpoint.

For a single static retriever that trades retrieval quality for microsecond embeds and a few megabytes of deployment size, see Lattice.

Open-weights models on your own GPU

To use a stronger retrieval model than the CPU menu offers, run it on a GPU in your own cluster. You build a container with the model’s weights inside it, declare it in InfraRules, and name the model in a schema with serving.prefer: worker. Layer runs the container on a GPU pool, scales it to zero when idle, and sends it every embedding for that model.

Build the container

The container must serve two endpoints on its declared port: GET /health, which succeeds once the model is loaded, and the OpenAI-compatible POST /v1/embeddings. The weights go inside the image. Layer downloads no weights and passes the container no model arguments, so the image’s own entrypoint starts the server with the model configured. The server must reject an input longer than the model’s limit, not truncate it.

Hugging Face text-embeddings-inference (TEI) serves both endpoints. Download the checkpoint at a pinned revision, keeping only the files TEI loads:

hf download BAAI/bge-m3 \
  --revision 5617a9f61b028005a4858fdac845db406aefb181 \
  --include "*.json" pytorch_model.bin sentencepiece.bpe.model \
  --exclude "onnx/*" \
  --local-dir ./bge-m3

Then copy it into the TEI image for your GPU:

# The tag prefix is the GPU's compute capability: 86 for A10G, 89 for L4.
FROM ghcr.io/huggingface/text-embeddings-inference:86-1.9
COPY bge-m3/ /model/
ENV HF_HUB_OFFLINE=1 \
    PORT=8080
CMD ["--model-id", "/model", "--served-model-name", "BAAI/bge-m3"]

--served-model-name must equal the model you declare. Layer rejects a response whose model field names anything else. HF_HUB_OFFLINE=1 makes a missing file fail at startup instead of triggering a download.

Push the image to a registry your cluster can pull from. The image is the model’s identity; see image identity.

Declare it

Each container is declared once, under spec.embedders in InfraRules/default. Add it to the existing object and keep its computePools: the object must list at least one pool, and the embedder’s pool must be one of them. See compute pools for the pool definitions.

apiVersion: hevlayer.com/v1alpha1
kind: InfraRules
metadata:
  name: default
spec:
  # computePools: unchanged. It must include the pool named below.
  embedders:
    - name: bge-m3
      model: BAAI/bge-m3
      image: <registry>/embed-bge-m3:2026-09-29
      port: 8080
      dims: 1024
      maxInputTokens: 8192
      scaling:
        pool: gpu
        mode: autoscale
        warmWindowSeconds: 300
        replicas:
          min: 0
          max: 2

fixed mode requires replicas.min to equal replicas.max. disabled removes the scaler and sets replicas to zero.

Adding or changing an embedder is an edit to this object. The gateway reads the declaration on demand, so it needs no gateway image rebuild or redeploy. Removing an entry deletes the Deployment, Service, and scaler the operator created for it.

Use it

Name the model in the schema. Compared with the CPU example, only model, dims, and serving.prefer change:

await client.write_namespace("articles", {
    "upsert_rows": [
        {"id": "planet-1", "text": "Jupiter is the biggest planet in the Solar System."},
    ],
    "distance_metric": "cosine_distance",
    "schema": {
        "text": {
            "type": "string",
            "embed": {
                "model": "BAAI/bge-m3",
                "dims": 1024,
                "serving": {"prefer": "worker"},
            },
        },
    },
})

_, err := client.WriteNamespace(ctx, "articles", hevlayer.TurbopufferWriteRequest{
    "upsert_rows": []map[string]any{
        {"id": "planet-1", "text": "Jupiter is the biggest planet in the Solar System."},
    },
    "distance_metric": "cosine_distance",
    "schema": map[string]any{
        "text": map[string]any{
            "type": "string",
            "embed": map[string]any{
                "model":   "BAAI/bge-m3",
                "dims":    1024,
                "serving": map[string]any{"prefer": "worker"},
            },
        },
    },
})

await client.writeNamespace("articles", {
  upsert_rows: [
    { id: "planet-1", text: "Jupiter is the biggest planet in the Solar System." },
  ],
  distance_metric: "cosine_distance",
  schema: {
    text: {
      type: "string",
      embed: {
        model: "BAAI/bge-m3",
        dims: 1024,
        serving: { prefer: "worker" },
      },
    },
  },
});

curl -X POST "$LAYER_GATEWAY_URL/v2/namespaces/articles" \
  -H "Authorization: Bearer $LAYER_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "upsert_rows": [
      {"id": "planet-1", "text": "Jupiter is the biggest planet in the Solar System."}
    ],
    "distance_metric": "cosine_distance",
    "schema": {
      "text": {
        "type": "string",
        "embed": {
          "model": "BAAI/bge-m3",
          "dims": 1024,
          "serving": { "prefer": "worker" }
        }
      }
    }
  }'

Queries are unchanged. Moving a namespace from a CPU model to a GPU model means editing those fields and re-indexing.

Status

The operator reports each embedder under status.embedders:

status:
  embedders:
    - name: bge-m3
      model: BAAI/bge-m3
      image: <registry>/embed-bge-m3:2026-09-29
      imageDigest: sha256:<digest>
      endpoint: http://layer-embedder-bge-m3.<namespace>.svc.cluster.local:8080
      conditions:
        - type: Ready
          status: "True"
          reason: Ready

Ready is always present. When it is False, a second condition says why:

Image identity

A declared @sha256: digest is recorded at once. For a tag, the operator records the digest Kubernetes pulled on the first wake and pins the Deployment to it. The digest survives scale to zero and operator restarts, so moving the tag in your registry does not change the model an embedder runs. Changing image in the declaration resets it.

The first write to a namespace stores the image digest and dims in its embedding profile. A later write or query that finds a different digest, dimension, model, or instructions returns 422, even when a vector for the input is cached. A rebuilt image is a new model: re-index into a fresh namespace, as in changing models.

Input limit

OpenAI-compatible servers expose no tokenizer endpoint, so the gateway cannot count tokens the way the model does. It checks a conservative budget instead: each prepared input’s length in UTF-8 bytes, plus two special tokens, must not exceed maxInputTokens. The budget includes any instruction prefix. A longer input returns 422 before anything is sent to the container.

A token is almost always longer than one byte, so this check rejects some inputs the model would accept. With maxInputTokens: 8192, an input may be at most 8,190 bytes. Set maxInputTokens to the model’s real limit, not a lower safety margin, and split long text with chunking.

Cold starts

One observed figure: on Layer’s production cluster on 2026-09-29, a bge-m3 embedder at zero replicas took about 3 min 54 s (233.6 s) from the first write to an HTTP 200 on retry. That included provisioning a g5.xlarge node and pulling the image. The first response was a 503, returned in 233 ms. Your cluster’s time depends on its node provisioning, registry, and image size; this is not a guarantee.

While an embedder is waking, Layer returns 503 with Retry-After: 5 and does not hold the request open. It returns the 503 even when the vector for that input is cached. Retry after the Retry-After interval.

Every request for an embedder, including one that gets a 503, counts toward hevlayer_embedder_demand{embedder}, the number of requests in the last 60 seconds. KEDA polls it every 15 seconds, targets one replica per counted request between replicas.min and replicas.max, and wakes a sleeping embedder when it rises above zero. After demand falls to zero, the embedder stays up for warmWindowSeconds, KEDA’s cooldown, before it returns to replicas.min.

For interactive search, keep one replica warm with replicas.min: 1, or set warmWindowSeconds long enough to cover the gaps between your bursts of traffic.

A bigger model

Bigger dense models follow the same steps. Build the container with the new weights, declare it on a pool with enough GPU memory, and re-index. For example, a 4B-parameter model that expects an instruction before each query:

    - name: qwen3-embed-4b
      model: Qwen/Qwen3-Embedding-4B
      image: <registry>/embed-qwen3-4b:2026-09-29
      dims: 2560
      maxInputTokens: 32768
      scaling:
        pool: gpu-48g   # not built in; declared under spec.computePools
        mode: autoscale
        warmWindowSeconds: 300
        replicas: { min: 0, max: 2 }

The built-in pools are cpu, cpu-large, and gpu. A pool with larger GPUs, here gpu-48g, is one you add under spec.computePools in the same object, with its own nodeSelector, gpuType, and resources. See compute pools.

"text": {
  "type": "string",
  "embed": {
    "model": "Qwen/Qwen3-Embedding-4B",
    "dims": 2560,
    "serving": { "prefer": "worker" },
    "instructions": {
      "query": "Instruct: Given a question, retrieve passages that answer it\nQuery: "
    }
  }
}

Layer applies the instruction to queries only, once, before sending them to the container. The container never adds its own.

Errors

Layer never falls back to another provider. An undeclared model is a configuration error you can fix, so Layer returns a 4xx that names the model. 503 means a declared model is not serving yet; the embedder’s condition on the InfraRules status says why.

Query with Embed

["Embed", text] computes a query vector from text using the model settings of the attribute being ranked. When rank_by names the source attribute, Layer reads the model from its schema:

response = await client.query_namespace("clinical-notes", {
    "rank_by": ["text", "ANN", ["Embed", "chest pain radiating to left arm"]],
    "top_k": 10,
})
print(response.rows)

response, err := client.QueryNamespace(ctx, "clinical-notes", &hevlayer.QueryRequest{
    RankBy: []any{"text", "ANN", []any{"Embed", "chest pain radiating to left arm"}},
    TopK:   10,
})

const response = await client.queryNamespace("clinical-notes", {
  rank_by: ["text", "ANN", ["Embed", "chest pain radiating to left arm"]],
  top_k: 10,
});

curl -X POST "$LAYER_GATEWAY_URL/v2/namespaces/clinical-notes/query" \
  -H "Authorization: Bearer $LAYER_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "rank_by": ["text", "ANN", ["Embed", "chest pain radiating to left arm"]],
    "top_k": 10
  }'

When rank_by names the derived vector attribute embed_<attr>, pass the model explicitly. Omitting it returns 422 with a model name must be provided.

response = await client.query_namespace("clinical-notes", {
    "rank_by": ["embed_text", "ANN", ["Embed", "chest pain radiating to left arm", {
        "model": "acme/clinical-retrieval-v3",
    }]],
    "top_k": 10,
})

response, err := client.QueryNamespace(ctx, "clinical-notes", &hevlayer.QueryRequest{
    RankBy: []any{"embed_text", "ANN", []any{"Embed", "chest pain radiating to left arm",
        map[string]any{"model": "acme/clinical-retrieval-v3"}}},
    TopK: 10,
})

const response = await client.queryNamespace("clinical-notes", {
  rank_by: ["embed_text", "ANN", ["Embed", "chest pain radiating to left arm", {
    model: "acme/clinical-retrieval-v3",
  }]],
  top_k: 10,
});

curl -X POST "$LAYER_GATEWAY_URL/v2/namespaces/clinical-notes/query" \
  -H "Authorization: Bearer $LAYER_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "rank_by": ["embed_text", "ANN", ["Embed", "chest pain radiating to left arm", {
      "model": "acme/clinical-retrieval-v3"
    }]],
    "top_k": 10
  }'

Query vectors are cached for 60 seconds by default. Set LAYER_EMBED_CACHE_TTL_MS to change the TTL. Instruction prefixes are part of the cache key. A missing provider returns 503 service_unavailable.

Embed with Auto

An inline Embed lets query routing run a semantic or fused query in one request:

response = await client.query_namespace("articles", {
    "rank_by": ["title", "Auto", "how plants turn sunlight into food", {
        "vector": ["Embed", "how plants turn sunlight into food", {"field": "text"}],
    }],
    "top_k": 10,
})

response, err := client.QueryNamespace(ctx, "articles", &hevlayer.QueryRequest{
    RankBy: []any{"title", "Auto", "how plants turn sunlight into food", map[string]any{
        "vector": []any{"Embed", "how plants turn sunlight into food",
            map[string]any{"field": "text"}},
    }},
    TopK: 10,
})

const response = await client.queryNamespace("articles", {
  rank_by: ["title", "Auto", "how plants turn sunlight into food", {
    vector: ["Embed", "how plants turn sunlight into food", { field: "text" }],
  }],
  top_k: 10,
});

curl -X POST "$LAYER_GATEWAY_URL/v2/namespaces/articles/query" \
  -H "Authorization: Bearer $LAYER_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "rank_by": ["title", "Auto", "how plants turn sunlight into food", {
      "vector": ["Embed", "how plants turn sunlight into food", {"field": "text"}]
    }],
    "top_k": 10
  }'

The first tuple field (title) is the attribute for the lexical legs. field inside Embed names the attribute whose schema supplies the embedding profile and vector column (text above). Omit field when both are the same attribute.

Layer chooses the route before it resolves Embed, so a short input routed to hybrid_text never calls the embedding provider. A semantic or fused response reports the routing decision (routing.policy: "v1", routing.executed: true) and merges embedding measurements into performance. Without a vector or an inline Embed, the router returns routing.executed: false and leaves embedding to the caller.

Model settings

The full embed block:

The first write that establishes an embedded schema must also set distance_metric.

revision, instructions, chunk, and modality are Layer extensions. Layer applies them itself and never forwards them to the store. A profile that sets any of them must use serving.prefer: turbopuffer or worker, or the write returns 422. The one exception is a CLIP profile with modality: image on local, which may not set the other three. A turbopuffer profile with an extension also needs a configured Turbopuffer embedding provider, and returns 503 without one. A worker profile embeds text only and rejects revision; bake the revision into the container image instead.

Chunking long documents

A single vector for a long document matches queries poorly, so long sources should be split into chunks and each chunk embedded as its own row.

Add embed.chunk to the schema and Layer splits the attribute on write. The split runs on the gateway’s CPU during the write request, and only the chunks go to the embedding provider. This profile splits text into windows of up to 512 tokens that overlap by 64:

await client.write_namespace("manuals", {
    "upsert_rows": [
        {"id": "pump-manual", "title": "Pump maintenance", "text": long_text},
    ],
    "distance_metric": "cosine_distance",
    "schema": {
        "text": {
            "type": "string",
            "embed": {
                "model": "BAAI/bge-m3",
                "dims": 1024,
                "serving": {"prefer": "worker"},
                "chunk": {
                    "strategy": "recursive",
                    "unit": "tokens",
                    "tokenizer": "BAAI/bge-m3",
                    "size": 512,
                    "overlap": 64,
                },
            },
        },
    },
})

_, err := client.WriteNamespace(ctx, "manuals", hevlayer.TurbopufferWriteRequest{
    "upsert_rows": []map[string]any{
        {"id": "pump-manual", "title": "Pump maintenance", "text": longText},
    },
    "distance_metric": "cosine_distance",
    "schema": map[string]any{
        "text": map[string]any{
            "type": "string",
            "embed": map[string]any{
                "model":   "BAAI/bge-m3",
                "dims":    1024,
                "serving": map[string]any{"prefer": "worker"},
                "chunk": map[string]any{
                    "strategy":  "recursive",
                    "unit":      "tokens",
                    "tokenizer": "BAAI/bge-m3",
                    "size":      512,
                    "overlap":   64,
                },
            },
        },
    },
})

await client.writeNamespace("manuals", {
  upsert_rows: [
    { id: "pump-manual", title: "Pump maintenance", text: longText },
  ],
  distance_metric: "cosine_distance",
  schema: {
    text: {
      type: "string",
      embed: {
        model: "BAAI/bge-m3",
        dims: 1024,
        serving: { prefer: "worker" },
        chunk: {
          strategy: "recursive",
          unit: "tokens",
          tokenizer: "BAAI/bge-m3",
          size: 512,
          overlap: 64,
        },
      },
    },
  },
});

curl -X POST "$LAYER_GATEWAY_URL/v2/namespaces/manuals" \
  -H "Authorization: Bearer $LAYER_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "upsert_rows": [
      {"id": "pump-manual", "title": "Pump maintenance", "text": "..."}
    ],
    "distance_metric": "cosine_distance",
    "schema": {
      "text": {
        "type": "string",
        "embed": {
          "model": "BAAI/bge-m3",
          "dims": 1024,
          "serving": { "prefer": "worker" },
          "chunk": {
            "strategy": "recursive",
            "unit": "tokens",
            "tokenizer": "BAAI/bge-m3",
            "size": 512,
            "overlap": 64
          }
        }
      }
    }
  }'

Layer keeps the original row and adds one row per chunk. A chunk row copies the original’s attributes, replaces the embedded attribute with the chunk text, and gets the id {id}#{n} plus two generated attributes: _hevlayer_parent_id (the original id) and _hevlayer_chunk_index (n). A query returns chunk rows, and _hevlayer_parent_id groups them back to their document. The original row has no chunk-sized text of its own. On Turbopuffer, which requires a vector on every row, it is stored with the vector of its first chunk, so it can also match a query. Postgres stores it without a vector, so queries return only chunk rows. Only chunk rows have _hevlayer_parent_id set. A row whose text fits in one chunk is written unchanged.

A chunked serving.prefer: turbopuffer profile is embedded on the Layer side, like an unchunked one, and needs a configured Turbopuffer embedding provider. Without one the write returns 503 and nothing is stored.

The section strategy splits a structured row by field before windowing. Set sectionSource: jsonFields and list the source attributes in fields, and each field becomes its own section. The chunk id becomes {id}#{field}#{n} and the field name is stored in sectionAttribute (default section). Put the window settings for each section in a nested split block.

Chunking has these limits:

• The profile must use serving.prefer: worker or turbopuffer, on any store.
• A namespace can have one chunked embedding attribute.
• Chunking cannot be combined with modality: image.
• Writes must use upsert_rows with a string id on every row. Columnar upsert_columns writes are rejected.

Because the split runs inside the write request, a large document makes that write slower and uses gateway CPU that is also serving queries. For bulk ingest, or for sources that need extraction first (PDFs, scans), use a two-stage pipeline. Its CPU worker extracts and chunks outside the gateway and scales separately, and its GPU stage embeds the staged chunks. The pipeline takes the same chunk settings; see chunking.

Performance accounting

Write and query responses report embedding work under performance:

{
  "rows": [ /* ... */ ],
  "performance": {
    "embedding_tokens": 8,
    "embedding_ms": 42
  }
}

Queries omit embedding_tokens on a cache hit. Layer merges provider measurements into the same object and exports the work as hevlayer_embed_tokens_total and hevlayer_embed_compute_seconds_total, labeled by namespace, store kind, model, and serving mode.