Move

PDF extraction

layer-extract is the stock extract worker for a Pipeline. Point it at an object-store prefix of PDFs or a cloud drive and it parses each document with LiteParse and stages one row per page into the target namespace. Each row carries the page’s markdown, its tables, the page number, the parser that produced it, and a link back to the page in the source document. Reruns parse only documents whose bytes changed.

The same image has a second command, write, which writes staged pages into the namespace. A PDF pipeline is two Pipeline resources that share one pipelineId: the extract stage stages pages, and the write stage writes them. The namespace’s schema embeds each page’s full text natively, so no stage computes vectors and no page is truncated.

Declare it

The extract stage:

apiVersion: hevlayer.com/v1alpha1
kind: Pipeline
metadata:
  name: contracts-extract
  namespace: layer
spec:
  pipelineId: contracts
  target:
    namespace: contracts-pages
  sourceRef:
    kind: objects
    uri: s3://your-bucket/contracts/
    cacheUri: s3://your-bucket/layer-extract-cache/
    parse:
      processes: 2
      timeoutSeconds: 120
    rescanSeconds: 900
  worker:
    image: <acct>.dkr.ecr.us-east-1.amazonaws.com/mesh:layer-extract-<version>
    computeClass: cpu
    podSpec:
      serviceAccountName: layer
      containers:
        - name: worker
          env:
            - name: HEVLAYER_SNAPSHOT_REGION
              value: us-east-1
          resources:
            requests:
              cpu: 1500m
              memory: 3Gi
            limits:
              memory: 6Gi
  scaling:
    pool: cpu
    mode: fixed
    replicas:
      min: 1
      max: 1

The write stage runs the same image with args: [write]:

apiVersion: hevlayer.com/v1alpha1
kind: Pipeline
metadata:
  name: contracts-write
  namespace: layer
spec:
  pipelineId: contracts
  target:
    namespace: contracts-pages
  worker:
    image: <acct>.dkr.ecr.us-east-1.amazonaws.com/mesh:layer-extract-<version>
    computeClass: cpu
    podSpec:
      containers:
        - name: worker
          args: [write]
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
  scaling:
    pool: cpu
    mode: fixed
    replicas:
      min: 1
      max: 1

Set worker.image on both resources. The image is published to the mesh repository with a layer-extract-<version> tag in the mesh-account ECR registry, next to the other stock workers.

Run the extract stage as one replica in mode: fixed. It sweeps the prefix on a timer rather than draining a queue, and sweeps do not coordinate across pods: two replicas would parse every document twice and overwrite each other’s manifest. Scale it with parse.processes and the CPU request instead. The write stage claims documents with leases, so it can run more replicas.

The worker reads S3 with the pod’s AWS credentials (the default provider chain, so IRSA on EKS). The service account needs read access to uri and read and write access to the cache location. The S3 client takes its region, and an optional S3-compatible endpoint, from HEVLAYER_SNAPSHOT_REGION and HEVLAYER_SNAPSHOT_ENDPOINT. The operator injects both when the chart has a snapshot bucket; otherwise set the region on the container as above.

Cache identity and unchanged pages

Conversion identities are scoped to each file format’s handler, Python and converter versions, font packages, and output libraries (PDF, image, font, XML and Office import libraries). The whole apt dependency closure is not identity: unrelated desktop, networking and multimedia package security patches leave every format’s identity unchanged. Adding email, HTML or Box Notes support also leaves unchanged Office handlers valid. Changes to a used handler, LibreOffice component, output library or font still invalidate relevant results. The complete installed package inventory is available as image provenance at /usr/local/lib/layer-extract/runtime-converter-profile.json.

Legacy global fingerprints cannot reveal their old package versions. Before rebuilding an image that used them, export its customer-data-free runtime profile with converter_identity.py --export-profile and supply the JSON as Docker build argument LEGACY_CONVERTER_PROFILE (the verification build and merged-image publishing workflows accept legacy_converter_profile). The extractor checks the historical handler, output dependencies and Python/library versions against this baseline before accepting its legacy identities. Unrelated OS updates therefore preserve those existing keys too. Keep this original profile in subsequent builds until the legacy receipts are retired; omitting it cannot recover old hashes after package updates. The rollout recipe includes export, build and acceptance commands.

When re-extraction produces identical page text, content hashes and all extractor-owned attributes, the extractor compares the gateway’s stored pipeline chunks and skips staging. It counts the document as skipped_unchanged, leaving embedding state and write-triggered Functions untouched. These reads use the pipeline cache or S3 backing, without querying the target namespace. A comparison failure stops staging and reports stage_error rather than risking a rewrite. Downstream derived attributes and vectors are outside the staged-chunk receipt.

Source

kind: objects is an open source kind: the operator hands spec.sourceRef to the worker verbatim as HEVLAYER_SOURCE_REF, and layer-extract validates it at startup.

`sourceRef` field	Default	Purpose
`kind`	required	Must be `objects`.
`uri`	required	The prefix to extract, `s3://bucket/prefix/`. `file:///absolute/dir/` works for local runs. The prefix is listed recursively.
`cacheUri`	see below	Where the manifest and parse cache live. The worker writes under `{cacheUri}/{pipelineId}/`.
`suffixes`	`[".pdf"]`	Object key suffixes to extract, matched case-insensitively. Every match is parsed as a PDF.
`parse.processes`	the pod’s available cores	Parse processes in the pool.
`parse.timeoutSeconds`	`120`	Hard limit per document. A parse over it is killed and named `timeout`.
`parse.maxPages`	`2000`	Pages parsed per document. Later pages are dropped.
`parse.maxBytes`	`536870912` (512 MiB)	Larger documents are named `too_large` without being downloaded.
`rescanSeconds`	`3600`	Seconds between sweeps of the prefix. The worker stays up between sweeps.

Without cacheUri, the cache goes to s3://$HEVLAYER_SNAPSHOT_BUCKET/pipelines/{pipelineId}/extract/, the chart’s snapshot bucket. If the operator injects no snapshot bucket, cacheUri is required and the worker exits at startup without it.

Page rows

Each page becomes one row with id {doc_id}#p{n}, where n is the 1-based page number. doc_id is the first 32 hex characters of the sha256 of the document’s source URI, so it is stable across runs and the page id stays under the store’s 64-byte id limit. A document that moves to a new key gets a new doc_id.

Attribute	Contents
`text`	The page as markdown. Full-text searchable (BM25) with fuzzy matching, not filterable.
`_hevlayer_parent_id`	The document id, the same parent convention as gateway chunking.
`page`	Page number, 1-based.
`page_count`	Pages in the source document, including any past `parse.maxPages`.
`page_label`	The page’s printed label (for example `iv`), when the PDF defines one.
`tables`	The page’s tables, each as a markdown table. Absent when the page has none. Not filterable.
`blocks`	Compact layout blocks as a JSON string: `t` (block type), `b` (bounding box in 72-DPI points, top-left origin) and `s` (byte span of the block’s text in `text`, when it can be located). Absent when the page has none, or when the JSON is over 32 KiB. Not filterable.
`ocr`	`none`.
`complexity`	LiteParse’s complexity reasons for the page, comma-separated (for example `scanned,no-text`). Empty for a plain text page.
`needs_ocr`	`true` when `complexity` is non-empty.
`source_uri`	The document’s URI with a page deep link, `s3://your-bucket/contracts/a/lease.pdf#page=3`.
`content_hash`	sha256 of the document bytes.
`parser`	The engine and version, `[email protected]`.
`path`	The object key relative to `uri`.
`name`	The last segment of `path`.
`source_bytes`	Document size in bytes.
`source_modified`	The object’s last-modified time, when the listing reports one.

Pages with no text layer, such as scans, are staged with empty text. LiteParse flags sparse-text and embedded-images as well as scans, so needs_ocr is much broader than “no text layer”. Count the reasons in complexity rather than the flag.

The worker accepts up to 1 KiB of junk before the %PDF- header, as PDF readers do, and strips it before parsing.

Source metadata is copied onto every page so filters work per page: filter on _hevlayer_parent_id for one document’s pages, or on path for a folder.

Incremental runs

Each sweep lists the prefix and compares it with a manifest, manifest.json under the cache location. The manifest maps each document id to the listing fingerprint (the S3 ETag and size), content hash and parser it was last staged or failed with.

Unchanged documents with the same fingerprint are skipped without being downloaded, parsed or restaged. A second run over an unchanged prefix parses nothing.
Touched documents (a new ETag, the same bytes) are downloaded and hashed, then skipped.
Changed documents are parsed and restaged. Restaging resets the document to pending, so the write stage writes it again and the namespace re-embeds its pages.
Duplicate documents parse once. The parse cache, under parses/ in the cache location, is keyed by parser and content hash, so identical bytes under different keys share one parse.
Removed documents have all their page rows deleted from the namespace.
Shorter documents drop their trailing pages: when a new version has fewer pages than the staged one, rows past its last page are deleted.

A new layer-extract release that changes the parser version re-parses every document. The manifest also records the parse limits each document was handled under, so raising a limit retries exactly the documents that hit it:

a larger timeoutSeconds retries timeout failures
a larger maxBytes retries too_large failures
a larger maxPages retries truncated documents

Other unchanged documents and other failures stay skipped. Entries written by an older layer-extract that did not record limits get one retry to record them. To reprocess everything, point cacheUri at a new prefix.

If a listing returns no documents at all while the manifest knows some, the worker keeps the existing rows and logs a warning. An empty listing is far more often a wrong prefix or a permissions change than a deliberate deletion.

The manifest also records the pipeline id and source URI. If either changes, the worker starts a fresh manifest.

Failures

A document that cannot be turned into rows is recorded with a named reason. Each failure is written to the manifest with status: failed, its reason and a message, and counted in the sweep report. Parse failures are also logged with the document’s key.

Reason	Meaning	Retried
`encrypted`	Password-protected, or a security handler PDFium cannot open.	When the bytes change.
`invalid_pdf`	Not a PDF, or damaged past what PDFium recovers.	When the bytes change.
`no_pages`	The document has no pages.	When the bytes change.
`incomplete_parse`	The parser omitted or duplicated pages within the requested range. Nothing is staged; prior rows stay searchable.	When the bytes change.
`too_large`	Over `parse.maxBytes`. Not downloaded.	When the object changes.
`timeout`	Over `parse.timeoutSeconds`. The parse process was killed.	When the bytes change.
`crashed`	The parse process died, from a PDFium crash or an OOM kill.	Once on the next sweep, then when the bytes change.
`parser_error`	Any other error the parser reported.	When the bytes change.
`source_error`	Reading the document from the source failed.	Every sweep.
`stage_error`	Staging the page rows through the gateway failed.	Every sweep.

Each parse runs in its own child process, so a document that hangs or crashes PDFium takes down only that process. Each child is reaped before another document takes its slot, and the sweep continues. On Linux each child has a 2 GiB address-space limit.

If a changed document fails to parse, its previous version’s page rows stay searchable. A later successful parse replaces them and removes any trailing pages the new version no longer has. Removing the document from the source deletes its rows, including rows kept from an earlier version, unless the whole listing came back empty.

Crashes and conversion failures

A crashed or conversion_error is usually the environment, not the document, so the next sweep retries each one exactly once (a persisted attempt count in the manifest; a second failure of the same bytes and parser is permanent until the bytes change). A process that cannot even start because the pod is out of processes, memory or descriptors is a transient source_error and does not use that retry. The report’s failure_causes names why, as reason:cause with fixed tokens only: crashed:exit_signal_9, crashed:exit_code_101, conversion_error:exit_code_1, conversion_error:output_invalid, source_error:spawn_resource, conversion_error:workspace_full. The same cause appears on each document failed log line. The image runs the worker under tini so helper processes that LibreOffice leaves behind are reaped.

OCR failure causes

An OCR request that fails marks the document ocr_error. Invalid images (http_400), persistent size rejections (http_413) and empty_text settle without another sweep attempt. With a fallback, both failures must be permanent. Other OCR failures retry for at most three sweeps for the same content and parser. A changed document or OCR configuration starts a new attempt budget. Settled failures remain in manifest.json with their cause and attempt count; settled_ocr and settled_ocr_causes in the sweep log count them separately from failed and skipped_failed. retry_exhausted means the three-attempt budget was consumed; the manifest retains the final service cause.

PDF rendering retains aspect ratio and caps page images below 16 megapixels before bitmap allocation. A size rejection retries once at half width and height. Known OCR server error bodies are logged without client pixels or text; arbitrary proxy bodies are suppressed. The report’s ocr_failure_causes says why, with fixed tokens only (connect_deadline, timeout, transport, http_<code>, empty_text, wrong_backend, malformed_reply, body_limit), and ,fallback=<cause> when the fallback backend failed too. A page the backend reads successfully and finds no text on (a blank, decorative or chart-only page, common after an Office conversion) is not a failure: the page keeps its native content with ocr: none and its complexity reasons, and the document stages. With a fallback configured, both backends must answer empty; an empty answer next to a failed fallback is still ocr_error, because the page’s text is then unknown. empty_text only appears in a cause as empty_text,fallback=<cause>. A reset or a 502/503/504 from a restarting backend is retried up to three times; a timeout is not, because the backend may still be working on it. A burst of concurrent pages queues behind the backend’s single inference lock and counts that wait against timeoutSeconds.

Process some documents first

A sweep works through the listing in order. To get chosen documents ahead of a long backlog without narrowing the selection, name their provider item ids (for a cloud drive, the Graph item id):

sourceRef:
  extract:
    priority:
      itemIdsFile: /etc/layer/priority-items.txt   # `<itemId>` or `<driveId> <itemId>` per line, # comments
      # itemIds: [<id>, ...]                       # inline ids rank first
      # items: [{driveId: <drive>, itemId: <id>}]  # drive-scoped, rank after itemIds

Only the order changes. Every selected document is still processed, in listing order after the prioritized ones, and ids, manifest, dedupe and OCR settings are untouched, so a prioritized sweep re-parses nothing an unprioritized one would skip. An id ranks by its position (itemIds, then items, then file lines; a repeat keeps its first rank), is matched on any location of a document, and applies to the current version of that item. An id not in the listing is counted, not an error. An unreadable itemIdsFile fails the sweep. Ids are as private as the documents: keep them in a mounted Secret or ConfigMap, never in git, and they are never logged. The report carries counts only: priority_matched, priority_absent (configured ids missing from this listing) and priority_failed (matched documents that failed this sweep).

Sweep report

Each sweep logs a sweep completed line with its counts: documents, staged, skipped unchanged, skipped known failures, failed, deleted, pages staged, pages parsed per second per busy core, and time spent parsing, downloading and staging. The full report adds failures by reason, the failed documents with their keys, truncated documents, and complexity reason counts.

Variable	Effect
`LAYER_EXTRACT_REPORT_PATH`	Write the full JSON report to this path after each sweep.
`LAYER_EXTRACT_ONCE`	`1` or `true`: run one sweep, print the report to stdout, and exit.
`LAYER_EXTRACT_DRY_RUN`	`1` or `true`: parse without staging, and imply `LAYER_EXTRACT_ONCE`. No gateway is needed.
`RUST_LOG`	Log filter. Defaults to `layer_extract=info`. Logs are JSON on stderr.

LAYER_EXTRACT_ONCE and LAYER_EXTRACT_DRY_RUN are for local runs; on a Deployment, a worker that exits is restarted. A dry run can share cacheUri with a real run. It warms the parse cache, which is keyed by content, but does not write the manifest, so a later real run still stages every document.

Write stage

layer-extract write claims staged documents from the pipeline and writes their pages as rows through PUT /v2/pipelines/{id}/documents/{doc_id}/rows. It sends no vectors. Once every batch of a document’s pages is written, it marks the document indexed.

Embedding. Each write carries the page schema, which declares text with embed: {model: "voyage/voyage-4"} and no serving. So the namespace embeds natively: on Turbopuffer, Turbopuffer computes a 1,024-dimension vector from the page’s full markdown. The model reads up to 32,000 tokens; the longest page in our test set was under 14,000. Query the namespace with Embed and it uses the same model.
Text. Every page with text is written byte-for-byte, with no truncation. Native embedding rejects empty strings, so a page with no text is written as {name}, page {n} (no text layer).
Schema. text declares full_text_search: true and fuzzy: true alongside native embedding. This supports BM25 and typo-tolerant HybridText queries. text, tables and blocks are declared non-filterable, because page content regularly exceeds Turbopuffer’s 4 KiB limit on filterable values. The distance metric is cosine_distance.
Failures. A document whose write fails is released back to pending for a later claim. Page ids are stable, so a retried write overwrites what an earlier attempt left. Leases are renewed while a batch is in flight.

Variable	Default	Purpose
`LAYER_EXTRACT_WRITE_MODEL`	`voyage/voyage-4`	The embedding model in the page schema. It must be one your store’s native embedding offers.
`LAYER_EXTRACT_WRITE_BATCH`	`8`	Pages per write request.
`LAYER_EXTRACT_WRITE_DOCS`	`1`	Documents per write request. Above 1, consecutive claimed documents share one bulk write, which spreads the fixed cost of a namespace write over them. Each document is completed only after the request that carries it succeeds; a failed request releases all its documents.
`LAYER_EXTRACT_WRITE_MAX_ROWS`	`256`	Most rows in a packed request. A document with more rows is written on its own, in `LAYER_EXTRACT_WRITE_BATCH` pages per request.
`LAYER_EXTRACT_WRITE_MAX_BYTES`	`4194304`	Most serialized bytes of page rows in one write request, for packed requests and for the batches of a single document alike (a batch also stops at `LAYER_EXTRACT_WRITE_BATCH` rows; one row over the cap goes alone). Requests of many long, dense pages can be refused by the store while smaller ones pass, and row count does not bound that: if writes to a namespace with large documents fail fast, lower this (for example 65536 to 131072) and read the refusal logs for the size that was refused. The default leaves the row-count split unchanged.
`LAYER_EXTRACT_WRITE_CLAIM`	`16`	Documents per claim.
`LAYER_EXTRACT_WRITE_LEASE_SECONDS`	`900`	Claim lease.
`LAYER_EXTRACT_WRITE_POLL_SECONDS`	`5`	Wait between claims when nothing is pending.
`LAYER_EXTRACT_WRITE_BACKOFF_SECONDS`	`5`	Pause before the next claim when more than half of the last claim was released, which means the store is refusing or losing writes. Doubles for each consecutive such claim, with up to 50% jitter, and resets as soon as a claim mostly succeeds. `0` turns it off.
`LAYER_EXTRACT_WRITE_BACKOFF_MAX_SECONDS`	`60`	The longest pause.

The stage connects with the variables the operator injects: HEVLAYER_BASE_URL, HEVLAYER_PIPELINE_ID, HEVLAYER_TARGET_NAMESPACE and LAYER_GATEWAY_API_KEY.

Enable fuzzy search on an existing page namespace

For Turbopuffer-backed namespaces, enable fuzzy: true on the existing text attribute with a schema-only namespace write. Preserve its current type, full-text search settings, filterable: false, and embedding configuration; fuzzy is a sibling of full_text_search, not a setting inside it. Turbopuffer supports this update in place, without rewriting rows or rerunning the extraction pipeline.

The new index builds asynchronously. Queries that require an index still being built can return HTTP 409; wait for indexing to finish before checking fuzzy or HybridText results. Changing full-text search parameters also rebuilds that index in the background, with queries using the previous settings during the rebuild. Adding fuzzy support does not change BM25’s ranking expression. Verify the stored schema and both BM25 and HybridText queries after the build. The Turbopuffer full-text search guide explains how fuzzy matching contributes to ranking.

Local embedding stage

The image also retains args: [embed] for deployments using a layer-embed CPU sidecar. It discovers the model at the sidecar’s /v1/models and computes vectors before writing them through the pipeline vector API. This stage uses LAYER_EMBED_URL (default http://127.0.0.1:8081) and LAYER_EXTRACT_EMBED_MODEL (default BAAI/bge-small-en-v1.5). LAYER_EXTRACT_EMBED_MAX_CHARS defaults to 1500: only that prefix is sent to the model, while the row retains the full markdown. Run either this stage or the native write stage for a pipeline, with the same pipelineId as its extractor.

Limits

Limit	Default	Behavior
Document size	512 MiB (`parse.maxBytes`)	Larger documents are named `too_large` and not downloaded.
Pages per document	2,000 (`parse.maxPages`)	Later pages are dropped. `page_count` still reports the full count, and the sweep report counts the document as truncated.
Parse time	120 s per document (`parse.timeoutSeconds`)	The parse process is killed and the document is named `timeout`.
Layout blocks	32 KiB of JSON per page	`blocks` is left off the row.
Embedded text	The model’s context: 32,000 tokens for `voyage/voyage-4`	A longer page can’t be embedded. Choose a model whose context fits your longest pages.

Up to twice parse.processes documents are in flight at once, and each is held in memory while it downloads and parses. Size the memory limit for your largest documents.

Files attached to REST rows

Some APIs hand out documents row by row: POST /api/invoices/{id}/showinvoice returns the invoice PDF. A rest source with an attachment block, run by layer-extract instead of the REST source worker, lists the rows the way a REST Warehouse does, fetches each row’s file with the Warehouse’s own auth (static token or login), and stages one row per page. Nothing else in the Pipeline changes: the same parse cache, OCR, write stage and retained originals apply as for object-store and cloud-drive sources.

apiVersion: hevlayer.com/v1alpha1
kind: Pipeline
metadata:
  name: ki-invoices-extract
spec:
  pipelineId: ki-invoices
  targetNamespace: ki-invoices
  worker:
    image: <your layer-extract image>
  sourceRef:
    kind: rest
    warehouseRef: ki
    request: { path: /api/invoices }
    pagination: { kind: page, pageSize: 100, pageSizeParam: pageSize, pageParam: page, totalsHeader: X-Pagination }
    response: { items: "$[*]" }
    cursor: { field: id }
    mapping:
      id: "{{id}}"
      attributes:
        invoice_number: invoiceNumber
        customer_id: customerId
        hasFile: hasFile
    attachment:
      request:
        method: POST
        path: /api/invoices/{id}/showinvoice
      when: hasFile
    extract:
      ocr: { url: pp-ocrv5 }

`sourceRef.attachment` field	Default	Purpose
`request.path`	required	The file’s path. `{name}` is replaced with that attribute of the mapped row (`{id}` is the row id), percent-encoded.
`request.method`	`GET`	`GET` or `POST`.
`request.body`	none	Optional JSON object sent with the request. String values may use `{name}`.
`when`	every row	A mapped attribute that must be present and not `false`, `null` or empty for the row to have a file. Rows without one are listed but produce no pages.
`extension`	`pdf`	The file type, used for the document key and, with `extract.normalize`, for conversion.

The rest of sourceRef is the REST source’s own (request, pagination, response, cursor, mapping, request.forEach), and extract, parse, cacheUri and rescanSeconds are this page’s. A listing that fails on any partition fails the sweep, so a partial listing never deletes pages. layer-extract is the only worker that runs an attachment; the REST source worker refuses a Pipeline that declares one.

Each document is rest://{warehouseRef}/{id}.{extension}, so page ids are stable across sweeps and doc_id follows the parent row. Bytes are checked against the file type (a pdf response must start with %PDF-), so an HTML error page is a named failure, not a page of garbage.

Pages carry the parent row’s mapped attributes (invoice_number, customer_id, …) next to the usual page attributes, plus source_row_id, the parent row id. A parent attribute never replaces a page attribute or a reserved _hevlayer_* or original_* attribute. When the row changes but the file does not, the pages are restaged from the cached parse, with no new download-and-parse or OCR. Identical bytes under two rows parse once and retain one original, keyed by content hash.

One document per file across a drive and a REST source

The same PDF often lives in two systems: an invoice attached to a REST row and a copy in a SharePoint library. Indexed twice, search returns both and neither knows about the other. Add the REST source to the cloud-drive Pipeline’s sourceRef.merge and layer-extract collapses the two by content, so a file found in both is one document with a location per source.

  sourceRef:
    kind: clouddrive
    warehouseRef: sharepoint
    drives: { sites: [{ site: "example.sharepoint.com:/sites/Finance", library: Docs }] }
    mapping: { id: id, version: file.hashes.quickXorHash }
    merge:
      - kind: rest
        warehouseRef: ki
        attributePrefix: ki_
        request: { path: /api/invoices }
        # pagination, response, cursor and mapping as for a REST source
        attachment:
          request: { method: POST, path: /api/invoices/{id}/showinvoice }
          when: hasFile

`sourceRef.merge[]`	Purpose
entries	Each is a `rest` sourceRef with an `attachment` of `extension: pdf`, as above, and its own `warehouseRef`. The primary must be `kind: clouddrive`. `extract`, `parse`, `cacheUri` and `request.forEach` belong to the primary and are rejected here.
`attributePrefix`	Lowercase letters, digits and underscores that namespace the row’s attributes on the pages. Default: the `warehouseRef` with other characters turned into `_`, plus a trailing `_` (`bcc-ki` becomes `bcc_ki_`).

The operator resolves each entry’s Warehouse (it must be Verified) and gives the worker its connection as HEVLAYER_WAREHOUSE_<n>, with the credential mounted under /var/run/hevlayer/warehouse-<n>/.

The key is the sha256 of the original bytes (original_sha256 on the rows). A REST file has no listing hash, so the worker downloads each row’s file once, hashes it, and remembers the result per row fingerprint in rest-hashes.json under the cache location. A settled listing downloads nothing for hashing. Then, per sweep:

A REST file whose sha256 matches a document the drive source has already processed joins that document. With dedupe: item (the default whenever extract.normalize is on) the drive source keeps one document per item, so when several items share the bytes, the REST copy joins the one with the lowest document id and the other items stay as they were.
REST rows that share a sha256 but have no drive copy form one document together.
A row whose file could not be fetched or hashed keeps its own document, which fails the usual way.

Document ids. A drive document keeps its existing id (the provider hash, as before), so merging changes none of its page ids and re-embeds nothing. A document only REST sources have takes the first 32 hex characters of the content sha256. The page ids are {doc_id}#p{n} in both cases. A REST-only document that later gains a drive copy moves to the drive id: the sweep that sees the match deletes the REST-only pages and the drive document gains the REST location from its cached parse. An unhashable row keeps the per-source id sha256(rest://{warehouseRef}/{id}.pdf), so existing REST-only Pipelines are unaffected and need no migration.

Locations. The locations attribute is a JSON list with one entry per copy. On a document that has a REST copy, every entry names its source (clouddrive or rest); a document with only drive copies is written as before, without source. The drive copies come first, then REST copies ordered by warehouse and row id, so the same listing always produces the same rows.

[
  {"source": "clouddrive", "path": "Invoices/a.pdf", "webUrl": "https://...", "folderUrl": "https://..."},
  {"source": "rest", "path": "ki/17.pdf",
   "rest": {"warehouse": "ki", "rowId": "17", "url": "rest://ki/17.pdf",
            "attributePrefix": "ki_", "attributes": {"invoice_id": 17, "vendor": "..."}}}
]

Attributes. The page’s links and provenance (webUrl, folderUrl, path, name, pdf, source_uri, original_*) come from the drive copy when there is one. The first REST row’s attributes are added under the prefix (ki_invoice_id, ki_vendor, ki_source_row_id), so they never collide with the drive source’s attributes; the other rows’ attributes are only in locations. A REST-only document keeps its rest:// links and has no webUrl. A change to a REST row’s attributes patches the existing rows from the cached parse with no new parse or embedding.

The sweep report counts merged_documents (found under both kinds of source) and rest_only_documents.

Office, text and document images

Set sourceRef.extract.normalize: true to convert originals before the stock LiteParse/PDFium extraction and OCR path. The stock image includes LibreOffice, img2pdf and pikepdf; consumers do not need a private parser. Existing PDF-only sources retain their selection, deduplication and parser behavior.

For a cloud drive, listing a non-PDF type in include.mimeTypes (for example the .docx, .xlsx and .pptx types) turns normalize on. With no list, a normalized source selects every file in its roots and counts unknown formats; with a list, only the listed types are selected and the rest are filtered, not counted. Omit include.maxBytes and use dedupe: item. Set parse.maxBytes to count oversized originals as too_large. Object sources ignore suffixes in this mode. Unknown formats become unsupported_format, not filtered files. Use only verified source roots; personal drives require separate selection and are excluded from the Story contract.

Original	Conversion and page identity
PDF	Original PDF pages; existing parser and renderer
DOC/DOCX/DOTX/ODT/RTF, PPT/PPTX/ODP	LibreOffice PDF export; converted PDF pages, with a fixed locale and font set
XLS/XLSX/XLSM/XLSB/ODS	LibreOffice PDF export with its print pagination: each sheet starts on a new page, in workbook order, and a wide or long sheet continues onto further pages. Page numbers are not sheet numbers
TXT/MD	Strict UTF-8 text, rendered literally by LibreOffice Writer; converted pages
Box Notes (`.boxnote`)	Strict JSON attributed text (`atext.text`), rendered as text pages
MSG/OFT/EML	Message headers and plain body (HTML body fallback), rendered as text pages; attachments excluded
HTML/HTM	Visible UTF-8 text, excluding scripts, styles and document head; rendered as text pages
MPP, Pages	`unsupported_format`: no bundled Project converter; Pages requires an exported rendition
CSV	UTF-8, comma-separated, double-quote escaping; LibreOffice Calc print pages
PNG/JPEG/TIFF/BMP/GIF/WebP	img2pdf; each supported raster frame becomes a page, then existing OCR routing
Camera RAW, including DNG/CR2/NEF/ARW	Unsupported originals; no automatic developing, embedded-thumbnail substitution or photo classification
Other formats	Counted `unsupported_format`; add stock converter support before processing

The run report counts bytes_downloaded and bytes_converted (original bytes sent to the converter). Both are 0 on a second sweep of unchanged files. Unsupported and failed files appear in failures_by_reason, never as silent drops.

Generic application/octet-stream is admitted for extension-based conversion. The converter validates JSON, OLE, ZIP, RTF, RFC822 or HTML content; a suffix alone does not turn arbitrary bytes into a document. Add text/html, application/vnd.ms-outlook, message/rfc822, application/vnd.openxmlformats-officedocument.wordprocessingml.template, application/vnd.ms-excel.sheet.macroEnabled.12, application/vnd.ms-excel.sheet.binary.macroEnabled.12, application/rtf, and application/octet-stream to a drive MIME list to select these types. Add application/vnd.ms-project to account for unsupported MPP originals. Generic MIME can also select unrelated files; those count as unsupported. Email attachments are excluded because embedded files require independent identities and selection. MSG/OFT supports Unicode property streams or strict UTF-8 ANSI streams, plain body or UTF-8 HTML; RTF-only message bodies fail conversion rather than silently publishing only headers. HTML conversion never fetches external resources.

Conversion errors, corrupt containers and undecodable text become conversion_error. Password-protected Office inputs that cannot export are conversion failures. PDF failures retain their existing reasons. The per-document process deadline covers conversion and parsing; its process group is killed and reaped, including converter descendants. Converted PDFs are bounded to 512 MiB. A raster is bounded to 100 million pixels. Converter diagnostics are suppressed.

Original bytes remain the content-hash authority. Parent IDs are based on the original item URI, and page IDs stay {parent}#p{number} across versions. Page rows retain original_uri (view link), original_source_uri (stable fetch identity), original_sha256, original_version, provider drive/item/MIME/ETag and locations. For non-PDF originals, source_uri opens the original file; page_basis: converted_pdf explicitly names the page coordinate system. A conversion JSON string records original format, converter recipe and normalized PDF SHA256. Metadata and trailer IDs are canonicalized; reproducibility requires the same immutable image, libraries, locale and fonts. Converted page numbers must not be presented as Office source pagination or camera capture numbers.

The parse cache separates format, declared MIME and converter stack fingerprint (script, installed package versions and fonts) as well as original byte hash. Identical bytes in two provider items can reuse parsing while retaining separate parents and provenance. Unsupported and failed originals remain in the manifest and report, with no page rows published for the failed version. A last complete prior version remains until removal or successful replacement.

Turning on normalize or adding imageMimeTypes to a source that already extracted PDFs does not re-parse them: a native PDF keeps its parse identity, so settled PDFs (and their OCR) are reused, and only the newly selected formats are converted and parsed. Each converted format keeps its own identity.

Scanned images from a drive

Scans often sit in a SharePoint library as images, not PDFs. A clouddrive source can read selected image types as documents without normalizing every file. List them in sourceRef.extract.imageMimeTypes; the choice of which images are documents is the Pipeline author’s, made by MIME type and folder.

sourceRef:
  kind: clouddrive
  warehouseRef: m365
  drives:
    sites:
      - site: contoso.sharepoint.com:/sites/Projects
        library: Documents
  include:
    paths: [/Scans, /Field Reports/Scanned]   # folders that hold documents
  extract:
    imageMimeTypes: [image/tiff, image/jpeg, image/png]
  mapping:
    id: id
    version: file.hashes.quickXorHash

imageMimeTypes accepts image/png, image/jpeg, image/tiff, image/bmp, image/gif and image/webp. include.mimeTypes defaults to application/pdf plus the listed types and may not name anything else. Deduplication stays content by default. Each image, or each frame of a multi-page TIFF, becomes one page row, converted by the stock converter described above and run through the existing OCR routing (PP-OCRv5, falling back to Tesseract). Rows carry locations, webUrl, folderUrl, path and name as PDF rows do. For an image, source_uri is the file’s webUrl with no #page= fragment, there is no pdf link, and page_basis is converted_pdf.

Failures use the reasons above and are counted in the sweep report. HEIC is not supported: image/heic is not an accepted imageMimeTypes value, so real HEIC files are filtered by include.mimeTypes and skipped, and a .heic name carrying a listed MIME type is unsupported_format. Any other file whose extension is not a supported raster is unsupported_format. A file whose bytes are not the format its extension and MIME type claim, or that cannot be decoded, is conversion_error. Files filtered out by include.paths or include.mimeTypes appear in the metadata worker’s skipped counts, not as failures.

This does not tell photos from documents. Images that should be searchable by content, not text, belong to a clouddrive-images source; choose between them with include.paths. Automatic classification of mixed folders is not part of this feature.

Verified cached originals and sealed manifests

For extensionless cached objects, use a manifest-backed objects source. The manifest supplies the original name and MIME; the fetch key remains unchanged. The worker makes no Graph requests and does not interpret a private SQLite ledger. An acquisition owner exports a complete JSONL selection joined to its current verified receipts, including pending and failed items, then seals the exact bytes with SHA256. Update the snapshot seal when receipts change. Never provide a success-only delta: absent items are reconciled as removals.

sourceRef:
  kind: objects
  uri: s3://example-bucket/selected/
  cacheUri: s3://example-bucket/extract-cache/
  extract:
    normalize: true
  dedupe: item
  manifest:
    uri: s3://example-bucket/manifests/selection.jsonl
    sha256: <64-lowercase-hex-SHA256-of-exact-JSONL-bytes>
    warehouseRef: example-sharepoint
    complete: true
    graphVersionKeys: true
    mapping:
      drive: drive_id
      item: item_id
      name: metadata.name
      mime: metadata.file.mimeType
      path: metadata.parentReference.path
      webUrl: metadata.webUrl
      etag: metadata.eTag
      size: metadata.size
      modifiedAt: metadata.lastModifiedDateTime
      bucket: artifact.bucket
      key: artifact.key
      sha256: artifact.sha256
      receiptEtag: artifact.etag
      receiptBytes: artifact.bytes
      verified: artifact.s3_readback_verified

Mappings are dot-separated JSON object paths. Without a mapping, the same canonical field names are read directly. driveMetadata, folderUrl and reuse are optional additional fields. path can be a provider parent path or a full original path; the original filename is appended when needed. Preserve provider links verbatim. Supply original identity, name, ETag and size even when a receipt is pending. An absent MIME is unspecified and still undergoes stock byte/container validation.

bucket must match the configured source bucket; key is the complete S3 key within its prefix. With graphVersionKeys: true, the relative key must equal versions/SHA256(drive + "/" + item + "/" + ETag) using UTF-8 bytes and lowercase hex. Receipt ETag and byte count must match original metadata, verified must be the JSON boolean true, and the receipt SHA256 must match fetched bytes. Missing, stale, corrupt or out-of-prefix receipts become counted source_error failures. Every sweep verifies bytes again, including settled/cache-hit items. Manifest SHA mismatch or duplicate item identity aborts the entire listing before removal reconciliation. The manifest is bounded to 512 MiB.

The stable source identity is clouddrive://{warehouseRef}/{drive}/{item}; original_source_uri and document/page IDs use this identity rather than a versioned S3 key. original_fetch_uri records the cached object URI. ETag and byte SHA identify the original version. An unchanged-byte version or link update patches provenance through existing gateway APIs without reparsing. Cache state binds destination, mode, source prefix, warehouse and mapping; changing the snapshot URI or SHA retains reconciliation history. Use a distinct warehouse identity and cache for a separately owned selected-root contract. All staging, publication and queries use gateway APIs.

Reusing native or OCR page outputs

A manifest may bind reuse to an existing stock ParsedDocument JSON blob (page_count, ordered pages, optional conversion). Set encoding: parse_worker_response to reuse the stock parse-worker stdout envelope {"ok": <ParsedDocument>} directly, including retained native or OCR responses. The default encoding is parsed_document. Error/image responses are rejected. Use encoding: page_rows for already-produced gateway page rows. rowsPath selects an array (empty means the root); default fields are text and metadata.page, metadata.page_count, metadata.original_sha256, optional metadata.ocr, metadata.complexity, metadata.tables, metadata.blocks, metadata.page_label and metadata.conversion. Stock comma-separated complexity and JSON-text blocks/conversion retain their existing wire encoding; native JSON arrays are also accepted. pageMapping overrides these with dot paths named markdown, number, pageCount, originalSha256, ocr, complexity, tables, blocks, label and conversion.

For a flat retained projection, a mapping such as {"number":"page","pageCount":"page_count","originalSha256":"content_sha256","complexity":"complexity"} keeps text as the default markdown field. Every row needs a valid original byte SHA; only the receipt’s matching hash is selected. Selected rows must agree on page count and conversion provenance, and sorted page numbers must form the required consecutive prefix. Old row IDs and vectors are ignored. Missing backend labels on a certified OCR recipe become retained_unknown; the importer does not infer a backend or pretend those pages are native. This reads existing text, not an original document parser or converter. Optional tables, blocks, complexity, labels and OCR flags absent from the projection are recorded in the published receipt’s projectionOmissions; the importer does not recover missing layout or claim lossless metadata reuse. Its receipt is:

{"version":1,"uri":"s3://example-bucket/outputs/item-version.json",
 "sha256":"<output-byte-SHA256>",
 "originalUri":"clouddrive://example-sharepoint/drive/item",
 "originalSha256":"<original-byte-SHA256>","etag":"<original-ETag>",
 "bytes":123,"format":"pdf","mime":"application/pdf",
 "parser":"<exact-effective-parser-recipe>","policy":"exact",
 "limits":{"timeoutSeconds":120,"maxPages":2000,"maxBytes":536870912}}

The receipt binds original identity, hash, ETag, size, format and MIME. exact binds the full active parser recipe including converter fingerprint, format, MIME hash and OCR routing fingerprint when applicable. Non-PDF outputs require matching stock conversion provenance and converted-PDF SHA256. pdf_pass_through permits unconverted PDF outputs from exactly [email protected] across the stock normalization wrapper, which passes PDF bytes unchanged. Existing PDF OCR output is reusable only when its old [email protected]+ocr-v2-<routing-SHA256> recipe matches the active engine and exact OCR routing fingerprint after removing only the stock wrapper. An OCR page without that recipe is rejected. Converted outputs cannot use this policy.

The output blob is bounded to 64 MiB, hash-checked on each sweep and validated for ordered page coverage under both recorded and current page limits. Missing or incompatible receipts fail the document; they do not silently trigger parsing. Changing an import receipt changes its cache identity. Accepted outputs enter the existing parse cache and per-document staging/publication path without a parser/OCR call. Each published page retains the verified reuse receipt as a JSON string, including the actual output engine, output hash and import policy; the active compatibility recipe does not replace this provenance. A partial old output cannot satisfy a larger current page limit. Retained outputs require a declared stock encoding or explicit page-row mapping and their actual engine, limits and original bindings. Unbound legacy payloads remain explicit intake gaps; a receipt does not certify a historical collection.

Independent text namespace contract

A Story text sibling uses a fresh Turbopuffer namespace, pov-bcc-story-pages, accessed through the gateway only. This name is a source contract, not evidence that the namespace exists. Use a distinct extractor/write pipeline ID and cache prefix, dedupe: item, the complete verified Story SharePoint roots, and the native args: [write] stage. Set LAYER_EXTRACT_WRITE_MODEL=voyage/voyage-4 and LAYER_EXTRACT_WRITE_DIMS=1024. The schema is:

{"text":{"type":"string","full_text_search":true,"fuzzy":true,"filterable":false,
         "embed":{"model":"voyage/voyage-4","dims":1024}}}

The profile binds source text to target embed_text, text modality, native serving, 1024 dimensions and cosine distance. Native Voyage is provider-managed: there is no local weight artifact SHA to invent or inherit. Pin the declared model/schema and provider readback; a requirement for immutable weight artifacts would require a separately selected local profile. The writer keeps full nonblank page text and labels textless pages. Query through the same gateway namespace with rank_by: ["text", "ANN", ["Embed", "question"]], BM25 on text, or HybridText on text; supplied vectors must match the exact selected space and dimensions.

The legacy BCC worker uses BGE-small 384-dimensional local vectors and a 1500-character input prefix. Its model discovery, artifact and query prefix do not apply to the native Story sibling. CLIP image-profile approval also does not approve this text profile. There is no copy, migration or inherited schema.

Before runtime activation, independently read back the gateway/store schema, profile binding, source selection, immutable extractor/writer images and in-image source versions. Test new writes and ANN/BM25/HybridText on synthetic rows, reject 384-dimensional vectors, and reconcile original/document/version/page identities and all failure counters. Retain the existing Function pauses and extract/OCR/embed holds until the owners explicitly authorize rollout. Per-document Function processing also requires LYR-261’s warm-cache query budget proof (at most two Turbopuffer queries per document); source support alone proves no runtime cost or search acceptance. All claims, page reads and writes go through existing gateway Pipeline APIs, never direct store access.

Retained originals

Every source keeps each original file it ingests in the target namespace’s private blob storage, so an authorized MCP reader can fetch the PDF itself (see Original files). This is on by default. Rows gain original_blob and original_bytes, next to the original_sha256, version and webUrl they already carry, and nothing else on the row changes.

sourceRef:
  extract:
    retainOriginals: false          # opt out
    # retainOriginals: {maxBytes: 5242880, backfill: true}   # tune

maxBytes defaults to 10 MiB; larger originals are skipped and counted in the sweep report’s originals.oversize, never truncated. Retention reuses bytes the sweep already downloaded, so it adds no provider traffic. Documents already settled before retention was on are filled by the layer-extract originals Function, or, with backfill: true, from sources whose bytes Layer already holds; a cloud drive is never re-downloaded for this and its settled documents are counted in originals.pending until they next change.

Validate a rollout against live receipts

Set LAYER_EXTRACT_VALIDATE_LIVE=1 on a one-shot extractor using the production Pipeline ID, source reference, gateway URL, target namespace and cache URI. This mode uses the production cache binding. It first audits every converted receipt, including documents outside the current selection, then runs a read-only sweep. LAYER_EXTRACT_EXPECT_CONVERTED_RECEIPTS optionally asserts the expected receipt count. A missing or differently bound manifest fails validation.

The gateway sink compares stored pipeline pages through the same cache/object storage read path used by production. It counts would_stage separately from would_rewrite of documents already in the manifest, and reports would_delete and would_patch. It never ensures/creates the Pipeline, stages rows, deletes or patches rows, retains originals, or writes connector cursors, parse caches or manifests. Counts are printed as JSON; no document identifiers are included in the validation reports. A nonzero migration gap or existing-document mutation fails the command. The isolated LAYER_EXTRACT_DRY_RUN mode is mutually exclusive and continues to use its separate cache binding.

The image build input legacy_converter_profile accepts either one exported runtime profile or an array of profiles from every historical immutable build represented in the live cache. Combine only package/version metadata and the platform normalization script; document content is unnecessary. Recipes and output dependencies must still match before an old fingerprint qualifies. LST, XLK and PPS remain unsupported; matching historical failure recipes retain their previous receipt identity.

Use the runtime-only helper to combine profiles with a shared inventory and per-build overrides, keeping workflow inputs below GitHub’s size limit:

python3 apps/layer-extract/tools/converter_identity.py --combine-profiles \
  /tmp/story-live-profile.json /tmp/story-history-one.json /tmp/story-history-two.json \
  > /tmp/story-legacy-converter-profiles.json

Supply that file with -F legacy_converter_profile=@/tmp/story-legacy-converter-profiles.json. The original single-profile input and raw profile arrays remain supported.

Fresh exports also include the old image’s normalization source. It is platform code, never document content, and is parsed only for recipe comparison and historical hashing; it is never executed by migration. This reproduces hashes from earlier script revisions absent from the bundled history. Re-export all historical profiles with the current helper before combining them. Older package-only profiles remain accepted for bundled historical revisions. Compact bundles deduplicate both inventories and normalization sources.