Retain Warehouse originals
Use clouddrive-originals to retain the PDFs selected by a clouddrive
Warehouse source, including files that have never been indexed. It shares the connector’s
selection rules. It does no OCR, embedding, indexing or Function
queue work. In the clouddrive-source image, set the container command to
/usr/local/bin/clouddrive-originals; the default command remains the metadata
worker. Both binaries provide --help without network or credential access.
Inventory first
Mount the same credentials and item-list files as the source worker. Supply
its HEVLAYER_SOURCE_REF and projected HEVLAYER_WAREHOUSE connection JSON;
the latter is not the full Kubernetes resource. Sources with keep retention also
need the verified history binding below. Then choose the destination:
export HEVLAYER_ORIGINAL_BUCKET=private-layer-bucket
export HEVLAYER_ORIGINAL_NAMESPACE=documents
export HEVLAYER_ORIGINAL_STATE_PREFIX=original-retention/documents
clouddrive-originals --inventory-onlyHEVLAYER_ORIGINAL_REGION defaults to us-east-1.
HEVLAYER_ORIGINAL_ENDPOINT selects an S3-compatible endpoint. The destination
must be the private bucket the gateway uses for that namespace’s blobs.
Use one writer per state prefix, and keep that prefix outside pipelines/,
blobs/ and original-catalog/.
Inventory reads metadata, without downloading source files. It follows every configured drive’s complete page chain and checks that roots resolve, including empty folders. Explicit include/exclude identities must resolve too. Root and multi-root selection, paths, mounted item lists, MIME/size rules and exclusions are shared with the metadata connector. A missing provider content hash does not exclude an otherwise selected PDF.
The report’s inventoryComplete must be true before treating its selected
count as the denominator. Declared and resolved roots, include identities and
exclude identities are counted separately. selectedBytes sums known source
sizes; unknownSizes records omissions. A null counter means it was not
recorded by an older checkpoint. Missing roots, identities or ancestry leave explicit
unresolved evidence and a nonzero exit. Metadata entries can repeat across
pages; selected identities are unique by drive/item.
Preserve prior source membership
A source with retention.deselected: keep includes still-current prior members,
not only files that match today’s literal folder paths. Supply a pinned Layer
source snapshot; this is drive-listing.json, not an extraction-success manifest
or a search-index export:
export HEVLAYER_ORIGINAL_HISTORY_KEY=pipelines/example/extract/cache/drive-listing.json
export HEVLAYER_ORIGINAL_HISTORY_VERSION_ID=immutable-s3-version
export HEVLAYER_ORIGINAL_HISTORY_SHA256=expected-full-object-sha256
export HEVLAYER_ORIGINAL_HISTORY_BYTES=expected-byte-count
export HEVLAYER_ORIGINAL_HISTORY_PIPELINE_ID=example
export HEVLAYER_ORIGINAL_HISTORY_TARGET=s3://private-layer-bucket/pipelines/example/extract/cache/
clouddrive-originals --inventory-only --reselect-inventoryThe importer verifies the complete object and each source row’s committed hash, generation and provider identity. Drive manifests must match the current Warehouse, mapping, MIME rules, configured roots and exact cache target. Item lists and retention are not part of that historical config hash: current explicit exclusions always win. Keep retention carries proven prior PDFs across later folder, item-list, MIME or size-filter changes; it does not turn a current non-PDF into a PDF. Prior membership alone proves neither current PDF bytes nor a successful copy. Current metadata, ancestry and versions determine what can be copied; missing members or those no longer selected as PDFs by current metadata remain explicit unresolved cases.
Pinned history reads first use an ordinary S3 GetObject request. The response
VersionId must equal HEVLAYER_ORIGINAL_HISTORY_VERSION_ID before any bytes
are read. This route needs no GetObjectVersion permission when the pin is still
current. If the current version differs, the worker tries an explicit-version
GET and checks its returned VersionId too. A missing VersionId or denied version
read fails closed. Full SHA-256 and size checks still apply. An existing authorized
operator can preserve the requested version at a dedicated key, verify its full
checksum and size, and supply a new binding for that preserved object; never
substitute the latest object for the pin. A saved generation retains its binding,
so preserve its evidence and reconcile any binding change before resuming.
Reselection creates a new generation from completed immutable metadata pages.
It preserves the old generation and acquired byte proofs and performs no new
Graph delta walk. Subsequent generations also carry the worker’s own source
membership, including new files selected on an earlier run. Missing members
remain unresolved across repeats. A missing literal root can be repaired only by a unique folder
identity proven in the verified old tree and present in the current tree. Similar
folder names or a broader parent are insufficient. Independent valid files can
still copy while scope gaps remain unresolved. Changed file versions require
an explicit new current inventory with --refresh-inventory; no stale version
is silently accepted.
Inspect source metadata
Use an existing completed inventory to diagnose a selected file without copying it or changing retention evidence:
umask 077
clouddrive-originals --inspect-item DRIVE ITEM > private-item-inspection.jsonKeep the same source configuration, projected Warehouse, namespace, bucket and state prefix. This mode requires the existing binding and fully committed selected inventory, with no unresolved inventory scope errors. A copy run may still be incomplete or have changed/missing source outcomes. Inspection reads binding, state and bounded selected shards, rejects an unselected identity before Graph access, then makes one logical authenticated item-metadata lookup. Existing Graph authentication, pacing and bounded HTTP retries apply; the JSON counts actual attempts and received metadata/auth bytes. There is no content request, source discovery, metadata traversal, source-history/hint import or sink write.
The JSON is private evidence: it contains the inventory generation, selected and current source metadata, observation time, ETag, version, QuickXor, size, provider-declared MIME, citation, changed fields, HTTP error class/status and request/byte counters. Authenticated download URLs, provider error bodies and credentials are excluded. Citation query strings and fragments are removed from this diagnostic output. Logs go to stderr; stdout contains the diagnostic JSON.
sourceStatus is unchanged, changed, missing or error. Missing 404/410 and
changed metadata are successful diagnostic observations, so the command exits
zero after recording them. Provider errors emit safe diagnostic JSON and exit
nonzero. retentionVerified is always false: this mode never verifies bytes,
resolves a retention gap or updates counts. It does not change binding, phase,
cursors, outcomes, acquired proofs, original objects, catalog or queues, and
never starts a new inventory even when the old copy phase is complete.
--inspect-item cannot be combined with inventory, refresh, reselection or copy
flags. History and hint import environment variables are not used. The selected
snapshot remains unchanged; any later source refresh or retention repair is a
separate explicit operation.
Typed source originals
Selection follows the configured PDF scope, including files whose provider calls
them PDFs. Retention does not change that scope or alter source bytes. If a
selected original is a valid PNG despite its PDF metadata, the worker retains
the exact PNG with actual MIME image/png, provider-declared MIME and explicit
sourceFormatMismatch provenance. It verifies the PNG container, chunk CRCs,
zlib stream and scanline framing without rendering pixels. Indexed-color PNGs
remain unsupported because palette index validation needs pixel reconstruction.
Unsupported or malformed formats, HTML responses, missing files and changed sources remain
unresolved. PNG validation uses bounded buffers and a 1 GiB inflated scanline
limit; exceeding that bound is unsupported, never successful retention.
The same before/after source checks, provider QuickXor check when present,
SHA-256, size and full S3 readback apply to typed originals, including checksum
reuse. A verified PNG resolves its original-copy gap but never counts as a
verified PDF. Reports expose candidateTotal, verifiedOriginals,
verifiedPdfs, verifiedNonPdfOriginalsWithFormatMismatch, pending, missing,
changed, corrupt, unsupported and failed. Successful original types are
disjoint, and pending plus verified originals plus unsuccessful candidates
conserves the candidate total. unresolved is the sum of pending and unsuccessful
candidates plus scopeUnresolved, the separate inventory scope errors. The legacy verified aggregate now equals verifiedOriginals, not
verifiedPdfs; invalidPdf retains its legacy outcome name for unsupported
selected bytes. Old reports without typed fields leave those counts unknown.
For one known selected identity, use clouddrive-originals --only-item DRIVE ITEM
on the existing state prefix. No new inventory is needed while its generation
is resumable. Historical invalid-PDF quarantine receipts remain in place; the
new verified outcome records the format mismatch separately. No conversion,
OCR, indexing or source metadata update takes place.
MCP returns the whole PNG as an embedded resource with its actual MIME and
source link. A pages request for a non-PDF explicitly fails. Namespace grants,
pins and source/version bindings are unchanged. Existing v1 PDF catalogs remain
readable; typed catalog fields are optional additions. The MCP arguments and
HTTP API are unchanged, so this capability needs no generated SDK changes.
Copy, resume and repeat
With the same environment, run:
clouddrive-originalsFor a bounded copy check after inventory, use clouddrive-originals --max-files 2.
It processes at most two selected identities in that invocation. Remaining files
stay pending and the command exits nonzero until the full inventory reconciles.
Remove the flag to process the entire selection.
The worker inventories first if needed, then copies its selected PDFs. Each file
must have a stable source ETag and size. It streams to scratch disk while
hashing, checks the source again, writes the object, and fully reads it back to
verify SHA-256, size and supported actual format. Originals over the older Function’s
10 MiB cap are copied too. Objects use blobs/<namespace>/<sha256> and the
existing blob://<namespace>/<sha256> reference format.
A failed run resumes from its committed pages and per-file evidence. A run that finished starts a fresh inventory on the next invocation. Unchanged source-version receipts can avoid another Graph download, but every reused S3 object is read and hashed. Size and S3 ETag alone never prove reuse. Files that changed, disappeared, failed verification or contain unsupported bytes remain unresolved. A run succeeds only when its complete inventory reconciles with verified objects and catalog references, with unresolved zero.
Use --refresh-inventory after repairing source scope or when a source change
invalidates the saved inventory. Old generations remain under the prefix.
Changing Warehouse, resolved selection or destination requires a new prefix.
To inspect one inventoried file without copying other members, run
clouddrive-originals --only-item DRIVE ITEM. Other files stay pending, so this
bounded diagnostic run exits nonzero. Invalid PDF bytes retain their observed
SHA-256, byte count and before/after metadata. A private quarantine/ receipt
classifies HTML-like, JSON-like or missing-PDF-header responses and points to
at most 64 KiB of raw bytes, with its own checksum and truncation flag. These
bytes are never published as an original or silently repaired.
State and resource use
Metadata pages are immutable shards, written before their compact cursor checkpoint. Selected shards contain at most 500 files. A disposable SQLite projection resolves ancestry; the accumulated tree is never rewritten per page or file. Outcomes are small per-file objects, and aggregate progress is saved in batches. Private reports and invocation meters remain under the state prefix, separate from extraction state. They contain source identities and citations; keep them private.
HEVLAYER_ORIGINAL_CONCURRENCY defaults to 2 and accepts 1–4 files. Each has
its own scratch directory; all share the Warehouse Graph rate limiter. Scratch
disk must hold current and historical metadata projections, the pinned history
file, optional hint manifest and up to that many originals at once.
Downloads do not buffer a whole file in memory. S3 multipart uploads use
8 MiB parts, increasing only to stay within 10,000 parts; the worker’s object
safety bound is 5 TiB. Known upload IDs are checkpointed before any parts and
reconciled on restart. A crash between initiation and saving that ID can leave
an empty upload for bucket lifecycle cleanup. Other writers’ uploads and
completed objects are not deleted.
Recovery requires AbortMultipartUpload for the exact owned key and upload ID; GetObject and PutObject alone do not grant it. If the worker cannot abort, it fails and preserves the marker. The error reports the S3 operation, safe service code/status, owned key, upload ID and marker location without SDK request URLs or response bodies. An existing authorized operator can read that private marker, abort only its recorded upload, and resume the same generation. NoSuchUpload is safe to reconcile after an operator abort; the worker then removes the marker. Do not delete markers to bypass denied cleanup. If initiation succeeded but checkpointing failed, the error carries the upload identity and intended marker location for operator recovery. Check this recovery capability before a large copy test; permission expansion is not required.
Reports separate copied, checksum-verified reuse, source changes, missing
files, invalid PDF signatures, failures and unrepaired corruption. Root aliases
and content aliases are separate counts. Graph/auth/content requests include
retries, and content bytes count received chunks. Successful metadata and auth response
payload bytes are counted separately. S3 counters report request attempts,
received object bytes and acknowledged write payloads, not wire billing.
reusedHints and reusedReceipts distinguish verified reuse through extraction
hints from the worker’s own acquired proofs; outcomes also identify checksum
reuse after a source download. Meters identify their invocation and checkpoint boundary; a hard crash can
leave later attempts unrecorded. Such intervals are not evidence of zero work.
Reuse an extraction receipt
An existing hev layer extraction manifest can supply optional source-version-to-SHA hints:
export HEVLAYER_ORIGINAL_HINT_MANIFEST_KEY=pipelines/example/extract/cache/manifest.json
export HEVLAYER_ORIGINAL_HINT_PIPELINE_ID=example
clouddrive-originalsThe manifest is streamed to scratch and imported into SQLite. Its version and
Pipeline identity must match. A hint requires the same namespace’s retained
blob reference and exact drive/item/ETag/provider version. The source version
uses the connector’s quickXor:<hash> spelling. Conflicting hints
are ignored. Parser status is not proof. A matching blob still receives full
SHA-256/size/PDF-signature validation; a mismatch falls back to source copying.
Fetch indexed and unindexed originals
Enable originals on the gateway’s MCP namespace.
Per-file outcomes contain a stable original:<source-id> catalog ID. Pass
that ID to fetch_original_<toolName>, or to generic fetch_original with the
namespace. It uses the same id and optional pages arguments as indexed
originals. The returned metadata includes source drive/item/version/ETag,
SHA-256 and the provider citation. SharePoint links are preserved.
Existing indexed IDs also work when their rows lack original_blob: the
gateway lazily matches the authorized row’s exact content hash or source
identity/version to a catalog receipt. It does not scan or patch rows. Unknown
legacy identity remains an error.
Namespace read authorization applies before catalog access. For direct catalog IDs, every pinned filter must be provable from catalog attributes; a missing value is denied. Indexed IDs retain their row authorization and pinned filters. Their fallback adds only the verified blob binding, preserving the row’s own citation and source fields. Catalog IDs cannot widen a tool’s scope. Retention does not raise MCP whole-file, page-render or response-size limits; oversized responses still return an error with the source citation.
Probe access without copying
clouddrive-source --preflight reads metadata from each discovered drive and
streams one selected PDF into SHA-256, checking byte count and before/after
source identity. It constructs no sink or state store and makes no S3, index
or queue writes. This establishes access, not a complete inventory or retention.
HEVLAYER_PREFLIGHT_MAX_PAGES defaults to 100 per drive (1–10,000), and
HEVLAYER_PREFLIGHT_MAX_BYTES to 512 MiB. Larger candidates are counted while
the probe searches for a sample within its cap; they are not skip-success.
No sample within the budgets, an unreadable drive or unstable bytes fails.
Do not combine --preflight with --once, --verify or --reset-cursors.
Set HEVLAYER_PREFLIGHT_RECEIPT to a new absolute file path for an atomic,
owner-only private sample receipt. It binds resolved selection, Warehouse,
drive/item/path/citation, before/after metadata and observed SHA/bytes.
Identifiers stay out of routine progress logs. Credentials and signed download
URLs are never written. Raw sourceRef parsing strips only existing worker-owned
parse, extract, cacheUri, rescanSeconds, suffixes, uri, merge and
dedupe settings (dedupe must be content or item); other unknown fields
fail validation.
Retained snapshot when the source is unavailable
After deploying a gateway that reads snapshot catalogs, quiesce the sole copy
worker before enabling --allow-historical-reuse. This opt-in policy preserves
an exact selected inventory revision whose authenticated metadata lookup returns
404 or 410. It does not select more files or substitute an older original for
known changed content. Start with one selected identity:
clouddrive-originals --allow-historical-reuse --only-item DRIVE ITEMThis mode requires an existing completed inventory binding and cannot be combined with inspection, refresh, reselection or inventory-only flags. A bound acquisition receipt, or verified source history plus a pinned extraction manifest, supplies a byte-reference lead. For the manifest path, keep the source-history variables from prior membership and supply:
export HEVLAYER_ORIGINAL_HINT_MANIFEST_KEY=pipelines/example/manifest.json
export HEVLAYER_ORIGINAL_HINT_PIPELINE_ID=example
export HEVLAYER_ORIGINAL_HINT_VERSION_ID=exact-provider-version-id
export HEVLAYER_ORIGINAL_HINT_SHA256=full-manifest-sha256
export HEVLAYER_ORIGINAL_HINT_BYTES=exact-manifest-byte-countPrivate per-file outcomes record objectVersionId only after accepted full S3 readback.
Legacy outcomes and unversioned live objects have an unknown version binding; historical
proof still requires a nonempty S3 VersionId. Empty bodies and entirely zero-filled
selected originals are established corruption, not unknown formats. They remain unresolved
and cannot use unclassified or historical retention; MCP refuses empty originals too.
Both evidence objects must match their pinned VersionId, full SHA and size. The receipt binds provider, tenant and client identity as well as selection, namespace and destination; a Warehouse name alone is insufficient. Layer must fully read the retained S3 object and verify its VersionId, SHA-256, size, actual format and inventory provider checksum. It checks source metadata again before publication; both observations must be 404 or 410. Missing checksums, ambiguous evidence, known newer content and corrupt or absent objects remain unresolved. No source body is read and no raw original is uploaded on this path.
The returned immutable original: reference identifies that inventory snapshot.
MCP reports retentionBasis=inventorySnapshot, sourceAvailability=unavailable,
the observed status and time, and deletionStatus=unknown. Its SourceLink is the
inventory citation, not a claim that the source still opens. Internal object and
evidence locations remain private. Existing namespace grants and pins apply.
Snapshot IDs use a canonical versioned tuple and exclude timestamps and attempt
IDs. Verification receipts, prior outcome archives and snapshots use conditional
create, with conflict validation. A retry after any publication boundary can
verify and finish the same snapshot without replacing a newer source alias.
Each validation writes a private receipt with its own invocation and both
status/time observations. A new proof lead or retained object version can create
a new receipt for the same frozen snapshot. The first snapshot and its first
receipt pointer stay unchanged; the current outcome names the new receipt.
The earlier missing outcome stays archived. Verified historical originals count
as originals and as their PDF, mismatched PNG or opted-in unclassified partition; source unavailability
remains a separate anomaly. Reports add verifiedHistoricalOriginals as a subset
of verifiedOriginals, and disjoint sourceAvailable, sourceUnavailable and
sourceAvailabilityUnknown counters. The availability counters conserve the
candidate total: authenticated 200 is available, 404/410 is unavailable, and
unvisited candidates or a failed metadata recheck are unknown. Earlier reports
without these fields remain unknown. historicalProofPolicy records the enabled
proof-policy revision.
Refreshing unchanged keep-retention selection must carry previously authenticated missing members, preserving their exact requested revision or an explicit unknown-version unresolved placeholder. It must not shrink the denominator or clear root and scope errors. Changed content requires fresh inventory and a copy of that newly requested version. Historical writing requires a compatible reader; old readers reject snapshot schema 2. Rollback disables writing and preserves the compatible reader and immutable evidence. Schema 1 aliases remain unchanged.
Isolated snapshot acceptance
Use an isolated MinIO instance through an HTTP loopback endpoint. An existing
authorized executor owns its lifecycle and supplies synthetic credentials. The
harness creates an owned scratch-original-snapshot-* bucket, enables versioning,
and creates a separate unversioned bucket to check refusal of missing VersionIds.
It never disables versioning on an existing bucket or uses ambient AWS credentials.
export LAYER_SNAPSHOT_TEST_ENDPOINT=http://127.0.0.1:9000
export LAYER_SNAPSHOT_TEST_BUCKET=scratch-original-snapshot-test
export LAYER_SNAPSHOT_TEST_ACCESS_KEY_ID=fixture-access
export LAYER_SNAPSHOT_TEST_SECRET_ACCESS_KEY=fixture-secret
cargo test -p clouddrive-source minio_real_ -- --ignored --nocapture --test-threads=1Use the fixture’s supplied test credentials. The client requires a loopback URL,
an owned bucket prefix and both explicit credential variables before any request.
For an executor’s custom scratch prefix, set LAYER_SNAPSHOT_TEST_BUCKET_PREFIX
to that exact owned prefix; it must start with scratch-.
Tests retain only fixed synthetic PDF/PNG bytes and metadata, with a 16 MiB
successful-PUT budget. Source-body calls and raw uploads during historical
verification are prohibited by the test adapter. Request timeouts and per-case
progress bound an interrupted run. The fixture audit lists at most 1,000 owned
keys and verifies producer receipts and their exact retained object versions;
LAYER_SNAPSHOT_TEST_REQUIRE_COMPLETE=1 also requires the complete matrix.
Preserve interrupted evidence and audit effects before a retry. Leave the fixture
available for independent inspection, then let its owner remove it.
The ordinary source tests cover unavailable observations, changed-content refusal, checksums, format and binding failures, refresh conservation, and interruption after each durable publication boundary. Gateway MCP integration tests separately cover schema 2 PDF/PNG, PDF pages, schema 1 compatibility, tuple tampering and namespace and pin denials. Passing synthetic tests does not establish live historical retention: validate the exact source candidate, its two authenticated unavailable observations, pinned proof, counters, restart and authenticated MCP response after the reader rollout. Keep earlier missing outcomes and acceptance receipts.
Preserve unclassified source bytes
--allow-unclassified-originals opts into exact-byte retention when the selected
source format cannot be established. It does not broaden selection or transform
content. It requires the existing complete inventory binding, keeps its
generation and rejects refresh, reselection and inventory-only combinations.
After the reader rollout, run clouddrive-originals --allow-unclassified-originals --only-item DRIVE ITEM with the same projected configuration and state prefix.
The source must still match its inventoried ETag, version and size; a
provider checksum is required. Layer checks the whole SHA-256 and checksum, reads
the complete S3 object back, and rechecks source metadata before publication.
The same policy can accompany --allow-historical-reuse with its exact-version
evidence and two unavailable observations.
These originals use application/octet-stream, preserve declared MIME, and
report formatValidation=unverified. Actual format and source-format mismatch
are unknown, not false. MCP serves whole bytes with the original citation and
refuses PDF pages. Namespace grants and pins still apply. Roll out the reader
before enabling this writer policy; older readers reject the new catalog shape.
verifiedUnclassifiedOriginals is a third disjoint partition:
verifiedOriginals = verifiedPdfs + verifiedNonPdfOriginalsWithFormatMismatch + verifiedUnclassifiedOriginals.
unverifiedFormatOriginals reports the format anomaly separately; raw retention
completion says nothing about container validity. Legacy reports without these
fields remain unknown.
Known malformed or truncated PNGs, HTML/JSON error responses, checksum or size mismatches, changed sources, and missing or unbound proof remain unresolved. A format outside validation limits is unassessed, not a validated image. Existing unsupported outcomes and quarantine evidence are archived before replacement. If later analysis establishes corruption, preserve an append-only reassessment and restore the unresolved outcome; never rewrite the original proof. The writer records established PNG corruption as a conditional, append-only assessment bound to the namespace and whole-object SHA. Source reuse, historical reuse and MCP refuse assessed bytes; rerunning restores an unresolved outcome while preserving the predecessor and quarantine. Assessment records are private operator evidence, not MCP object locations.