File storage

Uploaded files go to S3-compatible object storage: Garage locally, real S3 (or any S3-API service) in a deployed environment. One code path, one IFileStore; the endpoint, credentials and addressing style are configuration.

This note exists because the previous arrangement was never actually decided. Deliverable stored its bytes in a Postgres bytea column (#165), evidence followed the precedent (#51), and by the time anyone asked the question there were two aggregates depending on a default that no PRD, issue or note recorded. Storage wasn't even in PRD-003's Non-Goals. The point of writing it down is not to justify the answer — it is to make it falsifiable, so the next person can check the reasoning instead of re-deriving it from the code.

What bytea had going for it

Worth stating plainly, because it is the thing the move has to preserve rather than lose.

Evidence carries a chain of custody: a SHA-256 computed server-side at upload, recorded so a later change to the file is detectable. In Postgres the hash and the bytes are written in the same transaction — they cannot disagree. That is a real guarantee, not a nicety, and object storage does not offer it: writing an object and committing a row are two operations, and a crash between them is possible by construction.

The tenant story was also simpler. HasQueryFilter is fail-closed and covers every read of an EvidenceItem row. Object storage has no equivalent — bucket/prefix authorisation is hand-written per path, and a presigned URL escapes the application's authorisation entirely once issued.

That gap is not abstract, and it interacts badly with content-addressing. A flat {sha256} namespace means two companies uploading byte-identical files — a statutory annex, a standard-form contract, a circulated ACT notice — share one object. Reference counting runs through EF, under the same fail-closed filter, so "am I the last reference?" is answered within one company: it returns yes while another company still points at those bytes, and deleting them breaks that company's chain of custody with no action and no audit entry on their side. Hence the tenant prefix below — it is what makes the question answerable, not a tidiness measure.

Why we moved anyway

• WAL amplification. Postgres logs a change to the WAL before it touches the table file, which is already 2× on its own — but the real multiplier is the first write to a page after a checkpoint: a full-page write logs the entire 8 KB page, so a byte inside a large bytea value forces its whole page, repeatedly, through the log. Add a table's indexes (each one logs its own entry) and the flush's own block-alignment (a sub-4KB WAL flush still costs a full physical block on an SSD), and a modest file turns into a disproportionate write. That amplified stream also replicates and lands in every base backup, so backup size and restore time grow with the volume of evidence — which is the one thing guaranteed to grow. None of this shows up in a functional test; it shows up in IOPS billing and backup windows once there is real volume, which is exactly why it reads as a performance footnote until it doesn't.
• No streaming. byte[] on the entity means EF materialises the whole file in memory on read. With a download endpoint (SYS-REQ-404, #205) that is memory pinned proportional to file size × concurrency.
• Accidental reads. A repository over an entity with a bytea column pulls file bytes on any query that forgets to project. That is a footgun that has to be remembered rather than one the type system prevents.
• The Dossier (#98/#99) assembles many files at once.

How the custody guarantee is preserved

Two decisions carry it, and neither is optional:

1. Server-side upload, not presigned. The client posts the bytes to the API; the API hashes them and then writes to storage. The alternative — a presigned URL the client PUTs to directly — means the server never sees the bytes and can only trust a client-supplied hash. In a disciplinary process the uploader is precisely who the chain of custody protects against, so that is not a trade-off, it is the removal of the feature. (Presigned download URLs are a different question and remain open.)

At our sizes the choice is cheap: uploads are capped at 15 MiB (UploadLimits). Presigned uploads earn their complexity in the hundreds of megabytes, not here.

2. The object key is {companyId}/{sha256} — content-addressed, tenant-prefixed, built by ContentAddressedKey.For so no call site interpolates its own. Content-addressing is what makes the two-phase write safe. It cannot be made atomic, so the question is only whether a failure is detectable:

failure	with a content-addressed key	with an opaque key (a GUID)
row committed, object missing	detected — read it back, the hash doesn't verify	undetectable without a separate audit
object written, row rolled back	inert; de-duplicates against a later identical upload	orphan, unattributable, needs a GC job to even find
upload retried after a partial failure	idempotent — same bytes, same key, same object	forks the file the recorded hash refers to

So a crash becomes a garbage-collection problem instead of a correctness one. Without this, moving off bytea would genuinely weaken custody; with it, it doesn't. S3FileStoreIntegrationTests pins the third row — the retry — against a real Garage, along with the key layout, checked against the live bucket listing rather than against the helper that produced it.

The second row — object written, row rolled back — is pinned by the real caller: UploadEvidenceCommand writes to IFileStore before adding the row and committing, so A_failed_commit_after_a_successful_write_orphans_the_object_rather_than_dangling_the_row forces a commit failure after a successful write and asserts the write still happened exactly once. That ordering is the whole point — write then commit, never the reverse — and it is now enforced by a test, not just argued in this paragraph.

The first row — row committed, object missing — still has no test, because nothing yet reads evidence back to notice the mismatch. That is #205's territory (SYS-REQ-404), not #191's: #191 proves the object was written, not that a later reader would catch its absence. EvidenceEndpointHttpTests currently proves the positive case instead (upload, then read the same bytes back through IFileStore) — the negative case belongs with the read endpoint.

The prefix costs only cross-tenant de-duplication, which for confidential disciplinary files is a feature being removed rather than a price being paid. Within a company, identical bytes still de-duplicate and a retry is still idempotent. What it buys, beyond closing the delete hazard, is the bucket/{companyId}/* boundary that prefix-level authorisation needs in order to be expressible at all.

Shape

• IFileStore (Application) — Put / OpenRead / Exists / Delete. Deliberately dumb: it takes a key the caller chose and moves bytes. Note the asymmetry: OpenRead streams, but Put takes a byte[], so the write path still materialises the whole file. That is deliberate — server-side hashing has to see every byte anyway, and uploads are capped at 15 MiB — but it is the same limitation the read path was redesigned to remove, and it would need revisiting before the cap rises. It does not name objects or hash them, which is what lets one store serve evidence now and deliverables later, and why the key convention lives with the feature that owns the file's meaning.
• S3FileStore (Infrastructure) — the only implementation. Garage and S3 differ by FileStorageSettings, never by code path.
• FileStoreStartupCheck — an IHostedService that proves the bucket is reachable at boot, mirroring AiAccessTokenIssuerStartupCheck. A startup check rather than an IHealthCheck because /health here is one minimal endpoint with no HealthChecks pipeline behind it, and adding one to hang a bucket probe off would change a live endpoint's contract for an unrelated reason.
• Local stack: a single garage service in docker-compose.infra.yml, wired in the Tiltfile, with its configuration in infra/garage/garage.toml. --single-node --default-access-key --default-bucket creates the cluster layout, the access key, the bucket and the key's grant during that one container start, so there is no init job to sequence — the MinIO setup this replaced needed a second service to run mc mb.
• garage-webui (community image, not Deuxfleurs') gives a browser view of that cluster — buckets, keys, status — for local dev only. It authenticates to the admin API with GARAGE_ADMIN_TOKEN (now a committed dev default rather than unset, so both containers agree on it), and is bound to loopback like the admin API it talks to.

Deployment

Leave FileStorageSettings__Endpoint blank against real S3 — the SDK resolves the regional endpoint from Region. Leave AccessKey/SecretKey blank too, so the ambient AWS credential chain (instance role, environment, profile) is used and long-lived keys never sit in configuration. ForcePathStyle is required by Garage and unnecessary on S3, and Region must equal s3_region in infra/garage/garage.toml — Garage enforces the region a request is signed for and answers AuthorizationHeaderMalformed on a mismatch, where MinIO accepted anything.

Why Garage rather than MinIO

MinIO's own README states the repository is no longer maintained: the community edition is source-only, with no pre-compiled binaries or container images going forward and security fixes handled case by case. Nothing in our usage was broken by that — we drive the store through the SDK, never the console that was stripped in early 2025 — but both the test fixture and the e2e workflow pinned images from a project that had stopped publishing them.

Two things that are not reasons, and should not be cited as such: licence (Garage is AGPLv3, and MinIO has been AGPLv3 since 2021 — this is a wash, not an improvement) and capability (equivalent for our use, not better). The genuine engineering upside is operational and small: one container instead of two, and no init job.

Garage is AGPLv3, run as unmodified upstream infrastructure in a separate process, not linked into the application and not redistributed. Deuxfleurs publish no commercial/SaaS guidance, so there is no project position to cite; the licensing judgement for that deployment shape sits with whoever owns it here.

One thing the docs did not tell us

AWSSDK.S3 4.x defaults to RequestChecksumCalculation.WHEN_SUPPORTED, which sends the body as aws-chunked with a CRC32 trailer signed as STREAMING-UNSIGNED-PAYLOAD-TRAILER. Reading Garage's Rust source says it implements that path. It does not work. Every PutObject fails with Bad request: Invalid payload signature until the client is set to WHEN_REQUIRED. S3FileStoreIntegrationTests fails without it and passes with it — which is the reason the fixture runs against a real container rather than a mock.

Scoped to the self-hosted endpoint, not applied globally. It was first set unconditionally, which meant a limitation of the local store had switched off the upload-time integrity check in production — on a feature whose entire premise is chain of custody. Against real S3 the SDK default works, so real S3 keeps its checksum. S3ClientConfiguration is where that branching lives, and S3ClientConfigurationTests pins it, because the failure mode is silent: nothing observable breaks, the protection is just gone.

What the workaround costs locally: our SHA-256 remains the durable integrity record, but it is computed before the upload, so against Garage a corruption in transit is no longer caught at write time — it surfaces when the bytes are read back and re-hashed, which is #205's job.

A second thing the same branch needs: AuthenticationRegion. Setting ServiceURL alone leaves the SDK signing for a default region, and Garage enforces the match against its own s3_region — so every request fails. That one surfaced only when the test fixture was switched to build its client through S3ClientConfiguration instead of its own copy: the fixture had been setting the region itself, masking the gap for any real deployment pointed at a self-hosted store.

Deploying this — what must be overridden

Everything below is configurable by environment variable or appsettings; nothing needs a code change. The committed values are dev defaults chosen so tilt up works with no setup, which is exactly what makes them wrong anywhere else.

Application (FileStorageSettings__*, or an appsettings section):

Setting	Dev value	Deployed
Endpoint	http://localhost:3900	blank for real S3, else the store's https URL
AccessKey / SecretKey	mylegalteam / mylegalteamdevsecret	blank for the ambient AWS chain, else real credentials — both or neither
AllowInsecureEndpoint	true	leave unset. Defaults false; an http endpoint is refused at startup
Region	eu-west-1	must equal the store's own region — for Garage, s3_region in its toml
BucketName, ForcePathStyle, MaxUploadBytes	see backend/.env.example	as appropriate

Garage container (only if a self-hosted store is deployed at all):

• GARAGE_RPC_SECRET — the committed one in infra/garage/garage.toml is dev-only. Override it. Note an empty value overrides the file with nothing and stops Garage booting (Invalid RPC secret key), so set it properly or leave the variable unset entirely.
• Changing GARAGE_DEFAULT_SECRET_KEY against an existing volume stops the container starting, with "Access key … is associated with a secret key different than the one given in GARAGE_DEFAULT_SECRET_KEY". Garage refuses to overwrite rather than silently re-keying, which is the right call but reads as "Garage won't come up" with no obvious link to the line just edited. It fails at startup, not at first upload. Fix: delete the garage_meta volume. Same shape as the stale-Postgres-volume trap. This also applies to anyone pulling this branch with a volume from an earlier run.
• GARAGE_ADMIN_TOKEN — now a committed dev default (for garage-webui to work out of the box), not unset. Verified: every admin route except /health 403s without a matching bearer token, including a request with no Authorization header at all — there is no unauthenticated fallback to worry about. Override the default anywhere real, the same as GARAGE_RPC_SECRET.
• GARAGE_DEFAULT_ACCESS_KEY / _SECRET_KEY / _BUCKET — the container creates the key, bucket and grant on first start from these. They must match the application's AccessKey/SecretKey/BucketName.
• Its own garage.toml. replication_factor, the data and metadata directories, api_bind_addr and s3_region have no environment-variable equivalent — Garage only reads them from the file. The one in infra/garage/ is shaped for a laptop: replication_factor = 1 means one copy on one disk, with durability to match, and no backup story. That is fine for something rebuildable and is a decision, not a default, for anything that keeps evidence.

docker-compose.infra.yml is local tooling and is not the deployment artifact.

Still open

• Deliverable moved too. Originally deferred here as "later, one aggregate at a time" — reviewed and done in this same PR instead, once it was clear the change was mechanical: Content (byte[]) is now Sha256 (string), the produce/refresh handlers write to IFileStore before the row, and download reads back through it. The asymmetry noted below is still true and is why evidence moved first and kept the stronger guarantees, not a reason to leave deliverables on bytea once the pattern existed. Deliverables are generated and so regenerable from case data; evidence are uploaded originals and unrecoverable.
  • The Deliverables table was already on dev (#165) — unlike evidence, whose migration was unmerged and could simply be regenerated clean. This one is a real schema change on a live table, hand edited against the repo's own "never hand-edit a migration" rule for exactly that reason: the scaffold was a bare drop-and-add that would have thrown away every existing row's hash with no way to recompute it. Fixed by computing Sha256 = encode(sha256("Content"), 'hex') in the database before the column is dropped — sha256(bytea) is a Postgres 11+ built-in, so nothing new is added to the model for a one-off backfill. DeliverableContentMigrationBackfillTests proves it against a real container: starts a database mid-migration, inserts a row shaped like the pre-migration schema via raw SQL, runs the migration under test, and asserts the hash matches.
  • What that migration cannot do: copy the bytes themselves into object storage — a migration runs SQL against Postgres, not calls against Garage/S3. So a deliverable produced before this migration has a correct hash and no matching object; downloading it 404s until it is refreshed. Acceptable only because this table carries no production data yet. A deployed environment with real deliverables needs an out-of-band job copying Content into object storage under {CompanyId}/{Sha256} before this migration runs, not left to run after.
• Retention / RGPD deletion — object lifecycle rules are a much better fit than DELETE + VACUUM over TOAST, but the policy itself is not specified yet.
• Presigned download URLs for #205, and whether issuing one writes an access custody entry given it escapes the application's authorisation once issued.
• Virus scanning on upload — named out of scope in BEHAVIOUR.case-evidence; object storage makes an out-of-band scanner easier, if it is ever wanted.
• The e2e suite's InMemoryFileStore cannot fail. Swapping the real store out of ApiFixture is the right call — it matches the CapturingEmailSender precedent, and forcing a Garage container onto every HTTP test class for tests that never touch a file would be worse. But the fake has no network errors, no size limit and no 404-vs-throw, and #191's evidence upload now routes through IFileStore for real, so no e2e test covers a storage failure path for that endpoint. The failure paths live in S3FileStoreIntegrationTests and FileStoreStartupCheckIntegrationTests instead — which is fine, except that CI runs neither (see below).
• CI runs none of these tests. pr-test.yml runs only MyLegalTeam.Tests; no workflow references MyLegalTeam.IntegrationTests, so every test proving this store works is run by hand. The same gap makes MigrationsTests inert. Pre-existing and tracked separately, but it means a green CI says nothing about storage — the one exception being that pr-e2e boots the real backend, so it does exercise BuildS3Client and FileStoreStartupCheck's success path against a real Garage.