Make the production compose stack deployable and disclose the AI-generated catalogue (#329)

2026-10-14 17:54:42 +00:00 · 2026-10-08 00:16:37 -04:00
parent bd0e9e4a20
commit 0e51630439
11 changed files with 409 additions and 55 deletions
--- a/docs/web/docker.md
+++ b/docs/web/docker.md
@@ -134,27 +134,136 @@ containers, which would end the run prematurely. See
 - Production logging levels
 - No database clearing
 - Optimized build configurations
- Health checks enabled
- Restart policies (unless-stopped)
+- Health checks on every long-running service
+- Restart policies that survive a reboot (see
+  [Restart tolerance](#restart-tolerance))
 - Security hardening
- Only `frontend` publishes a port to the host; `sqlserver` and `api.core` are
-  reachable only from other containers on `prodnet`
+- Only `frontend` publishes a port to the host, bound to `127.0.0.1` so a reverse
+  proxy can sit in front of it; `sqlserver`, `api.core` and `seaweedfs` are
+  reachable only from other containers on `prodnet` (named `biergarten-prodnet`
+  so the seed job can join it from its own Compose project)
+- Seeding is a separate, explicitly run compose file
+  (`docker-compose.prod.seed.yaml`), so an `up` or a restart leaves the existing
+  catalogue as it is. See [Production data](#production-data) below.

 **Services**:

 ```yaml
-sqlserver          # Production SQL Server (internal only)
-database.migrations # Schema updates only
+sqlserver         # Production SQL Server (internal only)
 api.core          # Production API (internal only, no published ports)
-frontend          # Production website (port 3000, the only public binding)
+seaweedfs         # S3-compatible object storage (internal only)
+frontend          # Production website (127.0.0.1:5656, the only host binding)
 ```

+Migrating and seeding are run-once jobs kept outside this file, so an `up` or a
+restart touches neither the schema nor the catalogue. Run them explicitly at
+deploy time: see [Migrating production](#migrating-production) and
+[Seeding production](#seeding-production-docker-composeprodseedyaml).
+
 **Deploy Production**:

 ```bash
+# One time, before the first `up`: the data volumes are external, so Compose
+# expects them to exist already.
+docker volume create biergartendata-prod
+docker volume create seaweedfsdata-prod
+
 docker compose --env-file web/.env.prod -f web/docker-compose.prod.yaml up -d
 ```

+Docker seeds each volume with the ownership the image expects and labels it for
+SELinux, so the host needs no `chown` and no `:z`/`:Z` mount options on
+enforcing distributions such as RHEL, AlmaLinux and Fedora.
+
+#### Migrating production
+
+`database.migrations` is a run-once job rather than a service in the prod stack,
+so the schema is migrated when an operator asks for it instead of on every boot.
+Build the image and run it against the network the prod stack creates:
+
+```bash
+docker build -t database.migrations \
+  -f web/backend/Database/Database.Migrations/Dockerfile \
+  --build-arg BUILD_CONFIGURATION=Release --build-arg APP_UID=1000 \
+  web/backend
+
+docker run --rm --env-file web/.env.prod --network biergarten-prodnet \
+  -e DOTNET_RUNNING_IN_CONTAINER=true database.migrations
+```
+
+DbUp records every script it has applied, so a re-run applies only what is new.
+Run this with `sqlserver` already up and healthy, and before the seed job.
+
+#### Restart tolerance
+
+`sqlserver`, `api.core`, `seaweedfs` and `frontend` run under
+`restart: unless-stopped`, so the stack returns on its own after a container
+crash, a Docker daemon restart or a host reboot, provided Docker itself starts at
+boot (`systemctl enable docker`). The migration and seed jobs stay out of the
+stack, so a reboot replays neither.
+
+The daemon replays restart policies without replaying the `depends_on` graph, so
+after a reboot the containers come back in arbitrary order and each one tolerates
+a dependency that arrives late. `api.core` opens its database and S3 connections
+per request rather than at startup, and `frontend` answers with its own error
+states while `api.core` is still coming up.
+
+Health checks report when the stack has converged:
+
+| Service | Probe |
+| --- | --- |
+| `sqlserver` | `sqlcmd -Q 'SELECT 1'`, with a 30s start period and 12 retries |
+| `api.core` | `GET /health` spoken over bash's `/dev/tcp` — the .NET runtime image carries bash and no HTTP client |
+| `seaweedfs` | `curl http://localhost:8333/status`, the S3 gateway, which serves once the master, volume and filer servers are up |
+| `frontend` | `wget --spider http://127.0.0.1:5656/healthz`, answered by Express alone and so independent of the API |
+
+`sqlserver` also carries `stop_grace_period: 60s`, giving a checkpoint time to
+finish on shutdown instead of being recovered on the next start.
+
+Deploy and wait for the stack to report healthy, or check convergence after a
+reboot:
+
+```bash
+docker compose --env-file web/.env.prod -f web/docker-compose.prod.yaml up -d --wait
+docker compose --env-file web/.env.prod -f web/docker-compose.prod.yaml ps
+```
+
+#### Seeding production (`docker-compose.prod.seed.yaml`)
+
+`database.seed` lives in its own compose file, run on demand once the prod stack
+is up, `sqlserver` is healthy, and the schema is migrated:
+
+```bash
+docker compose --env-file web/.env.prod -f web/docker-compose.prod.seed.yaml \
+  run --rm --build database.seed
+```
+
+The job runs as the `biergarten-prod-seed` Compose project and attaches to the
+external `biergarten-prodnet` network, which the main prod stack creates, so it
+reaches `sqlserver` and `seaweedfs` under the same hostnames the API uses. It
+exits when seeding finishes; `--rm` removes the container. Bring the prod stack
+up first, because the seed project has no visibility into the other project's
+health checks, so a run started before SQL Server is accepting connections fails.
+
+#### Production data
+
+A production deployment seeded this way serves the AI-generated fixture data
+produced by the pipeline. Per
+[Pipeline ethics, bias, and known issues](../pipeline/ETHICS-AND-KNOWN-ISSUES.md),
+anyone interacting with an application seeded from it has to be told upfront that
+the content is AI-generated.
+
+The website meets that obligation with `AiDisclosureDialog`
+(`web/frontend/app/features/home/components/AiDisclosureDialog.tsx`), which
+the root route renders on every full page load, whichever route the visitor lands
+on. It is part of the server-rendered HTML rather than mounted after hydration, so
+the disclosure is on screen with the first paint. Acknowledging it hides the
+dialog for the remainder of that page load; a new load shows it again.
+
+Keep the seed job and the dialog together: any deployment that runs
+`docker-compose.prod.seed.yaml` needs the disclosure in place to satisfy the
+ethics documentation.
+
 ### 4. Database only (`docker-compose.db.yaml`)

 **Purpose**: Run just the database for local backend development outside Docker
@@ -215,6 +324,12 @@ graph TD
 In development, `api.core` also depends on `seaweedfs`
 (`condition: service_started`, not a health check) alongside `database.seed`.

+Production has a shorter chain, because the migration and seed jobs run outside
+the stack: `api.core` waits for `seaweedfs` to pass its health check, and
+`frontend` waits for `api.core` to report healthy. This ordering applies to
+`docker compose up`; see [Restart tolerance](#restart-tolerance) for what happens
+on a host reboot.
+
 **Health Check Example** (SQL Server):

 ```yaml
@@ -255,8 +370,10 @@ api.core:

 **Production**:

- `sqlserverdata-prod` - Production database files
- `nuget-cache-prod` - Production NuGet cache
+- `biergartendata-prod` - Database files, external; survives `down -v`
+- `seaweedfsdata-prod` - SeaweedFS object storage, external; survives `down -v`
+
+See [Production data volumes](#production-data-volumes).

 ### Mounted volumes

@@ -269,6 +386,41 @@ volumes:

 Test results are written to host filesystem for CI/CD integration.

+### Production data volumes
+
+`biergartendata-prod` and `seaweedfsdata-prod` are declared `external: true`.
+Compose removes only the volumes it owns, so `docker compose down -v` tears the
+stack down and leaves the database and the uploaded photos in place. Deleting
+them takes an explicit `docker volume rm`.
+
+The trade-off is that Compose creates neither one, and `up` fails with a clear
+error until both exist, hence the `docker volume create` step in
+[Deploy Production](#3-production-docker-composeprodyaml). External volumes also
+carry their literal name rather than a `<project>_` prefix, so `docker volume ls`
+shows `biergartendata-prod` and `seaweedfsdata-prod` unprefixed, unlike the
+`web_`-prefixed volumes of the dev and test stacks.
+
+Both live under `/var/lib/docker/volumes`. To put them on a specific disk, point
+the daemon's `data-root` at it in `/etc/docker/daemon.json`, or mount the disk at
+`/var/lib/docker`.
+
+Back up the database through SQL Server rather than by copying its files, since a
+file copy taken while the engine is running can be torn:
+
+```bash
+docker exec prod-env-sqlserver /opt/mssql-tools18/bin/sqlcmd \
+  -S localhost -U sa -P '<db-password>' -C \
+  -Q "BACKUP DATABASE [<db-name>] TO DISK = '/var/opt/mssql/data/backup.bak'"
+docker cp prod-env-sqlserver:/var/opt/mssql/data/backup.bak .
+```
+
+Object storage can be archived directly, with the stack stopped:
+
+```bash
+docker run --rm -v seaweedfsdata-prod:/data -v "$PWD:/backup" \
+  alpine tar czf /backup/seaweedfs.tar.gz -C /data .
+```
+
 ## Networks

 Each environment uses isolated bridge networks: