From 4ebd28c06b625c5af9329a414b97daf3818d1ef0 Mon Sep 17 00:00:00 2001 From: Aaron Po Date: Sun, 19 Jul 2026 04:09:37 -0400 Subject: [PATCH] format docs --- docs/pipeline/ETHICS-AND-KNOWN-ISSUES.md | 35 ++-- docs/pipeline/README.md | 219 ++++++++++--------- docs/pipeline/ROADMAP.md | 256 +++++++++++------------ tooling/pipeline/.clang-tidy | 10 +- 4 files changed, 249 insertions(+), 271 deletions(-) diff --git a/docs/pipeline/ETHICS-AND-KNOWN-ISSUES.md b/docs/pipeline/ETHICS-AND-KNOWN-ISSUES.md index 7892cf5c..ef0aa961 100644 --- a/docs/pipeline/ETHICS-AND-KNOWN-ISSUES.md +++ b/docs/pipeline/ETHICS-AND-KNOWN-ISSUES.md @@ -102,21 +102,19 @@ vendored verbatim, unmodified, from (the `common-forenames-by-country.json` / `common-surnames-by-country.json` release assets), released under **CC0** (public domain). That dataset's own forename/surname lists are pulled from Wikipedia's "Lists of most common -surnames" and "List of most popular given names" as of the week of -2023-07-08 — see that project's README for full provenance. Names are not -LLM-generated; this is curated fixture data per ROADMAP.md §2. Per-forename -gender from the source data is preserved through to the sampled `Name` -(rather than discarded during loading) so it's available for gender-aware -persona/bio generation later. +surnames" and "List of most popular given names" as of the week of 2023-07-08 — +see that project's README for full provenance. Names are not LLM-generated; this +is curated fixture data per ROADMAP.md §2. Per-forename gender from the source +data is preserved through to the sampled `Name` (rather than discarded during +loading) so it's available for gender-aware persona/bio generation later. -The full multinational dataset is kept as-is (106 countries for forenames, -75 for surnames) rather than trimmed to `locations.json`'s current country -list, so it doesn't need re-sourcing if more countries are added later. -`SampleName()` (a free helper in `generate_users.cc`) returns no result for -a country present in neither file; of the countries in `locations.json`, -that's currently `KE`, `SE`, `SG`, `TH`, `VN`, and `ZA` — `GenerateUsers` -skips cities in those countries the same way brewery generation skips -cities whose enrichment lookup fails. +The full multinational dataset is kept as-is (106 countries for forenames, 75 +for surnames) rather than trimmed to `locations.json`'s current country list, so +it doesn't need re-sourcing if more countries are added later. `SampleName()` (a +free helper in `generate_users.cc`) returns no result for a country present in +neither file; of the countries in `locations.json`, that's currently `KE`, `SE`, +`SG`, `TH`, `VN`, and `ZA` — `GenerateUsers` skips cities in those countries the +same way brewery generation skips cities whose enrichment lookup fails. --- @@ -273,8 +271,7 @@ For languages such as Welsh (Wales), Māori (Aotearoa/New Zealand), or Sicilian but is semantically incoherent. This comes from limited training-data coverage rather than prompt engineering. -Output sample: -[./french-cities.example](french-cities.example) +Output sample: [./french-cities.example](french-cities.example) #### Proposed Mitigations @@ -298,9 +295,9 @@ curated real-world examples. The generated code is **format-conformant by construction** — it always satisfies the regex it was generated from — but it is not looked up from any real postal registry. It does not correspond to a real address, business, or -resident, and should not be treated as accurate geocoding or delivery data. -This is consistent with the rest of the dataset: fixture data for a -proof-of-concept, not a source of truth. +resident, and should not be treated as accurate geocoding or delivery data. This +is consistent with the rest of the dataset: fixture data for a proof-of-concept, +not a source of truth. --- diff --git a/docs/pipeline/README.md b/docs/pipeline/README.md index eed2936c..e30b03f9 100644 --- a/docs/pipeline/README.md +++ b/docs/pipeline/README.md @@ -1,9 +1,9 @@ # Biergarten Pipeline A C++20 command-line pipeline that samples city records from local JSON, -enriches each with Wikipedia context, and generates bilingual brewery names -and descriptions plus locale-grounded user profiles via a local GGUF model or -a deterministic mock. +enriches each with Wikipedia context, and generates bilingual brewery names and +descriptions plus locale-grounded user profiles via a local GGUF model or a +deterministic mock. > **This pipeline produces AI-generated data.** It is not a source of truth for > brewing techniques, cultural representation, or local-language accuracy. See @@ -40,12 +40,12 @@ The pipeline is a data ingestion layer. It sits outside the web app runtime and produces seed records the app imports at startup or during a dedicated seed step. -| Planned app area | Pipeline contribution | -| -------------------------------- | ------------------------------------------------------------------ | -| Brewery discovery and management | Sampled city records, localized names, long-form descriptions | -| Beer reviews and ratings | Stable brewery fixtures with enough context to anchor review pages | -| Social follow relationships | Repeatable brewery entities for feeds, follows, and saved lists | -| Geospatial brewery experiences | Latitude, longitude, and country-level metadata | +| Planned app area | Pipeline contribution | +| -------------------------------- | ------------------------------------------------------------------------------------------------- | +| Brewery discovery and management | Sampled city records, localized names, long-form descriptions | +| Beer reviews and ratings | Stable brewery fixtures with enough context to anchor review pages | +| Social follow relationships | Repeatable brewery entities for feeds, follows, and saved lists | +| Geospatial brewery experiences | Latitude, longitude, and country-level metadata | | User accounts and profiles | Locale-grounded names, bios, and an auth-ready email/date-of-birth pair for seeding real accounts | --- @@ -65,10 +65,9 @@ cmake --build build CMake automatically detects whether a compatible llama.cpp installation is present on the system (`libllama`, `libggml`, `libggml-base`, and `llama.h` -visible on the default search paths). If found, it links against those -libraries and skips the FetchContent build. If not found, it fetches and builds -llama.cpp from source at tag `b9012`. No additional flags are required in -either case. +visible on the default search paths). If found, it links against those libraries +and skips the FetchContent build. If not found, it fetches and builds llama.cpp +from source at tag `b9012`. No additional flags are required in either case. Metal is enabled automatically on Apple Silicon. CUDA or HIP/ROCm is detected automatically on Linux when the relevant toolkit is present. @@ -138,17 +137,18 @@ NVIDIA GPU. The container uses a two-stage build. The builder stage installs CMake/Ninja, clones the matching llama.cpp release tag for its headers only (installed into `/usr/local/include`), and copies prebuilt shared libraries (`libllama`, -`libggml`, and CUDA/CPU backend plugins) from `ghcr.io/ggml-org/llama.cpp:full-cuda` -into `/usr/local/lib`. With both headers and libraries present, CMake's -system-library detection (see [Build](#build) above) finds them and skips the -FetchContent source build, keeping image build times short. +`libggml`, and CUDA/CPU backend plugins) from +`ghcr.io/ggml-org/llama.cpp:full-cuda` into `/usr/local/lib`. With both headers +and libraries present, CMake's system-library detection (see [Build](#build) +above) finds them and skips the FetchContent source build, keeping image build +times short. The runtime stage copies the compiled binary, the same prebuilt shared libraries, and config/prompt assets into a slim CUDA runtime image. It sets -`LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH` so the dynamic linker -resolves `libllama`/`libggml` at startup, and also co-locates -`libggml-cuda.so` and the CPU backend plugins next to the binary for -`ggml_backend_load_all()`'s `dlopen` scan. +`LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH` so the dynamic linker resolves +`libllama`/`libggml` at startup, and also co-locates `libggml-cuda.so` and the +CPU backend plugins next to the binary for `ggml_backend_load_all()`'s `dlopen` +scan. ### Build the image @@ -177,8 +177,8 @@ output to confirm the fast path was taken. The container always runs the model-backed path; there is no `--mocked` container mode (use a native build for that — see [Quick Start](#quick-start)). -The entrypoint, `runpod/start.sh`, downloads the GGUF model automatically if -it is not already present at the configured path. +The entrypoint, `runpod/start.sh`, downloads the GGUF model automatically if it +is not already present at the configured path. ```bash docker run --rm \ @@ -189,20 +189,20 @@ docker run --rm \ biergarten-pipeline:latest ``` -By default this downloads `google_gemma-4-E4B-it-Q6_K.gguf` to -`./models/` on first run if it isn't already there. To use a pre-downloaded -model, place it at that path first — see [Model](#model) above. +By default this downloads `google_gemma-4-E4B-it-Q6_K.gguf` to `./models/` on +first run if it isn't already there. To use a pre-downloaded model, place it at +that path first — see [Model](#model) above. #### Environment variables -| Variable | Purpose | -| ------------------------- | ----------------------------------------------------------------- | -| `BIERGARTEN_MODEL_PATH` | GGUF model path. Default: `/workspace/models/google_gemma-4-E4B-it-Q6_K.gguf`. | -| `BIERGARTEN_OUTPUT_DIR` | SQLite output directory. Default: `/workspace/output`. | -| `BIERGARTEN_LOG_PATH` | Log file path. Default: `/workspace/logs/pipeline.log`. | -| `BIERGARTEN_GL_LAYERS` | GPU layers to offload (`--n-gpu-layers`). Default: `40`. | -| `BIERGARTEN_TEMPERATURE`, `BIERGARTEN_TOP_P`, `BIERGARTEN_TOP_K`, `BIERGARTEN_N_CTX`, `BIERGARTEN_SEED` | Optional sampling overrides, unset by default (binary defaults apply). | -| `BIERGARTEN_EXTRA_ARGS` | Additional raw CLI args appended verbatim. | +| Variable | Purpose | +| ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | +| `BIERGARTEN_MODEL_PATH` | GGUF model path. Default: `/workspace/models/google_gemma-4-E4B-it-Q6_K.gguf`. | +| `BIERGARTEN_OUTPUT_DIR` | SQLite output directory. Default: `/workspace/output`. | +| `BIERGARTEN_LOG_PATH` | Log file path. Default: `/workspace/logs/pipeline.log`. | +| `BIERGARTEN_GL_LAYERS` | GPU layers to offload (`--n-gpu-layers`). Default: `40`. | +| `BIERGARTEN_TEMPERATURE`, `BIERGARTEN_TOP_P`, `BIERGARTEN_TOP_K`, `BIERGARTEN_N_CTX`, `BIERGARTEN_SEED` | Optional sampling overrides, unset by default (binary defaults apply). | +| `BIERGARTEN_EXTRA_ARGS` | Additional raw CLI args appended verbatim. | `--prompt-dir` is hardcoded to `/app/prompts` inside the container and is not configurable via environment variable. @@ -220,15 +220,15 @@ environment variables listed above to match your run. ### Pipeline Stages -| Stage | Implementation | -| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | -| Load | `ICuratedDataService` (`CuratedJsonDataService`) reads `locations.json`, `personas.json`, `forenames-by-country.json`, and `surnames-by-country.json` (paths supplied via a `CuratedDataFilePaths` DTO at construction) into typed records, caching each after its first load. `--mocked` runs use `MockCuratedDataService`'s fixed in-memory dataset instead. | -| Sample | `BiergartenPipelineOrchestrator::QueryCitiesWithCountries()` samples `--location-count` locations per run (default `10`). | -| Enrich | `WikipediaEnrichmentService` fetches brewing and beer-related context. Keeps going when a lookup fails. `--mocked` runs use `MockEnrichmentService` instead and skip Wikipedia entirely. | -| Generate Users | `GenerateUsers()` samples a persona and a forename/surname pair per enriched city (skipping countries with no name data), then `MockGenerator` or `LlamaGenerator` produces a username, bio, and activity weight around the sampled name. | -| Generate Breweries | `MockGenerator` or `LlamaGenerator` produces brewery names and descriptions in English and the local language. | -| Store | `SqliteExportService` writes each successful user and brewery into a fresh dated `.sqlite` database with normalized `locations`, `users`, and `breweries` tables. | -| Log | `spdlog` writes results and warnings to the console. | +| Stage | Implementation | +| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Load | `ICuratedDataService` (`CuratedJsonDataService`) reads `locations.json`, `personas.json`, `forenames-by-country.json`, and `surnames-by-country.json` (paths supplied via a `CuratedDataFilePaths` DTO at construction) into typed records, caching each after its first load. `--mocked` runs use `MockCuratedDataService`'s fixed in-memory dataset instead. | +| Sample | `BiergartenPipelineOrchestrator::QueryCitiesWithCountries()` samples `--location-count` locations per run (default `10`). | +| Enrich | `WikipediaEnrichmentService` fetches brewing and beer-related context. Keeps going when a lookup fails. `--mocked` runs use `MockEnrichmentService` instead and skip Wikipedia entirely. | +| Generate Users | `GenerateUsers()` samples a persona and a forename/surname pair per enriched city (skipping countries with no name data), then `MockGenerator` or `LlamaGenerator` produces a username, bio, and activity weight around the sampled name. | +| Generate Breweries | `MockGenerator` or `LlamaGenerator` produces brewery names and descriptions in English and the local language. | +| Store | `SqliteExportService` writes each successful user and brewery into a fresh dated `.sqlite` database with normalized `locations`, `users`, and `breweries` tables. | +| Log | `spdlog` writes results and warnings to the console. | If name sampling, enrichment, or generation fails for a city, that city is skipped and the pipeline continues. `GenerateUsers()` runs before @@ -240,33 +240,32 @@ skipped and the pipeline continues. `GenerateUsers()` runs before - `CuratedJsonDataService` — implements `ICuratedDataService`; takes a `CuratedDataFilePaths` DTO (locations/personas/forenames/surnames paths) in its constructor, then parses and validates curated location, persona, and - forename/surname JSON, memoizing each result after its first load on a - given instance. `MockCuratedDataService` is the in-memory substitute (4 - fixed locations, 3 personas, and name data for `US`/`DE`/`FR`/`BE`) used in + forename/surname JSON, memoizing each result after its first load on a given + instance. `MockCuratedDataService` is the in-memory substitute (4 fixed + locations, 3 personas, and name data for `US`/`DE`/`FR`/`BE`) used in `--mocked` runs. - `WikipediaEnrichmentService` — queries Wikipedia extracts, caches results, returns empty context on failure. `MockEnrichmentService` is the no-op substitute used in `--mocked` runs. -- `LlamaGenerator` — formats prompts for Gemma 4, validates JSON output for - both `GenerateBrewery` and `GenerateUser`, retries malformed responses up - to three times with corrective feedback in the retry prompt. The token - budget is fixed across attempts; it is not raised automatically on - truncation. -- `MockGenerator` — stable hash-based output so the same city/persona/name - input always produces the same brewery or user. -- `SqliteExportService` — creates a dated SQLite file per run and persists - each successful user and brewery into normalized tables. +- `LlamaGenerator` — formats prompts for Gemma 4, validates JSON output for both + `GenerateBrewery` and `GenerateUser`, retries malformed responses up to three + times with corrective feedback in the retry prompt. The token budget is fixed + across attempts; it is not raised automatically on truncation. +- `MockGenerator` — stable hash-based output so the same city/persona/name input + always produces the same brewery or user. +- `SqliteExportService` — creates a dated SQLite file per run and persists each + successful user and brewery into normalized tables. - Brewery payloads include English and local-language name and description fields. User payloads carry a sampled first/last name and gender, an - LLM-generated username/bio/activity weight, and a programmatically - generated (not LLM-authored) unique email and date of birth. + LLM-generated username/bio/activity weight, and a programmatically generated + (not LLM-authored) unique email and date of birth. ### Runtime Behaviour -`WikipediaEnrichmentService` fetches two Wikipedia extracts per city: a -generic "brewing" extract and a "beer in `{country}`" extract. It does not -currently query a city- or region-specific page. Each query string is cached -after its first successful (or empty) lookup. +`WikipediaEnrichmentService` fetches two Wikipedia extracts per city: a generic +"brewing" extract and a "beer in `{country}`" extract. It does not currently +query a city- or region-specific page. Each query string is cached after its +first successful (or empty) lookup. `GetLocationContext()` returns an empty string when the web client is unavailable or when lookup/parsing fails. @@ -281,15 +280,15 @@ runs. `CuratedJsonDataService` memoizes each of `LoadLocations()`, `LoadPersonas()`, `LoadForenamesByCountry()`, and `LoadSurnamesByCountry()` independently the -first time each is called, since `BiergartenPipelineOrchestrator` owns a -single `ICuratedDataService` instance for the whole run — later calls return -the cached result instead of re-parsing. +first time each is called, since `BiergartenPipelineOrchestrator` owns a single +`ICuratedDataService` instance for the whole run — later calls return the cached +result instead of re-parsing. `GenerateUsers()` samples a forename/surname pair per city via `SampleName()`, -keyed by the city's ISO 3166-1 code. Countries present in `locations.json` -but absent from either name fixture (currently `KE`, `SE`, `SG`, `TH`, `VN`, -`ZA`) are skipped, the same way a failed enrichment or generation call skips -a city — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section. +keyed by the city's ISO 3166-1 code. Countries present in `locations.json` but +absent from either name fixture (currently `KE`, `SE`, `SG`, `TH`, `VN`, `ZA`) +are skipped, the same way a failed enrichment or generation call skips a city — +see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section. ### Process Flow - Activity Diagram @@ -303,10 +302,10 @@ a city — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section. ## Generated Output -Each successful run stores a `BreweryRecord` pair with the source location -and a `BreweryResult` payload, and a `UserRecord` pair with the source -location and a `UserResult` payload. The same generated records are also -written to a fresh SQLite export file named with the current UTC timestamp. +Each successful run stores a `BreweryRecord` pair with the source location and a +`BreweryResult` payload, and a `UserRecord` pair with the source location and a +`UserResult` payload. The same generated records are also written to a fresh +SQLite export file named with the current UTC timestamp. | Field | Meaning | | ------------------- | ------------------------------------------ | @@ -315,16 +314,16 @@ written to a fresh SQLite export file named with the current UTC timestamp. | `name_local` | Brewery name in the local language. | | `description_local` | Brewery description in the local language. | -| Field | Meaning | -| ----------------- | ----------------------------------------------------------------- | -| `first_name` | Sampled forename, copied from the curated name data (not LLM-invented). | -| `last_name` | Sampled surname, copied from the curated name data (not LLM-invented). | -| `gender` | Gender associated with the sampled forename in the source dataset. | -| `username` | LLM-generated handle. | -| `bio` | LLM-generated short biography. | -| `activity_weight` | Relative check-in/activity weight, reserved for a future J-curve activity profile. | +| Field | Meaning | +| ----------------- | -------------------------------------------------------------------------------------- | +| `first_name` | Sampled forename, copied from the curated name data (not LLM-invented). | +| `last_name` | Sampled surname, copied from the curated name data (not LLM-invented). | +| `gender` | Gender associated with the sampled forename in the source dataset. | +| `username` | LLM-generated handle. | +| `bio` | LLM-generated short biography. | +| `activity_weight` | Relative check-in/activity weight, reserved for a future J-curve activity profile. | | `email` | Unique `@thebiergarten.app` address, generated programmatically from the sampled name. | -| `date_of_birth` | Randomized date of birth (age 19-48), generated programmatically. | +| `date_of_birth` | Randomized date of birth (age 19-48), generated programmatically. | The log dump also includes city, country, state or province, ISO subdivision code, latitude, and longitude for each entry. @@ -358,11 +357,9 @@ The build fetches Boost.DI, spdlog, and SQLite via CMake. llama.cpp is fetched only when a system installation is not detected. Metal is enabled on Apple Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present. -> **Code Style:** Modern C++20 throughout — RAII for ownership, -> `std::unique_ptr` for injected dependencies, `std::optional` for parse -> outcomes, `std::span` for read-only views over generated city data, structured -> bindings in pipeline loops. Formatting follows the Google C++ Style Guide via -> `.clang-format` with a narrow column limit and two-space indentation. +> **Code Style:** Code style generally follows the Google Style Guide and +> features modern C++20 throughout. Formatting follows the Google C++ Style +> Guide, with the exception of indentation width (3 spaces vs. Google's 2). --- @@ -404,34 +401,33 @@ Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present. ## Fixture Strategy - `--mocked` for stable fixtures, repeatable screenshots, and Storybook runs. - `MockCuratedDataService` swaps in for `CuratedJsonDataService`, so no - fixture files need to be present on disk. + `MockCuratedDataService` swaps in for `CuratedJsonDataService`, so no fixture + files need to be present on disk. - `--model` when geographically grounded content matters for demos. - Keep `locations.json` structured enough to support discovery and future filtering. -- `personas.json`, `forenames-by-country.json`, and - `surnames-by-country.json` are curated/vendored fixture data, not - LLM-generated — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset - section for provenance. -- Treat SQLite output as seed material for the app's brewery and user - domains, not production data. +- `personas.json`, `forenames-by-country.json`, and `surnames-by-country.json` + are curated/vendored fixture data, not LLM-generated — see + ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for provenance. +- Treat SQLite output as seed material for the app's brewery and user domains, + not production data. --- ## Repo Layout -| Path | Purpose | -| -------------------------------------- | -------------------------------------------------- | -| `tooling/pipeline/includes/` | Public headers and shared models. | -| `tooling/pipeline/src/` | Implementation files. | -| `tooling/pipeline/locations.json` | Curated city input copied into the build tree. | -| `tooling/pipeline/personas.json` | Curated user persona archetypes copied into the build tree. | -| `tooling/pipeline/forenames-by-country.json` | Vendored (CC0) forename data by ISO 3166-1 country code. | -| `tooling/pipeline/surnames-by-country.json` | Vendored (CC0) surname data by ISO 3166-1 country code. | -| `tooling/pipeline/prompts/` | System prompts used by the model-backed path. | -| `tooling/pipeline/runpod/` | Dockerfile, launcher, and RunPod pod template. | -| `docs/pipeline/diagrams/` | Architecture and pipeline diagrams. | -| `docs/pipeline/ETHICS-AND-KNOWN-ISSUES.md` | Ethics, bias, hallucination analysis, mitigations. | +| Path | Purpose | +| -------------------------------------------- | ----------------------------------------------------------- | +| `tooling/pipeline/includes/` | Public headers and shared models. | +| `tooling/pipeline/src/` | Implementation files. | +| `tooling/pipeline/locations.json` | Curated city input copied into the build tree. | +| `tooling/pipeline/personas.json` | Curated user persona archetypes copied into the build tree. | +| `tooling/pipeline/forenames-by-country.json` | Vendored (CC0) forename data by ISO 3166-1 country code. | +| `tooling/pipeline/surnames-by-country.json` | Vendored (CC0) surname data by ISO 3166-1 country code. | +| `tooling/pipeline/prompts/` | System prompts used by the model-backed path. | +| `tooling/pipeline/runpod/` | Dockerfile, launcher, and RunPod pod template. | +| `docs/pipeline/diagrams/` | Architecture and pipeline diagrams. | +| `docs/pipeline/ETHICS-AND-KNOWN-ISSUES.md` | Ethics, bias, hallucination analysis, mitigations. | --- @@ -456,12 +452,11 @@ Paths below are relative to `tooling/pipeline/`. ## Next Steps -The pipeline currently produces city-aware brewery and user records and -dated SQLite exports. The next passes add additional fixture types so the -app can exercise the full brewery and social domains without live data. For -the detailed engineering breakdown of what's needed to reach the -architecture in [`diagrams/planned/`](./diagrams/planned/), see -[ROADMAP.md](./ROADMAP.md). +The pipeline currently produces city-aware brewery and user records and dated +SQLite exports. The next passes add additional fixture types so the app can +exercise the full brewery and social domains without live data. For the detailed +engineering breakdown of what's needed to reach the architecture in +[`diagrams/planned/`](./diagrams/planned/), see [ROADMAP.md](./ROADMAP.md). ### Testing — Very High Priority diff --git a/docs/pipeline/ROADMAP.md b/docs/pipeline/ROADMAP.md index 0f2876f8..71e01f06 100644 --- a/docs/pipeline/ROADMAP.md +++ b/docs/pipeline/ROADMAP.md @@ -3,13 +3,12 @@ This is the engineering breakdown for closing the gap between the **current** pipeline and the **planned** architecture documented in [`diagrams/planned/class.puml`](./diagrams/planned/class.puml) and -[`diagrams/planned/activity.puml`](./diagrams/planned/activity.puml). Nothing -in `diagrams/planned/` is implemented yet — this file tracks what it would -take to get there. For the current implementation, see -[README.md](./README.md). +[`diagrams/planned/activity.puml`](./diagrams/planned/activity.puml). Nothing in +`diagrams/planned/` is implemented yet — this file tracks what it would take to +get there. For the current implementation, see [README.md](./README.md). -Items are grouped by layer and roughly ordered by dependency: later groups -build on types and services introduced earlier. +Items are grouped by layer and roughly ordered by dependency: later groups build +on types and services introduced earlier. The only concurrency in the current pipeline is the log dispatcher thread (`main.cc` spawns it to drain a `BoundedChannel` into spdlog for the @@ -25,31 +24,30 @@ architecture needs. `tooling/pipeline/includes/data_model/generated_models.h` and `models.h`. - [ ] Add `Completeness` enum (`Full`, `Partial`, `Absent`) and a - `LocationContext` struct (`text`, `completeness`, `char_count`). - Replace `EnrichedCity::region_context` (currently a plain - `std::string`) with `context : LocationContext`. -- [ ] Add `BeerStyle` (`name`, `description`, `min_abv`, `max_abv`, - `min_ibu`, `max_ibu`). + `LocationContext` struct (`text`, `completeness`, `char_count`). Replace + `EnrichedCity::region_context` (currently a plain `std::string`) with + `context : LocationContext`. +- [ ] Add `BeerStyle` (`name`, `description`, `min_abv`, `max_abv`, `min_ibu`, + `max_ibu`). - [ ] Add `BeerResult`, `CheckinResult`, `RatingResult` result payloads. - [ ] Add `GenerationMetadata` (`generation_id`, `generated_time`, `context_provided`, `generated_with`). - [x] Add `first_name`, `last_name`, `gender`, and `activity_weight` to `UserResult` (currently just `username`, `bio`). `first_name`/ - `last_name`/`gender` are copied from the sampled `Name` (see below), - not LLM-invented. + `last_name`/`gender` are copied from the sampled `Name` (see below), not + LLM-invented. - [x] Add `Name` (`first_name`, `last_name`, `gender`) — the sampled result handed to `DataGenerator::GenerateUser`. Add `ForenameEntry` (`name`, `gender`). Landed as a flatter shape than originally planned here: no - `NamesByCountry` wrapper class. `curated_data_service.h` instead - declares `ForenameList = std::vector` and - `SurnameList = std::vector`, and - `ICuratedDataService` exposes two maps keyed by ISO 3166-1 code — + `NamesByCountry` wrapper class. `curated_data_service.h` instead declares + `ForenameList = std::vector` and + `SurnameList = std::vector`, and `ICuratedDataService` + exposes two maps keyed by ISO 3166-1 code — `ForenamesByCountryMap = unordered_map` and - `SurnamesByCountryMap = unordered_map` — - directly (see §2). Sampling is - a free function, `SampleName(forenames_by_country, - surnames_by_country, iso3166_1, rng)`, in - `generate_users.cc`'s anonymous namespace, not a method on a class. + `SurnamesByCountryMap = unordered_map` — directly + (see §2). Sampling is a free function, + `SampleName(forenames_by_country, surnames_by_country, iso3166_1, rng)`, + in `generate_users.cc`'s anonymous namespace, not a method on a class. Pairing a forename with a surname happens at sample time so gender (present per-forename in the source data) is never discarded the way a pre-flattened `{first_name, last_name}` list would lose it; see §2. @@ -58,14 +56,14 @@ architecture needs. - [ ] Add `GeneratedBeer`, `GeneratedCheckin`, `GeneratedRating`, `GeneratedFollow` aggregate structs. - [x] Add `UserRecord` (`location`, `user : UserResult`, `email`, - `date_of_birth`) as the current stand-in for the planned - `GeneratedUser` — `email` and `date_of_birth` are programmatically - generated by the orchestrator, never LLM-authored, so a downstream - auth-account seeding consumer can register real accounts from the - pipeline's SQLite export. No `password`, `user_id`, or `metadata` - field yet, matching `BreweryRecord`'s equally pre-`metadata` shape. - Already wired into `IExportService`/`SqliteExportService`: a `users` - table exists and `GenerateUsers()` exports each successful record via + `date_of_birth`) as the current stand-in for the planned `GeneratedUser` — + `email` and `date_of_birth` are programmatically generated by the + orchestrator, never LLM-authored, so a downstream auth-account seeding + consumer can register real accounts from the pipeline's SQLite export. No + `password`, `user_id`, or `metadata` field yet, matching `BreweryRecord`'s + equally pre-`metadata` shape. Already wired into + `IExportService`/`SqliteExportService`: a `users` table exists and + `GenerateUsers()` exports each successful record via `ProcessRecord(const UserRecord&)` (see §5) in addition to logging and holding `generated_users_` in memory. Still missing vs. the planned `GeneratedUser`: `user_id`, `password`, and `metadata`. @@ -75,66 +73,62 @@ architecture needs. `tooling/pipeline/includes/services/curated_data/`, fixture files. -- [x] Extract an interface; have `CuratedJsonDataService` implement it. - Landed as `ICuratedDataService` - (`services/curated_data/curated_data_service.h`), not `DataPreloader` - — `LoadLocations()`, `LoadPersonas()`, `LoadForenamesByCountry()`, and - `LoadSurnamesByCountry()` are all virtual methods returning `const&` - and take no arguments. `CuratedJsonDataService` - (`services/curated_data/curated_json_data_service.h`, originally - landed as `JsonLoader` under `json_handling/` before being renamed and - moved) takes a `CuratedDataFilePaths` DTO (`locations_path`, - `personas_path`, `forenames_path`, `surnames_path`) in its - constructor rather than a path per `Load*()` call, and memoizes each - result in a private `cache` struct (`locations`, `personas`, - `forenames_by_country`, `surnames_by_country`), so a second call on - the same instance returns the cached result instead of re-parsing — - safe because `CuratedJsonDataService` outlives every call site (owned - by `BiergartenPipelineOrchestrator` via +- [x] Extract an interface; have `CuratedJsonDataService` implement it. Landed + as `ICuratedDataService` (`services/curated_data/curated_data_service.h`), + not `DataPreloader` — `LoadLocations()`, `LoadPersonas()`, + `LoadForenamesByCountry()`, and `LoadSurnamesByCountry()` are all virtual + methods returning `const&` and take no arguments. `CuratedJsonDataService` + (`services/curated_data/curated_json_data_service.h`, originally landed as + `JsonLoader` under `json_handling/` before being renamed and moved) takes + a `CuratedDataFilePaths` DTO (`locations_path`, `personas_path`, + `forenames_path`, `surnames_path`) in its constructor rather than a path + per `Load*()` call, and memoizes each result in a private `cache` struct + (`locations`, `personas`, `forenames_by_country`, `surnames_by_country`), + so a second call on the same instance returns the cached result instead of + re-parsing — safe because `CuratedJsonDataService` outlives every call + site (owned by `BiergartenPipelineOrchestrator` via `unique_ptr` for the whole run). `main.cc`'s DI - injector also gained a `MockCuratedDataService` (fixed in-memory - data: 4 locations across `US`/`DE`/`FR`/`BE`, 3 personas, and - matching forename/surname sets for those four countries), bound in - place of `CuratedJsonDataService` under `--mocked`, mirroring the - existing `MockEnrichmentService`/`MockGenerator` pattern. -- [ ] Add `LoadBeerStyles()`. `beer-styles.json` already exists in the repo - and is already copied into the Docker image - (`runpod/Dockerfile`), but no loader reads it yet, and the native - CMake build doesn't copy it into `build/` at all — `CMakeLists.txt`'s - "Runtime Assets" step only copies `locations.json` and `prompts/`. + injector also gained a `MockCuratedDataService` (fixed in-memory data: 4 + locations across `US`/`DE`/`FR`/`BE`, 3 personas, and matching + forename/surname sets for those four countries), bound in place of + `CuratedJsonDataService` under `--mocked`, mirroring the existing + `MockEnrichmentService`/`MockGenerator` pattern. +- [ ] Add `LoadBeerStyles()`. `beer-styles.json` already exists in the repo and + is already copied into the Docker image (`runpod/Dockerfile`), but no + loader reads it yet, and the native CMake build doesn't copy it into + `build/` at all — `CMakeLists.txt`'s "Runtime Assets" step only copies + `locations.json` and `prompts/`. - [x] Add `LoadPersonas()` and author `personas.json` (doesn't exist yet). - [x] Add name-by-country loading, parsing `tooling/pipeline/forenames-by-country.json` and `surnames-by-country.json`. Landed as two separate methods — - `LoadForenamesByCountry()` and `LoadSurnamesByCountry()` — each - returning a flat `ForenamesByCountryMap` / `SurnamesByCountryMap` - (aliases for `unordered_map` / + `LoadForenamesByCountry()` and `LoadSurnamesByCountry()` — each returning + a flat `ForenamesByCountryMap` / `SurnamesByCountryMap` (aliases for + `unordered_map` / `unordered_map`) keyed by ISO 3166-1 code, rather - than one combined `LoadNamesByCountry() : NamesByCountry` call. Both - files are vendored verbatim (unmodified, - full multinational coverage) from + than one combined `LoadNamesByCountry() : NamesByCountry` call. Both files + are vendored verbatim (unmodified, full multinational coverage) from `sigpwned/popular-names-by-country-dataset` (CC0) — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for provenance. Deliberately not pre-paired or filtered down to just the countries in `locations.json`: keeping the source shape (including - per-forename gender) intact means the loader can support more - countries later, or gender-aware persona/bio generation, without - re-sourcing data. + per-forename gender) intact means the loader can support more countries + later, or gender-aware persona/bio generation, without re-sourcing data. ## 3. Policy / Strategy Layer Entirely new — no `includes/policy/` (or equivalent) directory exists today. - [ ] `ContextStrategy` interface + `BreweryContextStrategy` / - `BeerContextStrategy`. Today `WikipediaEnrichmentService::GetLocationContext` - hardcodes a generic `"brewing"` query and a `"beer in {country}"` - query directly — no per-phase strategy selection. -- [ ] `SamplingStrategy` interface + `UniformSamplingStrategy`, replacing - the inline `std::ranges::sample(...)` call in + `BeerContextStrategy`. Today + `WikipediaEnrichmentService::GetLocationContext` hardcodes a generic + `"brewing"` query and a `"beer in {country}"` query directly — no + per-phase strategy selection. +- [ ] `SamplingStrategy` interface + `UniformSamplingStrategy`, replacing the + inline `std::ranges::sample(...)` call in `BiergartenPipelineOrchestrator::QueryCitiesWithCountries()`. -- [ ] `BeerSelectionStrategy` interface + `RandomBeerSelectionStrategy`, to - pick styles per brewery from the `BeerStyle` palette (depends on §1 - and §2). +- [ ] `BeerSelectionStrategy` interface + `RandomBeerSelectionStrategy`, to pick + styles per brewery from the `BeerStyle` palette (depends on §1 and §2). - [ ] `CheckinDistributionStrategy` interface + `JCurveCheckinStrategy` / `RandomCheckinStrategy` — this is the "Check-In System" item already called out in README.md's Next Steps, made concrete. @@ -150,88 +144,87 @@ Entirely new — no `includes/policy/` (or equivalent) directory exists today. `GenerateUser`). - [x] Implement `LlamaGenerator::GenerateUser` for real, with a retry loop mirroring `GenerateBrewery` (GBNF grammar, `ValidateUserJson`, up to 3 - attempts with corrective feedback) and a new - `prompts/USER_GENERATION.md`. Changed the `DataGenerator::GenerateUser` - signature to + attempts with corrective feedback) and a new `prompts/USER_GENERATION.md`. + Changed the `DataGenerator::GenerateUser` signature to `(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult`, - matching what the activity diagram actually passes. `MockGenerator::GenerateUser` - updated to match the new signature too. + matching what the activity diagram actually passes. + `MockGenerator::GenerateUser` updated to match the new signature too. - [ ] Implement `MockGenerator::GenerateBeer` / `GenerateCheckin` / `GenerateRating`. - [ ] Add `IPromptFormatter::ExpectedArchitecture()` and `LlamaGenerator::ValidateModelArchitecture()` so loading a GGUF that - doesn't match the configured chat template fails fast instead of - silently producing degraded output. -- [ ] Add prompt template files for beer/checkin/rating generation next to - the existing `prompts/BREWERY_GENERATION.md`. + doesn't match the configured chat template fails fast instead of silently + producing degraded output. +- [ ] Add prompt template files for beer/checkin/rating generation next to the + existing `prompts/BREWERY_GENERATION.md`. ## 5. Export Service `tooling/pipeline/includes/services/database/`, `src/services/sqlite/`. - [x] `IExportService::ProcessRecord(const UserRecord&)` and - `SqliteExportService`'s `users` table (see - `kCreateUsersTableSql` / `kInsertUserSql` in - `includes/services/database/sqlite_statement_helpers.h`) are - implemented — `GenerateUsers()` exports every successful user - alongside its resolved `location_id`. + `SqliteExportService`'s `users` table (see `kCreateUsersTableSql` / + `kInsertUserSql` in + `includes/services/database/sqlite_statement_helpers.h`) are implemented — + `GenerateUsers()` exports every successful user alongside its resolved + `location_id`. - [ ] Extend `IExportService` with `ProcessBeer`, `ProcessCheckin`, - `ProcessRating`, `ProcessFollow` (today `ProcessRecord(const - BreweryRecord&)` and `ProcessRecord(const UserRecord&)` exist). + `ProcessRating`, `ProcessFollow` (today + `ProcessRecord(const BreweryRecord&)` and + `ProcessRecord(const UserRecord&)` exist). - [ ] Add `beers`, `checkins`, `ratings`, `follows` tables to - `SqliteExportService::InitializeSchema()` (`locations`, `breweries`, - and `users` already exist) — see `kCreateLocationsTableSql` / + `SqliteExportService::InitializeSchema()` (`locations`, `breweries`, and + `users` already exist) — see `kCreateLocationsTableSql` / `kCreateBreweriesTableSql` / `kCreateUsersTableSql` in - `includes/services/database/sqlite_statement_helpers.h` for the - existing pattern to follow. -- [ ] Add a `brewery_cache_` alongside the existing `location_cache_`, and - move from one open transaction for the whole run to per-phase batched - commits (`BEGIN` / `COMMIT & BEGIN` on a batch-size threshold), as - shown in the planned activity diagram. + `includes/services/database/sqlite_statement_helpers.h` for the existing + pattern to follow. +- [ ] Add a `brewery_cache_` alongside the existing `location_cache_`, and move + from one open transaction for the whole run to per-phase batched commits + (`BEGIN` / `COMMIT & BEGIN` on a batch-size threshold), as shown in the + planned activity diagram. ## 6. Concurrency & Orchestration `tooling/pipeline/includes/concurrency/`, `src/biergarten_pipeline_orchestrator/`. -- [ ] `BoundedChannel` already exists and is production-tested — but - it's only wired to the single log channel today. Stand up the - per-phase producer/consumer channels (`loc_ch`, `exp_ch`, etc.) the - planned activity diagram describes, each with a dedicated LLM worker - thread and SQLite worker thread. -- [ ] Rewrite `BiergartenPipelineOrchestrator::Run()` from one synchronous - pass over `generated_breweries_` into phased methods — - `RunUserPhase` → `RunBreweryAndBeerPhase` (with a `RunBeerPhase` - sub-step once `brewery_pool_` is populated) → `RunCheckinPhase` / - `RunFollowPhase` (these two can run in parallel) → `RunRatingPhase` — - backed by `user_pool_`, `brewery_pool_`, `beer_pool_`, - `checkin_pool_`, `follow_pool_`. +- [ ] `BoundedChannel` already exists and is production-tested — but it's + only wired to the single log channel today. Stand up the per-phase + producer/consumer channels (`loc_ch`, `exp_ch`, etc.) the planned activity + diagram describes, each with a dedicated LLM worker thread and SQLite + worker thread. +- [ ] Rewrite `BiergartenPipelineOrchestrator::Run()` from one synchronous pass + over `generated_breweries_` into phased methods — `RunUserPhase` → + `RunBreweryAndBeerPhase` (with a `RunBeerPhase` sub-step once + `brewery_pool_` is populated) → `RunCheckinPhase` / `RunFollowPhase` + (these two can run in parallel) → `RunRatingPhase` — backed by + `user_pool_`, `brewery_pool_`, `beer_pool_`, `checkin_pool_`, + `follow_pool_`. - [ ] Honor the structural-concurrency requirement already called out in a comment on `BiergartenPipelineOrchestrator::Run()` in - `biergarten_pipeline_orchestrator.h`: once real worker threads exist, - they must be structurally joined (e.g. via `std::jthread`) before - `Run()` returns, so no worker logs to a closed channel during - teardown. + `biergarten_pipeline_orchestrator.h`: once real worker threads exist, they + must be structurally joined (e.g. via `std::jthread`) before `Run()` + returns, so no worker logs to a closed channel during teardown. ## 7. Enrichment -- [ ] Decide whether to restore the city/region-specific Wikipedia query - that's currently commented out in - `src/services/enrichment/wikipedia/get_summary.cc` - (`GetLocationContext`). The `ContextStrategy` work in §3 is a natural - place to reintroduce it via `BreweryContextStrategy::QueriesFor()`. +- [ ] Decide whether to restore the city/region-specific Wikipedia query that's + currently commented out in + `src/services/enrichment/wikipedia/get_summary.cc` (`GetLocationContext`). + The `ContextStrategy` work in §3 is a natural place to reintroduce it via + `BreweryContextStrategy::QueriesFor()`. - [ ] Pre-warm caches at startup (`PreWarmBeerStyleCache`, `PreWarmLocationCache` in the planned activity diagram) instead of - fetching lazily per record, so the streaming phases never block on a - cold cache mid-run. + fetching lazily per record, so the streaming phases never block on a cold + cache mid-run. ## 8. Fixtures / Build - [x] Author `personas.json`, `forenames-by-country.json`, and `surnames-by-country.json` (see §2). -- [ ] Fix the `beer-styles.json` build-tree gap: add it to the "Runtime - Assets" `configure_file` step in `CMakeLists.txt` so native builds and - the Docker image agree on what's available at runtime. +- [ ] Fix the `beer-styles.json` build-tree gap: add it to the "Runtime Assets" + `configure_file` step in `CMakeLists.txt` so native builds and the Docker + image agree on what's available at runtime. --- @@ -240,13 +233,12 @@ Entirely new — no `includes/policy/` (or equivalent) directory exists today. The planned activity diagram's own phase notes ("brewery*pool* is now fully populated. Phase 1b may begin.", etc.) imply a dependency order. Roughly: -1. Domain models (§1) and data preloading (§2) — nothing else compiles - without these. -2. Export schema (§5) — so every later generation phase has somewhere to - land its output. +1. Domain models (§1) and data preloading (§2) — nothing else compiles without + these. +2. Export schema (§5) — so every later generation phase has somewhere to land + its output. 3. Policy/strategy layer (§3) and generator interface (§4). -4. Concurrency/orchestration rewrite (§6), which is the only piece that - actually wires §1–§5 into the phased, parallel pipeline shown in the - planned diagrams. -5. Enrichment cache pre-warming and the city/region query decision (§7) — - useful at any point, but most valuable once phases run concurrently. +4. Concurrency/orchestration rewrite (§6), which is the only piece that actually + wires §1–§5 into the phased, parallel pipeline shown in the planned diagrams. +5. Enrichment cache pre-warming and the city/region query decision (§7) — useful + at any point, but most valuable once phases run concurrently. diff --git a/tooling/pipeline/.clang-tidy b/tooling/pipeline/.clang-tidy index c180f753..a6436fe3 100644 --- a/tooling/pipeline/.clang-tidy +++ b/tooling/pipeline/.clang-tidy @@ -1,12 +1,6 @@ Checks: > - -*, - bugprone-*, - google-*, - modernize-*, - readability-*, - cppcoreguidelines-*, - -modernize-use-trailing-return-type, - -google-runtime-references + -*, bugprone-*, google-*, modernize-*, readability-*, cppcoreguidelines-*, + -modernize-use-trailing-return-type, -google-runtime-references CheckOptions: # Enforce Google Naming Conventions with valid clang-tidy strings