# Pipeline Roadmap — Reaching the Planned Architecture This is the engineering breakdown for closing the gap between the **current** pipeline and the **planned** architecture documented in [`diagrams/planned/class.puml`](./diagrams/planned/class.puml) and [`diagrams/planned/activity.puml`](./diagrams/planned/activity.puml). Nothing in `diagrams/planned/` is implemented yet — this file tracks what it would take to get there. For the current implementation, see [README.md](./README.md). Items are grouped by layer and roughly ordered by dependency: later groups build on types and services introduced earlier. The only concurrency in the current pipeline is the log dispatcher thread (`main.cc` spawns it to drain a `BoundedChannel` into spdlog for the whole run). Sampling, enrichment, generation, and export all happen synchronously on the main thread inside `BiergartenPipelineOrchestrator::Run()` — §6 below is what introduces the additional worker threads the planned architecture needs. --- ## 1. Domain Models `tooling/pipeline/includes/data_model/generated_models.h` and `models.h`. - [ ] Add `Completeness` enum (`Full`, `Partial`, `Absent`) and a `LocationContext` struct (`text`, `completeness`, `char_count`). Replace `EnrichedCity::region_context` (currently a plain `std::string`) with `context : LocationContext`. - [ ] Add `BeerStyle` (`name`, `description`, `min_abv`, `max_abv`, `min_ibu`, `max_ibu`). - [ ] Add `BeerResult`, `CheckinResult`, `RatingResult` result payloads. - [ ] Add `GenerationMetadata` (`generation_id`, `generated_time`, `context_provided`, `generated_with`). - [x] Add `first_name`, `last_name`, `gender`, and `activity_weight` to `UserResult` (currently just `username`, `bio`). `first_name`/ `last_name`/`gender` are copied from the sampled `Name` (see below), not LLM-invented. - [x] Add `Name` (`first_name`, `last_name`, `gender`) — the sampled result handed to `DataGenerator::GenerateUser`. Add `ForenameEntry` (`name`, `gender`). Landed as a flatter shape than originally planned here: no `NamesByCountry` wrapper class. `curated_data_service.h` instead declares `ForenameList = std::vector` and `SurnameList = std::vector`, and `ICuratedDataService` exposes two maps keyed by ISO 3166-1 code — `ForenamesByCountryMap = unordered_map` and `SurnamesByCountryMap = unordered_map` — directly (see §2). Sampling is a free function, `SampleName(forenames_by_country, surnames_by_country, iso3166_1, rng)`, in `generate_users.cc`'s anonymous namespace, not a method on a class. Pairing a forename with a surname happens at sample time so gender (present per-forename in the source data) is never discarded the way a pre-flattened `{first_name, last_name}` list would lose it; see §2. - [ ] Extend `GeneratedBrewery` with `brewery_id`, `context_completeness`, `metadata` (currently `{location, address, brewery}` — see the `Location`→`City` rename and `Address` struct added in the `postal_regex`/§9 entries below). - [ ] Add `GeneratedBeer`, `GeneratedCheckin`, `GeneratedRating`, `GeneratedFollow` aggregate structs. - [x] Add `UserRecord` (`location`, `user : UserResult`, `email`, `date_of_birth`) as the current stand-in for the planned `GeneratedUser` — `email` and `date_of_birth` are programmatically generated by the orchestrator, never LLM-authored, so a downstream auth-account seeding consumer can register real accounts from the pipeline's SQLite export. No `password`, `user_id`, or `metadata` field yet, matching `BreweryRecord`'s equally pre-`metadata` shape. Already wired into `IExportService`/`SqliteExportService`: a `users` table exists and `GenerateUsers()` exports each successful record via `ProcessRecord(const UserRecord&)` (see §5) in addition to logging and holding `generated_users_` in memory. Still missing vs. the planned `GeneratedUser`: `user_id`, `password`, and `metadata`. - [x] Add `UserPersona` (`name`, `description`, `style_affinities`). - [x] Add `postal_regex` (`std::vector`) and `postal_code_examples` (`std::vector`) to `Location` (now `City` — see below), sourced from each `cities.json` entry's `postal_code.city_regex` / `postal_code.examples`. Not part of the originally planned `Location` shape in `diagrams/planned/class.puml` — added ahead of postal code generation work; see §9. - [x] Renamed `Location` → `City` throughout the pipeline (struct, all parameter/return types, `LocationsList`→`CityList`, `LoadLocations()`→`LoadCities()`) so the domain vocabulary matches the web backend's `City`/`CityId`. Field/variable names that merely hold a `City` instance (e.g. `EnrichedCity::location`, `BreweryRecord::location`) were left as-is — `City` itself has a `city` field (the city name string), so renaming the container field too would read as `enriched_city.city.city`. The SQLite export layer's own vocabulary (`locations` table, `location_id` column, `location_cache_`, etc.) was renamed to `cities`/`city_id` to match, since it's this pipeline's own throwaway export, not the backend's SQL Server schema (see §5). - [x] Added a new `Address` struct (`postal_code` only, for now) and a `BreweryRecord::address` member, mirroring the web backend's `BreweryPostLocation` (`AddressLine1`, `AddressLine2`, `PostalCode`, `CityId`). Only `postal_code` is populated — `address_line1`/ `address_line2` generation has no fixture data or service yet. Users do not get an `Address`: the backend's `UserAccount` has no address fields, only `BreweryPostLocation` does. ## 2. Data Preloading `tooling/pipeline/includes/services/curated_data/`, fixture files. - [x] Extract an interface; have `CuratedJsonDataService` implement it. Landed as `ICuratedDataService` (`services/curated_data/curated_data_service.h`), not `DataPreloader` — `LoadCities()`, `LoadPersonas()`, `LoadForenamesByCountry()`, and `LoadSurnamesByCountry()` are all virtual methods returning `const&` and take no arguments. `CuratedJsonDataService` (`services/curated_data/curated_json_data_service.h`, originally landed as `JsonLoader` under `json_handling/` before being renamed and moved) takes a `CuratedDataFilePaths` DTO (`locations_path`, `personas_path`, `forenames_path`, `surnames_path`) in its constructor rather than a path per `Load*()` call, and memoizes each result in a private `cache` struct (`locations`, `personas`, `forenames_by_country`, `surnames_by_country`), so a second call on the same instance returns the cached result instead of re-parsing — safe because `CuratedJsonDataService` outlives every call site (owned by `BiergartenPipelineOrchestrator` via `unique_ptr` for the whole run). `main.cc`'s DI injector also gained a `MockCuratedDataService` (fixed in-memory data: 4 locations across `US`/`DE`/`FR`/`BE`, 3 personas, and matching forename/surname sets for those four countries), bound in place of `CuratedJsonDataService` under `--mocked`, mirroring the existing `MockEnrichmentService`/`MockGenerator` pattern. - [ ] Add `LoadBeerStyles()`. `beer-styles.json` already exists in the repo and is already copied into the Docker image (`runpod/Dockerfile`), but no loader reads it yet, and the native CMake build doesn't copy it into `build/` at all — `CMakeLists.txt`'s "Runtime Assets" step only copies `cities.json` and `prompts/`. - [x] Add `LoadPersonas()` and author `personas.json` (doesn't exist yet). - [x] Add name-by-country loading, parsing `tooling/pipeline/forenames-by-country.json` and `surnames-by-country.json`. Landed as two separate methods — `LoadForenamesByCountry()` and `LoadSurnamesByCountry()` — each returning a flat `ForenamesByCountryMap` / `SurnamesByCountryMap` (aliases for `unordered_map` / `unordered_map`) keyed by ISO 3166-1 code, rather than one combined `LoadNamesByCountry() : NamesByCountry` call. Both files are vendored verbatim (unmodified, full multinational coverage) from `sigpwned/popular-names-by-country-dataset` (CC0) — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for provenance. Deliberately not pre-paired or filtered down to just the countries in `cities.json`: keeping the source shape (including per-forename gender) intact means the loader can support more countries later, or gender-aware persona/bio generation, without re-sourcing data. - [x] Parse `postal_code.city_regex` and `postal_code.examples` out of each `cities.json` entry's nested `postal_code` object in `CuratedJsonDataService::LoadCities()`, into `City::postal_regex` / `City::postal_code_examples` (see §1). `MockCuratedDataService`'s 4 fixed locations carry matching values pulled from the corresponding `cities.json` entries. ## 3. Policy / Strategy Layer Entirely new — no `includes/policy/` (or equivalent) directory exists today. - [ ] `ContextStrategy` interface + `BreweryContextStrategy` / `BeerContextStrategy`. Today `WikipediaEnrichmentService::GetLocationContext` hardcodes a generic `"brewing"` query and a `"beer in {country}"` query directly — no per-phase strategy selection. - [ ] `SamplingStrategy` interface + `UniformSamplingStrategy`, replacing the inline `std::ranges::sample(...)` call in `BiergartenPipelineOrchestrator::QueryCitiesWithCountries()`. - [ ] `BeerSelectionStrategy` interface + `RandomBeerSelectionStrategy`, to pick styles per brewery from the `BeerStyle` palette (depends on §1 and §2). - [ ] `CheckinDistributionStrategy` interface + `JCurveCheckinStrategy` / `RandomCheckinStrategy` — this is the "Check-In System" item already called out in README.md's Next Steps, made concrete. - [ ] `FollowGenerationStrategy` interface + `RandomFollowStrategy` / `ActivityWeightedFollowStrategy`. ## 4. Data Generation `tooling/pipeline/includes/data_generation/`, `src/data_generation/`. - [ ] Extend the `DataGenerator` interface with `GenerateBeer`, `GenerateCheckin`, `GenerateRating` (today: only `GenerateBrewery` and `GenerateUser`). - [x] Implement `LlamaGenerator::GenerateUser` for real, with a retry loop mirroring `GenerateBrewery` (GBNF grammar, `ValidateUserJson`, up to 3 attempts with corrective feedback) and a new `prompts/USER_GENERATION.md`. Changed the `DataGenerator::GenerateUser` signature to `(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult`, matching what the activity diagram actually passes. `MockGenerator::GenerateUser` updated to match the new signature too. - [ ] Implement `MockGenerator::GenerateBeer` / `GenerateCheckin` / `GenerateRating`. - [ ] Add `IPromptFormatter::ExpectedArchitecture()` and `LlamaGenerator::ValidateModelArchitecture()` so loading a GGUF that doesn't match the configured chat template fails fast instead of silently producing degraded output. - [ ] Add prompt template files for beer/checkin/rating generation next to the existing `prompts/BREWERY_GENERATION.md`. ## 5. Export Service `tooling/pipeline/includes/services/database/`, `src/services/sqlite/`. - [x] `IExportService::ProcessRecord(const UserRecord&)` and `SqliteExportService`'s `users` table (see `kCreateUsersTableSql` / `kInsertUserSql` in `includes/services/database/sqlite_statement_helpers.h`) are implemented — `GenerateUsers()` exports every successful user alongside its resolved `city_id`. - [x] `ProcessRecord(const BreweryRecord&)` now also binds `BreweryRecord::address.postal_code` into a `postal_code` column on `breweries` (see §9). - [ ] Extend `IExportService` with `ProcessBeer`, `ProcessCheckin`, `ProcessRating`, `ProcessFollow` (today `ProcessRecord(const BreweryRecord&)` and `ProcessRecord(const UserRecord&)` exist). - [ ] Add `beers`, `checkins`, `ratings`, `follows` tables to `SqliteExportService::InitializeSchema()` (`cities`, `breweries`, and `users` already exist) — see `kCreateCitiesTableSql` / `kCreateBreweriesTableSql` / `kCreateUsersTableSql` in `includes/services/database/sqlite_statement_helpers.h` for the existing pattern to follow. - [ ] Add a `brewery_cache_` alongside the existing `city_cache_`, and move from one open transaction for the whole run to per-phase batched commits (`BEGIN` / `COMMIT & BEGIN` on a batch-size threshold), as shown in the planned activity diagram. ## 6. Concurrency & Orchestration `tooling/pipeline/includes/concurrency/`, `src/biergarten_pipeline_orchestrator/`. - [ ] `BoundedChannel` already exists and is production-tested — but it's only wired to the single log channel today. Stand up the per-phase producer/consumer channels (`loc_ch`, `exp_ch`, etc.) the planned activity diagram describes, each with a dedicated LLM worker thread and SQLite worker thread. - [ ] Rewrite `BiergartenPipelineOrchestrator::Run()` from one synchronous pass over `generated_breweries_` into phased methods — `RunUserPhase` → `RunBreweryAndBeerPhase` (with a `RunBeerPhase` sub-step once `brewery_pool_` is populated) → `RunCheckinPhase` / `RunFollowPhase` (these two can run in parallel) → `RunRatingPhase` — backed by `user_pool_`, `brewery_pool_`, `beer_pool_`, `checkin_pool_`, `follow_pool_`. - [ ] Honor the structural-concurrency requirement already called out in a comment on `BiergartenPipelineOrchestrator::Run()` in `biergarten_pipeline_orchestrator.h`: once real worker threads exist, they must be structurally joined (e.g. via `std::jthread`) before `Run()` returns, so no worker logs to a closed channel during teardown. ## 7. Enrichment - [ ] Decide whether to restore the city/region-specific Wikipedia query that's currently commented out in `src/services/enrichment/wikipedia/get_summary.cc` (`GetLocationContext`). The `ContextStrategy` work in §3 is a natural place to reintroduce it via `BreweryContextStrategy::QueriesFor()`. - [ ] Pre-warm caches at startup (`PreWarmBeerStyleCache`, `PreWarmLocationCache` in the planned activity diagram) instead of fetching lazily per record, so the streaming phases never block on a cold cache mid-run. ## 8. Fixtures / Build - [x] Author `personas.json`, `forenames-by-country.json`, and `surnames-by-country.json` (see §2). - [ ] Fix the `beer-styles.json` build-tree gap: add it to the "Runtime Assets" `configure_file` step in `CMakeLists.txt` so native builds and the Docker image agree on what's available at runtime. ## 9. Postal Code Generation `tooling/pipeline/includes/services/postal_code/`. Not part of the original `diagrams/planned/class.puml` sketch — added once `Location` (now `City`) gained `postal_regex`/`postal_code_examples` (§1, §2). - [x] Add the `IPostalCodeService` interface — `GeneratePostalCode(location : const City&) : std::string`. - [x] Add `MockPostalCodeService`: ignores `location.postal_regex` entirely and returns `location.postal_code_examples.front()`. - [ ] **Implement a real, xeger-based `IPostalCodeService`.** A xeger generator turns a regex pattern (here, one of `City::postal_regex`) into a random string that matches it — the inverse of matching a string against a regex. Without it, every generated postal code is either a hardcoded mock value or absent entirely; a real implementation is required before postal codes can vary per generated brewery instead of always reusing the same curated example. - [x] Wire `IPostalCodeService` into `main.cc`'s Boost.DI injector — bound unconditionally to `MockPostalCodeService` (no `--mocked` branch yet, since a real implementation doesn't exist to switch to; add one once the xeger-based generator above lands, mirroring `IEnrichmentService`) — and into `BiergartenPipelineOrchestrator::GenerateBreweries()`, which calls `GeneratePostalCode(location)` per brewery and stores the result on `BreweryRecord::address` (a new `Address` struct — see §1), exported to the `breweries.postal_code` SQLite column (§5). Postal-code generation is per-brewery, not per-city or per-user: users don't have an address concept in the web backend, only breweries do (`BreweryPostLocation`), so `UserRecord` was deliberately not extended. --- ## Suggested build order The planned activity diagram's own phase notes ("brewery*pool* is now fully populated. Phase 1b may begin.", etc.) imply a dependency order. Roughly: 1. Domain models (§1) and data preloading (§2) — nothing else compiles without these. 2. Export schema (§5) — so every later generation phase has somewhere to land its output. 3. Policy/strategy layer (§3) and generator interface (§4). 4. Concurrency/orchestration rewrite (§6), which is the only piece that actually wires §1–§5 into the phased, parallel pipeline shown in the planned diagrams. 5. Enrichment cache pre-warming and the city/region query decision (§7) — useful at any point, but most valuable once phases run concurrently. 6. Postal code generation (§9) is independent of the phases above — the xeger-based generator just needs `City::postal_regex`, already in place. Wiring the mock into brewery output has already landed (§9); the real generator can be swapped in once built.