the-biergarten-app/docs/pipeline/ROADMAP.md

Pipeline Roadmap — Reaching the Planned Architecture

This is the engineering breakdown for closing the gap between the current pipeline and the planned architecture documented in diagrams/planned/class.puml and diagrams/planned/activity.puml. Nothing in diagrams/planned/ is implemented yet — this file tracks what it would take to get there. For the current implementation, see README.md.

Items are grouped by layer and roughly ordered by dependency: later groups build on types and services introduced earlier.

The only concurrency in the current pipeline is the log dispatcher thread (main.cc spawns it to drain a BoundedChannel<LogEntry> into spdlog for the whole run). Sampling, enrichment, generation, and export all happen synchronously on the main thread inside BiergartenPipelineOrchestrator::Run() — §6 below is what introduces the additional worker threads the planned architecture needs.

---

1. Domain Models

tooling/pipeline/includes/data_model/generated_models.h and models.h.

• Add Completeness enum (Full, Partial, Absent) and a LocationContext struct (text, completeness, char_count). Replace EnrichedCity::region_context (currently a plain std::string) with context : LocationContext.
• Add BeerStyle (name, description, min_abv, max_abv, min_ibu, max_ibu).
• Add BeerResult, CheckinResult, RatingResult result payloads.
• Add GenerationMetadata (generation_id, generated_time, context_provided, generated_with).
• Add first_name, last_name, gender, and activity_weight to UserResult (currently just username, bio). first_name/ last_name/gender are copied from the sampled Name (see below), not LLM-invented.
• Add Name (first_name, last_name, gender) — the sampled result handed to DataGenerator::GenerateUser. Add ForenameEntry (name, gender). Landed as a flatter shape than originally planned here: no NamesByCountry wrapper class. curated_data_service.h instead declares ForenameList = std::vector<ForenameEntry> and SurnameList = std::vector<std::string>, and ICuratedDataService exposes two maps keyed by ISO 3166-1 code — ForenamesByCountryMap = unordered_map<string, ForenameList> and SurnamesByCountryMap = unordered_map<string, SurnameList> — directly (see §2). Sampling is a free function, SampleName(forenames_by_country, surnames_by_country, iso3166_1, rng), in generate_users.cc's anonymous namespace, not a method on a class. Pairing a forename with a surname happens at sample time so gender (present per-forename in the source data) is never discarded the way a pre-flattened {first_name, last_name} list would lose it; see §2.
• Extend GeneratedBrewery with brewery_id, context_completeness, metadata (currently {location, address, brewery} — see the Location→City rename and Address struct added in the postal_regex/§9 entries below).
• Add GeneratedBeer, GeneratedCheckin, GeneratedRating, GeneratedFollow aggregate structs.
• Add UserRecord (location, user : UserResult, email, date_of_birth) as the current stand-in for the planned GeneratedUser — email and date_of_birth are programmatically generated by the orchestrator, never LLM-authored, so a downstream auth-account seeding consumer can register real accounts from the pipeline's SQLite export. No password, user_id, or metadata field yet, matching BreweryRecord's equally pre-metadata shape. Already wired into IExportService/SqliteExportService: a users table exists and GenerateUsers() exports each successful record via ProcessRecord(const UserRecord&) (see §5) in addition to logging and holding generated_users_ in memory. Still missing vs. the planned GeneratedUser: user_id, password, and metadata.
• Add UserPersona (name, description, style_affinities).
• Add postal_regex (std::vector<std::string>) and postal_code_examples (std::vector<std::string>) to Location (now City — see below), sourced from each cities.json entry's postal_code.city_regex / postal_code.examples. Not part of the originally planned Location shape in diagrams/planned/class.puml — added ahead of postal code generation work; see §9.
• Renamed Location → City throughout the pipeline (struct, all parameter/return types, LocationsList→CityList, LoadLocations()→LoadCities()) so the domain vocabulary matches the web backend's City/CityId. Field/variable names that merely hold a City instance (e.g. EnrichedCity::location, BreweryRecord::location) were left as-is — City itself has a city field (the city name string), so renaming the container field too would read as enriched_city.city.city. The SQLite export layer's own vocabulary (locations table, location_id column, location_cache_, etc.) was renamed to cities/city_id to match, since it's this pipeline's own throwaway export, not the backend's SQL Server schema (see §5).
• Added a new Address struct (postal_code only, for now) and a BreweryRecord::address member, mirroring the web backend's BreweryPostLocation (AddressLine1, AddressLine2, PostalCode, CityId). Only postal_code is populated — address_line1/ address_line2 generation has no fixture data or service yet. Users do not get an Address: the backend's UserAccount has no address fields, only BreweryPostLocation does.

2. Data Preloading

tooling/pipeline/includes/services/curated_data/, fixture files.

• Extract an interface; have CuratedJsonDataService implement it. Landed as ICuratedDataService (services/curated_data/curated_data_service.h), not DataPreloader — LoadCities(), LoadPersonas(), LoadForenamesByCountry(), and LoadSurnamesByCountry() are all virtual methods returning const& and take no arguments. CuratedJsonDataService (services/curated_data/curated_json_data_service.h, originally landed as JsonLoader under json_handling/ before being renamed and moved) takes a CuratedDataFilePaths DTO (locations_path, personas_path, forenames_path, surnames_path) in its constructor rather than a path per Load*() call, and memoizes each result in a private cache struct (locations, personas, forenames_by_country, surnames_by_country), so a second call on the same instance returns the cached result instead of re-parsing — safe because CuratedJsonDataService outlives every call site (owned by BiergartenPipelineOrchestrator via unique_ptr<ICuratedDataService> for the whole run). main.cc's DI injector also gained a MockCuratedDataService (fixed in-memory data: 4 locations across US/DE/FR/BE, 3 personas, and matching forename/surname sets for those four countries), bound in place of CuratedJsonDataService under --mocked, mirroring the existing MockEnrichmentService/MockGenerator pattern.
• Add LoadBeerStyles(). beer-styles.json already exists in the repo and is already copied into the Docker image (runpod/Dockerfile), but no loader reads it yet, and the native CMake build doesn't copy it into build/ at all — CMakeLists.txt's "Runtime Assets" step only copies cities.json and prompts/.
• Add LoadPersonas() and author personas.json (doesn't exist yet).
• Add name-by-country loading, parsing tooling/pipeline/forenames-by-country.json and surnames-by-country.json. Landed as two separate methods — LoadForenamesByCountry() and LoadSurnamesByCountry() — each returning a flat ForenamesByCountryMap / SurnamesByCountryMap (aliases for unordered_map<string, ForenameList> / unordered_map<string, SurnameList>) keyed by ISO 3166-1 code, rather than one combined LoadNamesByCountry() : NamesByCountry call. Both files are vendored verbatim (unmodified, full multinational coverage) from sigpwned/popular-names-by-country-dataset (CC0) — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for provenance. Deliberately not pre-paired or filtered down to just the countries in cities.json: keeping the source shape (including per-forename gender) intact means the loader can support more countries later, or gender-aware persona/bio generation, without re-sourcing data.
• Parse postal_code.city_regex and postal_code.examples out of each cities.json entry's nested postal_code object in CuratedJsonDataService::LoadCities(), into City::postal_regex / City::postal_code_examples (see §1). MockCuratedDataService's 4 fixed locations carry matching values pulled from the corresponding cities.json entries.

3. Policy / Strategy Layer

Entirely new — no includes/policy/ (or equivalent) directory exists today.

• ContextStrategy interface + BreweryContextStrategy / BeerContextStrategy. Today WikipediaEnrichmentService::GetLocationContext hardcodes a generic "brewing" query and a "beer in {country}" query directly — no per-phase strategy selection.
• SamplingStrategy interface + UniformSamplingStrategy, replacing the inline std::ranges::sample(...) call in BiergartenPipelineOrchestrator::QueryCitiesWithCountries().
• BeerSelectionStrategy interface + RandomBeerSelectionStrategy, to pick styles per brewery from the BeerStyle palette (depends on §1 and §2).
• CheckinDistributionStrategy interface + JCurveCheckinStrategy / RandomCheckinStrategy — this is the "Check-In System" item already called out in README.md's Next Steps, made concrete.
• FollowGenerationStrategy interface + RandomFollowStrategy / ActivityWeightedFollowStrategy.

4. Data Generation

tooling/pipeline/includes/data_generation/, src/data_generation/.

• Extend the DataGenerator interface with GenerateBeer, GenerateCheckin, GenerateRating (today: only GenerateBrewery and GenerateUser).
• Implement LlamaGenerator::GenerateUser for real, with a retry loop mirroring GenerateBrewery (GBNF grammar, ValidateUserJson, up to 3 attempts with corrective feedback) and a new prompts/USER_GENERATION.md. Changed the DataGenerator::GenerateUser signature to (city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult, matching what the activity diagram actually passes. MockGenerator::GenerateUser updated to match the new signature too.
• Implement MockGenerator::GenerateBeer / GenerateCheckin / GenerateRating.
• Add IPromptFormatter::ExpectedArchitecture() and LlamaGenerator::ValidateModelArchitecture() so loading a GGUF that doesn't match the configured chat template fails fast instead of silently producing degraded output.
• Add prompt template files for beer/checkin/rating generation next to the existing prompts/BREWERY_GENERATION.md.

5. Export Service

tooling/pipeline/includes/services/database/, src/services/sqlite/.

• IExportService::ProcessRecord(const UserRecord&) and SqliteExportService's users table (see kCreateUsersTableSql / kInsertUserSql in includes/services/database/sqlite_statement_helpers.h) are implemented — GenerateUsers() exports every successful user alongside its resolved city_id.
• ProcessRecord(const BreweryRecord&) now also binds BreweryRecord::address.postal_code into a postal_code column on breweries (see §9).
• Extend IExportService with ProcessBeer, ProcessCheckin, ProcessRating, ProcessFollow (today ProcessRecord(const BreweryRecord&) and ProcessRecord(const UserRecord&) exist).
• Add beers, checkins, ratings, follows tables to SqliteExportService::InitializeSchema() (cities, breweries, and users already exist) — see kCreateCitiesTableSql / kCreateBreweriesTableSql / kCreateUsersTableSql in includes/services/database/sqlite_statement_helpers.h for the existing pattern to follow.
• Add a brewery_cache_ alongside the existing city_cache_, and move from one open transaction for the whole run to per-phase batched commits (BEGIN / COMMIT & BEGIN on a batch-size threshold), as shown in the planned activity diagram.

6. Concurrency & Orchestration

tooling/pipeline/includes/concurrency/, src/biergarten_pipeline_orchestrator/.

• BoundedChannel<T> already exists and is production-tested — but it's only wired to the single log channel today. Stand up the per-phase producer/consumer channels (loc_ch, exp_ch, etc.) the planned activity diagram describes, each with a dedicated LLM worker thread and SQLite worker thread.
• Rewrite BiergartenPipelineOrchestrator::Run() from one synchronous pass over generated_breweries_ into phased methods — RunUserPhase → RunBreweryAndBeerPhase (with a RunBeerPhase sub-step once brewery_pool_ is populated) → RunCheckinPhase / RunFollowPhase (these two can run in parallel) → RunRatingPhase — backed by user_pool_, brewery_pool_, beer_pool_, checkin_pool_, follow_pool_.
• Honor the structural-concurrency requirement already called out in a comment on BiergartenPipelineOrchestrator::Run() in biergarten_pipeline_orchestrator.h: once real worker threads exist, they must be structurally joined (e.g. via std::jthread) before Run() returns, so no worker logs to a closed channel during teardown.

7. Enrichment

• Decide whether to restore the city/region-specific Wikipedia query that's currently commented out in src/services/enrichment/wikipedia/get_summary.cc (GetLocationContext). The ContextStrategy work in §3 is a natural place to reintroduce it via BreweryContextStrategy::QueriesFor().
• Pre-warm caches at startup (PreWarmBeerStyleCache, PreWarmLocationCache in the planned activity diagram) instead of fetching lazily per record, so the streaming phases never block on a cold cache mid-run.

8. Fixtures / Build

• Author personas.json, forenames-by-country.json, and surnames-by-country.json (see §2).
• Fix the beer-styles.json build-tree gap: add it to the "Runtime Assets" configure_file step in CMakeLists.txt so native builds and the Docker image agree on what's available at runtime.

9. Postal Code Generation

tooling/pipeline/includes/services/postal_code/. Not part of the original diagrams/planned/class.puml sketch — added once Location (now City) gained postal_regex/postal_code_examples (§1, §2).

• Add the IPostalCodeService interface — GeneratePostalCode(location : const City&) : std::string.
• Add MockPostalCodeService: ignores location.postal_regex entirely and returns location.postal_code_examples.front().
• Implement a real, xeger-based IPostalCodeService. A xeger generator turns a regex pattern (here, one of City::postal_regex) into a random string that matches it — the inverse of matching a string against a regex. Without it, every generated postal code is either a hardcoded mock value or absent entirely; a real implementation is required before postal codes can vary per generated brewery instead of always reusing the same curated example.
• Wire IPostalCodeService into main.cc's Boost.DI injector — bound unconditionally to MockPostalCodeService (no --mocked branch yet, since a real implementation doesn't exist to switch to; add one once the xeger-based generator above lands, mirroring IEnrichmentService) — and into BiergartenPipelineOrchestrator::GenerateBreweries(), which calls GeneratePostalCode(location) per brewery and stores the result on BreweryRecord::address (a new Address struct — see §1), exported to the breweries.postal_code SQLite column (§5). Postal-code generation is per-brewery, not per-city or per-user: users don't have an address concept in the web backend, only breweries do (BreweryPostLocation), so UserRecord was deliberately not extended.

---

Suggested build order

The planned activity diagram's own phase notes ("brewerypool is now fully populated. Phase 1b may begin.", etc.) imply a dependency order. Roughly:

1. Domain models (§1) and data preloading (§2) — nothing else compiles without these.
2. Export schema (§5) — so every later generation phase has somewhere to land its output.
3. Policy/strategy layer (§3) and generator interface (§4).
4. Concurrency/orchestration rewrite (§6), which is the only piece that actually wires §1–§5 into the phased, parallel pipeline shown in the planned diagrams.
5. Enrichment cache pre-warming and the city/region query decision (§7) — useful at any point, but most valuable once phases run concurrently.
6. Postal code generation (§9) is independent of the phases above — the xeger-based generator just needs City::postal_regex, already in place. Wiring the mock into brewery output has already landed (§9); the real generator can be swapped in once built.