the-biergarten-app/docs/pipeline/ROADMAP.md

# Pipeline Roadmap — Reaching the Planned Architecture

This is the engineering breakdown for closing the gap between the **current**
pipeline and the **planned** architecture documented in
[`diagrams/planned/class.puml`](./diagrams/planned/class.puml) and
[`diagrams/planned/activity.puml`](./diagrams/planned/activity.puml). Nothing in
`diagrams/planned/` is implemented yet — this file tracks what it would take to
get there. For the current implementation, see [README.md](./README.md).

Items are grouped by layer and roughly ordered by dependency: later groups build
on types and services introduced earlier.

The only concurrency in the current pipeline is the log dispatcher thread
(`main.cc` spawns it to drain a `BoundedChannel<LogEntry>` into spdlog for the
whole run). Sampling, enrichment, generation, and export all happen
synchronously on the main thread inside `BiergartenPipelineOrchestrator::Run()`
— §6 below is what introduces the additional worker threads the planned
architecture needs.

---

## 1. Domain Models

`tooling/pipeline/includes/data_model/generated_models.h` and `models.h`.

- [ ] Add `Completeness` enum (`Full`, `Partial`, `Absent`) and a
      `LocationContext` struct (`text`, `completeness`, `char_count`). Replace
      `EnrichedCity::region_context` (currently a plain `std::string`) with
      `context : LocationContext`.
- [ ] Add `BeerStyle` (`name`, `description`, `min_abv`, `max_abv`, `min_ibu`,
      `max_ibu`).
- [ ] Add `BeerResult`, `CheckinResult`, `RatingResult` result payloads.
- [ ] Add `GenerationMetadata` (`generation_id`, `generated_time`,
      `context_provided`, `generated_with`).
- [x] Add `first_name`, `last_name`, `gender`, and `activity_weight` to
      `UserResult` (currently just `username`, `bio`). `first_name`/
      `last_name`/`gender` are copied from the sampled `Name` (see below), not
      LLM-invented.
- [x] Add `Name` (`first_name`, `last_name`, `gender`) — the sampled result
      handed to `DataGenerator::GenerateUser`. Add `ForenameEntry` (`name`,
      `gender`). Landed as a flatter shape than originally planned here: no
      `NamesByCountry` wrapper class. `curated_data_service.h` instead declares
      `ForenameList = std::vector<ForenameEntry>` and
      `SurnameList = std::vector<std::string>`, and `ICuratedDataService`
      exposes two maps keyed by ISO 3166-1 code —
      `ForenamesByCountryMap = unordered_map<string, ForenameList>` and
      `SurnamesByCountryMap = unordered_map<string, SurnameList>` — directly
      (see §2). Sampling is a free function,
      `SampleName(forenames_by_country,     surnames_by_country, iso3166_1, rng)`,
      in `generate_users.cc`'s anonymous namespace, not a method on a class.
      Pairing a forename with a surname happens at sample time so gender
      (present per-forename in the source data) is never discarded the way a
      pre-flattened `{first_name, last_name}` list would lose it; see §2.
- [ ] Extend `GeneratedBrewery` with `brewery_id`, `context_completeness`,
      `metadata` (currently just `{location, brewery}`).
- [ ] Add `GeneratedBeer`, `GeneratedCheckin`, `GeneratedRating`,
      `GeneratedFollow` aggregate structs.
- [x] Add `UserRecord` (`location`, `user : UserResult`, `email`,
      `date_of_birth`) as the current stand-in for the planned `GeneratedUser` —
      `email` and `date_of_birth` are programmatically generated by the
      orchestrator, never LLM-authored, so a downstream auth-account seeding
      consumer can register real accounts from the pipeline's SQLite export. No
      `password`, `user_id`, or `metadata` field yet, matching `BreweryRecord`'s
      equally pre-`metadata` shape. Already wired into
      `IExportService`/`SqliteExportService`: a `users` table exists and
      `GenerateUsers()` exports each successful record via
      `ProcessRecord(const UserRecord&)` (see §5) in addition to logging and
      holding `generated_users_` in memory. Still missing vs. the planned
      `GeneratedUser`: `user_id`, `password`, and `metadata`.
- [x] Add `UserPersona` (`name`, `description`, `style_affinities`).

## 2. Data Preloading

`tooling/pipeline/includes/services/curated_data/`, fixture files.

- [x] Extract an interface; have `CuratedJsonDataService` implement it. Landed
      as `ICuratedDataService` (`services/curated_data/curated_data_service.h`),
      not `DataPreloader` — `LoadLocations()`, `LoadPersonas()`,
      `LoadForenamesByCountry()`, and `LoadSurnamesByCountry()` are all virtual
      methods returning `const&` and take no arguments. `CuratedJsonDataService`
      (`services/curated_data/curated_json_data_service.h`, originally landed as
      `JsonLoader` under `json_handling/` before being renamed and moved) takes
      a `CuratedDataFilePaths` DTO (`locations_path`, `personas_path`,
      `forenames_path`, `surnames_path`) in its constructor rather than a path
      per `Load*()` call, and memoizes each result in a private `cache` struct
      (`locations`, `personas`, `forenames_by_country`, `surnames_by_country`),
      so a second call on the same instance returns the cached result instead of
      re-parsing — safe because `CuratedJsonDataService` outlives every call
      site (owned by `BiergartenPipelineOrchestrator` via
      `unique_ptr<ICuratedDataService>` for the whole run). `main.cc`'s DI
      injector also gained a `MockCuratedDataService` (fixed in-memory data: 4
      locations across `US`/`DE`/`FR`/`BE`, 3 personas, and matching
      forename/surname sets for those four countries), bound in place of
      `CuratedJsonDataService` under `--mocked`, mirroring the existing
      `MockEnrichmentService`/`MockGenerator` pattern.
- [ ] Add `LoadBeerStyles()`. `beer-styles.json` already exists in the repo and
      is already copied into the Docker image (`runpod/Dockerfile`), but no
      loader reads it yet, and the native CMake build doesn't copy it into
      `build/` at all — `CMakeLists.txt`'s "Runtime Assets" step only copies
      `locations.json` and `prompts/`.
- [x] Add `LoadPersonas()` and author `personas.json` (doesn't exist yet).
- [x] Add name-by-country loading, parsing
      `tooling/pipeline/forenames-by-country.json` and
      `surnames-by-country.json`. Landed as two separate methods —
      `LoadForenamesByCountry()` and `LoadSurnamesByCountry()` — each returning
      a flat `ForenamesByCountryMap` / `SurnamesByCountryMap` (aliases for
      `unordered_map<string, ForenameList>` /
      `unordered_map<string, SurnameList>`) keyed by ISO 3166-1 code, rather
      than one combined `LoadNamesByCountry() : NamesByCountry` call. Both files
      are vendored verbatim (unmodified, full multinational coverage) from
      `sigpwned/popular-names-by-country-dataset` (CC0) — see
      ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for
      provenance. Deliberately not pre-paired or filtered down to just the
      countries in `locations.json`: keeping the source shape (including
      per-forename gender) intact means the loader can support more countries
      later, or gender-aware persona/bio generation, without re-sourcing data.

## 3. Policy / Strategy Layer

Entirely new — no `includes/policy/` (or equivalent) directory exists today.

- [ ] `ContextStrategy` interface + `BreweryContextStrategy` /
      `BeerContextStrategy`. Today
      `WikipediaEnrichmentService::GetLocationContext` hardcodes a generic
      `"brewing"` query and a `"beer in {country}"` query directly — no
      per-phase strategy selection.
- [ ] `SamplingStrategy` interface + `UniformSamplingStrategy`, replacing the
      inline `std::ranges::sample(...)` call in
      `BiergartenPipelineOrchestrator::QueryCitiesWithCountries()`.
- [ ] `BeerSelectionStrategy` interface + `RandomBeerSelectionStrategy`, to pick
      styles per brewery from the `BeerStyle` palette (depends on §1 and §2).
- [ ] `CheckinDistributionStrategy` interface + `JCurveCheckinStrategy` /
      `RandomCheckinStrategy` — this is the "Check-In System" item already
      called out in README.md's Next Steps, made concrete.
- [ ] `FollowGenerationStrategy` interface + `RandomFollowStrategy` /
      `ActivityWeightedFollowStrategy`.

## 4. Data Generation

`tooling/pipeline/includes/data_generation/`, `src/data_generation/`.

- [ ] Extend the `DataGenerator` interface with `GenerateBeer`,
      `GenerateCheckin`, `GenerateRating` (today: only `GenerateBrewery` and
      `GenerateUser`).
- [x] Implement `LlamaGenerator::GenerateUser` for real, with a retry loop
      mirroring `GenerateBrewery` (GBNF grammar, `ValidateUserJson`, up to 3
      attempts with corrective feedback) and a new `prompts/USER_GENERATION.md`.
      Changed the `DataGenerator::GenerateUser` signature to
      `(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult`,
      matching what the activity diagram actually passes.
      `MockGenerator::GenerateUser` updated to match the new signature too.
- [ ] Implement `MockGenerator::GenerateBeer` / `GenerateCheckin` /
      `GenerateRating`.
- [ ] Add `IPromptFormatter::ExpectedArchitecture()` and
      `LlamaGenerator::ValidateModelArchitecture()` so loading a GGUF that
      doesn't match the configured chat template fails fast instead of silently
      producing degraded output.
- [ ] Add prompt template files for beer/checkin/rating generation next to the
      existing `prompts/BREWERY_GENERATION.md`.

## 5. Export Service

`tooling/pipeline/includes/services/database/`, `src/services/sqlite/`.

- [x] `IExportService::ProcessRecord(const UserRecord&)` and
      `SqliteExportService`'s `users` table (see `kCreateUsersTableSql` /
      `kInsertUserSql` in
      `includes/services/database/sqlite_statement_helpers.h`) are implemented —
      `GenerateUsers()` exports every successful user alongside its resolved
      `location_id`.
- [ ] Extend `IExportService` with `ProcessBeer`, `ProcessCheckin`,
      `ProcessRating`, `ProcessFollow` (today
      `ProcessRecord(const     BreweryRecord&)` and
      `ProcessRecord(const UserRecord&)` exist).
- [ ] Add `beers`, `checkins`, `ratings`, `follows` tables to
      `SqliteExportService::InitializeSchema()` (`locations`, `breweries`, and
      `users` already exist) — see `kCreateLocationsTableSql` /
      `kCreateBreweriesTableSql` / `kCreateUsersTableSql` in
      `includes/services/database/sqlite_statement_helpers.h` for the existing
      pattern to follow.
- [ ] Add a `brewery_cache_` alongside the existing `location_cache_`, and move
      from one open transaction for the whole run to per-phase batched commits
      (`BEGIN` / `COMMIT & BEGIN` on a batch-size threshold), as shown in the
      planned activity diagram.

## 6. Concurrency & Orchestration

`tooling/pipeline/includes/concurrency/`,
`src/biergarten_pipeline_orchestrator/`.

- [ ] `BoundedChannel<T>` already exists and is production-tested — but it's
      only wired to the single log channel today. Stand up the per-phase
      producer/consumer channels (`loc_ch`, `exp_ch`, etc.) the planned activity
      diagram describes, each with a dedicated LLM worker thread and SQLite
      worker thread.
- [ ] Rewrite `BiergartenPipelineOrchestrator::Run()` from one synchronous pass
      over `generated_breweries_` into phased methods — `RunUserPhase` →
      `RunBreweryAndBeerPhase` (with a `RunBeerPhase` sub-step once
      `brewery_pool_` is populated) → `RunCheckinPhase` / `RunFollowPhase`
      (these two can run in parallel) → `RunRatingPhase` — backed by
      `user_pool_`, `brewery_pool_`, `beer_pool_`, `checkin_pool_`,
      `follow_pool_`.
- [ ] Honor the structural-concurrency requirement already called out in a
      comment on `BiergartenPipelineOrchestrator::Run()` in
      `biergarten_pipeline_orchestrator.h`: once real worker threads exist, they
      must be structurally joined (e.g. via `std::jthread`) before `Run()`
      returns, so no worker logs to a closed channel during teardown.

## 7. Enrichment

- [ ] Decide whether to restore the city/region-specific Wikipedia query that's
      currently commented out in
      `src/services/enrichment/wikipedia/get_summary.cc` (`GetLocationContext`).
      The `ContextStrategy` work in §3 is a natural place to reintroduce it via
      `BreweryContextStrategy::QueriesFor()`.
- [ ] Pre-warm caches at startup (`PreWarmBeerStyleCache`,
      `PreWarmLocationCache` in the planned activity diagram) instead of
      fetching lazily per record, so the streaming phases never block on a cold
      cache mid-run.

## 8. Fixtures / Build

- [x] Author `personas.json`, `forenames-by-country.json`, and
      `surnames-by-country.json` (see §2).
- [ ] Fix the `beer-styles.json` build-tree gap: add it to the "Runtime Assets"
      `configure_file` step in `CMakeLists.txt` so native builds and the Docker
      image agree on what's available at runtime.

---

## Suggested build order

The planned activity diagram's own phase notes ("brewery*pool* is now fully
populated. Phase 1b may begin.", etc.) imply a dependency order. Roughly:

1. Domain models (§1) and data preloading (§2) — nothing else compiles without
   these.
2. Export schema (§5) — so every later generation phase has somewhere to land
   its output.
3. Policy/strategy layer (§3) and generator interface (§4).
4. Concurrency/orchestration rewrite (§6), which is the only piece that actually
   wires §1–§5 into the phased, parallel pipeline shown in the planned diagrams.
5. Enrichment cache pre-warming and the city/region query decision (§7) — useful
   at any point, but most valuable once phases run concurrently.