Pipeline: Add LLM and mocked user generation to the pipeline (#230)

2026-07-16 09:37:23 +00:00 · 2026-07-01 07:47:24 -04:00
parent 880f73e004
commit 2b8a900d12
90 changed files with 57254 additions and 800 deletions
--- a/docs/pipeline/ROADMAP.md
+++ b/docs/pipeline/ROADMAP.md
@@ -33,28 +33,93 @@ architecture needs.
 - [ ] Add `BeerResult`, `CheckinResult`, `RatingResult` result payloads.
 - [ ] Add `GenerationMetadata` (`generation_id`, `generated_time`,
      `context_provided`, `generated_with`).
- [ ] Add `activity_weight` to `UserResult` (currently just `username`,
-      `bio`).
+- [x] Add `first_name`, `last_name`, `gender`, and `activity_weight` to
+      `UserResult` (currently just `username`, `bio`). `first_name`/
+      `last_name`/`gender` are copied from the sampled `Name` (see below),
+      not LLM-invented.
+- [x] Add `Name` (`first_name`, `last_name`, `gender`) — the sampled result
+      handed to `DataGenerator::GenerateUser`. Add `ForenameEntry` (`name`,
+      `gender`). Landed as a flatter shape than originally planned here: no
+      `NamesByCountry` wrapper class. `curated_data_service.h` instead
+      declares `ForenameList = std::vector<ForenameEntry>` and
+      `SurnameList = std::vector<std::string>`, and
+      `ICuratedDataService` exposes two maps keyed by ISO 3166-1 code —
+      `ForenamesByCountryMap = unordered_map<string, ForenameList>` and
+      `SurnamesByCountryMap = unordered_map<string, SurnameList>` —
+      directly (see §2). Sampling is
+      a free function, `SampleName(forenames_by_country,
+      surnames_by_country, iso3166_1, rng)`, in
+      `generate_users.cc`'s anonymous namespace, not a method on a class.
+      Pairing a forename with a surname happens at sample time so gender
+      (present per-forename in the source data) is never discarded the way a
+      pre-flattened `{first_name, last_name}` list would lose it; see §2.
 - [ ] Extend `GeneratedBrewery` with `brewery_id`, `context_completeness`,
      `metadata` (currently just `{location, brewery}`).
- [ ] Add `GeneratedBeer`, `GeneratedUser`, `GeneratedCheckin`,
-      `GeneratedRating`, `GeneratedFollow` aggregate structs.
- [ ] Add `UserPersona` (`name`, `description`, `style_affinities`).
+- [ ] Add `GeneratedBeer`, `GeneratedCheckin`, `GeneratedRating`,
+      `GeneratedFollow` aggregate structs.
+- [x] Add `UserRecord` (`location`, `user : UserResult`, `email`,
+      `date_of_birth`) as the current stand-in for the planned
+      `GeneratedUser` — `email` and `date_of_birth` are programmatically
+      generated by the orchestrator, never LLM-authored, so a downstream
+      auth-account seeding consumer can register real accounts from the
+      pipeline's SQLite export. No `password`, `user_id`, or `metadata`
+      field yet, matching `BreweryRecord`'s equally pre-`metadata` shape.
+      Already wired into `IExportService`/`SqliteExportService`: a `users`
+      table exists and `GenerateUsers()` exports each successful record via
+      `ProcessRecord(const UserRecord&)` (see §5) in addition to logging and
+      holding `generated_users_` in memory. Still missing vs. the planned
+      `GeneratedUser`: `user_id`, `password`, and `metadata`.
+- [x] Add `UserPersona` (`name`, `description`, `style_affinities`).

 ## 2. Data Preloading

-`tooling/pipeline/includes/json_handling/`, fixture files.
+`tooling/pipeline/includes/services/curated_data/`, fixture files.

- [ ] Extract a `DataPreloader` interface; have `JsonLoader` implement it
-      instead of exposing only a static `LoadLocations()`.
+- [x] Extract an interface; have `CuratedJsonDataService` implement it.
+      Landed as `ICuratedDataService`
+      (`services/curated_data/curated_data_service.h`), not `DataPreloader`
+      — `LoadLocations()`, `LoadPersonas()`, `LoadForenamesByCountry()`, and
+      `LoadSurnamesByCountry()` are all virtual methods returning `const&`
+      and take no arguments. `CuratedJsonDataService`
+      (`services/curated_data/curated_json_data_service.h`, originally
+      landed as `JsonLoader` under `json_handling/` before being renamed and
+      moved) takes a `CuratedDataFilePaths` DTO (`locations_path`,
+      `personas_path`, `forenames_path`, `surnames_path`) in its
+      constructor rather than a path per `Load*()` call, and memoizes each
+      result in a private `cache` struct (`locations`, `personas`,
+      `forenames_by_country`, `surnames_by_country`), so a second call on
+      the same instance returns the cached result instead of re-parsing —
+      safe because `CuratedJsonDataService` outlives every call site (owned
+      by `BiergartenPipelineOrchestrator` via
+      `unique_ptr<ICuratedDataService>` for the whole run). `main.cc`'s DI
+      injector also gained a `MockCuratedDataService` (fixed in-memory
+      data: 4 locations across `US`/`DE`/`FR`/`BE`, 3 personas, and
+      matching forename/surname sets for those four countries), bound in
+      place of `CuratedJsonDataService` under `--mocked`, mirroring the
+      existing `MockEnrichmentService`/`MockGenerator` pattern.
 - [ ] Add `LoadBeerStyles()`. `beer-styles.json` already exists in the repo
      and is already copied into the Docker image
      (`runpod/Dockerfile`), but no loader reads it yet, and the native
      CMake build doesn't copy it into `build/` at all — `CMakeLists.txt`'s
      "Runtime Assets" step only copies `locations.json` and `prompts/`.
- [ ] Add `LoadPersonas()` and author `personas.json` (doesn't exist yet).
- [ ] Add `LoadNamesByCountry()` and author `names-by-country.json` (doesn't
-      exist yet).
+- [x] Add `LoadPersonas()` and author `personas.json` (doesn't exist yet).
+- [x] Add name-by-country loading, parsing
+      `tooling/pipeline/forenames-by-country.json` and
+      `surnames-by-country.json`. Landed as two separate methods —
+      `LoadForenamesByCountry()` and `LoadSurnamesByCountry()` — each
+      returning a flat `ForenamesByCountryMap` / `SurnamesByCountryMap`
+      (aliases for `unordered_map<string, ForenameList>` /
+      `unordered_map<string, SurnameList>`) keyed by ISO 3166-1 code, rather
+      than one combined `LoadNamesByCountry() : NamesByCountry` call. Both
+      files are vendored verbatim (unmodified,
+      full multinational coverage) from
+      `sigpwned/popular-names-by-country-dataset` (CC0) — see
+      ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for
+      provenance. Deliberately not pre-paired or filtered down to just the
+      countries in `locations.json`: keeping the source shape (including
+      per-forename gender) intact means the loader can support more
+      countries later, or gender-aware persona/bio generation, without
+      re-sourcing data.

 ## 3. Policy / Strategy Layer

@@ -83,10 +148,14 @@ Entirely new — no `includes/policy/` (or equivalent) directory exists today.
 - [ ] Extend the `DataGenerator` interface with `GenerateBeer`,
      `GenerateCheckin`, `GenerateRating` (today: only `GenerateBrewery` and
      `GenerateUser`).
- [ ] Implement `LlamaGenerator::GenerateUser` for real. It currently
-      returns a hardcoded `{"test_user", "This is a test user profile from
-    {locale}."}` regardless of input — see the `// TODO` at the top of
-      `src/data_generation/llama/generate_user.cc`.
+- [x] Implement `LlamaGenerator::GenerateUser` for real, with a retry loop
+      mirroring `GenerateBrewery` (GBNF grammar, `ValidateUserJson`, up to 3
+      attempts with corrective feedback) and a new
+      `prompts/USER_GENERATION.md`. Changed the `DataGenerator::GenerateUser`
+      signature to
+      `(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult`,
+      matching what the activity diagram actually passes. `MockGenerator::GenerateUser`
+      updated to match the new signature too.
 - [ ] Implement `MockGenerator::GenerateBeer` / `GenerateCheckin` /
      `GenerateRating`.
 - [ ] Add `IPromptFormatter::ExpectedArchitecture()` and
@@ -100,12 +169,19 @@ Entirely new — no `includes/policy/` (or equivalent) directory exists today.

 `tooling/pipeline/includes/services/database/`, `src/services/sqlite/`.

- [ ] Extend `IExportService` with `ProcessBeer`, `ProcessUser`,
-      `ProcessCheckin`, `ProcessRating`, `ProcessFollow` (today only
-      `ProcessRecord(const GeneratedBrewery&)` exists).
- [ ] Add `beers`, `users`, `checkins`, `ratings`, `follows` tables to
-      `SqliteExportService::InitializeSchema()` — see
-      `kCreateLocationsTableSql` / `kCreateBreweriesTableSql` in
+- [x] `IExportService::ProcessRecord(const UserRecord&)` and
+      `SqliteExportService`'s `users` table (see
+      `kCreateUsersTableSql` / `kInsertUserSql` in
+      `includes/services/database/sqlite_statement_helpers.h`) are
+      implemented — `GenerateUsers()` exports every successful user
+      alongside its resolved `location_id`.
+- [ ] Extend `IExportService` with `ProcessBeer`, `ProcessCheckin`,
+      `ProcessRating`, `ProcessFollow` (today `ProcessRecord(const
+      BreweryRecord&)` and `ProcessRecord(const UserRecord&)` exist).
+- [ ] Add `beers`, `checkins`, `ratings`, `follows` tables to
+      `SqliteExportService::InitializeSchema()` (`locations`, `breweries`,
+      and `users` already exist) — see `kCreateLocationsTableSql` /
+      `kCreateBreweriesTableSql` / `kCreateUsersTableSql` in
      `includes/services/database/sqlite_statement_helpers.h` for the
      existing pattern to follow.
 - [ ] Add a `brewery_cache_` alongside the existing `location_cache_`, and
@@ -151,7 +227,8 @@ Entirely new — no `includes/policy/` (or equivalent) directory exists today.

 ## 8. Fixtures / Build

- [ ] Author `personas.json` and `names-by-country.json` (see §2).
+- [x] Author `personas.json`, `forenames-by-country.json`, and
+      `surnames-by-country.json` (see §2).
 - [ ] Fix the `beer-styles.json` build-tree gap: add it to the "Runtime
      Assets" `configure_file` step in `CMakeLists.txt` so native builds and
      the Docker image agree on what's available at runtime.