Formatting

This commit is contained in:
Aaron Po
2026-07-19 04:05:30 -04:00
parent 8e7ecf2fc3
commit ec770aa724
78 changed files with 13388 additions and 32143 deletions

View File

@@ -3,13 +3,12 @@
This is the engineering breakdown for closing the gap between the **current**
pipeline and the **planned** architecture documented in
[`diagrams/planned/class.puml`](./diagrams/planned/class.puml) and
[`diagrams/planned/activity.puml`](./diagrams/planned/activity.puml). Nothing
in `diagrams/planned/` is implemented yet — this file tracks what it would
take to get there. For the current implementation, see
[README.md](./README.md).
[`diagrams/planned/activity.puml`](./diagrams/planned/activity.puml). Nothing in
`diagrams/planned/` is implemented yet — this file tracks what it would take to
get there. For the current implementation, see [README.md](./README.md).
Items are grouped by layer and roughly ordered by dependency: later groups
build on types and services introduced earlier.
Items are grouped by layer and roughly ordered by dependency: later groups build
on types and services introduced earlier.
The only concurrency in the current pipeline is the log dispatcher thread
(`main.cc` spawns it to drain a `BoundedChannel<LogEntry>` into spdlog for the
@@ -25,31 +24,30 @@ architecture needs.
`tooling/pipeline/includes/data_model/generated_models.h` and `models.h`.
- [ ] Add `Completeness` enum (`Full`, `Partial`, `Absent`) and a
`LocationContext` struct (`text`, `completeness`, `char_count`).
Replace `EnrichedCity::region_context` (currently a plain
`std::string`) with `context : LocationContext`.
- [ ] Add `BeerStyle` (`name`, `description`, `min_abv`, `max_abv`,
`min_ibu`, `max_ibu`).
`LocationContext` struct (`text`, `completeness`, `char_count`). Replace
`EnrichedCity::region_context` (currently a plain `std::string`) with
`context : LocationContext`.
- [ ] Add `BeerStyle` (`name`, `description`, `min_abv`, `max_abv`, `min_ibu`,
`max_ibu`).
- [ ] Add `BeerResult`, `CheckinResult`, `RatingResult` result payloads.
- [ ] Add `GenerationMetadata` (`generation_id`, `generated_time`,
`context_provided`, `generated_with`).
- [x] Add `first_name`, `last_name`, `gender`, and `activity_weight` to
`UserResult` (currently just `username`, `bio`). `first_name`/
`last_name`/`gender` are copied from the sampled `Name` (see below),
not LLM-invented.
`last_name`/`gender` are copied from the sampled `Name` (see below), not
LLM-invented.
- [x] Add `Name` (`first_name`, `last_name`, `gender`) — the sampled result
handed to `DataGenerator::GenerateUser`. Add `ForenameEntry` (`name`,
`gender`). Landed as a flatter shape than originally planned here: no
`NamesByCountry` wrapper class. `curated_data_service.h` instead
declares `ForenameList = std::vector<ForenameEntry>` and
`SurnameList = std::vector<std::string>`, and
`ICuratedDataService` exposes two maps keyed by ISO 3166-1 code —
`NamesByCountry` wrapper class. `curated_data_service.h` instead declares
`ForenameList = std::vector<ForenameEntry>` and
`SurnameList = std::vector<std::string>`, and `ICuratedDataService`
exposes two maps keyed by ISO 3166-1 code —
`ForenamesByCountryMap = unordered_map<string, ForenameList>` and
`SurnamesByCountryMap = unordered_map<string, SurnameList>` —
directly (see §2). Sampling is
a free function, `SampleName(forenames_by_country,
surnames_by_country, iso3166_1, rng)`, in
`generate_users.cc`'s anonymous namespace, not a method on a class.
`SurnamesByCountryMap = unordered_map<string, SurnameList>` — directly
(see §2). Sampling is a free function,
`SampleName(forenames_by_country, surnames_by_country, iso3166_1, rng)`,
in `generate_users.cc`'s anonymous namespace, not a method on a class.
Pairing a forename with a surname happens at sample time so gender
(present per-forename in the source data) is never discarded the way a
pre-flattened `{first_name, last_name}` list would lose it; see §2.
@@ -58,14 +56,14 @@ architecture needs.
- [ ] Add `GeneratedBeer`, `GeneratedCheckin`, `GeneratedRating`,
`GeneratedFollow` aggregate structs.
- [x] Add `UserRecord` (`location`, `user : UserResult`, `email`,
`date_of_birth`) as the current stand-in for the planned
`GeneratedUser` — `email` and `date_of_birth` are programmatically
generated by the orchestrator, never LLM-authored, so a downstream
auth-account seeding consumer can register real accounts from the
pipeline's SQLite export. No `password`, `user_id`, or `metadata`
field yet, matching `BreweryRecord`'s equally pre-`metadata` shape.
Already wired into `IExportService`/`SqliteExportService`: a `users`
table exists and `GenerateUsers()` exports each successful record via
`date_of_birth`) as the current stand-in for the planned `GeneratedUser` —
`email` and `date_of_birth` are programmatically generated by the
orchestrator, never LLM-authored, so a downstream auth-account seeding
consumer can register real accounts from the pipeline's SQLite export. No
`password`, `user_id`, or `metadata` field yet, matching `BreweryRecord`'s
equally pre-`metadata` shape. Already wired into
`IExportService`/`SqliteExportService`: a `users` table exists and
`GenerateUsers()` exports each successful record via
`ProcessRecord(const UserRecord&)` (see §5) in addition to logging and
holding `generated_users_` in memory. Still missing vs. the planned
`GeneratedUser`: `user_id`, `password`, and `metadata`.
@@ -75,66 +73,62 @@ architecture needs.
`tooling/pipeline/includes/services/curated_data/`, fixture files.
- [x] Extract an interface; have `CuratedJsonDataService` implement it.
Landed as `ICuratedDataService`
(`services/curated_data/curated_data_service.h`), not `DataPreloader`
— `LoadLocations()`, `LoadPersonas()`, `LoadForenamesByCountry()`, and
`LoadSurnamesByCountry()` are all virtual methods returning `const&`
and take no arguments. `CuratedJsonDataService`
(`services/curated_data/curated_json_data_service.h`, originally
landed as `JsonLoader` under `json_handling/` before being renamed and
moved) takes a `CuratedDataFilePaths` DTO (`locations_path`,
`personas_path`, `forenames_path`, `surnames_path`) in its
constructor rather than a path per `Load*()` call, and memoizes each
result in a private `cache` struct (`locations`, `personas`,
`forenames_by_country`, `surnames_by_country`), so a second call on
the same instance returns the cached result instead of re-parsing —
safe because `CuratedJsonDataService` outlives every call site (owned
by `BiergartenPipelineOrchestrator` via
- [x] Extract an interface; have `CuratedJsonDataService` implement it. Landed
as `ICuratedDataService` (`services/curated_data/curated_data_service.h`),
not `DataPreloader` — `LoadLocations()`, `LoadPersonas()`,
`LoadForenamesByCountry()`, and `LoadSurnamesByCountry()` are all virtual
methods returning `const&` and take no arguments. `CuratedJsonDataService`
(`services/curated_data/curated_json_data_service.h`, originally landed as
`JsonLoader` under `json_handling/` before being renamed and moved) takes
a `CuratedDataFilePaths` DTO (`locations_path`, `personas_path`,
`forenames_path`, `surnames_path`) in its constructor rather than a path
per `Load*()` call, and memoizes each result in a private `cache` struct
(`locations`, `personas`, `forenames_by_country`, `surnames_by_country`),
so a second call on the same instance returns the cached result instead of
re-parsing — safe because `CuratedJsonDataService` outlives every call
site (owned by `BiergartenPipelineOrchestrator` via
`unique_ptr<ICuratedDataService>` for the whole run). `main.cc`'s DI
injector also gained a `MockCuratedDataService` (fixed in-memory
data: 4 locations across `US`/`DE`/`FR`/`BE`, 3 personas, and
matching forename/surname sets for those four countries), bound in
place of `CuratedJsonDataService` under `--mocked`, mirroring the
existing `MockEnrichmentService`/`MockGenerator` pattern.
- [ ] Add `LoadBeerStyles()`. `beer-styles.json` already exists in the repo
and is already copied into the Docker image
(`runpod/Dockerfile`), but no loader reads it yet, and the native
CMake build doesn't copy it into `build/` at all — `CMakeLists.txt`'s
"Runtime Assets" step only copies `locations.json` and `prompts/`.
injector also gained a `MockCuratedDataService` (fixed in-memory data: 4
locations across `US`/`DE`/`FR`/`BE`, 3 personas, and matching
forename/surname sets for those four countries), bound in place of
`CuratedJsonDataService` under `--mocked`, mirroring the existing
`MockEnrichmentService`/`MockGenerator` pattern.
- [ ] Add `LoadBeerStyles()`. `beer-styles.json` already exists in the repo and
is already copied into the Docker image (`runpod/Dockerfile`), but no
loader reads it yet, and the native CMake build doesn't copy it into
`build/` at all — `CMakeLists.txt`'s "Runtime Assets" step only copies
`locations.json` and `prompts/`.
- [x] Add `LoadPersonas()` and author `personas.json` (doesn't exist yet).
- [x] Add name-by-country loading, parsing
`tooling/pipeline/forenames-by-country.json` and
`surnames-by-country.json`. Landed as two separate methods —
`LoadForenamesByCountry()` and `LoadSurnamesByCountry()` — each
returning a flat `ForenamesByCountryMap` / `SurnamesByCountryMap`
(aliases for `unordered_map<string, ForenameList>` /
`LoadForenamesByCountry()` and `LoadSurnamesByCountry()` — each returning
a flat `ForenamesByCountryMap` / `SurnamesByCountryMap` (aliases for
`unordered_map<string, ForenameList>` /
`unordered_map<string, SurnameList>`) keyed by ISO 3166-1 code, rather
than one combined `LoadNamesByCountry() : NamesByCountry` call. Both
files are vendored verbatim (unmodified,
full multinational coverage) from
than one combined `LoadNamesByCountry() : NamesByCountry` call. Both files
are vendored verbatim (unmodified, full multinational coverage) from
`sigpwned/popular-names-by-country-dataset` (CC0) — see
ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for
provenance. Deliberately not pre-paired or filtered down to just the
countries in `locations.json`: keeping the source shape (including
per-forename gender) intact means the loader can support more
countries later, or gender-aware persona/bio generation, without
re-sourcing data.
per-forename gender) intact means the loader can support more countries
later, or gender-aware persona/bio generation, without re-sourcing data.
## 3. Policy / Strategy Layer
Entirely new — no `includes/policy/` (or equivalent) directory exists today.
- [ ] `ContextStrategy` interface + `BreweryContextStrategy` /
`BeerContextStrategy`. Today `WikipediaEnrichmentService::GetLocationContext`
hardcodes a generic `"brewing"` query and a `"beer in {country}"`
query directly — no per-phase strategy selection.
- [ ] `SamplingStrategy` interface + `UniformSamplingStrategy`, replacing
the inline `std::ranges::sample(...)` call in
`BeerContextStrategy`. Today
`WikipediaEnrichmentService::GetLocationContext` hardcodes a generic
`"brewing"` query and a `"beer in {country}"` query directly — no
per-phase strategy selection.
- [ ] `SamplingStrategy` interface + `UniformSamplingStrategy`, replacing the
inline `std::ranges::sample(...)` call in
`BiergartenPipelineOrchestrator::QueryCitiesWithCountries()`.
- [ ] `BeerSelectionStrategy` interface + `RandomBeerSelectionStrategy`, to
pick styles per brewery from the `BeerStyle` palette (depends on §1
and §2).
- [ ] `BeerSelectionStrategy` interface + `RandomBeerSelectionStrategy`, to pick
styles per brewery from the `BeerStyle` palette (depends on §1 and §2).
- [ ] `CheckinDistributionStrategy` interface + `JCurveCheckinStrategy` /
`RandomCheckinStrategy` — this is the "Check-In System" item already
called out in README.md's Next Steps, made concrete.
@@ -150,88 +144,87 @@ Entirely new — no `includes/policy/` (or equivalent) directory exists today.
`GenerateUser`).
- [x] Implement `LlamaGenerator::GenerateUser` for real, with a retry loop
mirroring `GenerateBrewery` (GBNF grammar, `ValidateUserJson`, up to 3
attempts with corrective feedback) and a new
`prompts/USER_GENERATION.md`. Changed the `DataGenerator::GenerateUser`
signature to
attempts with corrective feedback) and a new `prompts/USER_GENERATION.md`.
Changed the `DataGenerator::GenerateUser` signature to
`(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult`,
matching what the activity diagram actually passes. `MockGenerator::GenerateUser`
updated to match the new signature too.
matching what the activity diagram actually passes.
`MockGenerator::GenerateUser` updated to match the new signature too.
- [ ] Implement `MockGenerator::GenerateBeer` / `GenerateCheckin` /
`GenerateRating`.
- [ ] Add `IPromptFormatter::ExpectedArchitecture()` and
`LlamaGenerator::ValidateModelArchitecture()` so loading a GGUF that
doesn't match the configured chat template fails fast instead of
silently producing degraded output.
- [ ] Add prompt template files for beer/checkin/rating generation next to
the existing `prompts/BREWERY_GENERATION.md`.
doesn't match the configured chat template fails fast instead of silently
producing degraded output.
- [ ] Add prompt template files for beer/checkin/rating generation next to the
existing `prompts/BREWERY_GENERATION.md`.
## 5. Export Service
`tooling/pipeline/includes/services/database/`, `src/services/sqlite/`.
- [x] `IExportService::ProcessRecord(const UserRecord&)` and
`SqliteExportService`'s `users` table (see
`kCreateUsersTableSql` / `kInsertUserSql` in
`includes/services/database/sqlite_statement_helpers.h`) are
implemented — `GenerateUsers()` exports every successful user
alongside its resolved `location_id`.
`SqliteExportService`'s `users` table (see `kCreateUsersTableSql` /
`kInsertUserSql` in
`includes/services/database/sqlite_statement_helpers.h`) are implemented —
`GenerateUsers()` exports every successful user alongside its resolved
`location_id`.
- [ ] Extend `IExportService` with `ProcessBeer`, `ProcessCheckin`,
`ProcessRating`, `ProcessFollow` (today `ProcessRecord(const
BreweryRecord&)` and `ProcessRecord(const UserRecord&)` exist).
`ProcessRating`, `ProcessFollow` (today
`ProcessRecord(const BreweryRecord&)` and
`ProcessRecord(const UserRecord&)` exist).
- [ ] Add `beers`, `checkins`, `ratings`, `follows` tables to
`SqliteExportService::InitializeSchema()` (`locations`, `breweries`,
and `users` already exist) — see `kCreateLocationsTableSql` /
`SqliteExportService::InitializeSchema()` (`locations`, `breweries`, and
`users` already exist) — see `kCreateLocationsTableSql` /
`kCreateBreweriesTableSql` / `kCreateUsersTableSql` in
`includes/services/database/sqlite_statement_helpers.h` for the
existing pattern to follow.
- [ ] Add a `brewery_cache_` alongside the existing `location_cache_`, and
move from one open transaction for the whole run to per-phase batched
commits (`BEGIN` / `COMMIT & BEGIN` on a batch-size threshold), as
shown in the planned activity diagram.
`includes/services/database/sqlite_statement_helpers.h` for the existing
pattern to follow.
- [ ] Add a `brewery_cache_` alongside the existing `location_cache_`, and move
from one open transaction for the whole run to per-phase batched commits
(`BEGIN` / `COMMIT & BEGIN` on a batch-size threshold), as shown in the
planned activity diagram.
## 6. Concurrency & Orchestration
`tooling/pipeline/includes/concurrency/`,
`src/biergarten_pipeline_orchestrator/`.
- [ ] `BoundedChannel<T>` already exists and is production-tested — but
it's only wired to the single log channel today. Stand up the
per-phase producer/consumer channels (`loc_ch`, `exp_ch`, etc.) the
planned activity diagram describes, each with a dedicated LLM worker
thread and SQLite worker thread.
- [ ] Rewrite `BiergartenPipelineOrchestrator::Run()` from one synchronous
pass over `generated_breweries_` into phased methods —
`RunUserPhase` → `RunBreweryAndBeerPhase` (with a `RunBeerPhase`
sub-step once `brewery_pool_` is populated) → `RunCheckinPhase` /
`RunFollowPhase` (these two can run in parallel) → `RunRatingPhase` —
backed by `user_pool_`, `brewery_pool_`, `beer_pool_`,
`checkin_pool_`, `follow_pool_`.
- [ ] `BoundedChannel<T>` already exists and is production-tested — but it's
only wired to the single log channel today. Stand up the per-phase
producer/consumer channels (`loc_ch`, `exp_ch`, etc.) the planned activity
diagram describes, each with a dedicated LLM worker thread and SQLite
worker thread.
- [ ] Rewrite `BiergartenPipelineOrchestrator::Run()` from one synchronous pass
over `generated_breweries_` into phased methods — `RunUserPhase` →
`RunBreweryAndBeerPhase` (with a `RunBeerPhase` sub-step once
`brewery_pool_` is populated) → `RunCheckinPhase` / `RunFollowPhase`
(these two can run in parallel) → `RunRatingPhase` — backed by
`user_pool_`, `brewery_pool_`, `beer_pool_`, `checkin_pool_`,
`follow_pool_`.
- [ ] Honor the structural-concurrency requirement already called out in a
comment on `BiergartenPipelineOrchestrator::Run()` in
`biergarten_pipeline_orchestrator.h`: once real worker threads exist,
they must be structurally joined (e.g. via `std::jthread`) before
`Run()` returns, so no worker logs to a closed channel during
teardown.
`biergarten_pipeline_orchestrator.h`: once real worker threads exist, they
must be structurally joined (e.g. via `std::jthread`) before `Run()`
returns, so no worker logs to a closed channel during teardown.
## 7. Enrichment
- [ ] Decide whether to restore the city/region-specific Wikipedia query
that's currently commented out in
`src/services/enrichment/wikipedia/get_summary.cc`
(`GetLocationContext`). The `ContextStrategy` work in §3 is a natural
place to reintroduce it via `BreweryContextStrategy::QueriesFor()`.
- [ ] Decide whether to restore the city/region-specific Wikipedia query that's
currently commented out in
`src/services/enrichment/wikipedia/get_summary.cc` (`GetLocationContext`).
The `ContextStrategy` work in §3 is a natural place to reintroduce it via
`BreweryContextStrategy::QueriesFor()`.
- [ ] Pre-warm caches at startup (`PreWarmBeerStyleCache`,
`PreWarmLocationCache` in the planned activity diagram) instead of
fetching lazily per record, so the streaming phases never block on a
cold cache mid-run.
fetching lazily per record, so the streaming phases never block on a cold
cache mid-run.
## 8. Fixtures / Build
- [x] Author `personas.json`, `forenames-by-country.json`, and
`surnames-by-country.json` (see §2).
- [ ] Fix the `beer-styles.json` build-tree gap: add it to the "Runtime
Assets" `configure_file` step in `CMakeLists.txt` so native builds and
the Docker image agree on what's available at runtime.
- [ ] Fix the `beer-styles.json` build-tree gap: add it to the "Runtime Assets"
`configure_file` step in `CMakeLists.txt` so native builds and the Docker
image agree on what's available at runtime.
---
@@ -240,13 +233,12 @@ Entirely new — no `includes/policy/` (or equivalent) directory exists today.
The planned activity diagram's own phase notes ("brewery*pool* is now fully
populated. Phase 1b may begin.", etc.) imply a dependency order. Roughly:
1. Domain models (§1) and data preloading (§2) — nothing else compiles
without these.
2. Export schema (§5) — so every later generation phase has somewhere to
land its output.
1. Domain models (§1) and data preloading (§2) — nothing else compiles without
these.
2. Export schema (§5) — so every later generation phase has somewhere to land
its output.
3. Policy/strategy layer (§3) and generator interface (§4).
4. Concurrency/orchestration rewrite (§6), which is the only piece that
actually wires §1–§5 into the phased, parallel pipeline shown in the
planned diagrams.
5. Enrichment cache pre-warming and the city/region query decision (§7) —
useful at any point, but most valuable once phases run concurrently.
4. Concurrency/orchestration rewrite (§6), which is the only piece that actually
wires §1–§5 into the phased, parallel pipeline shown in the planned diagrams.
5. Enrichment cache pre-warming and the city/region query decision (§7) — useful
at any point, but most valuable once phases run concurrently.