mirror of
https://github.com/aaronpo97/the-biergarten-app.git
synced 2026-07-16 17:47:22 +00:00
212 lines
11 KiB
Markdown
212 lines
11 KiB
Markdown
# Pipeline Roadmap — Reaching the Planned Architecture
|
||
|
||
This is the engineering breakdown for closing the gap between the **current**
|
||
pipeline and the **planned** architecture documented in
|
||
[`diagrams/planned/class.puml`](./diagrams/planned/class.puml) and
|
||
[`diagrams/planned/activity.puml`](./diagrams/planned/activity.puml). Nothing
|
||
in `diagrams/planned/` is implemented yet — this file tracks what it would
|
||
take to get there. For the current implementation, see
|
||
[README.md](./README.md).
|
||
|
||
Items are grouped by layer and roughly ordered by dependency: later groups
|
||
build on types and services introduced earlier.
|
||
|
||
The only concurrency in the current pipeline is the log dispatcher thread
|
||
(`main.cc` spawns it to drain a `BoundedChannel<LogEntry>` into spdlog for the
|
||
whole run). Sampling, enrichment, generation, and export all happen
|
||
synchronously on the main thread inside `BiergartenPipelineOrchestrator::Run()`
|
||
— §6 below is what introduces the additional worker threads the planned
|
||
architecture needs.
|
||
|
||
---
|
||
|
||
## 1. Domain Models
|
||
|
||
`tooling/pipeline/includes/data_model/generated_models.h` and `models.h`.
|
||
|
||
- [ ] Add `Completeness` enum (`Full`, `Partial`, `Absent`) and a
|
||
`LocationContext` struct (`text`, `completeness`, `char_count`).
|
||
Replace `EnrichedCity::region_context` (currently a plain
|
||
`std::string`) with `context : LocationContext`.
|
||
- [ ] Add `BeerStyle` (`name`, `description`, `min_abv`, `max_abv`,
|
||
`min_ibu`, `max_ibu`).
|
||
- [ ] Add `BeerResult`, `CheckinResult`, `RatingResult` result payloads.
|
||
- [ ] Add `GenerationMetadata` (`generation_id`, `generated_time`,
|
||
`context_provided`, `generated_with`).
|
||
- [x] Add `first_name`, `last_name`, `gender`, and `activity_weight` to
|
||
`UserResult` (currently just `username`, `bio`). `first_name`/
|
||
`last_name`/`gender` are copied from the sampled `Name` (see below),
|
||
not LLM-invented.
|
||
- [x] Add `Name` (`first_name`, `last_name`, `gender`) — the sampled result
|
||
handed to `DataGenerator::GenerateUser`. Add `ForenameEntry` (`name`,
|
||
`gender`) and a `NamesByCountry` class (not a bare typedef) that owns
|
||
two maps — `std::unordered_map<std::string, std::vector<ForenameEntry>>`
|
||
and `std::unordered_map<std::string, std::vector<std::string>>`, both
|
||
keyed by ISO 3166-1 code — and exposes `SampleName(iso3166_1, rng)`.
|
||
Pairing a forename with a surname happens at sample time so gender
|
||
(present per-forename in the source data) is never discarded the way a
|
||
pre-flattened `{first_name, last_name}` list would lose it; see
|
||
`LoadNamesByCountry()` below.
|
||
- [ ] Extend `GeneratedBrewery` with `brewery_id`, `context_completeness`,
|
||
`metadata` (currently just `{location, brewery}`).
|
||
- [ ] Add `GeneratedBeer`, `GeneratedCheckin`, `GeneratedRating`,
|
||
`GeneratedFollow` aggregate structs.
|
||
- [x] Add `GeneratedUser` aggregate struct. Carries `email`, `date_of_birth`,
|
||
and `password` — programmatically generated by the orchestrator, never
|
||
LLM-authored — so a downstream auth-account seeding consumer (outside
|
||
this repo tree's docs) can register real accounts from the pipeline's
|
||
SQLite export. Skips `user_id`/`metadata` for now, matching
|
||
`GeneratedBrewery`'s current (pre-`brewery_id`/`metadata`) shape. Not
|
||
yet wired into `IExportService`/`SqliteExportService` — `GenerateUsers`
|
||
currently only logs and holds `generated_users_` in memory; no `users`
|
||
table exists in the SQLite export yet (see §5).
|
||
- [x] Add `UserPersona` (`name`, `description`, `style_affinities`).
|
||
|
||
## 2. Data Preloading
|
||
|
||
`tooling/pipeline/includes/json_handling/`, fixture files.
|
||
|
||
- [ ] Extract a `DataPreloader` interface; have `JsonLoader` implement it.
|
||
`JsonLoader` now also exposes static `LoadPersonas()` and
|
||
`LoadNamesByCountry()` (see below), but as additional static methods,
|
||
not yet behind a formal interface.
|
||
- [ ] Add `LoadBeerStyles()`. `beer-styles.json` already exists in the repo
|
||
and is already copied into the Docker image
|
||
(`runpod/Dockerfile`), but no loader reads it yet, and the native
|
||
CMake build doesn't copy it into `build/` at all — `CMakeLists.txt`'s
|
||
"Runtime Assets" step only copies `locations.json` and `prompts/`.
|
||
- [x] Add `LoadPersonas()` and author `personas.json` (doesn't exist yet).
|
||
- [x] Add `LoadNamesByCountry(forenames_filepath, surnames_filepath)`,
|
||
parsing `tooling/pipeline/forenames-by-country.json` and
|
||
`surnames-by-country.json` into a `NamesByCountry`. Both files are
|
||
vendored verbatim (unmodified, full multinational coverage) from
|
||
`sigpwned/popular-names-by-country-dataset` (CC0) — see
|
||
ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for
|
||
provenance. Deliberately not pre-paired or filtered down to just the
|
||
countries in `locations.json`: keeping the source shape (including
|
||
per-forename gender) intact means the loader can support more
|
||
countries later, or gender-aware persona/bio generation, without
|
||
re-sourcing data.
|
||
|
||
## 3. Policy / Strategy Layer
|
||
|
||
Entirely new — no `includes/policy/` (or equivalent) directory exists today.
|
||
|
||
- [ ] `ContextStrategy` interface + `BreweryContextStrategy` /
|
||
`BeerContextStrategy`. Today `WikipediaEnrichmentService::GetLocationContext`
|
||
hardcodes a generic `"brewing"` query and a `"beer in {country}"`
|
||
query directly — no per-phase strategy selection.
|
||
- [ ] `SamplingStrategy` interface + `UniformSamplingStrategy`, replacing
|
||
the inline `std::ranges::sample(...)` call in
|
||
`BiergartenPipelineOrchestrator::QueryCitiesWithCountries()`.
|
||
- [ ] `BeerSelectionStrategy` interface + `RandomBeerSelectionStrategy`, to
|
||
pick styles per brewery from the `BeerStyle` palette (depends on §1
|
||
and §2).
|
||
- [ ] `CheckinDistributionStrategy` interface + `JCurveCheckinStrategy` /
|
||
`RandomCheckinStrategy` — this is the "Check-In System" item already
|
||
called out in README.md's Next Steps, made concrete.
|
||
- [ ] `FollowGenerationStrategy` interface + `RandomFollowStrategy` /
|
||
`ActivityWeightedFollowStrategy`.
|
||
|
||
## 4. Data Generation
|
||
|
||
`tooling/pipeline/includes/data_generation/`, `src/data_generation/`.
|
||
|
||
- [ ] Extend the `DataGenerator` interface with `GenerateBeer`,
|
||
`GenerateCheckin`, `GenerateRating` (today: only `GenerateBrewery` and
|
||
`GenerateUser`).
|
||
- [x] Implement `LlamaGenerator::GenerateUser` for real, with a retry loop
|
||
mirroring `GenerateBrewery` (GBNF grammar, `ValidateUserJson`, up to 3
|
||
attempts with corrective feedback) and a new
|
||
`prompts/USER_GENERATION.md`. Changed the `DataGenerator::GenerateUser`
|
||
signature to
|
||
`(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult`,
|
||
matching what the activity diagram actually passes. `MockGenerator::GenerateUser`
|
||
updated to match the new signature too.
|
||
- [ ] Implement `MockGenerator::GenerateBeer` / `GenerateCheckin` /
|
||
`GenerateRating`.
|
||
- [ ] Add `IPromptFormatter::ExpectedArchitecture()` and
|
||
`LlamaGenerator::ValidateModelArchitecture()` so loading a GGUF that
|
||
doesn't match the configured chat template fails fast instead of
|
||
silently producing degraded output.
|
||
- [ ] Add prompt template files for beer/checkin/rating generation next to
|
||
the existing `prompts/BREWERY_GENERATION.md`.
|
||
|
||
## 5. Export Service
|
||
|
||
`tooling/pipeline/includes/services/database/`, `src/services/sqlite/`.
|
||
|
||
- [ ] Extend `IExportService` with `ProcessBeer`, `ProcessUser`,
|
||
`ProcessCheckin`, `ProcessRating`, `ProcessFollow` (today only
|
||
`ProcessRecord(const GeneratedBrewery&)` exists).
|
||
- [ ] Add `beers`, `users`, `checkins`, `ratings`, `follows` tables to
|
||
`SqliteExportService::InitializeSchema()` — see
|
||
`kCreateLocationsTableSql` / `kCreateBreweriesTableSql` in
|
||
`includes/services/database/sqlite_statement_helpers.h` for the
|
||
existing pattern to follow.
|
||
- [ ] Add a `brewery_cache_` alongside the existing `location_cache_`, and
|
||
move from one open transaction for the whole run to per-phase batched
|
||
commits (`BEGIN` / `COMMIT & BEGIN` on a batch-size threshold), as
|
||
shown in the planned activity diagram.
|
||
|
||
## 6. Concurrency & Orchestration
|
||
|
||
`tooling/pipeline/includes/concurrency/`,
|
||
`src/biergarten_pipeline_orchestrator/`.
|
||
|
||
- [ ] `BoundedChannel<T>` already exists and is production-tested — but
|
||
it's only wired to the single log channel today. Stand up the
|
||
per-phase producer/consumer channels (`loc_ch`, `exp_ch`, etc.) the
|
||
planned activity diagram describes, each with a dedicated LLM worker
|
||
thread and SQLite worker thread.
|
||
- [ ] Rewrite `BiergartenPipelineOrchestrator::Run()` from one synchronous
|
||
pass over `generated_breweries_` into phased methods —
|
||
`RunUserPhase` → `RunBreweryAndBeerPhase` (with a `RunBeerPhase`
|
||
sub-step once `brewery_pool_` is populated) → `RunCheckinPhase` /
|
||
`RunFollowPhase` (these two can run in parallel) → `RunRatingPhase` —
|
||
backed by `user_pool_`, `brewery_pool_`, `beer_pool_`,
|
||
`checkin_pool_`, `follow_pool_`.
|
||
- [ ] Honor the structural-concurrency requirement already called out in a
|
||
comment on `BiergartenPipelineOrchestrator::Run()` in
|
||
`biergarten_pipeline_orchestrator.h`: once real worker threads exist,
|
||
they must be structurally joined (e.g. via `std::jthread`) before
|
||
`Run()` returns, so no worker logs to a closed channel during
|
||
teardown.
|
||
|
||
## 7. Enrichment
|
||
|
||
- [ ] Decide whether to restore the city/region-specific Wikipedia query
|
||
that's currently commented out in
|
||
`src/services/enrichment/wikipedia/get_summary.cc`
|
||
(`GetLocationContext`). The `ContextStrategy` work in §3 is a natural
|
||
place to reintroduce it via `BreweryContextStrategy::QueriesFor()`.
|
||
- [ ] Pre-warm caches at startup (`PreWarmBeerStyleCache`,
|
||
`PreWarmLocationCache` in the planned activity diagram) instead of
|
||
fetching lazily per record, so the streaming phases never block on a
|
||
cold cache mid-run.
|
||
|
||
## 8. Fixtures / Build
|
||
|
||
- [ ] Author `personas.json` and `names-by-country.json` (see §2).
|
||
- [ ] Fix the `beer-styles.json` build-tree gap: add it to the "Runtime
|
||
Assets" `configure_file` step in `CMakeLists.txt` so native builds and
|
||
the Docker image agree on what's available at runtime.
|
||
|
||
---
|
||
|
||
## Suggested build order
|
||
|
||
The planned activity diagram's own phase notes ("brewery*pool* is now fully
|
||
populated. Phase 1b may begin.", etc.) imply a dependency order. Roughly:
|
||
|
||
1. Domain models (§1) and data preloading (§2) — nothing else compiles
|
||
without these.
|
||
2. Export schema (§5) — so every later generation phase has somewhere to
|
||
land its output.
|
||
3. Policy/strategy layer (§3) and generator interface (§4).
|
||
4. Concurrency/orchestration rewrite (§6), which is the only piece that
|
||
actually wires §1–§5 into the phased, parallel pipeline shown in the
|
||
planned diagrams.
|
||
5. Enrichment cache pre-warming and the city/region query decision (§7) —
|
||
useful at any point, but most valuable once phases run concurrently.
|