AaronPo/the-biergarten-app
1
0
Fork 0
mirror of https://github.com/aaronpo97/the-biergarten-app.git synced 2026-10-15 10:04:44 +00:00
Code Issues Packages Projects Releases Wiki Activity

Refactor documentation and diagrams for web/pipeline. (#272)

Browse Source 
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> for assistance with diagram creation and documentation prose updates.
This commit is contained in:
Aaron Po
2026-09-02 01:58:40 -04:00
committed by GitHub
parent 1fe35a2ea6
commit 30317326ce
83 changed files with 4291 additions and 4497 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
docs/pipeline/diagrams/planned/activity.md Normal file
View File

@@ -0,0 +1,16 @@
# The Biergarten Data Pipeline — Activity Diagram (Planned)
Each generation phase runs a producer/consumer chain concurrently: an
**Orchestrator** producer feeds a bounded channel, an **LLM Worker** consumes it
and generates records (closing its output channel when the input is drained and
closed), and a **SQLite Worker** consumes that output channel and commits in
batches. The brewery phase inserts a fourth stage, a **Geocode Worker**, between
the LLM Worker and the SQLite Worker: reverse-geocoding is network-bound and
rate-limited independently of LLM inference (Nominatim's 1-request-per-second
policy), so keeping it off the LLM Worker's thread means neither stage stalls
waiting on the other's bottleneck. The Checkin and Follow phases run in parallel
with each other once the user and brewery pools are populated.
```mermaid
--8<-- "pipeline/diagrams/planned/activity.mmd"
```

208
docs/pipeline/diagrams/planned/activity.mmd Normal file
View File

@@ -0,0 +1,208 @@
flowchart TD
Start([start]) --> ParseArgs["Main: ParseArguments(argc, argv)"]
ParseArgs --> InvalidArgs{Invalid args?}
InvalidArgs -- yes --> LogErr["spdlog::error"] --> Stop1([stop])
InvalidArgs -- no --> InitOpenSSL["Main: Init OpenSSL global state and LlamaBackendState"]
InitOpenSSL --> BuildDI["Main: Build DI injector"]
BuildDI --> InitSqlite["Main: Initialize SqliteExportService<br/>opens SQLite connection; transactions now managed per-phase via batching"]
InitSqlite --> LogSetup["Main: create BoundedChannel log_ch, spawn Log Worker thread<br/>drains log_ch for the pipeline lifetime; all workers emit LogEntry via PipelineLogger, never spdlog directly"]
LogSetup --> OrchRun["BiergartenPipelineOrchestrator::Run()"]
OrchRun --> PL1["JsonLoader.LoadBeerStyles → EnrichmentService.PreWarmBeerStyleCache"]
OrchRun --> PL2["JsonLoader.LoadLocations → EnrichmentService.PreWarmLocationCache"]
OrchRun --> PL3["JsonLoader.LoadForenamesByCountry"]
OrchRun --> PL4["JsonLoader.LoadSurnamesByCountry"]
OrchRun --> PL5["JsonLoader.LoadPersonas"]
PL1 & PL2 & PL3 & PL4 & PL5 --> UserPhase["Orchestrator: RunUserPhase(sampled_locations)<br/>each Load* call memoized after first parse; MockCuratedDataService (--mocked) returns a fixed in-memory dataset instead, skipping these JSON files"]
UserPhase --> UCreate["Create BoundedChannels (loc_ch, exp_ch)"]
subgraph USERGEN["Phase 0 — User Generation (concurrent)"]
direction LR
subgraph U_PROD["Orchestrator (producer)"]
UP1["Loop: send Location → loc_ch"]
UP1 --> UP2["Close loc_ch"]
end
subgraph U_LLM["LLM Worker"]
UL1{"loc_ch has items?"}
UL1 -- yes --> UL2["Receive Location"]
UL2 --> UL3["GetLocationContextFromCache(location) — guaranteed cache hit from startup"]
UL3 --> UL4["IPersonaSelectionStrategy.SelectPersona(personas_palette_) — guaranteed cache hit; returns Persona{style_affinities, abv_range, ibu_preference, checkin_weight}"]
UL4 --> UL5["SampleName(forenames_by_country, surnames_by_country, iso3166_1, rng)<br/>deterministic, no LLM; free function; skips the city if either map has no entry for iso3166_1"]
UL5 --> UL6["GenerateUser(enriched_city, persona, sampled_name) via DataGenerator<br/>LLM receives EnrichedCity context + persona description + sampled name"]
UL6 --> UL7["PipelineLogger.Log(Info, UserGeneration, city, user_id, llm)"]
UL7 --> UL8["Send GeneratedUser → exp_ch"]
UL8 --> UL1
UL1 -- no --> UL9["Close exp_ch"]
end
subgraph U_SQL["SQLite Worker"]
US1["BEGIN TRANSACTION"] --> US2{"exp_ch has items?"}
US2 -- yes --> US3["Receive GeneratedUser"]
US3 --> US4["ProcessUser(user)"]
US4 --> US5["PipelineLogger.Log(Info, UserGeneration, city, user_id, sqlite)"]
US5 --> US6["Append → user_pool_"]
US6 --> US7{"Batch size reached?"}
US7 -- yes --> US8["COMMIT and BEGIN"] --> US2
US7 -- no --> US2
US2 -- no --> US9["COMMIT (Final)"]
end
end
UCreate --> UP1
UP2 -.->|loc_ch| UL1
UL9 -.->|exp_ch| US2
UL9 & US9 --> UserJoin["Orchestrator: Join LLM Worker, SQLite Worker"]
UserJoin --> BreweryPhase["Orchestrator: RunBreweryPhase(sampled_locations)"]
BreweryPhase --> BCreate["Create BoundedChannels (loc_ch, geo_ch, exp_ch)"]
subgraph BREWGEN["Phase 1a — Brewery Generation (concurrent)"]
direction LR
subgraph B_PROD["Orchestrator (producer)"]
BP1["Loop: sample User from user_pool_, pair with Location"]
BP1 --> BP2["Send BreweryTask(Location, User) → loc_ch"]
BP2 --> BP3["Close loc_ch"]
end
subgraph B_LLM["LLM Worker"]
BL1{"loc_ch has items?"}
BL1 -- yes --> BL2["Receive BreweryTask(Location, User)"]
BL2 --> BL3["GetLocationContextFromCache(task.location) — guaranteed cache hit"]
BL3 --> BL4["GenerateBrewery(enriched_city, context, task.user) via DataGenerator<br/>KV cache stays warm; brewery linked to sampled owner_user_id"]
BL4 --> BL5["PipelineLogger.Log(Info, BreweryGeneration, city, brewery_id, llm)"]
BL5 --> BL6["Send GeneratedBrewery → geo_ch"]
BL6 --> BL1
BL1 -- no --> BL7["Close geo_ch"]
end
subgraph B_GEO["Geocode Worker"]
BG1{"geo_ch has items?"}
BG1 -- yes --> BG2["Receive GeneratedBrewery"]
BG2 --> BG3["AddressService.ReverseGeocode(brewery.longitude, brewery.latitude)<br/>rate-limited independently of LLM throughput — NominatimAddressService sleeps 1s per request; MockAddressService (--mocked) returns a fixed placeholder, no network call"]
BG3 --> BG4{"Address resolved?"}
BG4 -- yes --> BG5["Attach resolved AddressLookupResult to brewery"]
BG4 -- no --> BG6["Log warning: could not reverse geocode; attach placeholder address_line1, empty postal_code — brewery is never dropped"]
BG5 --> BG7["PipelineLogger.Log(Info, BreweryGeneration, city, brewery_id, geocode)"]
BG6 --> BG7
BG7 --> BG8["Send GeneratedBrewery → exp_ch"]
BG8 --> BG1
BG1 -- no --> BG9["Close exp_ch"]
end
subgraph B_SQL["SQLite Worker"]
BS1["BEGIN TRANSACTION"] --> BS2{"exp_ch has items?"}
BS2 -- yes --> BS3["Receive GeneratedBrewery"]
BS3 --> BS4["ProcessBrewery(brewery)"]
BS4 --> BS5["PipelineLogger.Log(Info, BreweryGeneration, city, brewery_id, sqlite)"]
BS5 --> BS6["Append → brewery_pool_"]
BS6 --> BS7{"Batch size reached?"}
BS7 -- yes --> BS8["COMMIT and BEGIN"] --> BS2
BS7 -- no --> BS2
BS2 -- no --> BS9["COMMIT (Final)"]
end
end
BCreate --> BP1
BP3 -.->|loc_ch| BL1
BL7 -.->|geo_ch| BG1
BG9 -.->|exp_ch| BS2
BL7 & BG9 & BS9 --> BrewJoin["Orchestrator: Join LLM Worker, Geocode Worker, SQLite Worker<br/>brewery_pool_ is now fully populated"]
BrewJoin --> BeerPhase["Orchestrator: RunBeerPhase()"]
BeerPhase --> BeCreate["Create BoundedChannels (brew_ch, exp_ch)"]
subgraph BEERGEN["Phase 1b — Beer Generation (concurrent)"]
direction LR
subgraph BE_PROD["Orchestrator (producer)"]
BeP1["Loop: send Breweries → brew_ch"]
BeP1 --> BeP2["Close brew_ch"]
end
subgraph BE_LLM["LLM Worker"]
BeL1{"brew_ch has items?"}
BeL1 -- yes --> BeL2["Receive GeneratedBrewery"]
BeL2 --> BeL3["IBeerSelectionStrategy.SelectStyles(brewery, beer_style_palette_)"]
BeL3 --> BeL4{"For each selected BeerStyle"}
BeL4 -- remaining --> BeL5["GetStyleContextFromCache(style) — guaranteed cache hit; KV cache stays warm across all beer generations in this phase"]
BeL5 --> BeL6["GenerateBeer(brewery, style_context) via DataGenerator"]
BeL6 --> BeL7["Attach GeneratedBeer to bundle"] --> BeL4
BeL4 -- done --> BeL8["PipelineLogger.Log(Info, BeerGeneration, city, brewery_id, llm)"]
BeL8 --> BeL9["Send BeersBundle → exp_ch"]
BeL9 --> BeL1
BeL1 -- no --> BeL10["Close exp_ch"]
end
subgraph BE_SQL["SQLite Worker"]
BeS1["BEGIN TRANSACTION"] --> BeS2{"exp_ch has items?"}
BeS2 -- yes --> BeS3["Receive BeersBundle"]
BeS3 --> BeS4{"For each beer in bundle"}
BeS4 -- remaining --> BeS5["Set beer.brewery_id from bundle"]
BeS5 --> BeS6["ProcessBeer(beer)"]
BeS6 --> BeS7["Append → beer_pool_"] --> BeS4
BeS4 -- done --> BeS8["PipelineLogger.Log(Info, BeerGeneration, city, brewery_id, sqlite)"]
BeS8 --> BeS9{"Batch size reached?"}
BeS9 -- yes --> BeS10["COMMIT and BEGIN"] --> BeS2
BeS9 -- no --> BeS2
BeS2 -- no --> BeS11["COMMIT (Final)"]
end
end
BeCreate --> BeP1
BeP2 -.->|brew_ch| BeL1
BeL10 -.->|exp_ch| BeS2
BeL10 & BeS11 --> BeerJoin["Orchestrator: Join LLM Worker, SQLite Worker<br/>brewery_pool_ and beer_pool_ are now completely populated — Checkin and Follow phases may run in parallel"]
BeerJoin --> CheckinPhase["Orchestrator: RunCheckinPhase()"]
BeerJoin --> FollowPhase["Orchestrator: RunFollowPhase()"]
subgraph CHECKIN["Checkin Generation"]
CK1["ICheckinDistributionStrategy.AssignActivityWeights(user_pool_)<br/>weights seeded from persona.checkin_weight; J-curve profile emerges from persona distribution"]
CK1 --> CK2["BEGIN TRANSACTION"]
CK2 --> CK3{"For each GeneratedUser in user_pool_"}
CK3 -- remaining --> CK4["CheckinsForUser(user, brewery_pool_.size())"]
CK4 --> CK5{"For each checkin index"}
CK5 -- remaining --> CK6["TimestampFor(user, index)"]
CK6 --> CK7["Select brewery from brewery_pool_"]
CK7 --> CK8["GenerateCheckin(user, brewery, timestamp) via DataGenerator"]
CK8 --> CK9["ProcessCheckin(checkin)"]
CK9 --> CK10["PipelineLogger.Log(Info, CheckinGeneration, nullopt, checkin_id, sqlite)"]
CK10 --> CK11["Append → checkin_pool_"]
CK11 --> CK12{"Batch size reached?"}
CK12 -- yes --> CK13["COMMIT and BEGIN"] --> CK5
CK12 -- no --> CK5
CK5 -- done --> CK3
CK3 -- done --> CK14["COMMIT (Final)"]
end
subgraph FOLLOW["Follow Generation"]
FL1["IFollowGenerationStrategy.AssignFollowWeights(user_pool_)<br/>RandomFollowStrategy: uniform weights; ActivityWeightedFollowStrategy: derived from user.activity_weight so high-activity users attract more followers"]
FL1 --> FL2["BEGIN TRANSACTION"]
FL2 --> FL3["IFollowGenerationStrategy.GenerateFollows(user_pool_)<br/>self-follow constraint (follower_id != followed_id) enforced here and at the DB schema level"]
FL3 --> FL4{"For each GeneratedFollow"}
FL4 -- remaining --> FL5["ProcessFollow(follow)"]
FL5 --> FL6["PipelineLogger.Log(Info, FollowGeneration, nullopt, follower_id, sqlite)"]
FL6 --> FL7["Append → follow_pool_"]
FL7 --> FL8{"Batch size reached?"}
FL8 -- yes --> FL9["COMMIT and BEGIN"] --> FL4
FL8 -- no --> FL4
FL4 -- done --> FL10["COMMIT (Final)"]
end
CheckinPhase --> CK1
FollowPhase --> FL1
CK14 & FL10 --> PhaseJoin["Orchestrator: Join CheckinPhase, FollowPhase<br/>checkin_pool_ and follow_pool_ are now fully populated"]
PhaseJoin --> RatingPhase["Orchestrator: RunRatingPhase()<br/>beer selection biased by user.persona.style_affinities and abv_range; rating skew modulated per persona"]
RatingPhase --> RT1["BEGIN TRANSACTION"]
RT1 --> RT2{"For each GeneratedCheckin in checkin_pool_"}
RT2 -- remaining --> RT3["Match brewery_id, select beer from beer_pool_ (same brewery_id, biased by persona affinities)"]
RT3 --> RT4{"Beer exists for brewery?"}
RT4 -- yes --> RT5["GenerateRating(user, beer, checkin_id) via DataGenerator"]
RT5 --> RT6["ProcessRating(rating)"]
RT6 --> RT7["PipelineLogger.Log(Info, RatingGeneration, nullopt, rating_id, sqlite)"]
RT7 --> RT8{"Batch size reached?"}
RT8 -- yes --> RT9["COMMIT and BEGIN"] --> RT2
RT8 -- no --> RT2
RT4 -- no --> RT10["Log warning; skip — brewery has no beers"] --> RT2
RT2 -- done --> RT11["COMMIT (Final)"]
RT11 --> Teardown["Orchestrator: Finalize SqliteExportService — safely closes the DB connection"]
Teardown --> CloseLogCh["Orchestrator: close log_ch"]
CloseLogCh --> MainDone["Main: spdlog::info 'Pipeline complete in X ms'"]
MainDone --> JoinLog["Main: Join Log Worker — drain guarantees no LogEntry is dropped at shutdown"]
JoinLog --> Stop2([stop])

372
docs/pipeline/diagrams/planned/activity.puml
View File

@@ -1,372 +0,0 @@
@startuml biergarten_activity
!include ../biergarten-weizen-theme.puml
skinparam defaultFontSize 13
skinparam titleFontSize 20
title The Biergarten Data Pipeline — Activity Diagram
|Main|
start
:ParseArguments(argc, argv);
if (Invalid args?) then (yes)
:spdlog::error;
stop
else (no)
endif
:Init OpenSSL global state & LlamaBackendState;
:Build DI injector;
:Initialize SqliteExportService;
note right
Opens SQLite connection.
(Transactions are now managed
per-phase via batching).
end note
:Create BoundedChannel<LogEntry> log_ch;
:Spawn Log Worker thread;
note right
Log worker drains log_ch for the
entire pipeline lifetime.
All workers emit LogEntry structs
via PipelineLogger -- never spdlog directly.
end note
:BiergartenPipelineOrchestrator::Run();
|BiergartenPipelineOrchestrator::Run()|
fork
:JsonLoader::LoadBeerStyles("beer-styles.json");
:EnrichmentService::PreWarmBeerStyleCache(beer_styles);
fork again
:JsonLoader::LoadLocations("locations.json");
:EnrichmentService::PreWarmLocationCache(sampled_locations);
end fork
fork
:JsonLoader::LoadForenamesByCountry(\n "forenames-by-country.json");
fork again
:JsonLoader::LoadSurnamesByCountry(\n "surnames-by-country.json");
fork again
:JsonLoader::LoadPersonas("personas.json");
end fork
note right
Each call is memoized after its first parse
(implemented today). MockCuratedDataService
returns a fixed in-memory dataset instead
under --mocked, skipping these JSON files.
end note
' ═══════════════════════════════════════════
' PHASE 0 — USER GENERATION
' ═══════════════════════════════════════════
|Orchestrator|
:RunUserPhase(sampled_locations);
:Create BoundedChannels\n(loc_ch, exp_ch);
fork
|Orchestrator|
:Loop: Send Locations -> loc_ch;
:Close loc_ch;
note right
Producer closes loc_ch.
LLM Worker while loop
terminates on empty + closed.
end note
fork again
|LLM Worker|
while (loc_ch has items?) is (yes)
:Receive Location;
:GetLocationContextFromCache(location);
note right
Guaranteed cache hit from startup.
end note
:IPersonaSelectionStrategy::SelectPersona(\n personas_palette_);
note right
Guaranteed cache hit from startup.
Returns a Persona struct carrying
style_affinities, abv_range,
ibu_preference, checkin_weight.
end note
:SampleName(forenames_by_country,\n surnames_by_country, location.iso3166_1, rng);
note right
Deterministic lookup -- no LLM involved.
Free function (implemented today in
generate_users.cc), not a class method.
Name selected from pre-keyed maps
and passed into the generation prompt.
Skips the city if either map has no
entry for iso3166_1.
end note
:GenerateUser(enriched_city, persona, sampled_name)\nvia DataGenerator;
note right
LLM receives: EnrichedCity context + persona
description + sampled name. Generates
bio and preference signals grounded
in locale and persona.
end note
:PipelineLogger::Log(Info, UserGeneration,\n city, user_id, "llm");
:Send GeneratedUser -> exp_ch;
endwhile (no)
:Close exp_ch;
note right
Producer closes exp_ch.
SQLite Worker while loop
terminates on empty + closed.
end note
fork again
|SQLite Worker|
:BEGIN TRANSACTION;
while (exp_ch has items?) is (yes)
:Receive GeneratedUser;
:ProcessUser(user);
:PipelineLogger::Log(Info, UserGeneration,\n city, user_id, "sqlite");
:Append -> user_pool_;
if (Batch size reached?) then (yes)
:COMMIT & BEGIN;
else (no)
endif
endwhile (no)
:COMMIT (Final);
end fork
|Orchestrator|
:Join LLM Worker, SQLite Worker;
' ═══════════════════════════════════════════
' PHASE 1a — BREWERY GENERATION
' ═══════════════════════════════════════════
:RunBreweryPhase(sampled_locations);
:Create BoundedChannels\n(loc_ch, exp_ch);
fork
|Orchestrator|
:Loop: Sample User from user_pool_
and pair with Location;
:Send BreweryTask(Location, User) -> loc_ch;
:Close loc_ch;
fork again
|LLM Worker|
while (loc_ch has items?) is (yes)
:Receive BreweryTask(Location, User);
:GetLocationContextFromCache(task.location);
note right
Guaranteed cache hit from startup.
end note
:GenerateBrewery(enriched_city, context, task.user)\nvia DataGenerator;
note right
KV cache stays warm.
Brewery is linked to the sampled owner_user_id.
end note
:PipelineLogger::Log(Info,\n BreweryGeneration,\n city, brewery_id, "llm");
:Send GeneratedBrewery -> exp_ch;
endwhile (no)
:Close exp_ch;
fork again
|SQLite Worker|
:BEGIN TRANSACTION;
while (exp_ch has items?) is (yes)
:Receive GeneratedBrewery;
:ProcessBrewery(brewery);
:PipelineLogger::Log(Info,\n BreweryGeneration,\n city, brewery_id, "sqlite");
:Append -> brewery_pool_;
if (Batch size reached?) then (yes)
:COMMIT & BEGIN;
else (no)
endif
endwhile (no)
:COMMIT (Final);
end fork
|Orchestrator|
:Join LLM Worker, SQLite Worker;
note right
brewery_pool_ is now fully populated.
Phase 1b may begin.
end note
' ═══════════════════════════════════════════
' PHASE 1b — BEER GENERATION
' ═══════════════════════════════════════════
:RunBeerPhase();
:Create BoundedChannels\n(brew_ch, exp_ch);
fork
|Orchestrator|
:Loop: Send Breweries -> brew_ch;
:Close brew_ch;
fork again
|LLM Worker|
while (brew_ch has items?) is (yes)
:Receive GeneratedBrewery;
:IBeerSelectionStrategy::SelectStyles(\n brewery, beer_style_palette_);
while (For each selected BeerStyle?) is (remaining)
:GetStyleContextFromCache(style);
note right
Guaranteed cache hit from startup.
KV cache stays warm across all
beer generations -- system prompt
does not change within this phase.
end note
:GenerateBeer(brewery, style_context)\nvia DataGenerator;
:Attach GeneratedBeer to bundle;
endwhile (done)
:PipelineLogger::Log(Info,\n BeerGeneration,\n city, brewery_id, "llm");
:Send BeersBundle -> exp_ch;
endwhile (no)
:Close exp_ch;
fork again
|SQLite Worker|
:BEGIN TRANSACTION;
while (exp_ch has items?) is (yes)
:Receive BeersBundle;
while (For each beer in bundle?) is (remaining)
:Set beer.brewery_id from bundle;
:ProcessBeer(beer);
:Append -> beer_pool_;
endwhile (done)
:PipelineLogger::Log(Info,\n BeerGeneration,\n city, brewery_id, "sqlite");
if (Batch size reached?) then (yes)
:COMMIT & BEGIN;
else (no)
endif
endwhile (no)
:COMMIT (Final);
end fork
|Orchestrator|
:Join LLM Worker, SQLite Worker;
note right
Both brewery_pool_ and beer_pool_
are now completely populated.
Checkin and Follow phases may
now run in parallel.
end note
' ═══════════════════════════════════════════
' PHASE 2 — CHECKIN + FOLLOW GENERATION
' (parallel — both depend only on user_pool_
' and brewery_pool_ being fully populated)
' ═══════════════════════════════════════════
fork
|Orchestrator|
:RunCheckinPhase();
:ICheckinDistributionStrategy::\nAssignActivityWeights(user_pool_);
note right
Weights seeded from each user's
persona.checkin_weight. J-curve profile
emerges from persona distribution.
end note
:BEGIN TRANSACTION;
while (For each GeneratedUser in user_pool_?) is (remaining)
:CheckinsForUser(user, brewery_pool_.size());
while (For each checkin index?) is (remaining)
:TimestampFor(user, index);
:Select brewery from brewery_pool_;
:GenerateCheckin(user, brewery, timestamp)\nvia DataGenerator;
:ProcessCheckin(checkin);
:PipelineLogger::Log(Info, CheckinGeneration,\n nullopt, checkin_id, "sqlite");
:Append -> checkin_pool_;
if (Batch size reached?) then (yes)
:COMMIT & BEGIN;
else (no)
endif
endwhile (done)
endwhile (done)
:COMMIT (Final);
fork again
|Orchestrator|
:RunFollowPhase();
:IFollowGenerationStrategy::\nAssignFollowWeights(user_pool_);
note right
For RandomFollowStrategy, weights
are uniform. For ActivityWeightedFollowStrategy,
weights derived from user.activity_weight
so high-activity users attract more followers.
end note
:BEGIN TRANSACTION;
:IFollowGenerationStrategy::\nGenerateFollows(user_pool_);
note right
Self-follow constraint (follower_id != followed_id)
enforced here and at the DB schema level.
end note
while (For each GeneratedFollow?) is (remaining)
:ProcessFollow(follow);
:PipelineLogger::Log(Info, FollowGeneration,\n nullopt, follower_id, "sqlite");
:Append -> follow_pool_;
if (Batch size reached?) then (yes)
:COMMIT & BEGIN;
else (no)
endif
endwhile (done)
:COMMIT (Final);
end fork
|Orchestrator|
:Join CheckinPhase, FollowPhase;
note right
checkin_pool_ and follow_pool_
are now fully populated.
Rating phase may begin.
end note
' ═══════════════════════════════════════════
' PHASE 3 — RATING GENERATION
' ═══════════════════════════════════════════
:RunRatingPhase();
note right
Beer selection biased by
user.persona.style_affinities and abv_range.
Rating skew modulated per persona.
end note
:BEGIN TRANSACTION;
while (For each GeneratedCheckin in checkin_pool_?) is (remaining)
:Match brewery_id, select beer from beer_pool_\n(same brewery_id, biased by persona affinities);
if (Beer exists for brewery?) then (yes)
:GenerateRating(user, beer, checkin_id)\nvia DataGenerator;
:ProcessRating(rating);
:PipelineLogger::Log(Info, RatingGeneration,\n nullopt, rating_id, "sqlite");
if (Batch size reached?) then (yes)
:COMMIT & BEGIN;
else (no)
endif
else (no)
:PipelineLogger::Log(Warn, RatingGeneration,\n nullopt, brewery_id, "sqlite");
:Skip -- brewery has no beers;
endif
endwhile (done)
:COMMIT (Final);
' ═══════════════════════════════════════════
' TEARDOWN
' ═══════════════════════════════════════════
|Orchestrator|
:Finalize SqliteExportService;
note right
Safely closes the DB connection.
end note
:Close log_ch;
|Main|
:spdlog::info "Pipeline complete in X ms";
:Join Log Worker;
note right
Drain guarantees no LogEntry is
dropped at shutdown.
end note
stop
@enduml

17
docs/pipeline/diagrams/planned/class.md Normal file
View File

@@ -0,0 +1,17 @@
# Biergarten Data Pipeline — Class Diagram (Planned)
```mermaid
--8<-- "pipeline/diagrams/planned/class.mmd"
```
## Notes
- `Logger`, `PipelineLogger`, and `LogWorker` are referenced only in
relationships in the source PlantUML (not given their own class blocks there
either) — they appear here as implicit nodes for the same reason.
- Package grouping from the PlantUML source (`Domain: Models`,
`Domain: Application Configuration`, `Domain: Policy`,
`Infrastructure: Logging`, `Infrastructure: Pipeline Channel`,
`Infrastructure: Data Preloading`, `Infrastructure: Enrichment`,
`Infrastructure: Prompting`, `Infrastructure: Data Generation`,
`Infrastructure: Data Export`) is preserved as Mermaid `namespace` blocks.

643
docs/pipeline/diagrams/planned/class.mmd Normal file
View File

@@ -0,0 +1,643 @@
classDiagram
namespace DomainModels {
class Location {
+string city
+string state_province
+string iso3166_2
+string country
+string iso3166_1
+vector~string~ local_languages
+double latitude
+double longitude
}
class LocationContext {
+string text
+Completeness completeness
+size_t char_count
}
class Completeness {
<<enumeration>>
Full
Partial
Absent
}
class EnrichedCity {
+Location location
+LocationContext context
}
class BeerStyle {
+string name
+string description
+float min_abv
+float max_abv
+int min_ibu
+int max_ibu
}
class BreweryResult {
+string name_en
+string description_en
+string name_local
+string description_local
}
class BeerResult {
+string name_en
+string description_en
+string name_local
+string description_local
+string style
+float abv
+int ibu
}
class UserResult {
+string first_name
+string last_name
+string gender
+string username
+string bio
+float activity_weight
}
class Name {
+string first_name
+string last_name
+string gender
}
class ForenameEntry {
+string name
+string gender
}
class CheckinResult {
+string checked_in_at
+string note
}
class RatingResult {
+float score
+string note
}
class GenerationMetadata {
+uint64_t generation_id
+string generated_time
+bool context_provided
+string generated_with
}
class GeneratedBrewery {
+uint64_t brewery_id
+Location location
+BreweryResult brewery
+Completeness context_completeness
+GenerationMetadata metadata
}
class GeneratedBeer {
+uint64_t beer_id
+uint64_t brewery_id
+Location location
+BeerStyle style
+BeerResult beer
+GenerationMetadata metadata
}
class GeneratedUser {
+uint64_t user_id
+Location location
+UserResult user
+string email
+string date_of_birth
+string password
+GenerationMetadata metadata
}
class GeneratedCheckin {
+uint64_t checkin_id
+uint64_t user_id
+uint64_t brewery_id
+CheckinResult checkin
+GenerationMetadata metadata
}
class GeneratedRating {
+uint64_t user_id
+uint64_t beer_id
+uint64_t checkin_id
+RatingResult rating
+GenerationMetadata metadata
}
class GeneratedFollow {
+uint64_t follower_id
+uint64_t followed_id
+GenerationMetadata metadata
}
class UserPersona {
+string name
+string description
+vector~string~ style_affinities
}
}
note for ForenameEntry "forename_list/surname_list are unordered_set aliases, keyed by ISO 3166-1 country code via ICuratedDataService. SampleName() is a free function, not a method, sourced from popular-names-by-country-dataset (CC0)."
note for GeneratedUser "email, date_of_birth, and password are generated programmatically by the orchestrator (not LLM-authored) so a downstream auth-account seeding consumer can register real accounts from this export."
ForenameEntry ..> Name : SampleName() produces
LocationContext *-- Completeness
namespace DomainApplicationConfiguration {
class SamplingOptions {
+float temperature
+float top_p
+uint32_t top_k
+uint32_t n_ctx
+int seed
}
class GeneratorMode {
<<enumeration>>
kMock
kLlama
kOpenAI
}
class GeneratorOptions {
+path model_path
+GeneratorMode mode
+optional~SamplingOptions~ sampling
+string openai_api_key
+string openai_model
}
class PipelineOptions {
+path output_path
+path log_path
}
class ApplicationOptions {
+GeneratorOptions generator
+PipelineOptions pipeline
}
}
note for GeneratorOptions "Already implemented (models.h), not planned: the three-way GeneratorMode split (kMock/kLlama/kOpenAI) and the openai_api_key/openai_model fields ship today, ahead of the rest of this diagram."
ApplicationOptions *-- GeneratorOptions
ApplicationOptions *-- PipelineOptions
GeneratorOptions *-- GeneratorMode
GeneratorOptions o-- SamplingOptions
namespace DomainPolicy {
class ContextStrategy {
<<interface>>
+QueriesFor(loc) vector~string~
+MaxContextChars() size_t
}
class BreweryContextStrategy {
+QueriesFor(loc) vector~string~
+MaxContextChars() size_t
}
class BeerContextStrategy {
+QueriesFor(loc) vector~string~
+MaxContextChars() size_t
}
class SamplingStrategy {
<<interface>>
+Sample(locations) vector~Location~
}
class UniformSamplingStrategy {
-size_t sample_size_
+Sample(locations) vector~Location~
}
class BeerSelectionStrategy {
<<interface>>
+SelectStyles(brewery, palette) vector~BeerStyle~
}
class RandomBeerSelectionStrategy {
-mt19937 rng_
-size_t min_beers_
-size_t max_beers_
+SelectStyles(brewery, palette) vector~BeerStyle~
}
class CheckinDistributionStrategy {
<<interface>>
+AssignActivityWeights(users) void
+CheckinsForUser(user, brewery_count) size_t
+TimestampFor(user, index) string
}
class JCurveCheckinStrategy {
-mt19937 rng_
+AssignActivityWeights(users) void
+CheckinsForUser(user, brewery_count) size_t
+TimestampFor(user, index) string
}
class RandomCheckinStrategy {
-mt19937 rng_
-size_t min_checkins_
-size_t max_checkins_
+AssignActivityWeights(users) void
+CheckinsForUser(user, brewery_count) size_t
+TimestampFor(user, index) string
}
class FollowGenerationStrategy {
<<interface>>
+GenerateFollows(users) vector~GeneratedFollow~
}
class RandomFollowStrategy {
-mt19937 rng_
-size_t min_follows_
-size_t max_follows_
+GenerateFollows(users) vector~GeneratedFollow~
}
class ActivityWeightedFollowStrategy {
-mt19937 rng_
-size_t min_follows_
-size_t max_follows_
+GenerateFollows(users) vector~GeneratedFollow~
}
}
namespace InfrastructureLogging {
class LogLevel {
<<enumeration>>
Debug
Info
Warn
Error
}
class PipelinePhase {
<<enumeration>>
Startup
UserGeneration
BreweryAndBeerGeneration
CheckinGeneration
RatingGeneration
FollowGeneration
Teardown
}
class LogEntry {
+time_point timestamp
+LogLevel level
+PipelinePhase phase
+string message
+optional~string~ worker
}
class ILogger {
<<interface>>
+Log(entry) void
}
class LogProducer {
-BoundedChannel~LogEntry~ channel_
+Log(entry) void
}
class LogDispatcher {
-BoundedChannel~LogEntry~ channel_
+Run() void
-ToSpdlogLevel(level) level_enum
}
}
LogEntry *-- LogLevel
LogEntry *-- PipelinePhase
ILogger <|.. LogProducer
LogProducer ..> LogEntry : emits
LogDispatcher ..> LogEntry : consumes
namespace InfrastructurePipelineChannel {
class BoundedChannel~T~ {
-queue~T~ queue_
-mutex mutex_
-condition_variable not_full_
-condition_variable not_empty_
-size_t capacity_
-bool closed_
+Send(item) void
+Receive() optional~T~
+Close() void
}
}
namespace InfrastructureDataPreloading {
class ICuratedDataService {
<<interface>>
+LoadLocations(filepath) vector~Location~
+LoadBeerStyles(filepath) vector~BeerStyle~
+LoadPersonas(filepath) vector~UserPersona~
+LoadForenamesByCountry(filepath) unordered_map~string, forename_list~
+LoadSurnamesByCountry(filepath) unordered_map~string, surname_list~
}
class JsonLoader {
+LoadLocations(filepath) vector~Location~
+LoadBeerStyles(filepath) vector~BeerStyle~
+LoadPersonas(filepath) vector~UserPersona~
+LoadForenamesByCountry(filepath) unordered_map~string, forename_list~
+LoadSurnamesByCountry(filepath) unordered_map~string, surname_list~
}
class MockCuratedDataService {
+LoadLocations(filepath) vector~Location~
+LoadBeerStyles(filepath) vector~BeerStyle~
+LoadPersonas(filepath) vector~UserPersona~
+LoadForenamesByCountry(filepath) unordered_map~string, forename_list~
+LoadSurnamesByCountry(filepath) unordered_map~string, surname_list~
}
}
note for JsonLoader "Each Load* call is memoized after its first parse — implemented today, ahead of LoadBeerStyles which is still planned."
note for MockCuratedDataService "Fixed in-memory dataset for --mocked mode, implemented today for Locations/Personas/Forenames/Surnames; would need a LoadBeerStyles fixture too once that method is implemented."
namespace InfrastructureEnrichment {
class EnrichmentService {
<<interface>>
+GetLocationContext(loc, strategy) LocationContext
}
class WikipediaService {
-unique_ptr~WebClient~ client_
-unordered_map~string, string~ extract_cache_
+GetLocationContext(loc, strategy) LocationContext
-FetchExtract(query) string
}
class WebClient {
<<interface>>
+Get(url) string
+UrlEncode(value) string
}
class HttpWebClient {
+Get(url) string
+UrlEncode(value) string
}
}
namespace InfrastructureGeocoding {
class AddressLookupResult {
+string address_line1
+string postal_code
}
class AddressService {
<<interface>>
+ReverseGeocode(longitude, latitude) optional~AddressLookupResult~
}
class NominatimAddressService {
-unique_ptr~WebClient~ client_
-Logger logger_
+ReverseGeocode(longitude, latitude) optional~AddressLookupResult~
}
class MockAddressService {
+ReverseGeocode(longitude, latitude) optional~AddressLookupResult~
}
}
note for NominatimAddressService "Already implemented (services/address/), not planned: queries the public Nominatim API, sleeping 1s per request per its usage policy; returns nullopt on failure or when no usable street address is found. Ships today, ahead of the rest of this diagram."
note for MockAddressService "Already implemented, not planned: returns a fixed placeholder address ('123 Mock Street', '00000') for --mocked runs, no network call."
AddressService <|.. NominatimAddressService
AddressService <|.. MockAddressService
AddressService ..> AddressLookupResult : returns
NominatimAddressService *-- WebClient
namespace InfrastructurePrompting {
class IPromptDirectory {
<<interface>>
+Load(key) string
}
class PromptDirectory {
-path prompt_dir_
-unordered_map~string, string~ cache_
+PromptDirectory(prompt_dir)
+Load(key) string
}
}
IPromptDirectory <|.. PromptDirectory
namespace InfrastructureDataGeneration {
class DataGenerator {
<<interface>>
+GenerateBrewery(location, context) BreweryResult
+GenerateBeer(brewery_id, location, context, style) BeerResult
+GenerateUser(city, persona, name) UserResult
+GenerateCheckin(user, brewery, timestamp) CheckinResult
+GenerateRating(user, beer, checkin_id) RatingResult
}
class MockGenerator {
+GenerateBrewery(...) BreweryResult
+GenerateBeer(...) BeerResult
+GenerateUser(...) UserResult
+GenerateCheckin(...) CheckinResult
+GenerateRating(...) RatingResult
-DeterministicHash(location) size_t
}
class LlamaGenerator {
-ModelHandle model_
-ContextHandle context_
-unique_ptr~PromptFormatter~ prompt_formatter_
-unique_ptr~IPromptDirectory~ prompt_directory_
-mt19937 rng_
+GenerateBrewery(...) BreweryResult
+GenerateBeer(...) BeerResult
+GenerateUser(...) UserResult
+GenerateCheckin(...) CheckinResult
+GenerateRating(...) RatingResult
-Load(opts) void
-Infer(system_prompt, user_prompt, max_tokens, grammar) string
-ValidateModelArchitecture() void
}
class PromptFormatter {
<<interface>>
+Format(system_prompt, user_prompt) string
+ExpectedArchitecture() string_view
}
class Gemma4JinjaPromptFormatter {
+Format(system_prompt, user_prompt) string
+ExpectedArchitecture() string_view
}
class OpenAIGenerator {
-string api_key_
-string model_
-Logger logger_
-unique_ptr~IPromptDirectory~ prompt_directory_
-unique_ptr~WebClient~ web_client_
+GenerateBrewery(location, context) BreweryResult
+GenerateBeer(brewery_id, location, context, style) BeerResult
+GenerateUser(city, persona, name) UserResult
+GenerateCheckin(user, brewery, timestamp) CheckinResult
+GenerateRating(user, beer, checkin_id) RatingResult
-CallChatCompletionsApi(...) string
}
}
note for OpenAIGenerator "Already implemented (data_generation/openai/), not planned: calls the OpenAI Chat Completions API with Structured Outputs (json_schema, strict) to guarantee schema-valid JSON, needing no IPromptFormatter or GBNF grammar. Ships today for GenerateBrewery/GenerateUser only, ahead of the rest of this diagram — GenerateBeer/GenerateCheckin/GenerateRating are shown here for interface completeness but not yet implemented on any backend."
namespace InfrastructureDataExport {
class ExportService {
<<interface>>
+Initialize() void
+ProcessBrewery(brewery) uint64_t
+ProcessBeer(beer) uint64_t
+ProcessUser(user) uint64_t
+ProcessCheckin(checkin) uint64_t
+ProcessRating(rating) void
+ProcessFollow(follow) void
+Finalize() void
}
class SqliteExportService {
-unique_ptr~DateTimeProvider~ date_time_provider_
-SqliteDatabaseHandle db_handle_
-SqliteStatementHandle insert_location_stmt_
-SqliteStatementHandle insert_brewery_stmt_
-SqliteStatementHandle insert_beer_stmt_
-SqliteStatementHandle insert_user_stmt_
-SqliteStatementHandle insert_checkin_stmt_
-SqliteStatementHandle insert_rating_stmt_
-SqliteStatementHandle insert_follow_stmt_
-bool transaction_open_
-unordered_map~string, uint64_t~ location_cache_
-unordered_map~string, uint64_t~ brewery_cache_
+Initialize() void
+ProcessRecord(brewery) uint64_t
+ProcessRecord(beer) uint64_t
+ProcessRecord(user) uint64_t
+ProcessRecord(checkin) uint64_t
+ProcessRecord(rating) void
+ProcessRecord(follow) void
+Finalize() void
-InitializeSchema() void
-PrepareStatements() void
-RollbackAndCloseNoThrow() void
-FinalizeStatements() void
}
class DateTimeProvider {
<<interface>>
+GetUtcTimestamp() string
}
class SystemDateTimeProvider {
+GetUtcTimestamp() string
}
}
class BiergartenPipelineOrchestrator {
-unique_ptr~ICuratedDataService~ curated_data_service_
-unique_ptr~EnrichmentService~ enrichment_service_
-unique_ptr~DataGenerator~ generator_
-unique_ptr~Logger~ logger_
-unique_ptr~ExportService~ exporter_
-unique_ptr~AddressService~ address_service_
-unique_ptr~ContextStrategy~ brewery_context_strategy_
-unique_ptr~SamplingStrategy~ sampling_strategy_
-unique_ptr~BeerSelectionStrategy~ beer_selection_strategy_
-unique_ptr~CheckinDistributionStrategy~ checkin_strategy_
-unique_ptr~FollowGenerationStrategy~ follow_strategy_
-vector~BeerStyle~ beer_style_palette_
-ApplicationOptions options_
-vector~GeneratedUser~ user_pool_
-vector~GeneratedBrewery~ brewery_pool_
-vector~GeneratedBeer~ beer_pool_
-vector~GeneratedCheckin~ checkin_pool_
-vector~GeneratedFollow~ follow_pool_
+Run() bool
-RunUserPhase(locations) void
-RunBreweryAndBeerPhase(locations) void
-RunCheckinPhase() void
-RunRatingPhase() void
-RunFollowPhase() void
}
BiergartenPipelineOrchestrator *-- ICuratedDataService
BiergartenPipelineOrchestrator *-- EnrichmentService
BiergartenPipelineOrchestrator *-- DataGenerator
BiergartenPipelineOrchestrator *-- ExportService
BiergartenPipelineOrchestrator *-- AddressService
BiergartenPipelineOrchestrator *-- CheckinDistributionStrategy
BiergartenPipelineOrchestrator *-- FollowGenerationStrategy
BiergartenPipelineOrchestrator *-- SamplingStrategy
BiergartenPipelineOrchestrator *-- BeerSelectionStrategy
BiergartenPipelineOrchestrator *-- ApplicationOptions
BiergartenPipelineOrchestrator *-- Logger
BiergartenPipelineOrchestrator "1" *-- "0..*" GeneratedUser : user_pool_
BiergartenPipelineOrchestrator "1" *-- "0..*" GeneratedBrewery : brewery_pool_
BiergartenPipelineOrchestrator "1" *-- "0..*" GeneratedBeer : beer_pool_
BiergartenPipelineOrchestrator "1" *-- "0..*" GeneratedCheckin : checkin_pool_
BiergartenPipelineOrchestrator "1" *-- "0..*" GeneratedFollow : follow_pool_
ICuratedDataService <|.. JsonLoader
ICuratedDataService <|.. MockCuratedDataService
Logger <|.. PipelineLogger
ContextStrategy <|.. BreweryContextStrategy
ContextStrategy <|.. BeerContextStrategy
SamplingStrategy <|.. UniformSamplingStrategy
BeerSelectionStrategy <|.. RandomBeerSelectionStrategy
CheckinDistributionStrategy <|.. JCurveCheckinStrategy
CheckinDistributionStrategy <|.. RandomCheckinStrategy
FollowGenerationStrategy <|.. RandomFollowStrategy
FollowGenerationStrategy <|.. ActivityWeightedFollowStrategy
EnrichmentService <|.. WikipediaService
WebClient <|.. HttpWebClient
DataGenerator <|.. MockGenerator
DataGenerator <|.. LlamaGenerator
DataGenerator <|.. OpenAIGenerator
PromptFormatter <|.. Gemma4JinjaPromptFormatter
ExportService <|.. SqliteExportService
DateTimeProvider <|.. SystemDateTimeProvider
WikipediaService *-- WebClient
WikipediaService ..> ContextStrategy
LlamaGenerator *-- PromptFormatter
LlamaGenerator *-- IPromptDirectory
LlamaGenerator ..> GeneratorOptions
OpenAIGenerator *-- IPromptDirectory
OpenAIGenerator *-- WebClient
SqliteExportService *-- DateTimeProvider
PipelineLogger o-- BoundedChannel : logs to
LogWorker o-- BoundedChannel : drains from
EnrichedCity *-- Location
EnrichedCity *-- LocationContext
GeneratedBrewery *-- Location
GeneratedBrewery *-- BreweryResult
GeneratedBrewery *-- GenerationMetadata
GeneratedBeer *-- Location
GeneratedBeer *-- BeerStyle
GeneratedBeer *-- BeerResult
GeneratedBeer *-- GenerationMetadata
GeneratedUser *-- Location
GeneratedUser *-- UserResult
GeneratedUser *-- GenerationMetadata
GeneratedCheckin *-- CheckinResult
GeneratedCheckin *-- GenerationMetadata
GeneratedRating *-- RatingResult
GeneratedRating *-- GenerationMetadata
GeneratedFollow *-- GenerationMetadata

646
docs/pipeline/diagrams/planned/class.puml
View File

@@ -1,646 +0,0 @@
@startuml class_diagram
' ==========================================
' CONFIGURATION & STYLING
' ==========================================
!include ../biergarten-weizen-theme.puml
skinparam classAttributeFontSize 9
skinparam defaultFontSize 25
skinparam titleFontSize 30
title Biergarten Data Pipeline — Class Diagram
package "Domain: Models" {
class Location {
+ city : std::string
+ state_province : std::string
+ iso3166_2 : std::string
+ country : std::string
+ iso3166_1 : std::string
+ local_languages : std::vector<std::string>
+ latitude : double
+ longitude : double
}
class LocationContext {
+ text : std::string
+ completeness : Completeness
+ char_count : size_t
}
enum Completeness {
Full
Partial
Absent
}
class EnrichedCity {
+ location : Location
+ context : LocationContext
}
class BeerStyle {
+ name : std::string
+ description : std::string
+ min_abv : float
+ max_abv : float
+ min_ibu : int
+ max_ibu : int
}
class BreweryResult {
+ name_en : std::string
+ description_en : std::string
+ name_local : std::string
+ description_local : std::string
}
class BeerResult {
+ name_en : std::string
+ description_en : std::string
+ name_local : std::string
+ description_local : std::string
+ style : std::string
+ abv : float
+ ibu : int
}
class UserResult {
+ first_name : std::string
+ last_name : std::string
+ gender : std::string
+ username : std::string
+ bio : std::string
+ activity_weight : float
}
class Name {
+ first_name : std::string
+ last_name : std::string
+ gender : std::string
}
class ForenameEntry {
+ name : std::string
+ gender : std::string
}
note right of ForenameEntry
forename_list = std::unordered_set<ForenameEntry>
surname_list = std::unordered_set<std::string>
(aliases declared in curated_data_service.h).
ICuratedDataService::LoadForenamesByCountry() /
LoadSurnamesByCountry() each return a flat
std::unordered_map<std::string, forename_list>
or <std::string, surname_list> keyed by ISO
3166-1 country code -- no NamesByCountry
wrapper class. SampleName(forenames_by_country,
surnames_by_country, iso3166_1, rng) is a free
function, not a method, so it can be called by
any per-city worker without owning the maps.
Sourced from popular-names-by-country-dataset
(CC0) -- see ETHICS-AND-KNOWN-ISSUES.md. Gender
is preserved per forename so it can be threaded
through to persona/bio generation later; surnames
carry no gender in the source data. Pairing
happens at sample time, not at fixture-authoring
time, so no information from the source dataset
is discarded up front.
end note
ForenameEntry ..> Name : SampleName() produces
class CheckinResult {
+ checked_in_at : std::string
+ note : std::string
}
class RatingResult {
+ score : float
+ note : std::string
}
class GenerationMetadata {
+ generation_id : uint64_t
+ generated_time : std::string
+ context_provided : bool
+ generated_with : std::string
}
class GeneratedBrewery {
+ brewery_id : uint64_t
+ location : Location
+ brewery : BreweryResult
+ context_completeness : LocationContext::Completeness
+ metadata : GenerationMetadata
}
class GeneratedBeer {
+ beer_id : uint64_t
+ brewery_id : uint64_t
+ location : Location
+ style : BeerStyle
+ beer : BeerResult
+ metadata : GenerationMetadata
}
class GeneratedUser {
+ user_id : uint64_t
+ location : Location
+ user : UserResult
+ email : std::string
+ date_of_birth : std::string
+ password : std::string
+ metadata : GenerationMetadata
}
note right of GeneratedUser
email, date_of_birth, and password are
programmatically generated by the orchestrator
(not LLM-authored) so a downstream auth-account
seeding consumer can register real accounts from
this export. See ROADMAP.md §1.
end note
class GeneratedCheckin {
+ checkin_id : uint64_t
+ user_id : uint64_t
+ brewery_id : uint64_t
+ checkin : CheckinResult
+ metadata : GenerationMetadata
}
class GeneratedRating {
+ user_id : uint64_t
+ beer_id : uint64_t
+ checkin_id : uint64_t
+ rating : RatingResult
+ metadata : GenerationMetadata
}
class GeneratedFollow {
+ follower_id : uint64_t
+ followed_id : uint64_t
+ metadata : GenerationMetadata
}
class UserPersona {
+ name: std::string
+ description: std::string
+ style_affinities: std::vector<std::string>
}
LocationContext *-- Completeness
}
package "Domain: Application Configuration" {
class SamplingOptions {
+ temperature: float = 1.0F
+ top_p: float = 0.95F
+ top_k: uint32_t = 64
+ n_ctx: uint32_t = 8192
+ seed: int = -1
}
class GeneratorOptions {
+ model_path: std::filesystem::path
+ use_mocked: bool = false
+ sampling: std::optional<SamplingOptions>
}
class PipelineOptions {
+ output_path: std::filesystem::path
+ log_path: std::filesystem::path
}
class ApplicationOptions {
+ generator: GeneratorOptions
+ pipeline: PipelineOptions
}
ApplicationOptions *-- GeneratorOptions
ApplicationOptions *-- PipelineOptions
GeneratorOptions o-- SamplingOptions
}
package "Domain: Policy" {
interface ContextStrategy <<interface>> {
+ QueriesFor(loc : const Location&) : std::vector<std::string>
+ MaxContextChars() : size_t
}
class BreweryContextStrategy {
+ QueriesFor(loc : const Location&) : std::vector<std::string>
+ MaxContextChars() : size_t
}
class BeerContextStrategy {
+ QueriesFor(loc : const Location&) : std::vector<std::string>
+ MaxContextChars() : size_t
}
interface SamplingStrategy <<interface>> {
+ Sample(locations : const std::vector<Location>&) : std::vector<Location>
}
class UniformSamplingStrategy {
- sample_size_ : size_t
+ Sample(locations : const std::vector<Location>&) : std::vector<Location>
}
interface BeerSelectionStrategy <<interface>> {
+ SelectStyles(brewery : const GeneratedBrewery&,\n palette : std::span<const BeerStyle>) : std::vector<BeerStyle>
}
class RandomBeerSelectionStrategy {
- rng_ : std::mt19937
- min_beers_ : size_t
- max_beers_ : size_t
+ SelectStyles(brewery : const GeneratedBrewery&,\n palette : std::span<const BeerStyle>) : std::vector<BeerStyle>
}
interface CheckinDistributionStrategy <<interface>> {
+ AssignActivityWeights(users : std::vector<GeneratedUser>&) : void
+ CheckinsForUser(user : const GeneratedUser&,\n brewery_count : size_t) : size_t
+ TimestampFor(user : const GeneratedUser&,\n index : size_t) : std::string
}
class JCurveCheckinStrategy {
- rng_ : std::mt19937
+ AssignActivityWeights(users : std::vector<GeneratedUser>&) : void
+ CheckinsForUser(user : const GeneratedUser&,\n brewery_count : size_t) : size_t
+ TimestampFor(user : const GeneratedUser&,\n index : size_t) : std::string
}
class RandomCheckinStrategy {
- rng_ : std::mt19937
- min_checkins_ : size_t
- max_checkins_ : size_t
+ AssignActivityWeights(users : std::vector<GeneratedUser>&) : void
+ CheckinsForUser(user : const GeneratedUser&,\n brewery_count : size_t) : size_t
+ TimestampFor(user : const GeneratedUser&,\n index : size_t) : std::string
}
interface FollowGenerationStrategy <<interface>> {
+ GenerateFollows(users : const std::vector<GeneratedUser>&) : std::vector<GeneratedFollow>
}
class RandomFollowStrategy {
- rng_ : std::mt19937
- min_follows_ : size_t
- max_follows_ : size_t
+ GenerateFollows(users : const std::vector<GeneratedUser>&) : std::vector<GeneratedFollow>
}
class ActivityWeightedFollowStrategy {
- rng_ : std::mt19937
- min_follows_ : size_t
- max_follows_ : size_t
+ GenerateFollows(users : const std::vector<GeneratedUser>&) : std::vector<GeneratedFollow>
}
}
package "Infrastructure: Logging" {
enum LogLevel {
Debug
Info
Warn
Error
}
enum PipelinePhase {
Startup
UserGeneration
BreweryAndBeerGeneration
CheckinGeneration
RatingGeneration
FollowGeneration
Teardown
}
class LogEntry {
+ timestamp : std::chrono::system_clock::time_point
+ level : LogLevel
+ phase : PipelinePhase
+ message : std::string
+ worker : std::optional<std::string>
}
interface ILogger <<interface>> {
+ Log(entry : const LogEntry&) : void
}
class LogProducer {
- channel_ : BoundedChannel<LogEntry>&
+ Log(entry : const LogEntry&) : void
}
class LogDispatcher {
- channel_ : BoundedChannel<LogEntry>&
+ Run() : void
- ToSpdlogLevel(level) : spdlog::level::level_enum
}
LogEntry *-- LogLevel
LogEntry *-- PipelinePhase
ILogger <|.. LogProducer
LogProducer ..> LogEntry : emits
LogDispatcher ..> LogEntry : consumes
}
package "Infrastructure: Pipeline Channel" {
class "BoundedChannel<T>" as BoundedChannel {
- queue_ : std::queue<T>
- mutex_ : std::mutex
- not_full_ : std::condition_variable
- not_empty_ : std::condition_variable
- capacity_ : size_t
- closed_ : bool
+ Send(item : T) : void
+ Receive() : std::optional<T>
+ Close() : void
}
}
package "Infrastructure: Data Preloading" {
interface ICuratedDataService <<interface>> {
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
+ LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
}
class JsonLoader {
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
+ LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
}
note right of JsonLoader
Each Load* call is memoized after its first
parse -- implemented today, ahead of the rest
of this package (LoadBeerStyles is still planned).
end note
class MockCuratedDataService {
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
+ LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
}
note right of MockCuratedDataService
Fixed in-memory dataset used in --mocked mode,
implemented today for Locations/Personas/
Forenames/Surnames; would need a LoadBeerStyles
fixture too once that method is implemented.
end note
}
package "Infrastructure: Enrichment" {
interface EnrichmentService <<interface>> {
+ GetLocationContext(loc : const Location&,\n strategy : const ContextStrategy&) : LocationContext
}
class WikipediaService {
- client_ : std::unique_ptr<WebClient>
- extract_cache_ : std::unordered_map<std::string, std::string>
+ GetLocationContext(loc : const Location&,\n strategy : const ContextStrategy&) : LocationContext
- FetchExtract(query : std::string_view) : std::string
}
interface WebClient <<interface>> {
+ Get(url : const std::string&) : std::string
+ UrlEncode(value : const std::string&) : std::string
}
class HttpWebClient {
+ Get(url : const std::string&) : std::string
+ UrlEncode(value : const std::string&) : std::string
}
}
package "Infrastructure: Prompting" {
interface IPromptDirectory <<interface>> {
+ Load(key : std::string_view) : std::string
}
class PromptDirectory {
- prompt_dir_ : std::filesystem::path
- cache_ : std::unordered_map<std::string, std::string>
+ PromptDirectory(prompt_dir : const std::filesystem::path&)
+ Load(key : std::string_view) : std::string
}
IPromptDirectory <|.. PromptDirectory
}
package "Infrastructure: Data Generation" {
interface DataGenerator <<interface>> {
+ GenerateBrewery(location : const Location&,\n context : const LocationContext&) : BreweryResult
+ GenerateBeer(brewery_id : uint64_t,\n location : const Location&,\n context : const LocationContext&,\n style : const BeerStyle&) : BeerResult
+ GenerateUser(city : const EnrichedCity&,\n persona : const UserPersona&,\n name : const Name&) : UserResult
+ GenerateCheckin(user : const GeneratedUser&,\n brewery : const GeneratedBrewery&,\n timestamp : const std::string&) : CheckinResult
+ GenerateRating(user : const GeneratedUser&,\n beer : const GeneratedBeer&,\n checkin_id : uint64_t) : RatingResult
}
class MockGenerator {
+ GenerateBrewery(...) : BreweryResult
+ GenerateBeer(...) : BeerResult
+ GenerateUser(...) : UserResult
+ GenerateCheckin(...) : CheckinResult
+ GenerateRating(...) : RatingResult
- DeterministicHash(location : const Location&) : size_t
}
class LlamaGenerator {
- model_ : ModelHandle
- context_ : ContextHandle
- prompt_formatter_ : std::unique_ptr<PromptFormatter>
- prompt_directory_ : std::unique_ptr<IPromptDirectory>
- rng_ : std::mt19937
+ GenerateBrewery(...) : BreweryResult
+ GenerateBeer(...) : BeerResult
+ GenerateUser(...) : UserResult
+ GenerateCheckin(...) : CheckinResult
+ GenerateRating(...) : RatingResult
- Load(opts : const GeneratorOptions&) : void
- Infer(system_prompt, user_prompt,\n max_tokens, grammar) : std::string
- ValidateModelArchitecture() : void
}
interface PromptFormatter <<interface>> {
+ Format(system_prompt : std::string_view,\n user_prompt : std::string_view) : std::string
+ ExpectedArchitecture() : std::string_view
}
class Gemma4JinjaPromptFormatter {
+ Format(...) : std::string
+ ExpectedArchitecture() : std::string_view
}
}
package "Infrastructure: Data Export" {
interface ExportService <<interface>> {
+ Initialize() : void
+ ProcessBrewery(brewery : const GeneratedBrewery&) : uint64_t
+ ProcessBeer(beer : const GeneratedBeer&) : uint64_t
+ ProcessUser(user : const GeneratedUser&) : uint64_t
+ ProcessCheckin(checkin : const GeneratedCheckin&) : uint64_t
+ ProcessRating(rating : const GeneratedRating&) : void
+ ProcessFollow(follow : const GeneratedFollow&) : void
+ Finalize() : void
}
class SqliteExportService {
- date_time_provider_ : std::unique_ptr<DateTimeProvider>
- db_handle_ : SqliteDatabaseHandle
- insert_location_stmt_ : SqliteStatementHandle
- insert_brewery_stmt_ : SqliteStatementHandle
- insert_beer_stmt_ : SqliteStatementHandle
- insert_user_stmt_ : SqliteStatementHandle
- insert_checkin_stmt_ : SqliteStatementHandle
- insert_rating_stmt_ : SqliteStatementHandle
- insert_follow_stmt_ : SqliteStatementHandle
- transaction_open_ : bool
- location_cache_ : std::unordered_map<std::string, uint64_t>
- brewery_cache_ : std::unordered_map<std::string, uint64_t>
+ Initialize() : void
+ ProcessRecord(brewery : const GeneratedBrewery&) : uint64_t
+ ProcessRecord(beer : const GeneratedBeer&) : uint64_t
+ ProcessRecord(user : const GeneratedUser&) : uint64_t
+ ProcessRecord(checkin : const GeneratedCheckin&) : uint64_t
+ ProcessRecord(rating : const GeneratedRating&) : void
+ ProcessRecord(follow : const GeneratedFollow&) : void
+ Finalize() : void
- InitializeSchema() : void
- PrepareStatements() : void
- RollbackAndCloseNoThrow() : void
- FinalizeStatements() : void
}
interface DateTimeProvider <<interface>> {
+ GetUtcTimestamp() : std::string
}
class SystemDateTimeProvider {
+ GetUtcTimestamp() : std::string
}
}
class BiergartenPipelineOrchestrator {
- curated_data_service_ : std::unique_ptr<ICuratedDataService>
- enrichment_service_ : std::unique_ptr<EnrichmentService>
- generator_ : std::unique_ptr<DataGenerator>
- logger_ : std::unique_ptr<Logger>
- exporter_ : std::unique_ptr<ExportService>
- brewery_context_strategy_ : std::unique_ptr<ContextStrategy>
- sampling_strategy_ : std::unique_ptr<SamplingStrategy>
- beer_selection_strategy_ : std::unique_ptr<BeerSelectionStrategy>
- checkin_strategy_ : std::unique_ptr<CheckinDistributionStrategy>
- follow_strategy_ : std::unique_ptr<FollowGenerationStrategy>
- beer_style_palette_ : std::vector<BeerStyle>
- options_ : ApplicationOptions
--
- user_pool_ : std::vector<GeneratedUser>
- brewery_pool_ : std::vector<GeneratedBrewery>
- beer_pool_ : std::vector<GeneratedBeer>
- checkin_pool_ : std::vector<GeneratedCheckin>
- follow_pool_ : std::vector<GeneratedFollow>
--
+ Run() : bool
- RunUserPhase(locations : const std::vector<Location>&) : void
- RunBreweryAndBeerPhase(locations : const std::vector<Location>&) : void
- RunCheckinPhase() : void
- RunRatingPhase() : void
- RunFollowPhase() : void
}
' --- Orchestration Aggregations (Services & Strategies) ---
BiergartenPipelineOrchestrator *-- ICuratedDataService
BiergartenPipelineOrchestrator *-- EnrichmentService
BiergartenPipelineOrchestrator *-- DataGenerator
BiergartenPipelineOrchestrator *-- ExportService
BiergartenPipelineOrchestrator *-- CheckinDistributionStrategy
BiergartenPipelineOrchestrator *-- FollowGenerationStrategy
BiergartenPipelineOrchestrator *-- SamplingStrategy
BiergartenPipelineOrchestrator *-- BeerSelectionStrategy
BiergartenPipelineOrchestrator *-- ApplicationOptions
BiergartenPipelineOrchestrator *-- Logger
' --- Orchestration Aggregations (Data Pools) ---
BiergartenPipelineOrchestrator *-- "0..*" GeneratedUser : user_pool_
BiergartenPipelineOrchestrator *-- "0..*" GeneratedBrewery : brewery_pool_
BiergartenPipelineOrchestrator *-- "0..*" GeneratedBeer : beer_pool_
BiergartenPipelineOrchestrator *-- "0..*" GeneratedCheckin : checkin_pool_
BiergartenPipelineOrchestrator *-- "0..*" GeneratedFollow : follow_pool_
' --- Interfaces & Implementations ---
ICuratedDataService <|.. JsonLoader
ICuratedDataService <|.. MockCuratedDataService
Logger <|.. PipelineLogger
ContextStrategy <|.. BreweryContextStrategy
ContextStrategy <|.. BeerContextStrategy
SamplingStrategy <|.. UniformSamplingStrategy
BeerSelectionStrategy <|.. RandomBeerSelectionStrategy
CheckinDistributionStrategy <|.. JCurveCheckinStrategy
CheckinDistributionStrategy <|.. RandomCheckinStrategy
FollowGenerationStrategy <|.. RandomFollowStrategy
FollowGenerationStrategy <|.. ActivityWeightedFollowStrategy
EnrichmentService <|.. WikipediaService
WebClient <|.. HttpWebClient
DataGenerator <|.. MockGenerator
DataGenerator <|.. LlamaGenerator
PromptFormatter <|.. Gemma4JinjaPromptFormatter
ExportService <|.. SqliteExportService
DateTimeProvider <|.. SystemDateTimeProvider
' --- Service Compositions & Dependencies ---
WikipediaService *-- WebClient
WikipediaService ..> ContextStrategy
LlamaGenerator *-- PromptFormatter
LlamaGenerator *-- IPromptDirectory
LlamaGenerator ..> GeneratorOptions
SqliteExportService *-- DateTimeProvider
' --- Cross-Component Aggregations (Held References) ---
PipelineLogger o-- BoundedChannel : logs to
LogWorker o-- BoundedChannel : drains from
' --- Domain Containment ---
EnrichedCity *-- Location
EnrichedCity *-- LocationContext
GeneratedBrewery *-- Location
GeneratedBrewery *-- BreweryResult
GeneratedBrewery *-- GenerationMetadata
GeneratedBeer *-- Location
GeneratedBeer *-- BeerStyle
GeneratedBeer *-- BeerResult
GeneratedBeer *-- GenerationMetadata
GeneratedUser *-- Location
GeneratedUser *-- UserResult
GeneratedUser *-- GenerationMetadata
GeneratedCheckin *-- CheckinResult
GeneratedCheckin *-- GenerationMetadata
GeneratedRating *-- RatingResult
GeneratedRating *-- GenerationMetadata
GeneratedFollow *-- GenerationMetadata
@enduml

1
docs/pipeline/diagrams/planned/output/biergarten_activity.svg
View File

File diff suppressed because one or more lines are too long
Side by Side

Before

Width:  |  Height:  |  Size: 142 KiB

1
docs/pipeline/diagrams/planned/output/class.svg
View File

File diff suppressed because one or more lines are too long
Side by Side

Before

Width:  |  Height:  |  Size: 240 KiB