mirror of
https://github.com/aaronpo97/the-biergarten-app.git
synced 2026-10-15 01:54:43 +00:00
Refactor documentation and diagrams for web/pipeline. (#272)
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> for assistance with diagram creation and documentation prose updates.
This commit is contained in:
Aaron Po
@@ -1,34 +0,0 @@
|
||||
skinparam shadowing false
|
||||
skinparam backgroundColor #FCFCF7
|
||||
skinparam defaultFontName "DM Sans"
|
||||
skinparam defaultFontColor #14180C
|
||||
skinparam titleFontName "Volkhov"
|
||||
skinparam titleFontColor #14180C
|
||||
skinparam ArrowColor #656F33
|
||||
skinparam NoteBackgroundColor #DBEEDD
|
||||
skinparam NoteFontColor #14180C
|
||||
skinparam NoteBorderColor #4A5837
|
||||
skinparam SwimlaneBorderColor #4A5837
|
||||
skinparam SwimlaneBorderThickness 1
|
||||
skinparam activityStartColor #EBECE3
|
||||
skinparam activityEndColor #4A5837
|
||||
skinparam activityStopColor #4A5837
|
||||
skinparam ActivityBackgroundColor #EBECE3
|
||||
skinparam ActivityBorderColor #4A5837
|
||||
skinparam ActivityDiamondBackgroundColor #CBD2B5
|
||||
skinparam ActivityDiamondBorderColor #4A5837
|
||||
skinparam packageStyle rectangle
|
||||
skinparam packageBackgroundColor #F1F3EA
|
||||
skinparam packageBorderColor #4A5837
|
||||
skinparam packageFontColor #14180C
|
||||
skinparam classBackgroundColor #EBECE3
|
||||
skinparam classBorderColor #4A5837
|
||||
skinparam classFontColor #14180C
|
||||
skinparam classAttributeFontColor #3F4724
|
||||
skinparam classStereotypeFontColor #4A5837
|
||||
skinparam interfaceBackgroundColor #DBEEDD
|
||||
skinparam interfaceBorderColor #4A5837
|
||||
skinparam interfaceFontColor #14180C
|
||||
skinparam enumBackgroundColor #E4E6D8
|
||||
skinparam enumBorderColor #4A5837
|
||||
skinparam enumFontColor #14180C
|
||||
11
docs/pipeline/diagrams/current/activity.md
Normal file
11
docs/pipeline/diagrams/current/activity.md
Normal file
@@ -0,0 +1,11 @@
|
||||
# The Biergarten Data Pipeline (Current — Synchronous Data Path)
|
||||
|
||||
The only concurrency in the current pipeline is a dedicated log dispatcher
|
||||
thread that drains `log_channel` and forwards entries to spdlog for the entire
|
||||
run (joined during shutdown after `log_channel` closes).
|
||||
`BiergartenPipelineOrchestrator::Run()` itself is synchronous — sampling,
|
||||
enrichment, generation, and export all happen on the main thread.
|
||||
|
||||
```mermaid
|
||||
--8<-- "pipeline/diagrams/current/activity.mmd"
|
||||
```
|
||||
86
docs/pipeline/diagrams/current/activity.mmd
Normal file
86
docs/pipeline/diagrams/current/activity.mmd
Normal file
@@ -0,0 +1,86 @@
|
||||
flowchart TD
|
||||
Start([start]) --> ParseArgs["main.cc: ParseArguments(argc, argv)"]
|
||||
ParseArgs --> ArgsValid{Arguments valid?}
|
||||
ArgsValid -- no --> LogErr[Log error / usage info] --> Stop1([stop])
|
||||
ArgsValid -- yes --> Init["main.cc: Init OpenSSL global state and LlamaBackendState"]
|
||||
Init --> DI["main.cc: di::make_injector(...)<br/>binds ILogger, IEnrichmentService, DataGenerator, IExportService, IPromptFormatter, WebClient based on --mocked vs. model-backed mode"]
|
||||
DI --> LogThread["main.cc: create BoundedChannel log_channel, spawn log dispatcher thread (LogDispatcher::Run)"]
|
||||
LogThread --> CreateOrch["main.cc: injector.create BiergartenPipelineOrchestrator"]
|
||||
CreateOrch --> Run["BiergartenPipelineOrchestrator::Run()"]
|
||||
|
||||
Run --> SqliteInit["SqliteExportService.Initialize()<br/>builds biergarten_seed_<UTC datetime>.sqlite (numeric suffix if it exists), opens DB connection, executes schema DDL, begins transaction"]
|
||||
SqliteInit --> QueryLoc[Orchestrator: QueryLocations]
|
||||
QueryLoc --> LoadLoc["CuratedJsonDataService.LoadLocations()"]
|
||||
LoadLoc --> SampleLoc["std::ranges::sample(all_locations, --location-count) — defaults to 10"]
|
||||
|
||||
SampleLoc --> EnrichLoop{For each sampled Location}
|
||||
EnrichLoop --> GetContext["IEnrichmentService.GetLocationContext(loc)<br/>WikipediaEnrichmentService fetches a generic 'brewing' extract and a 'beer in {country}' extract (not city/region-specific); MockEnrichmentService (--mocked) returns an empty string"]
|
||||
GetContext --> LookupFailed{Lookup failed?}
|
||||
LookupFailed -- yes --> SkipCity1[Log warning, skip city] --> EnrichLoop
|
||||
LookupFailed -- no --> StoreEnriched["Store EnrichedCity{Location, region_context}"] --> EnrichLoop
|
||||
EnrichLoop -- done --> GenUsers[Orchestrator: GenerateUsers enriched_cities]
|
||||
SkipCity1 ~~~ GenUsers
|
||||
StoreEnriched ~~~ GenUsers
|
||||
|
||||
GenUsers --> LoadUserData["CuratedJsonDataService: LoadPersonas / LoadForenamesByCountry / LoadSurnamesByCountry<br/>each memoized after first parse; MockCuratedDataService (--mocked) returns a fixed in-memory dataset instead, skipping these JSON files"]
|
||||
|
||||
LoadUserData --> UserLoop{For each EnrichedCity}
|
||||
UserLoop --> SampleName["SampleName(forenames_by_country, surnames_by_country, location.iso3166_1)"]
|
||||
SampleName --> NamesAvail{Names available for country?}
|
||||
NamesAvail -- no --> SkipCity2[Log warning, skip city] --> UserLoop
|
||||
NamesAvail -- yes --> SamplePersona["Sample UserPersona (uniform random)"]
|
||||
SamplePersona --> UserGenMode{Generator mode}
|
||||
|
||||
UserGenMode -- MockGenerator --> UserMockGen[DeterministicHash and Format]
|
||||
UserGenMode -- LlamaGenerator --> UserLlamaLoad["prompt_directory_.Load(USER_GENERATION) → USER_GENERATION.md in --prompt-dir"]
|
||||
UserLlamaLoad --> UserLlamaRetry["Infer(...) with kUserJsonGrammar → ValidateUserJson<br/>retry up to 3 attempts, appending validation error to the retry prompt on failure"]
|
||||
UserGenMode -- OpenAIGenerator --> UserOAILoad["prompt_directory_.Load(USER_GENERATION) — same prompt file as LlamaGenerator, content is backend-agnostic"]
|
||||
UserOAILoad --> UserOAIRetry["CallChatCompletionsApi(...) with kUserJsonSchema → ValidateUserJson<br/>retry up to 2 attempts — Structured Outputs (json_schema, strict) already guarantees schema shape, so retries only cover transient failures / placeholder text"]
|
||||
|
||||
UserMockGen --> UserGenDone{Generation successful?}
|
||||
UserLlamaRetry --> UserGenDone
|
||||
UserOAIRetry --> UserGenDone
|
||||
|
||||
UserGenDone -- yes --> UserPostProc["GetRandomCoordsWithinRange (5km jitter), BuildEmail, GenerateDateOfBirth<br/>email and date_of_birth are generated programmatically, never LLM-authored; generation + coordinate jitter run inside one try/catch per record"]
|
||||
UserGenDone -- no --> UserGenFail["Log warning: generation failed, skipping"] --> UserLoop
|
||||
|
||||
UserPostProc --> UserExport["SqliteExportService.ProcessRecord(UserRecord)<br/>reuse cached city_id or insert City and cache its id; insert User + UserAddress (FK city_id)"]
|
||||
UserExport --> UserExportFail{Exception during insert?}
|
||||
UserExportFail -- yes --> UserExportWarn["Log warning: SQLite export failed"] --> UserLoop
|
||||
UserExportFail -- no --> UserLoop
|
||||
UserLoop -- done --> GenBrew[Orchestrator: GenerateBreweries enriched_cities]
|
||||
SkipCity2 ~~~ GenBrew
|
||||
UserGenFail ~~~ GenBrew
|
||||
UserExportWarn ~~~ GenBrew
|
||||
UserExportFail ~~~ GenBrew
|
||||
|
||||
GenBrew --> BrewLoop{For each EnrichedCity}
|
||||
BrewLoop --> BrewGenMode{Generator mode}
|
||||
|
||||
BrewGenMode -- MockGenerator --> BrewMockGen[DeterministicHash and Format]
|
||||
BrewGenMode -- LlamaGenerator --> BrewLlamaLoad["PrepareRegionContext, prompt_directory_.Load(BREWERY_GENERATION) → BREWERY_GENERATION.md"]
|
||||
BrewLlamaLoad --> BrewLlamaRetry["Infer(...) with kBreweryJsonGrammar → ValidateBreweryJson<br/>retry up to 3 attempts; max_tokens is fixed across retries, not raised on truncation"]
|
||||
BrewGenMode -- OpenAIGenerator --> BrewOAILoad["PrepareRegionContext, prompt_directory_.Load(BREWERY_GENERATION) — same prompt file as LlamaGenerator"]
|
||||
BrewOAILoad --> BrewOAIRetry["CallChatCompletionsApi(...) with kBreweryJsonSchema → ValidateBreweryJson<br/>retry up to 2 attempts — Structured Outputs already guarantees schema shape"]
|
||||
|
||||
BrewMockGen --> BrewGenDone{Generation successful?}
|
||||
BrewLlamaRetry --> BrewGenDone
|
||||
BrewOAIRetry --> BrewGenDone
|
||||
|
||||
BrewGenDone -- yes --> BrewPostProc["GetRandomCoordsWithinRange (5km jitter)<br/>generation + jitter run inside one try/catch per record"]
|
||||
BrewGenDone -- no --> BrewGenFail["Log warning: generation failed, skipping"] --> BrewLoop
|
||||
|
||||
BrewPostProc --> BrewGeocode["IAddressService.ReverseGeocode(longitude, latitude)<br/>NominatimAddressService hits the live Nominatim API (1 req/sec) and returns nullopt on failure; MockAddressService (--mocked) returns a fixed placeholder address instead"]
|
||||
BrewGeocode --> BrewGeocodeFailed{Geocode resolved?}
|
||||
BrewGeocodeFailed -- no --> BrewPlaceholder["Log warning: could not reverse geocode; use placeholder address_line1, empty postal_code"] --> BrewExport
|
||||
BrewGeocodeFailed -- yes --> BrewExport["SqliteExportService.ProcessRecord(BreweryRecord)<br/>reuse cached city_id or insert City and cache its id; insert Brewery + BreweryAddress (FK city_id)"]
|
||||
BrewExport --> BrewExportFail{Exception during insert?}
|
||||
BrewExportFail -- yes --> BrewExportWarn["Log warning: SQLite export failed (data loss prevented per-record; pipeline continues)"] --> BrewLoop
|
||||
BrewExportFail -- no --> BrewLoop
|
||||
BrewLoop -- done --> Finalize["SqliteExportService.Finalize()<br/>commits transaction, closes database connection"]
|
||||
BrewGenFail ~~~ Finalize
|
||||
BrewExportWarn ~~~ Finalize
|
||||
BrewExportFail ~~~ Finalize
|
||||
|
||||
Finalize --> CloseLog["main.cc: close log_channel, join log dispatcher thread"]
|
||||
CloseLog --> Stop2(["return 0 / stop"])
|
||||
@@ -1,293 +0,0 @@
|
||||
@startuml
|
||||
skinparam style strictuml
|
||||
skinparam defaultFontName "DM Sans"
|
||||
skinparam defaultFontSize 14
|
||||
skinparam titleFontName "Volkhov"
|
||||
skinparam titleFontSize 20
|
||||
skinparam backgroundColor #FAFCF9
|
||||
skinparam defaultFontColor #28342A
|
||||
skinparam titleFontColor #28342A
|
||||
skinparam ArrowColor #628A5B
|
||||
skinparam NoteBackgroundColor #EAF0E8
|
||||
skinparam NoteBorderColor #547461
|
||||
skinparam ActivityBackgroundColor #FAFCF9
|
||||
skinparam ActivityBorderColor #547461
|
||||
skinparam ActivityDiamondBackgroundColor #FAFCF9
|
||||
skinparam ActivityDiamondBorderColor #628A5B
|
||||
skinparam ActivityBarColor #628A5B
|
||||
skinparam SwimlaneBorderColor #547461
|
||||
skinparam SwimlaneBorderThickness 0.3
|
||||
|
||||
title The Biergarten Data Pipeline (Current — Synchronous Data Path)
|
||||
|
||||
|#F2F6F0|main.cc|
|
||||
start
|
||||
:Create BoundedChannel<LogEntry> log_channel;
|
||||
:Spawn log dispatcher thread\n(LogDispatcher::Run());
|
||||
note right
|
||||
The only concurrency in the current pipeline:
|
||||
a dedicated thread drains log_channel and
|
||||
forwards entries to spdlog for the entire run.
|
||||
Joined during shutdown after log_channel closes.
|
||||
end note
|
||||
:ParseArguments(argc, argv);
|
||||
if (Are arguments valid?) then (no)
|
||||
:Log error / usage info;
|
||||
stop
|
||||
else (yes)
|
||||
endif
|
||||
|
||||
:Init OpenSSL global state & LlamaBackendState;
|
||||
:di::make_injector(...);
|
||||
note right
|
||||
Binds ILogger, IEnrichmentService, DataGenerator,
|
||||
IExportService, IPromptFormatter, WebClient based
|
||||
on --mocked vs. model-backed mode.
|
||||
end note
|
||||
:injector.create<std::unique_ptr<BiergartenPipelineOrchestrator>>();
|
||||
:BiergartenPipelineOrchestrator::Run();
|
||||
note right
|
||||
Run() itself is synchronous — sampling, enrichment,
|
||||
generation, and export all happen on this thread.
|
||||
end note
|
||||
|
||||
|#EAF0E8|BiergartenPipelineOrchestrator|
|
||||
:Initialize SQLite export;
|
||||
|
||||
|#E0EAE0|SqliteExportService|
|
||||
:GetUtcTimestamp() from SystemDateTimeProvider;
|
||||
:Initialize();
|
||||
note right
|
||||
Builds a fresh biergarten_seed_<UTC datetime>.sqlite filename
|
||||
Appends a numeric suffix if the timestamp already exists
|
||||
Opens DB Connection
|
||||
Executes Schema DDL
|
||||
Begins Transaction
|
||||
end note
|
||||
|
||||
|#EAF0E8|BiergartenPipelineOrchestrator|
|
||||
:QueryLocations();
|
||||
|
||||
|#E2EBDC|JsonLoader|
|
||||
:JsonLoader::LoadLocations("locations.json");
|
||||
:std::ranges::sample(all_locations, --location-count);
|
||||
note right
|
||||
--location-count defaults to 10.
|
||||
end note
|
||||
|
||||
|#EAF0E8|BiergartenPipelineOrchestrator|
|
||||
while (For each sampled Location?) is (Remaining cities)
|
||||
|#DCE8D8|IEnrichmentService|
|
||||
:GetLocationContext(loc);
|
||||
note right
|
||||
WikipediaEnrichmentService fetches a generic
|
||||
"brewing" extract and a "beer in {country}" extract
|
||||
(not a city/region-specific page). MockEnrichmentService
|
||||
(--mocked) returns an empty string instead.
|
||||
end note
|
||||
|#EAF0E8|BiergartenPipelineOrchestrator|
|
||||
if (Lookup failed?) then (yes)
|
||||
:Log warning, skip city;
|
||||
else (no)
|
||||
:Store EnrichedCity{Location, region_context};
|
||||
endif
|
||||
endwhile (Done)
|
||||
|
||||
|#EAF0E8|BiergartenPipelineOrchestrator|
|
||||
:GenerateUsers(enriched_cities);
|
||||
|
||||
|#E2EBDC|JsonLoader|
|
||||
:LoadPersonas("personas.json");
|
||||
:LoadForenamesByCountry("forenames-by-country.json");
|
||||
:LoadSurnamesByCountry("surnames-by-country.json");
|
||||
note right
|
||||
Each call is cached after its first parse.
|
||||
MockCuratedDataService (--mocked) returns a
|
||||
fixed in-memory dataset instead and skips
|
||||
these three JSON files entirely.
|
||||
end note
|
||||
|
||||
|#EAF0E8|BiergartenPipelineOrchestrator|
|
||||
while (For each EnrichedCity?) is (Remaining cities)
|
||||
:SampleName(forenames_by_country,\n surnames_by_country, location.iso3166_1);
|
||||
if (Names available for country?) then (no)
|
||||
:Log warning, skip city;
|
||||
else (yes)
|
||||
:Sample UserPersona (uniform random);
|
||||
|
||||
|#E5EDE1|DataGenerator|
|
||||
if (Generator Mode) then (MockGenerator)
|
||||
:DeterministicHash & Format;
|
||||
elseif (Generator Mode) then (LlamaGenerator)
|
||||
:prompt_directory_->Load("USER_GENERATION");
|
||||
note right
|
||||
Resolves to USER_GENERATION.md inside --prompt-dir.
|
||||
end note
|
||||
repeat
|
||||
:Infer(system_prompt, user_prompt, max_tokens, kUserJsonGrammar);
|
||||
:ValidateUserJson(raw, user);
|
||||
if (Is JSON Valid?) then (yes)
|
||||
break
|
||||
else (no)
|
||||
:Attempt++;
|
||||
:Append validation error to retry prompt;
|
||||
endif
|
||||
repeat while (Attempt < 3?) is (yes)
|
||||
else (OpenAIGenerator)
|
||||
:prompt_directory_->Load("USER_GENERATION");
|
||||
note right
|
||||
Same prompt file as LlamaGenerator - content
|
||||
is backend-agnostic.
|
||||
end note
|
||||
repeat
|
||||
:CallChatCompletionsApi(system_prompt, user_prompt,\nkUserJsonSchema, max_tokens);
|
||||
:ValidateUserJson(raw, user);
|
||||
if (Is JSON Valid?) then (yes)
|
||||
break
|
||||
else (no)
|
||||
:Attempt++;
|
||||
:Append validation error to retry prompt;
|
||||
endif
|
||||
repeat while (Attempt < 2?) is (yes)
|
||||
note right
|
||||
Structured Outputs (response_format:
|
||||
json_schema, strict: true) already
|
||||
guarantees schema shape, so only 2
|
||||
attempts are needed - retries here cover
|
||||
transient failures / placeholder text,
|
||||
not malformed JSON.
|
||||
end note
|
||||
endif
|
||||
|
||||
|#EAF0E8|BiergartenPipelineOrchestrator|
|
||||
if (Generation successful?) then (yes)
|
||||
:GetRandomCoordsWithinRange(city_centre, 5km, rng);
|
||||
:BuildEmail(name, used_email_local_parts);
|
||||
:GenerateDateOfBirth(rng);
|
||||
note right
|
||||
email and date_of_birth are generated
|
||||
programmatically, never LLM-authored.
|
||||
Generation + coordinate jitter run inside
|
||||
one try/catch per record. UserRecord.address
|
||||
now carries {city, longitude, latitude}.
|
||||
end note
|
||||
|
||||
|#E0EAE0|SqliteExportService|
|
||||
:ProcessRecord(UserRecord);
|
||||
if (Location in cache?) then (yes)
|
||||
:Reuse location_id;
|
||||
else (no)
|
||||
:Insert Location & Cache ID;
|
||||
endif
|
||||
:Insert User (FK: location_id);
|
||||
|
||||
if (Exception caught during insert?) then (yes)
|
||||
|#EAF0E8|BiergartenPipelineOrchestrator|
|
||||
:Log warning "SQLite export failed";
|
||||
else (no)
|
||||
endif
|
||||
else (no)
|
||||
:Log warning "Generation failed, skipping...";
|
||||
endif
|
||||
endif
|
||||
endwhile (Done)
|
||||
|
||||
|#EAF0E8|BiergartenPipelineOrchestrator|
|
||||
:GenerateBreweries(enriched_cities);
|
||||
|
||||
|#E5EDE1|DataGenerator|
|
||||
while (For each EnrichedCity?) is (Remaining cities)
|
||||
if (Generator Mode) then (MockGenerator)
|
||||
:DeterministicHash & Format;
|
||||
elseif (Generator Mode) then (LlamaGenerator)
|
||||
:PrepareRegionContext;
|
||||
:prompt_directory_->Load("BREWERY_GENERATION");
|
||||
note right
|
||||
Resolves to BREWERY_GENERATION.md inside --prompt-dir.
|
||||
end note
|
||||
repeat
|
||||
:Infer(system_prompt, user_prompt, max_tokens, kBreweryJsonGrammar);
|
||||
:ValidateBreweryJson(raw, brewery);
|
||||
if (Is JSON Valid?) then (yes)
|
||||
break
|
||||
else (no)
|
||||
:Attempt++;
|
||||
:Append validation error to retry prompt;
|
||||
endif
|
||||
repeat while (Attempt < 3?) is (yes)
|
||||
note right
|
||||
max_tokens is fixed across retries —
|
||||
it is not raised on truncation.
|
||||
end note
|
||||
else (OpenAIGenerator)
|
||||
:PrepareRegionContext;
|
||||
:prompt_directory_->Load("BREWERY_GENERATION");
|
||||
note right
|
||||
Same prompt file as LlamaGenerator - content
|
||||
is backend-agnostic.
|
||||
end note
|
||||
repeat
|
||||
:CallChatCompletionsApi(system_prompt, user_prompt,\nkBreweryJsonSchema, max_tokens);
|
||||
:ValidateBreweryJson(raw, brewery);
|
||||
if (Is JSON Valid?) then (yes)
|
||||
break
|
||||
else (no)
|
||||
:Attempt++;
|
||||
:Append validation error to retry prompt;
|
||||
endif
|
||||
repeat while (Attempt < 2?) is (yes)
|
||||
note right
|
||||
Structured Outputs already guarantees schema
|
||||
shape, so only 2 attempts are needed - retries
|
||||
here cover transient failures / placeholder
|
||||
text, not malformed JSON.
|
||||
end note
|
||||
endif
|
||||
|
||||
|#EAF0E8|BiergartenPipelineOrchestrator|
|
||||
if (Generation successful?) then (yes)
|
||||
:GetRandomCoordsWithinRange(city_centre, 5km, rng);
|
||||
note right
|
||||
Generation + coordinate jitter run inside one
|
||||
try/catch per record. BreweryRecord.address
|
||||
now carries {city, longitude, latitude}.
|
||||
end note
|
||||
|
||||
|#E0EAE0|SqliteExportService|
|
||||
:ProcessRecord(GeneratedBrewery);
|
||||
if (Location in cache?) then (yes)
|
||||
:Reuse location_id;
|
||||
else (no)
|
||||
:Insert Location & Cache ID;
|
||||
endif
|
||||
:Insert Brewery (FK: location_id);
|
||||
|
||||
if (Exception caught during insert?) then (yes)
|
||||
|#EAF0E8|BiergartenPipelineOrchestrator|
|
||||
:Log warning "SQLite export failed";
|
||||
note right
|
||||
Data loss is prevented per-record.
|
||||
The pipeline continues running.
|
||||
end note
|
||||
else (no)
|
||||
endif
|
||||
else (no)
|
||||
:Log warning "Generation failed, skipping...";
|
||||
endif
|
||||
|#E5EDE1|DataGenerator|
|
||||
endwhile (Done)
|
||||
|
||||
|#E0EAE0|SqliteExportService|
|
||||
:Finalize();
|
||||
note right
|
||||
Commits Transaction
|
||||
Closes Database Connection
|
||||
end note
|
||||
|
||||
|#F2F6F0|main.cc|
|
||||
:Close log_channel;
|
||||
:Join log dispatcher thread;
|
||||
:Return 0;
|
||||
stop
|
||||
|
||||
@enduml
|
||||
5
docs/pipeline/diagrams/current/class.md
Normal file
5
docs/pipeline/diagrams/current/class.md
Normal file
@@ -0,0 +1,5 @@
|
||||
# The Biergarten Data Pipeline — Class Diagram (Current)
|
||||
|
||||
```mermaid
|
||||
--8<-- "pipeline/diagrams/current/class.mmd"
|
||||
```
|
||||
285
docs/pipeline/diagrams/current/class.mmd
Normal file
285
docs/pipeline/diagrams/current/class.mmd
Normal file
@@ -0,0 +1,285 @@
|
||||
classDiagram
|
||||
class BiergartenPipelineOrchestrator {
|
||||
-shared_ptr~ILogger~ logger_
|
||||
-unique_ptr~IEnrichmentService~ context_service_
|
||||
-unique_ptr~DataGenerator~ generator_
|
||||
-unique_ptr~IExportService~ exporter_
|
||||
-unique_ptr~ICuratedDataService~ curated_data_service_
|
||||
-unique_ptr~IAddressService~ address_service_
|
||||
-ApplicationOptions application_options_
|
||||
-vector~BreweryRecord~ generated_breweries_
|
||||
-vector~UserRecord~ generated_users_
|
||||
+Run() bool
|
||||
-QueryLocations() vector~City~
|
||||
-GenerateBreweries(cities) void
|
||||
-GenerateUsers(cities) void
|
||||
-LogResults() void
|
||||
}
|
||||
note for BiergartenPipelineOrchestrator "GenerateBreweries/GenerateUsers jitter each address via GetRandomCoordsWithinRange(), uniformly within 5km of the city centre. GenerateBreweries additionally reverse-geocodes the jittered coordinates via address_service_."
|
||||
|
||||
class LogLevel {
|
||||
<<enumeration>>
|
||||
Debug
|
||||
Info
|
||||
Warn
|
||||
Error
|
||||
}
|
||||
|
||||
class PipelinePhase {
|
||||
<<enumeration>>
|
||||
Startup
|
||||
Enrichment
|
||||
UserGeneration
|
||||
BreweryAndBeerGeneration
|
||||
CheckinGeneration
|
||||
RatingGeneration
|
||||
FollowGeneration
|
||||
Teardown
|
||||
}
|
||||
|
||||
class LogDTO {
|
||||
<<struct>>
|
||||
+LogLevel level
|
||||
+PipelinePhase phase
|
||||
+string message
|
||||
}
|
||||
|
||||
class LogEntry {
|
||||
<<struct>>
|
||||
+time_point timestamp
|
||||
+source_location origin
|
||||
+thread::id thread_id
|
||||
+LogLevel level
|
||||
+PipelinePhase phase
|
||||
+string message
|
||||
}
|
||||
|
||||
class ILogger {
|
||||
<<interface>>
|
||||
+Log(payload) void
|
||||
-DoLog(entry)* void
|
||||
}
|
||||
|
||||
class LogProducer {
|
||||
-BoundedChannel~LogEntry~ channel_
|
||||
-DoLog(entry) void
|
||||
}
|
||||
|
||||
class LogDispatcher {
|
||||
-BoundedChannel~LogEntry~ channel_
|
||||
+Run() void
|
||||
-ToSpdlogLevel(level) level_enum
|
||||
}
|
||||
|
||||
class IEnrichmentService {
|
||||
<<interface>>
|
||||
+GetLocationContext(loc) string
|
||||
}
|
||||
|
||||
class MockEnrichmentService {
|
||||
+GetLocationContext(loc) string
|
||||
}
|
||||
|
||||
class WikipediaEnrichmentService {
|
||||
-unique_ptr~WebClient~ client_
|
||||
-unordered_map~string, string~ extract_cache_
|
||||
+GetLocationContext(loc) string
|
||||
-FetchExtract(query) string
|
||||
}
|
||||
|
||||
class WebClient {
|
||||
<<interface>>
|
||||
+Get(url) string
|
||||
+UrlEncode(value) string
|
||||
}
|
||||
|
||||
class HttpWebClient {
|
||||
+Get(url) string
|
||||
+UrlEncode(value) string
|
||||
}
|
||||
|
||||
class DataGenerator {
|
||||
<<interface>>
|
||||
+GenerateBrewery(city) BreweryResult
|
||||
+GenerateUser(city, persona, name) UserResult
|
||||
}
|
||||
|
||||
class MockGenerator {
|
||||
+GenerateBrewery(city) BreweryResult
|
||||
+GenerateUser(city, persona, name) UserResult
|
||||
-DeterministicHash(location) size_t
|
||||
-DeterministicHash(location, persona, name) size_t
|
||||
}
|
||||
|
||||
class LlamaGenerator {
|
||||
-ModelHandle model_
|
||||
-ContextHandle context_
|
||||
-unique_ptr~IPromptFormatter~ prompt_formatter_
|
||||
-unique_ptr~IPromptDirectory~ prompt_directory_
|
||||
-mt19937 rng_
|
||||
+GenerateBrewery(...) BreweryResult
|
||||
+GenerateUser(...) UserResult
|
||||
-Load(model_path) void
|
||||
-Infer(...) string
|
||||
-InferFormatted(...) string
|
||||
}
|
||||
|
||||
class OpenAIGenerator {
|
||||
-string api_key_
|
||||
-string model_
|
||||
-shared_ptr~ILogger~ logger_
|
||||
-unique_ptr~IPromptDirectory~ prompt_directory_
|
||||
-unique_ptr~WebClient~ web_client_
|
||||
+GenerateBrewery(city) BreweryResult
|
||||
+GenerateUser(city, persona, name) UserResult
|
||||
-CallChatCompletionsApi(...) string
|
||||
}
|
||||
note for OpenAIGenerator "Chat Completions with Structured Outputs (json_schema, strict) guarantees schema-valid JSON, replacing llama.cpp's GBNF grammar — no IPromptFormatter needed"
|
||||
|
||||
class IPromptFormatter {
|
||||
<<interface>>
|
||||
+Format(system_prompt, user_prompt) string
|
||||
}
|
||||
|
||||
class Gemma4JinjaPromptFormatter {
|
||||
+Format(system_prompt, user_prompt) string
|
||||
}
|
||||
|
||||
class IPromptDirectory {
|
||||
<<interface>>
|
||||
+Load(key) string
|
||||
}
|
||||
|
||||
class PromptDirectory {
|
||||
-path prompt_dir_
|
||||
-unordered_map~string, string~ cache_
|
||||
+Load(key) string
|
||||
}
|
||||
|
||||
class ICuratedDataService {
|
||||
<<interface>>
|
||||
+LoadLocations() vector~Location~
|
||||
+LoadPersonas() vector~UserPersona~
|
||||
+LoadForenamesByCountry() unordered_map~string, forename_list~
|
||||
+LoadSurnamesByCountry() unordered_map~string, surname_list~
|
||||
}
|
||||
|
||||
class CuratedJsonDataService {
|
||||
-CuratedDataFilePaths filepaths_
|
||||
-cache cache_
|
||||
+LoadLocations() vector~Location~
|
||||
+LoadPersonas() vector~UserPersona~
|
||||
+LoadForenamesByCountry() unordered_map~string, forename_list~
|
||||
+LoadSurnamesByCountry() unordered_map~string, surname_list~
|
||||
}
|
||||
note for CuratedJsonDataService "Takes a CuratedDataFilePaths DTO (locations/personas/forenames/surnames paths) once at construction, not per Load* call. Each Load* method memoizes its result in cache_ after its first call."
|
||||
|
||||
class MockCuratedDataService {
|
||||
-vector~Location~ locations_
|
||||
-vector~UserPersona~ personas_
|
||||
-unordered_map~string, forename_list~ forenames_by_country_
|
||||
-unordered_map~string, surname_list~ surnames_by_country_
|
||||
+LoadLocations() vector~Location~
|
||||
+LoadPersonas() vector~UserPersona~
|
||||
+LoadForenamesByCountry() unordered_map~string, forename_list~
|
||||
+LoadSurnamesByCountry() unordered_map~string, surname_list~
|
||||
}
|
||||
note for MockCuratedDataService "Fixed 4-location/3-persona dataset (US, DE, FR, BE) for --mocked runs"
|
||||
|
||||
class IAddressService {
|
||||
<<interface>>
|
||||
+ReverseGeocode(longitude, latitude) optional~AddressLookupResult~
|
||||
}
|
||||
|
||||
class NominatimAddressService {
|
||||
-unique_ptr~WebClient~ client_
|
||||
-shared_ptr~ILogger~ logger_
|
||||
+ReverseGeocode(longitude, latitude) optional~AddressLookupResult~
|
||||
}
|
||||
note for NominatimAddressService "Queries the public Nominatim API; sleeps 1s per request per its usage policy; returns nullopt on failure or when no usable street address is found"
|
||||
|
||||
class MockAddressService {
|
||||
+ReverseGeocode(longitude, latitude) optional~AddressLookupResult~
|
||||
}
|
||||
note for MockAddressService "Returns a fixed placeholder address ('123 Mock Street', '00000') for --mocked runs, no network call"
|
||||
|
||||
class IExportService {
|
||||
<<interface>>
|
||||
+Initialize() void
|
||||
+ProcessRecord(brewery) uint64_t
|
||||
+ProcessRecord(user) uint64_t
|
||||
+Finalize() void
|
||||
}
|
||||
|
||||
class SqliteExportService {
|
||||
-unique_ptr~IDateTimeProvider~ date_time_provider_
|
||||
-string run_timestamp_utc_
|
||||
-path database_path_
|
||||
-SqliteDatabaseHandle db_handle_
|
||||
-SqliteStatementHandle insert_city_stmt_
|
||||
-SqliteStatementHandle insert_brewery_stmt_
|
||||
-SqliteStatementHandle insert_brewery_address_stmt_
|
||||
-SqliteStatementHandle insert_user_stmt_
|
||||
-SqliteStatementHandle insert_user_address_stmt_
|
||||
-bool transaction_open_
|
||||
-unordered_map~string, sqlite3_int64~ city_cache_
|
||||
+Initialize() void
|
||||
+ProcessRecord(brewery) uint64_t
|
||||
+ProcessRecord(user) uint64_t
|
||||
+Finalize() void
|
||||
-InitializeSchema() void
|
||||
-ResolveCityId(city) sqlite3_int64
|
||||
}
|
||||
|
||||
class IDateTimeProvider {
|
||||
<<interface>>
|
||||
+GetUtcTimestamp() string
|
||||
}
|
||||
|
||||
class SystemDateTimeProvider {
|
||||
+GetUtcTimestamp() string
|
||||
}
|
||||
|
||||
BiergartenPipelineOrchestrator *-- ILogger : owns
|
||||
BiergartenPipelineOrchestrator *-- IEnrichmentService : owns
|
||||
BiergartenPipelineOrchestrator *-- DataGenerator : owns
|
||||
BiergartenPipelineOrchestrator *-- IExportService : owns
|
||||
BiergartenPipelineOrchestrator *-- ICuratedDataService : owns
|
||||
BiergartenPipelineOrchestrator *-- IAddressService : owns
|
||||
|
||||
LogEntry *-- LogLevel
|
||||
LogEntry *-- PipelinePhase
|
||||
LogDTO *-- LogLevel
|
||||
LogDTO *-- PipelinePhase
|
||||
ILogger <|.. LogProducer : implements
|
||||
LogProducer ..> LogEntry : emits
|
||||
LogDispatcher ..> LogEntry : consumes
|
||||
|
||||
IEnrichmentService <|.. WikipediaEnrichmentService : implements
|
||||
IEnrichmentService <|.. MockEnrichmentService : implements
|
||||
WikipediaEnrichmentService *-- WebClient : owns
|
||||
|
||||
WebClient <|.. HttpWebClient : implements
|
||||
|
||||
DataGenerator <|.. MockGenerator : implements
|
||||
DataGenerator <|.. LlamaGenerator : implements
|
||||
DataGenerator <|.. OpenAIGenerator : implements
|
||||
|
||||
LlamaGenerator *-- IPromptFormatter : uses
|
||||
LlamaGenerator *-- IPromptDirectory : uses
|
||||
|
||||
OpenAIGenerator *-- IPromptDirectory : uses
|
||||
OpenAIGenerator *-- WebClient : uses
|
||||
|
||||
IPromptFormatter <|.. Gemma4JinjaPromptFormatter : implements
|
||||
IPromptDirectory <|.. PromptDirectory : implements
|
||||
|
||||
ICuratedDataService <|.. CuratedJsonDataService : implements
|
||||
ICuratedDataService <|.. MockCuratedDataService : implements
|
||||
|
||||
IAddressService <|.. NominatimAddressService : implements
|
||||
IAddressService <|.. MockAddressService : implements
|
||||
|
||||
IExportService <|.. SqliteExportService : implements
|
||||
SqliteExportService *-- IDateTimeProvider : owns
|
||||
IDateTimeProvider <|.. SystemDateTimeProvider : implements
|
||||
@@ -1,301 +0,0 @@
|
||||
@startuml
|
||||
skinparam style strictuml
|
||||
skinparam defaultFontName "DM Sans"
|
||||
skinparam defaultFontSize 14
|
||||
skinparam titleFontName "Volkhov"
|
||||
skinparam titleFontSize 20
|
||||
skinparam backgroundColor #FAFCF9
|
||||
skinparam defaultFontColor #28342A
|
||||
skinparam titleFontColor #28342A
|
||||
skinparam ArrowColor #628A5B
|
||||
|
||||
skinparam class {
|
||||
BackgroundColor #FAFCF9
|
||||
HeaderBackgroundColor #EAF0E8
|
||||
BorderColor #547461
|
||||
ArrowColor #628A5B
|
||||
FontColor #28342A
|
||||
}
|
||||
|
||||
skinparam note {
|
||||
BackgroundColor #EAF0E8
|
||||
BorderColor #547461
|
||||
FontColor #28342A
|
||||
}
|
||||
|
||||
title The Biergarten Data Pipeline - Class Diagram
|
||||
|
||||
class BiergartenPipelineOrchestrator {
|
||||
- logger_ : std::shared_ptr<ILogger>
|
||||
- context_service_ : std::unique_ptr<IEnrichmentService>
|
||||
- generator_ : std::unique_ptr<DataGenerator>
|
||||
- exporter_ : std::unique_ptr<IExportService>
|
||||
- curated_data_service_ : std::unique_ptr<ICuratedDataService>
|
||||
- application_options_ : ApplicationOptions
|
||||
- generated_breweries_ : std::vector<BreweryRecord>
|
||||
- generated_users_ : std::vector<UserRecord>
|
||||
+ Run() : bool
|
||||
- QueryLocations() : std::vector<City>
|
||||
- GenerateBreweries(cities : std::span<const EnrichedCity>) : void
|
||||
- GenerateUsers(cities : std::span<const EnrichedCity>) : void
|
||||
- LogResults() const : void
|
||||
}
|
||||
|
||||
note right of BiergartenPipelineOrchestrator
|
||||
GenerateBreweries and GenerateUsers both call the
|
||||
free function GetRandomCoordsWithinRange()
|
||||
(biergarten_pipeline_orchestrator/coords.h) to jitter
|
||||
a coordinate pair uniformly within 5km of the city
|
||||
centre for each generated record's address.
|
||||
end note
|
||||
|
||||
class LogLevel <<enumeration>> {
|
||||
Debug
|
||||
Info
|
||||
Warn
|
||||
Error
|
||||
}
|
||||
|
||||
class PipelinePhase <<enumeration>> {
|
||||
Startup
|
||||
Enrichment
|
||||
UserGeneration
|
||||
BreweryAndBeerGeneration
|
||||
CheckinGeneration
|
||||
RatingGeneration
|
||||
FollowGeneration
|
||||
Teardown
|
||||
}
|
||||
|
||||
struct LogDTO {
|
||||
+ level : LogLevel
|
||||
+ phase : PipelinePhase
|
||||
+ message : std::string
|
||||
}
|
||||
|
||||
struct LogEntry {
|
||||
+ timestamp : std::chrono::system_clock::time_point
|
||||
+ origin : std::source_location
|
||||
+ thread_id : std::thread::id
|
||||
+ level : LogLevel
|
||||
+ phase : PipelinePhase
|
||||
+ message : std::string
|
||||
}
|
||||
|
||||
interface ILogger <<interface>> {
|
||||
+ Log(payload : LogDTO) : void
|
||||
- {abstract} DoLog(entry : LogEntry) : void
|
||||
}
|
||||
|
||||
class LogProducer {
|
||||
- channel_ : BoundedChannel<LogEntry>&
|
||||
- DoLog(entry : LogEntry) : void
|
||||
}
|
||||
|
||||
class LogDispatcher {
|
||||
- channel_ : BoundedChannel<LogEntry>&
|
||||
+ Run() : void
|
||||
- ToSpdlogLevel(level) : spdlog::level::level_enum
|
||||
}
|
||||
|
||||
interface IEnrichmentService <<interface>> {
|
||||
+ GetLocationContext(loc : const Location&) : std::string
|
||||
}
|
||||
|
||||
class MockEnrichmentService {
|
||||
+ GetLocationContext(loc : const Location&) : std::string
|
||||
}
|
||||
|
||||
class WikipediaEnrichmentService {
|
||||
- client_ : std::unique_ptr<WebClient>
|
||||
- extract_cache_ : std::unordered_map<std::string, std::string>
|
||||
+ GetLocationContext(loc : const Location&) : std::string
|
||||
- FetchExtract(query : std::string_view) : std::string
|
||||
}
|
||||
|
||||
interface WebClient <<interface>> {
|
||||
+ Get(url : const std::string&) : std::string
|
||||
+ UrlEncode(value : const std::string&) : std::string
|
||||
}
|
||||
|
||||
class HttpWebClient {
|
||||
+ Get(url : const std::string&) : std::string
|
||||
+ UrlEncode(value : const std::string&) : std::string
|
||||
}
|
||||
|
||||
interface DataGenerator <<interface>> {
|
||||
+ GenerateBrewery(city : const EnrichedCity&) : BreweryResult
|
||||
+ GenerateUser(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult
|
||||
}
|
||||
|
||||
class MockGenerator {
|
||||
+ GenerateBrewery(city : const EnrichedCity&) : BreweryResult
|
||||
+ GenerateUser(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult
|
||||
- DeterministicHash(location : const Location&) : size_t
|
||||
- DeterministicHash(location : const Location&, persona : const UserPersona&, name : const Name&) : size_t
|
||||
}
|
||||
|
||||
class LlamaGenerator {
|
||||
- model_ : ModelHandle
|
||||
- context_ : ContextHandle
|
||||
- prompt_formatter_ : std::unique_ptr<IPromptFormatter>
|
||||
- prompt_directory_ : std::unique_ptr<IPromptDirectory>
|
||||
- rng_ : std::mt19937
|
||||
+ GenerateBrewery(...) : BreweryResult
|
||||
+ GenerateUser(...) : UserResult
|
||||
- Load(model_path : const std::string&) : void
|
||||
- Infer(...) : std::string
|
||||
- InferFormatted(...) : std::string
|
||||
}
|
||||
|
||||
class OpenAIGenerator {
|
||||
- api_key_ : std::string
|
||||
- model_ : std::string
|
||||
- logger_ : std::shared_ptr<ILogger>
|
||||
- prompt_directory_ : std::unique_ptr<IPromptDirectory>
|
||||
- web_client_ : std::unique_ptr<WebClient>
|
||||
+ GenerateBrewery(city : const EnrichedCity&) : BreweryResult
|
||||
+ GenerateUser(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult
|
||||
- CallChatCompletionsApi(...) : std::string
|
||||
}
|
||||
|
||||
note right of OpenAIGenerator
|
||||
Chat Completions with Structured Outputs
|
||||
(response_format: json_schema, strict: true)
|
||||
guarantees schema-valid JSON, replacing
|
||||
llama.cpp's GBNF grammar constraint -
|
||||
no IPromptFormatter needed.
|
||||
end note
|
||||
|
||||
interface IPromptFormatter <<interface>> {
|
||||
+ Format(system_prompt : std::string_view, user_prompt : std::string_view) : std::string
|
||||
}
|
||||
|
||||
class Gemma4JinjaPromptFormatter {
|
||||
+ Format(system_prompt : std::string_view, user_prompt : std::string_view) : std::string
|
||||
}
|
||||
|
||||
interface IPromptDirectory <<interface>> {
|
||||
+ Load(key : std::string_view) : std::string
|
||||
}
|
||||
|
||||
class PromptDirectory {
|
||||
- prompt_dir_ : std::filesystem::path
|
||||
- cache_ : std::unordered_map<std::string, std::string>
|
||||
+ Load(key : std::string_view) : std::string
|
||||
}
|
||||
|
||||
interface ICuratedDataService <<interface>> {
|
||||
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
|
||||
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
|
||||
+ LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
|
||||
+ LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
|
||||
}
|
||||
|
||||
class JsonLoader {
|
||||
- cache_ : cache
|
||||
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
|
||||
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
|
||||
+ LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
|
||||
+ LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
|
||||
}
|
||||
|
||||
note right of JsonLoader
|
||||
Each Load* method memoizes its result in
|
||||
cache_ on first call; later calls on the
|
||||
same instance return the cached value
|
||||
without re-parsing.
|
||||
end note
|
||||
|
||||
class MockCuratedDataService {
|
||||
- locations_ : std::vector<Location>
|
||||
- personas_ : std::vector<UserPersona>
|
||||
- forenames_by_country_ : std::unordered_map<std::string, forename_list>
|
||||
- surnames_by_country_ : std::unordered_map<std::string, surname_list>
|
||||
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
|
||||
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
|
||||
+ LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
|
||||
+ LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
|
||||
}
|
||||
|
||||
note right of MockCuratedDataService
|
||||
Fixed 4-location / 3-persona dataset
|
||||
(US, DE, FR, BE) used in --mocked runs;
|
||||
filepath arguments are ignored.
|
||||
end note
|
||||
|
||||
interface IExportService <<interface>> {
|
||||
+ Initialize() : void
|
||||
+ ProcessRecord(brewery : const BreweryRecord&) : uint64_t
|
||||
+ ProcessRecord(user : const UserRecord&) : uint64_t
|
||||
+ Finalize() : void
|
||||
}
|
||||
|
||||
class SqliteExportService {
|
||||
- date_time_provider_ : std::unique_ptr<IDateTimeProvider>
|
||||
- run_timestamp_utc_ : std::string
|
||||
- database_path_ : std::filesystem::path
|
||||
- db_handle_ : SqliteDatabaseHandle
|
||||
- insert_location_stmt_ : SqliteStatementHandle
|
||||
- insert_brewery_stmt_ : SqliteStatementHandle
|
||||
- insert_user_stmt_ : SqliteStatementHandle
|
||||
- transaction_open_ : bool
|
||||
- location_cache_ : std::unordered_map<std::string, sqlite3_int64>
|
||||
+ Initialize() : void
|
||||
+ ProcessRecord(brewery : const BreweryRecord&) : uint64_t
|
||||
+ ProcessRecord(user : const UserRecord&) : uint64_t
|
||||
+ Finalize() : void
|
||||
- InitializeSchema() : void
|
||||
- ResolveLocationId(location : const Location&) : sqlite3_int64
|
||||
}
|
||||
|
||||
interface IDateTimeProvider <<interface>> {
|
||||
+ GetUtcTimestamp() : std::string
|
||||
}
|
||||
|
||||
class SystemDateTimeProvider {
|
||||
+ GetUtcTimestamp() : std::string
|
||||
}
|
||||
|
||||
' Structural Relationships / Dependency Injection
|
||||
BiergartenPipelineOrchestrator *-- ILogger : owns
|
||||
BiergartenPipelineOrchestrator *-- IEnrichmentService : owns
|
||||
BiergartenPipelineOrchestrator *-- DataGenerator : owns
|
||||
BiergartenPipelineOrchestrator *-- IExportService : owns
|
||||
BiergartenPipelineOrchestrator *-- ICuratedDataService : owns
|
||||
|
||||
LogEntry *-- LogLevel
|
||||
LogEntry *-- PipelinePhase
|
||||
LogDTO *-- LogLevel
|
||||
LogDTO *-- PipelinePhase
|
||||
ILogger <|.. LogProducer : implements
|
||||
LogProducer ..> LogEntry : emits
|
||||
LogDispatcher ..> LogEntry : consumes
|
||||
|
||||
IEnrichmentService <|.. WikipediaEnrichmentService : implements
|
||||
IEnrichmentService <|.. MockEnrichmentService : implements
|
||||
WikipediaEnrichmentService *-- WebClient : owns
|
||||
|
||||
WebClient <|.. HttpWebClient : implements
|
||||
|
||||
DataGenerator <|.. MockGenerator : implements
|
||||
DataGenerator <|.. LlamaGenerator : implements
|
||||
DataGenerator <|.. OpenAIGenerator : implements
|
||||
|
||||
LlamaGenerator *-- IPromptFormatter : uses
|
||||
LlamaGenerator *-- IPromptDirectory : uses
|
||||
|
||||
OpenAIGenerator *-- IPromptDirectory : uses
|
||||
OpenAIGenerator *-- WebClient : uses
|
||||
|
||||
IPromptFormatter <|.. Gemma4JinjaPromptFormatter : implements
|
||||
IPromptDirectory <|.. PromptDirectory : implements
|
||||
|
||||
ICuratedDataService <|.. JsonLoader : implements
|
||||
ICuratedDataService <|.. MockCuratedDataService : implements
|
||||
|
||||
IExportService <|.. SqliteExportService : implements
|
||||
SqliteExportService *-- IDateTimeProvider : owns
|
||||
IDateTimeProvider <|.. SystemDateTimeProvider : implements
|
||||
|
||||
@enduml
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
16
docs/pipeline/diagrams/planned/activity.md
Normal file
16
docs/pipeline/diagrams/planned/activity.md
Normal file
@@ -0,0 +1,16 @@
|
||||
# The Biergarten Data Pipeline — Activity Diagram (Planned)
|
||||
|
||||
Each generation phase runs a producer/consumer chain concurrently: an
|
||||
**Orchestrator** producer feeds a bounded channel, an **LLM Worker** consumes it
|
||||
and generates records (closing its output channel when the input is drained and
|
||||
closed), and a **SQLite Worker** consumes that output channel and commits in
|
||||
batches. The brewery phase inserts a fourth stage, a **Geocode Worker**, between
|
||||
the LLM Worker and the SQLite Worker: reverse-geocoding is network-bound and
|
||||
rate-limited independently of LLM inference (Nominatim's 1-request-per-second
|
||||
policy), so keeping it off the LLM Worker's thread means neither stage stalls
|
||||
waiting on the other's bottleneck. The Checkin and Follow phases run in parallel
|
||||
with each other once the user and brewery pools are populated.
|
||||
|
||||
```mermaid
|
||||
--8<-- "pipeline/diagrams/planned/activity.mmd"
|
||||
```
|
||||
208
docs/pipeline/diagrams/planned/activity.mmd
Normal file
208
docs/pipeline/diagrams/planned/activity.mmd
Normal file
@@ -0,0 +1,208 @@
|
||||
flowchart TD
|
||||
Start([start]) --> ParseArgs["Main: ParseArguments(argc, argv)"]
|
||||
ParseArgs --> InvalidArgs{Invalid args?}
|
||||
InvalidArgs -- yes --> LogErr["spdlog::error"] --> Stop1([stop])
|
||||
InvalidArgs -- no --> InitOpenSSL["Main: Init OpenSSL global state and LlamaBackendState"]
|
||||
InitOpenSSL --> BuildDI["Main: Build DI injector"]
|
||||
BuildDI --> InitSqlite["Main: Initialize SqliteExportService<br/>opens SQLite connection; transactions now managed per-phase via batching"]
|
||||
InitSqlite --> LogSetup["Main: create BoundedChannel log_ch, spawn Log Worker thread<br/>drains log_ch for the pipeline lifetime; all workers emit LogEntry via PipelineLogger, never spdlog directly"]
|
||||
LogSetup --> OrchRun["BiergartenPipelineOrchestrator::Run()"]
|
||||
|
||||
OrchRun --> PL1["JsonLoader.LoadBeerStyles → EnrichmentService.PreWarmBeerStyleCache"]
|
||||
OrchRun --> PL2["JsonLoader.LoadLocations → EnrichmentService.PreWarmLocationCache"]
|
||||
OrchRun --> PL3["JsonLoader.LoadForenamesByCountry"]
|
||||
OrchRun --> PL4["JsonLoader.LoadSurnamesByCountry"]
|
||||
OrchRun --> PL5["JsonLoader.LoadPersonas"]
|
||||
PL1 & PL2 & PL3 & PL4 & PL5 --> UserPhase["Orchestrator: RunUserPhase(sampled_locations)<br/>each Load* call memoized after first parse; MockCuratedDataService (--mocked) returns a fixed in-memory dataset instead, skipping these JSON files"]
|
||||
|
||||
UserPhase --> UCreate["Create BoundedChannels (loc_ch, exp_ch)"]
|
||||
|
||||
subgraph USERGEN["Phase 0 — User Generation (concurrent)"]
|
||||
direction LR
|
||||
subgraph U_PROD["Orchestrator (producer)"]
|
||||
UP1["Loop: send Location → loc_ch"]
|
||||
UP1 --> UP2["Close loc_ch"]
|
||||
end
|
||||
subgraph U_LLM["LLM Worker"]
|
||||
UL1{"loc_ch has items?"}
|
||||
UL1 -- yes --> UL2["Receive Location"]
|
||||
UL2 --> UL3["GetLocationContextFromCache(location) — guaranteed cache hit from startup"]
|
||||
UL3 --> UL4["IPersonaSelectionStrategy.SelectPersona(personas_palette_) — guaranteed cache hit; returns Persona{style_affinities, abv_range, ibu_preference, checkin_weight}"]
|
||||
UL4 --> UL5["SampleName(forenames_by_country, surnames_by_country, iso3166_1, rng)<br/>deterministic, no LLM; free function; skips the city if either map has no entry for iso3166_1"]
|
||||
UL5 --> UL6["GenerateUser(enriched_city, persona, sampled_name) via DataGenerator<br/>LLM receives EnrichedCity context + persona description + sampled name"]
|
||||
UL6 --> UL7["PipelineLogger.Log(Info, UserGeneration, city, user_id, llm)"]
|
||||
UL7 --> UL8["Send GeneratedUser → exp_ch"]
|
||||
UL8 --> UL1
|
||||
UL1 -- no --> UL9["Close exp_ch"]
|
||||
end
|
||||
subgraph U_SQL["SQLite Worker"]
|
||||
US1["BEGIN TRANSACTION"] --> US2{"exp_ch has items?"}
|
||||
US2 -- yes --> US3["Receive GeneratedUser"]
|
||||
US3 --> US4["ProcessUser(user)"]
|
||||
US4 --> US5["PipelineLogger.Log(Info, UserGeneration, city, user_id, sqlite)"]
|
||||
US5 --> US6["Append → user_pool_"]
|
||||
US6 --> US7{"Batch size reached?"}
|
||||
US7 -- yes --> US8["COMMIT and BEGIN"] --> US2
|
||||
US7 -- no --> US2
|
||||
US2 -- no --> US9["COMMIT (Final)"]
|
||||
end
|
||||
end
|
||||
|
||||
UCreate --> UP1
|
||||
UP2 -.->|loc_ch| UL1
|
||||
UL9 -.->|exp_ch| US2
|
||||
UL9 & US9 --> UserJoin["Orchestrator: Join LLM Worker, SQLite Worker"]
|
||||
|
||||
UserJoin --> BreweryPhase["Orchestrator: RunBreweryPhase(sampled_locations)"]
|
||||
BreweryPhase --> BCreate["Create BoundedChannels (loc_ch, geo_ch, exp_ch)"]
|
||||
|
||||
subgraph BREWGEN["Phase 1a — Brewery Generation (concurrent)"]
|
||||
direction LR
|
||||
subgraph B_PROD["Orchestrator (producer)"]
|
||||
BP1["Loop: sample User from user_pool_, pair with Location"]
|
||||
BP1 --> BP2["Send BreweryTask(Location, User) → loc_ch"]
|
||||
BP2 --> BP3["Close loc_ch"]
|
||||
end
|
||||
subgraph B_LLM["LLM Worker"]
|
||||
BL1{"loc_ch has items?"}
|
||||
BL1 -- yes --> BL2["Receive BreweryTask(Location, User)"]
|
||||
BL2 --> BL3["GetLocationContextFromCache(task.location) — guaranteed cache hit"]
|
||||
BL3 --> BL4["GenerateBrewery(enriched_city, context, task.user) via DataGenerator<br/>KV cache stays warm; brewery linked to sampled owner_user_id"]
|
||||
BL4 --> BL5["PipelineLogger.Log(Info, BreweryGeneration, city, brewery_id, llm)"]
|
||||
BL5 --> BL6["Send GeneratedBrewery → geo_ch"]
|
||||
BL6 --> BL1
|
||||
BL1 -- no --> BL7["Close geo_ch"]
|
||||
end
|
||||
subgraph B_GEO["Geocode Worker"]
|
||||
BG1{"geo_ch has items?"}
|
||||
BG1 -- yes --> BG2["Receive GeneratedBrewery"]
|
||||
BG2 --> BG3["AddressService.ReverseGeocode(brewery.longitude, brewery.latitude)<br/>rate-limited independently of LLM throughput — NominatimAddressService sleeps 1s per request; MockAddressService (--mocked) returns a fixed placeholder, no network call"]
|
||||
BG3 --> BG4{"Address resolved?"}
|
||||
BG4 -- yes --> BG5["Attach resolved AddressLookupResult to brewery"]
|
||||
BG4 -- no --> BG6["Log warning: could not reverse geocode; attach placeholder address_line1, empty postal_code — brewery is never dropped"]
|
||||
BG5 --> BG7["PipelineLogger.Log(Info, BreweryGeneration, city, brewery_id, geocode)"]
|
||||
BG6 --> BG7
|
||||
BG7 --> BG8["Send GeneratedBrewery → exp_ch"]
|
||||
BG8 --> BG1
|
||||
BG1 -- no --> BG9["Close exp_ch"]
|
||||
end
|
||||
subgraph B_SQL["SQLite Worker"]
|
||||
BS1["BEGIN TRANSACTION"] --> BS2{"exp_ch has items?"}
|
||||
BS2 -- yes --> BS3["Receive GeneratedBrewery"]
|
||||
BS3 --> BS4["ProcessBrewery(brewery)"]
|
||||
BS4 --> BS5["PipelineLogger.Log(Info, BreweryGeneration, city, brewery_id, sqlite)"]
|
||||
BS5 --> BS6["Append → brewery_pool_"]
|
||||
BS6 --> BS7{"Batch size reached?"}
|
||||
BS7 -- yes --> BS8["COMMIT and BEGIN"] --> BS2
|
||||
BS7 -- no --> BS2
|
||||
BS2 -- no --> BS9["COMMIT (Final)"]
|
||||
end
|
||||
end
|
||||
|
||||
BCreate --> BP1
|
||||
BP3 -.->|loc_ch| BL1
|
||||
BL7 -.->|geo_ch| BG1
|
||||
BG9 -.->|exp_ch| BS2
|
||||
BL7 & BG9 & BS9 --> BrewJoin["Orchestrator: Join LLM Worker, Geocode Worker, SQLite Worker<br/>brewery_pool_ is now fully populated"]
|
||||
|
||||
BrewJoin --> BeerPhase["Orchestrator: RunBeerPhase()"]
|
||||
BeerPhase --> BeCreate["Create BoundedChannels (brew_ch, exp_ch)"]
|
||||
|
||||
subgraph BEERGEN["Phase 1b — Beer Generation (concurrent)"]
|
||||
direction LR
|
||||
subgraph BE_PROD["Orchestrator (producer)"]
|
||||
BeP1["Loop: send Breweries → brew_ch"]
|
||||
BeP1 --> BeP2["Close brew_ch"]
|
||||
end
|
||||
subgraph BE_LLM["LLM Worker"]
|
||||
BeL1{"brew_ch has items?"}
|
||||
BeL1 -- yes --> BeL2["Receive GeneratedBrewery"]
|
||||
BeL2 --> BeL3["IBeerSelectionStrategy.SelectStyles(brewery, beer_style_palette_)"]
|
||||
BeL3 --> BeL4{"For each selected BeerStyle"}
|
||||
BeL4 -- remaining --> BeL5["GetStyleContextFromCache(style) — guaranteed cache hit; KV cache stays warm across all beer generations in this phase"]
|
||||
BeL5 --> BeL6["GenerateBeer(brewery, style_context) via DataGenerator"]
|
||||
BeL6 --> BeL7["Attach GeneratedBeer to bundle"] --> BeL4
|
||||
BeL4 -- done --> BeL8["PipelineLogger.Log(Info, BeerGeneration, city, brewery_id, llm)"]
|
||||
BeL8 --> BeL9["Send BeersBundle → exp_ch"]
|
||||
BeL9 --> BeL1
|
||||
BeL1 -- no --> BeL10["Close exp_ch"]
|
||||
end
|
||||
subgraph BE_SQL["SQLite Worker"]
|
||||
BeS1["BEGIN TRANSACTION"] --> BeS2{"exp_ch has items?"}
|
||||
BeS2 -- yes --> BeS3["Receive BeersBundle"]
|
||||
BeS3 --> BeS4{"For each beer in bundle"}
|
||||
BeS4 -- remaining --> BeS5["Set beer.brewery_id from bundle"]
|
||||
BeS5 --> BeS6["ProcessBeer(beer)"]
|
||||
BeS6 --> BeS7["Append → beer_pool_"] --> BeS4
|
||||
BeS4 -- done --> BeS8["PipelineLogger.Log(Info, BeerGeneration, city, brewery_id, sqlite)"]
|
||||
BeS8 --> BeS9{"Batch size reached?"}
|
||||
BeS9 -- yes --> BeS10["COMMIT and BEGIN"] --> BeS2
|
||||
BeS9 -- no --> BeS2
|
||||
BeS2 -- no --> BeS11["COMMIT (Final)"]
|
||||
end
|
||||
end
|
||||
|
||||
BeCreate --> BeP1
|
||||
BeP2 -.->|brew_ch| BeL1
|
||||
BeL10 -.->|exp_ch| BeS2
|
||||
BeL10 & BeS11 --> BeerJoin["Orchestrator: Join LLM Worker, SQLite Worker<br/>brewery_pool_ and beer_pool_ are now completely populated — Checkin and Follow phases may run in parallel"]
|
||||
|
||||
BeerJoin --> CheckinPhase["Orchestrator: RunCheckinPhase()"]
|
||||
BeerJoin --> FollowPhase["Orchestrator: RunFollowPhase()"]
|
||||
|
||||
subgraph CHECKIN["Checkin Generation"]
|
||||
CK1["ICheckinDistributionStrategy.AssignActivityWeights(user_pool_)<br/>weights seeded from persona.checkin_weight; J-curve profile emerges from persona distribution"]
|
||||
CK1 --> CK2["BEGIN TRANSACTION"]
|
||||
CK2 --> CK3{"For each GeneratedUser in user_pool_"}
|
||||
CK3 -- remaining --> CK4["CheckinsForUser(user, brewery_pool_.size())"]
|
||||
CK4 --> CK5{"For each checkin index"}
|
||||
CK5 -- remaining --> CK6["TimestampFor(user, index)"]
|
||||
CK6 --> CK7["Select brewery from brewery_pool_"]
|
||||
CK7 --> CK8["GenerateCheckin(user, brewery, timestamp) via DataGenerator"]
|
||||
CK8 --> CK9["ProcessCheckin(checkin)"]
|
||||
CK9 --> CK10["PipelineLogger.Log(Info, CheckinGeneration, nullopt, checkin_id, sqlite)"]
|
||||
CK10 --> CK11["Append → checkin_pool_"]
|
||||
CK11 --> CK12{"Batch size reached?"}
|
||||
CK12 -- yes --> CK13["COMMIT and BEGIN"] --> CK5
|
||||
CK12 -- no --> CK5
|
||||
CK5 -- done --> CK3
|
||||
CK3 -- done --> CK14["COMMIT (Final)"]
|
||||
end
|
||||
|
||||
subgraph FOLLOW["Follow Generation"]
|
||||
FL1["IFollowGenerationStrategy.AssignFollowWeights(user_pool_)<br/>RandomFollowStrategy: uniform weights; ActivityWeightedFollowStrategy: derived from user.activity_weight so high-activity users attract more followers"]
|
||||
FL1 --> FL2["BEGIN TRANSACTION"]
|
||||
FL2 --> FL3["IFollowGenerationStrategy.GenerateFollows(user_pool_)<br/>self-follow constraint (follower_id != followed_id) enforced here and at the DB schema level"]
|
||||
FL3 --> FL4{"For each GeneratedFollow"}
|
||||
FL4 -- remaining --> FL5["ProcessFollow(follow)"]
|
||||
FL5 --> FL6["PipelineLogger.Log(Info, FollowGeneration, nullopt, follower_id, sqlite)"]
|
||||
FL6 --> FL7["Append → follow_pool_"]
|
||||
FL7 --> FL8{"Batch size reached?"}
|
||||
FL8 -- yes --> FL9["COMMIT and BEGIN"] --> FL4
|
||||
FL8 -- no --> FL4
|
||||
FL4 -- done --> FL10["COMMIT (Final)"]
|
||||
end
|
||||
|
||||
CheckinPhase --> CK1
|
||||
FollowPhase --> FL1
|
||||
CK14 & FL10 --> PhaseJoin["Orchestrator: Join CheckinPhase, FollowPhase<br/>checkin_pool_ and follow_pool_ are now fully populated"]
|
||||
|
||||
PhaseJoin --> RatingPhase["Orchestrator: RunRatingPhase()<br/>beer selection biased by user.persona.style_affinities and abv_range; rating skew modulated per persona"]
|
||||
RatingPhase --> RT1["BEGIN TRANSACTION"]
|
||||
RT1 --> RT2{"For each GeneratedCheckin in checkin_pool_"}
|
||||
RT2 -- remaining --> RT3["Match brewery_id, select beer from beer_pool_ (same brewery_id, biased by persona affinities)"]
|
||||
RT3 --> RT4{"Beer exists for brewery?"}
|
||||
RT4 -- yes --> RT5["GenerateRating(user, beer, checkin_id) via DataGenerator"]
|
||||
RT5 --> RT6["ProcessRating(rating)"]
|
||||
RT6 --> RT7["PipelineLogger.Log(Info, RatingGeneration, nullopt, rating_id, sqlite)"]
|
||||
RT7 --> RT8{"Batch size reached?"}
|
||||
RT8 -- yes --> RT9["COMMIT and BEGIN"] --> RT2
|
||||
RT8 -- no --> RT2
|
||||
RT4 -- no --> RT10["Log warning; skip — brewery has no beers"] --> RT2
|
||||
RT2 -- done --> RT11["COMMIT (Final)"]
|
||||
|
||||
RT11 --> Teardown["Orchestrator: Finalize SqliteExportService — safely closes the DB connection"]
|
||||
Teardown --> CloseLogCh["Orchestrator: close log_ch"]
|
||||
CloseLogCh --> MainDone["Main: spdlog::info 'Pipeline complete in X ms'"]
|
||||
MainDone --> JoinLog["Main: Join Log Worker — drain guarantees no LogEntry is dropped at shutdown"]
|
||||
JoinLog --> Stop2([stop])
|
||||
@@ -1,372 +0,0 @@
|
||||
@startuml biergarten_activity
|
||||
!include ../biergarten-weizen-theme.puml
|
||||
skinparam defaultFontSize 13
|
||||
skinparam titleFontSize 20
|
||||
|
||||
title The Biergarten Data Pipeline — Activity Diagram
|
||||
|
||||
|Main|
|
||||
start
|
||||
:ParseArguments(argc, argv);
|
||||
if (Invalid args?) then (yes)
|
||||
:spdlog::error;
|
||||
stop
|
||||
else (no)
|
||||
endif
|
||||
:Init OpenSSL global state & LlamaBackendState;
|
||||
:Build DI injector;
|
||||
|
||||
:Initialize SqliteExportService;
|
||||
note right
|
||||
Opens SQLite connection.
|
||||
(Transactions are now managed
|
||||
per-phase via batching).
|
||||
end note
|
||||
|
||||
:Create BoundedChannel<LogEntry> log_ch;
|
||||
:Spawn Log Worker thread;
|
||||
note right
|
||||
Log worker drains log_ch for the
|
||||
entire pipeline lifetime.
|
||||
All workers emit LogEntry structs
|
||||
via PipelineLogger -- never spdlog directly.
|
||||
end note
|
||||
|
||||
:BiergartenPipelineOrchestrator::Run();
|
||||
|BiergartenPipelineOrchestrator::Run()|
|
||||
|
||||
fork
|
||||
:JsonLoader::LoadBeerStyles("beer-styles.json");
|
||||
:EnrichmentService::PreWarmBeerStyleCache(beer_styles);
|
||||
fork again
|
||||
:JsonLoader::LoadLocations("locations.json");
|
||||
:EnrichmentService::PreWarmLocationCache(sampled_locations);
|
||||
end fork
|
||||
fork
|
||||
:JsonLoader::LoadForenamesByCountry(\n "forenames-by-country.json");
|
||||
fork again
|
||||
:JsonLoader::LoadSurnamesByCountry(\n "surnames-by-country.json");
|
||||
fork again
|
||||
:JsonLoader::LoadPersonas("personas.json");
|
||||
end fork
|
||||
note right
|
||||
Each call is memoized after its first parse
|
||||
(implemented today). MockCuratedDataService
|
||||
returns a fixed in-memory dataset instead
|
||||
under --mocked, skipping these JSON files.
|
||||
end note
|
||||
|
||||
' ═══════════════════════════════════════════
|
||||
' PHASE 0 — USER GENERATION
|
||||
' ═══════════════════════════════════════════
|
||||
|Orchestrator|
|
||||
:RunUserPhase(sampled_locations);
|
||||
:Create BoundedChannels\n(loc_ch, exp_ch);
|
||||
|
||||
fork
|
||||
|Orchestrator|
|
||||
:Loop: Send Locations -> loc_ch;
|
||||
:Close loc_ch;
|
||||
note right
|
||||
Producer closes loc_ch.
|
||||
LLM Worker while loop
|
||||
terminates on empty + closed.
|
||||
end note
|
||||
fork again
|
||||
|LLM Worker|
|
||||
while (loc_ch has items?) is (yes)
|
||||
:Receive Location;
|
||||
|
||||
:GetLocationContextFromCache(location);
|
||||
note right
|
||||
Guaranteed cache hit from startup.
|
||||
end note
|
||||
|
||||
:IPersonaSelectionStrategy::SelectPersona(\n personas_palette_);
|
||||
note right
|
||||
Guaranteed cache hit from startup.
|
||||
Returns a Persona struct carrying
|
||||
style_affinities, abv_range,
|
||||
ibu_preference, checkin_weight.
|
||||
end note
|
||||
|
||||
:SampleName(forenames_by_country,\n surnames_by_country, location.iso3166_1, rng);
|
||||
note right
|
||||
Deterministic lookup -- no LLM involved.
|
||||
Free function (implemented today in
|
||||
generate_users.cc), not a class method.
|
||||
Name selected from pre-keyed maps
|
||||
and passed into the generation prompt.
|
||||
Skips the city if either map has no
|
||||
entry for iso3166_1.
|
||||
end note
|
||||
|
||||
:GenerateUser(enriched_city, persona, sampled_name)\nvia DataGenerator;
|
||||
note right
|
||||
LLM receives: EnrichedCity context + persona
|
||||
description + sampled name. Generates
|
||||
bio and preference signals grounded
|
||||
in locale and persona.
|
||||
end note
|
||||
|
||||
:PipelineLogger::Log(Info, UserGeneration,\n city, user_id, "llm");
|
||||
:Send GeneratedUser -> exp_ch;
|
||||
endwhile (no)
|
||||
:Close exp_ch;
|
||||
note right
|
||||
Producer closes exp_ch.
|
||||
SQLite Worker while loop
|
||||
terminates on empty + closed.
|
||||
end note
|
||||
fork again
|
||||
|SQLite Worker|
|
||||
:BEGIN TRANSACTION;
|
||||
while (exp_ch has items?) is (yes)
|
||||
:Receive GeneratedUser;
|
||||
:ProcessUser(user);
|
||||
:PipelineLogger::Log(Info, UserGeneration,\n city, user_id, "sqlite");
|
||||
:Append -> user_pool_;
|
||||
if (Batch size reached?) then (yes)
|
||||
:COMMIT & BEGIN;
|
||||
else (no)
|
||||
endif
|
||||
endwhile (no)
|
||||
:COMMIT (Final);
|
||||
end fork
|
||||
|
||||
|Orchestrator|
|
||||
:Join LLM Worker, SQLite Worker;
|
||||
|
||||
' ═══════════════════════════════════════════
|
||||
' PHASE 1a — BREWERY GENERATION
|
||||
' ═══════════════════════════════════════════
|
||||
:RunBreweryPhase(sampled_locations);
|
||||
:Create BoundedChannels\n(loc_ch, exp_ch);
|
||||
|
||||
fork
|
||||
|Orchestrator|
|
||||
:Loop: Sample User from user_pool_
|
||||
and pair with Location;
|
||||
:Send BreweryTask(Location, User) -> loc_ch;
|
||||
:Close loc_ch;
|
||||
fork again
|
||||
|LLM Worker|
|
||||
while (loc_ch has items?) is (yes)
|
||||
:Receive BreweryTask(Location, User);
|
||||
|
||||
:GetLocationContextFromCache(task.location);
|
||||
note right
|
||||
Guaranteed cache hit from startup.
|
||||
end note
|
||||
|
||||
:GenerateBrewery(enriched_city, context, task.user)\nvia DataGenerator;
|
||||
note right
|
||||
KV cache stays warm.
|
||||
Brewery is linked to the sampled owner_user_id.
|
||||
end note
|
||||
:PipelineLogger::Log(Info,\n BreweryGeneration,\n city, brewery_id, "llm");
|
||||
:Send GeneratedBrewery -> exp_ch;
|
||||
endwhile (no)
|
||||
:Close exp_ch;
|
||||
fork again
|
||||
|SQLite Worker|
|
||||
:BEGIN TRANSACTION;
|
||||
while (exp_ch has items?) is (yes)
|
||||
:Receive GeneratedBrewery;
|
||||
:ProcessBrewery(brewery);
|
||||
:PipelineLogger::Log(Info,\n BreweryGeneration,\n city, brewery_id, "sqlite");
|
||||
:Append -> brewery_pool_;
|
||||
if (Batch size reached?) then (yes)
|
||||
:COMMIT & BEGIN;
|
||||
else (no)
|
||||
endif
|
||||
endwhile (no)
|
||||
:COMMIT (Final);
|
||||
end fork
|
||||
|
||||
|Orchestrator|
|
||||
:Join LLM Worker, SQLite Worker;
|
||||
note right
|
||||
brewery_pool_ is now fully populated.
|
||||
Phase 1b may begin.
|
||||
end note
|
||||
|
||||
' ═══════════════════════════════════════════
|
||||
' PHASE 1b — BEER GENERATION
|
||||
' ═══════════════════════════════════════════
|
||||
:RunBeerPhase();
|
||||
:Create BoundedChannels\n(brew_ch, exp_ch);
|
||||
|
||||
fork
|
||||
|Orchestrator|
|
||||
:Loop: Send Breweries -> brew_ch;
|
||||
:Close brew_ch;
|
||||
fork again
|
||||
|LLM Worker|
|
||||
while (brew_ch has items?) is (yes)
|
||||
:Receive GeneratedBrewery;
|
||||
:IBeerSelectionStrategy::SelectStyles(\n brewery, beer_style_palette_);
|
||||
|
||||
while (For each selected BeerStyle?) is (remaining)
|
||||
:GetStyleContextFromCache(style);
|
||||
note right
|
||||
Guaranteed cache hit from startup.
|
||||
KV cache stays warm across all
|
||||
beer generations -- system prompt
|
||||
does not change within this phase.
|
||||
end note
|
||||
:GenerateBeer(brewery, style_context)\nvia DataGenerator;
|
||||
:Attach GeneratedBeer to bundle;
|
||||
endwhile (done)
|
||||
|
||||
:PipelineLogger::Log(Info,\n BeerGeneration,\n city, brewery_id, "llm");
|
||||
:Send BeersBundle -> exp_ch;
|
||||
endwhile (no)
|
||||
:Close exp_ch;
|
||||
fork again
|
||||
|SQLite Worker|
|
||||
:BEGIN TRANSACTION;
|
||||
while (exp_ch has items?) is (yes)
|
||||
:Receive BeersBundle;
|
||||
while (For each beer in bundle?) is (remaining)
|
||||
:Set beer.brewery_id from bundle;
|
||||
:ProcessBeer(beer);
|
||||
:Append -> beer_pool_;
|
||||
endwhile (done)
|
||||
:PipelineLogger::Log(Info,\n BeerGeneration,\n city, brewery_id, "sqlite");
|
||||
if (Batch size reached?) then (yes)
|
||||
:COMMIT & BEGIN;
|
||||
else (no)
|
||||
endif
|
||||
endwhile (no)
|
||||
:COMMIT (Final);
|
||||
end fork
|
||||
|
||||
|Orchestrator|
|
||||
:Join LLM Worker, SQLite Worker;
|
||||
note right
|
||||
Both brewery_pool_ and beer_pool_
|
||||
are now completely populated.
|
||||
Checkin and Follow phases may
|
||||
now run in parallel.
|
||||
end note
|
||||
|
||||
' ═══════════════════════════════════════════
|
||||
' PHASE 2 — CHECKIN + FOLLOW GENERATION
|
||||
' (parallel — both depend only on user_pool_
|
||||
' and brewery_pool_ being fully populated)
|
||||
' ═══════════════════════════════════════════
|
||||
fork
|
||||
|Orchestrator|
|
||||
:RunCheckinPhase();
|
||||
:ICheckinDistributionStrategy::\nAssignActivityWeights(user_pool_);
|
||||
note right
|
||||
Weights seeded from each user's
|
||||
persona.checkin_weight. J-curve profile
|
||||
emerges from persona distribution.
|
||||
end note
|
||||
|
||||
:BEGIN TRANSACTION;
|
||||
while (For each GeneratedUser in user_pool_?) is (remaining)
|
||||
:CheckinsForUser(user, brewery_pool_.size());
|
||||
while (For each checkin index?) is (remaining)
|
||||
:TimestampFor(user, index);
|
||||
:Select brewery from brewery_pool_;
|
||||
:GenerateCheckin(user, brewery, timestamp)\nvia DataGenerator;
|
||||
:ProcessCheckin(checkin);
|
||||
:PipelineLogger::Log(Info, CheckinGeneration,\n nullopt, checkin_id, "sqlite");
|
||||
:Append -> checkin_pool_;
|
||||
if (Batch size reached?) then (yes)
|
||||
:COMMIT & BEGIN;
|
||||
else (no)
|
||||
endif
|
||||
endwhile (done)
|
||||
endwhile (done)
|
||||
:COMMIT (Final);
|
||||
|
||||
fork again
|
||||
|Orchestrator|
|
||||
:RunFollowPhase();
|
||||
:IFollowGenerationStrategy::\nAssignFollowWeights(user_pool_);
|
||||
note right
|
||||
For RandomFollowStrategy, weights
|
||||
are uniform. For ActivityWeightedFollowStrategy,
|
||||
weights derived from user.activity_weight
|
||||
so high-activity users attract more followers.
|
||||
end note
|
||||
|
||||
:BEGIN TRANSACTION;
|
||||
:IFollowGenerationStrategy::\nGenerateFollows(user_pool_);
|
||||
note right
|
||||
Self-follow constraint (follower_id != followed_id)
|
||||
enforced here and at the DB schema level.
|
||||
end note
|
||||
while (For each GeneratedFollow?) is (remaining)
|
||||
:ProcessFollow(follow);
|
||||
:PipelineLogger::Log(Info, FollowGeneration,\n nullopt, follower_id, "sqlite");
|
||||
:Append -> follow_pool_;
|
||||
if (Batch size reached?) then (yes)
|
||||
:COMMIT & BEGIN;
|
||||
else (no)
|
||||
endif
|
||||
endwhile (done)
|
||||
:COMMIT (Final);
|
||||
|
||||
end fork
|
||||
|
||||
|Orchestrator|
|
||||
:Join CheckinPhase, FollowPhase;
|
||||
note right
|
||||
checkin_pool_ and follow_pool_
|
||||
are now fully populated.
|
||||
Rating phase may begin.
|
||||
end note
|
||||
|
||||
' ═══════════════════════════════════════════
|
||||
' PHASE 3 — RATING GENERATION
|
||||
' ═══════════════════════════════════════════
|
||||
:RunRatingPhase();
|
||||
note right
|
||||
Beer selection biased by
|
||||
user.persona.style_affinities and abv_range.
|
||||
Rating skew modulated per persona.
|
||||
end note
|
||||
|
||||
:BEGIN TRANSACTION;
|
||||
while (For each GeneratedCheckin in checkin_pool_?) is (remaining)
|
||||
:Match brewery_id, select beer from beer_pool_\n(same brewery_id, biased by persona affinities);
|
||||
if (Beer exists for brewery?) then (yes)
|
||||
:GenerateRating(user, beer, checkin_id)\nvia DataGenerator;
|
||||
:ProcessRating(rating);
|
||||
:PipelineLogger::Log(Info, RatingGeneration,\n nullopt, rating_id, "sqlite");
|
||||
if (Batch size reached?) then (yes)
|
||||
:COMMIT & BEGIN;
|
||||
else (no)
|
||||
endif
|
||||
else (no)
|
||||
:PipelineLogger::Log(Warn, RatingGeneration,\n nullopt, brewery_id, "sqlite");
|
||||
:Skip -- brewery has no beers;
|
||||
endif
|
||||
endwhile (done)
|
||||
:COMMIT (Final);
|
||||
|
||||
' ═══════════════════════════════════════════
|
||||
' TEARDOWN
|
||||
' ═══════════════════════════════════════════
|
||||
|Orchestrator|
|
||||
:Finalize SqliteExportService;
|
||||
note right
|
||||
Safely closes the DB connection.
|
||||
end note
|
||||
:Close log_ch;
|
||||
|
||||
|Main|
|
||||
:spdlog::info "Pipeline complete in X ms";
|
||||
:Join Log Worker;
|
||||
note right
|
||||
Drain guarantees no LogEntry is
|
||||
dropped at shutdown.
|
||||
end note
|
||||
stop
|
||||
|
||||
@enduml
|
||||
17
docs/pipeline/diagrams/planned/class.md
Normal file
17
docs/pipeline/diagrams/planned/class.md
Normal file
@@ -0,0 +1,17 @@
|
||||
# Biergarten Data Pipeline — Class Diagram (Planned)
|
||||
|
||||
```mermaid
|
||||
--8<-- "pipeline/diagrams/planned/class.mmd"
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- `Logger`, `PipelineLogger`, and `LogWorker` are referenced only in
|
||||
relationships in the source PlantUML (not given their own class blocks there
|
||||
either) — they appear here as implicit nodes for the same reason.
|
||||
- Package grouping from the PlantUML source (`Domain: Models`,
|
||||
`Domain: Application Configuration`, `Domain: Policy`,
|
||||
`Infrastructure: Logging`, `Infrastructure: Pipeline Channel`,
|
||||
`Infrastructure: Data Preloading`, `Infrastructure: Enrichment`,
|
||||
`Infrastructure: Prompting`, `Infrastructure: Data Generation`,
|
||||
`Infrastructure: Data Export`) is preserved as Mermaid `namespace` blocks.
|
||||
643
docs/pipeline/diagrams/planned/class.mmd
Normal file
643
docs/pipeline/diagrams/planned/class.mmd
Normal file
@@ -0,0 +1,643 @@
|
||||
classDiagram
|
||||
namespace DomainModels {
|
||||
class Location {
|
||||
+string city
|
||||
+string state_province
|
||||
+string iso3166_2
|
||||
+string country
|
||||
+string iso3166_1
|
||||
+vector~string~ local_languages
|
||||
+double latitude
|
||||
+double longitude
|
||||
}
|
||||
|
||||
class LocationContext {
|
||||
+string text
|
||||
+Completeness completeness
|
||||
+size_t char_count
|
||||
}
|
||||
|
||||
class Completeness {
|
||||
<<enumeration>>
|
||||
Full
|
||||
Partial
|
||||
Absent
|
||||
}
|
||||
|
||||
class EnrichedCity {
|
||||
+Location location
|
||||
+LocationContext context
|
||||
}
|
||||
|
||||
class BeerStyle {
|
||||
+string name
|
||||
+string description
|
||||
+float min_abv
|
||||
+float max_abv
|
||||
+int min_ibu
|
||||
+int max_ibu
|
||||
}
|
||||
|
||||
class BreweryResult {
|
||||
+string name_en
|
||||
+string description_en
|
||||
+string name_local
|
||||
+string description_local
|
||||
}
|
||||
|
||||
class BeerResult {
|
||||
+string name_en
|
||||
+string description_en
|
||||
+string name_local
|
||||
+string description_local
|
||||
+string style
|
||||
+float abv
|
||||
+int ibu
|
||||
}
|
||||
|
||||
class UserResult {
|
||||
+string first_name
|
||||
+string last_name
|
||||
+string gender
|
||||
+string username
|
||||
+string bio
|
||||
+float activity_weight
|
||||
}
|
||||
|
||||
class Name {
|
||||
+string first_name
|
||||
+string last_name
|
||||
+string gender
|
||||
}
|
||||
|
||||
class ForenameEntry {
|
||||
+string name
|
||||
+string gender
|
||||
}
|
||||
|
||||
class CheckinResult {
|
||||
+string checked_in_at
|
||||
+string note
|
||||
}
|
||||
|
||||
class RatingResult {
|
||||
+float score
|
||||
+string note
|
||||
}
|
||||
|
||||
class GenerationMetadata {
|
||||
+uint64_t generation_id
|
||||
+string generated_time
|
||||
+bool context_provided
|
||||
+string generated_with
|
||||
}
|
||||
|
||||
class GeneratedBrewery {
|
||||
+uint64_t brewery_id
|
||||
+Location location
|
||||
+BreweryResult brewery
|
||||
+Completeness context_completeness
|
||||
+GenerationMetadata metadata
|
||||
}
|
||||
|
||||
class GeneratedBeer {
|
||||
+uint64_t beer_id
|
||||
+uint64_t brewery_id
|
||||
+Location location
|
||||
+BeerStyle style
|
||||
+BeerResult beer
|
||||
+GenerationMetadata metadata
|
||||
}
|
||||
|
||||
class GeneratedUser {
|
||||
+uint64_t user_id
|
||||
+Location location
|
||||
+UserResult user
|
||||
+string email
|
||||
+string date_of_birth
|
||||
+string password
|
||||
+GenerationMetadata metadata
|
||||
}
|
||||
|
||||
class GeneratedCheckin {
|
||||
+uint64_t checkin_id
|
||||
+uint64_t user_id
|
||||
+uint64_t brewery_id
|
||||
+CheckinResult checkin
|
||||
+GenerationMetadata metadata
|
||||
}
|
||||
|
||||
class GeneratedRating {
|
||||
+uint64_t user_id
|
||||
+uint64_t beer_id
|
||||
+uint64_t checkin_id
|
||||
+RatingResult rating
|
||||
+GenerationMetadata metadata
|
||||
}
|
||||
|
||||
class GeneratedFollow {
|
||||
+uint64_t follower_id
|
||||
+uint64_t followed_id
|
||||
+GenerationMetadata metadata
|
||||
}
|
||||
|
||||
class UserPersona {
|
||||
+string name
|
||||
+string description
|
||||
+vector~string~ style_affinities
|
||||
}
|
||||
}
|
||||
note for ForenameEntry "forename_list/surname_list are unordered_set aliases, keyed by ISO 3166-1 country code via ICuratedDataService. SampleName() is a free function, not a method, sourced from popular-names-by-country-dataset (CC0)."
|
||||
note for GeneratedUser "email, date_of_birth, and password are generated programmatically by the orchestrator (not LLM-authored) so a downstream auth-account seeding consumer can register real accounts from this export."
|
||||
ForenameEntry ..> Name : SampleName() produces
|
||||
LocationContext *-- Completeness
|
||||
|
||||
namespace DomainApplicationConfiguration {
|
||||
class SamplingOptions {
|
||||
+float temperature
|
||||
+float top_p
|
||||
+uint32_t top_k
|
||||
+uint32_t n_ctx
|
||||
+int seed
|
||||
}
|
||||
|
||||
class GeneratorMode {
|
||||
<<enumeration>>
|
||||
kMock
|
||||
kLlama
|
||||
kOpenAI
|
||||
}
|
||||
|
||||
class GeneratorOptions {
|
||||
+path model_path
|
||||
+GeneratorMode mode
|
||||
+optional~SamplingOptions~ sampling
|
||||
+string openai_api_key
|
||||
+string openai_model
|
||||
}
|
||||
|
||||
class PipelineOptions {
|
||||
+path output_path
|
||||
+path log_path
|
||||
}
|
||||
|
||||
class ApplicationOptions {
|
||||
+GeneratorOptions generator
|
||||
+PipelineOptions pipeline
|
||||
}
|
||||
}
|
||||
note for GeneratorOptions "Already implemented (models.h), not planned: the three-way GeneratorMode split (kMock/kLlama/kOpenAI) and the openai_api_key/openai_model fields ship today, ahead of the rest of this diagram."
|
||||
ApplicationOptions *-- GeneratorOptions
|
||||
ApplicationOptions *-- PipelineOptions
|
||||
GeneratorOptions *-- GeneratorMode
|
||||
GeneratorOptions o-- SamplingOptions
|
||||
|
||||
namespace DomainPolicy {
|
||||
class ContextStrategy {
|
||||
<<interface>>
|
||||
+QueriesFor(loc) vector~string~
|
||||
+MaxContextChars() size_t
|
||||
}
|
||||
|
||||
class BreweryContextStrategy {
|
||||
+QueriesFor(loc) vector~string~
|
||||
+MaxContextChars() size_t
|
||||
}
|
||||
|
||||
class BeerContextStrategy {
|
||||
+QueriesFor(loc) vector~string~
|
||||
+MaxContextChars() size_t
|
||||
}
|
||||
|
||||
class SamplingStrategy {
|
||||
<<interface>>
|
||||
+Sample(locations) vector~Location~
|
||||
}
|
||||
|
||||
class UniformSamplingStrategy {
|
||||
-size_t sample_size_
|
||||
+Sample(locations) vector~Location~
|
||||
}
|
||||
|
||||
class BeerSelectionStrategy {
|
||||
<<interface>>
|
||||
+SelectStyles(brewery, palette) vector~BeerStyle~
|
||||
}
|
||||
|
||||
class RandomBeerSelectionStrategy {
|
||||
-mt19937 rng_
|
||||
-size_t min_beers_
|
||||
-size_t max_beers_
|
||||
+SelectStyles(brewery, palette) vector~BeerStyle~
|
||||
}
|
||||
|
||||
class CheckinDistributionStrategy {
|
||||
<<interface>>
|
||||
+AssignActivityWeights(users) void
|
||||
+CheckinsForUser(user, brewery_count) size_t
|
||||
+TimestampFor(user, index) string
|
||||
}
|
||||
|
||||
class JCurveCheckinStrategy {
|
||||
-mt19937 rng_
|
||||
+AssignActivityWeights(users) void
|
||||
+CheckinsForUser(user, brewery_count) size_t
|
||||
+TimestampFor(user, index) string
|
||||
}
|
||||
|
||||
class RandomCheckinStrategy {
|
||||
-mt19937 rng_
|
||||
-size_t min_checkins_
|
||||
-size_t max_checkins_
|
||||
+AssignActivityWeights(users) void
|
||||
+CheckinsForUser(user, brewery_count) size_t
|
||||
+TimestampFor(user, index) string
|
||||
}
|
||||
|
||||
class FollowGenerationStrategy {
|
||||
<<interface>>
|
||||
+GenerateFollows(users) vector~GeneratedFollow~
|
||||
}
|
||||
|
||||
class RandomFollowStrategy {
|
||||
-mt19937 rng_
|
||||
-size_t min_follows_
|
||||
-size_t max_follows_
|
||||
+GenerateFollows(users) vector~GeneratedFollow~
|
||||
}
|
||||
|
||||
class ActivityWeightedFollowStrategy {
|
||||
-mt19937 rng_
|
||||
-size_t min_follows_
|
||||
-size_t max_follows_
|
||||
+GenerateFollows(users) vector~GeneratedFollow~
|
||||
}
|
||||
}
|
||||
|
||||
namespace InfrastructureLogging {
|
||||
class LogLevel {
|
||||
<<enumeration>>
|
||||
Debug
|
||||
Info
|
||||
Warn
|
||||
Error
|
||||
}
|
||||
|
||||
class PipelinePhase {
|
||||
<<enumeration>>
|
||||
Startup
|
||||
UserGeneration
|
||||
BreweryAndBeerGeneration
|
||||
CheckinGeneration
|
||||
RatingGeneration
|
||||
FollowGeneration
|
||||
Teardown
|
||||
}
|
||||
|
||||
class LogEntry {
|
||||
+time_point timestamp
|
||||
+LogLevel level
|
||||
+PipelinePhase phase
|
||||
+string message
|
||||
+optional~string~ worker
|
||||
}
|
||||
|
||||
class ILogger {
|
||||
<<interface>>
|
||||
+Log(entry) void
|
||||
}
|
||||
|
||||
class LogProducer {
|
||||
-BoundedChannel~LogEntry~ channel_
|
||||
+Log(entry) void
|
||||
}
|
||||
|
||||
class LogDispatcher {
|
||||
-BoundedChannel~LogEntry~ channel_
|
||||
+Run() void
|
||||
-ToSpdlogLevel(level) level_enum
|
||||
}
|
||||
}
|
||||
LogEntry *-- LogLevel
|
||||
LogEntry *-- PipelinePhase
|
||||
ILogger <|.. LogProducer
|
||||
LogProducer ..> LogEntry : emits
|
||||
LogDispatcher ..> LogEntry : consumes
|
||||
|
||||
namespace InfrastructurePipelineChannel {
|
||||
class BoundedChannel~T~ {
|
||||
-queue~T~ queue_
|
||||
-mutex mutex_
|
||||
-condition_variable not_full_
|
||||
-condition_variable not_empty_
|
||||
-size_t capacity_
|
||||
-bool closed_
|
||||
+Send(item) void
|
||||
+Receive() optional~T~
|
||||
+Close() void
|
||||
}
|
||||
}
|
||||
|
||||
namespace InfrastructureDataPreloading {
|
||||
class ICuratedDataService {
|
||||
<<interface>>
|
||||
+LoadLocations(filepath) vector~Location~
|
||||
+LoadBeerStyles(filepath) vector~BeerStyle~
|
||||
+LoadPersonas(filepath) vector~UserPersona~
|
||||
+LoadForenamesByCountry(filepath) unordered_map~string, forename_list~
|
||||
+LoadSurnamesByCountry(filepath) unordered_map~string, surname_list~
|
||||
}
|
||||
|
||||
class JsonLoader {
|
||||
+LoadLocations(filepath) vector~Location~
|
||||
+LoadBeerStyles(filepath) vector~BeerStyle~
|
||||
+LoadPersonas(filepath) vector~UserPersona~
|
||||
+LoadForenamesByCountry(filepath) unordered_map~string, forename_list~
|
||||
+LoadSurnamesByCountry(filepath) unordered_map~string, surname_list~
|
||||
}
|
||||
|
||||
class MockCuratedDataService {
|
||||
+LoadLocations(filepath) vector~Location~
|
||||
+LoadBeerStyles(filepath) vector~BeerStyle~
|
||||
+LoadPersonas(filepath) vector~UserPersona~
|
||||
+LoadForenamesByCountry(filepath) unordered_map~string, forename_list~
|
||||
+LoadSurnamesByCountry(filepath) unordered_map~string, surname_list~
|
||||
}
|
||||
}
|
||||
note for JsonLoader "Each Load* call is memoized after its first parse — implemented today, ahead of LoadBeerStyles which is still planned."
|
||||
note for MockCuratedDataService "Fixed in-memory dataset for --mocked mode, implemented today for Locations/Personas/Forenames/Surnames; would need a LoadBeerStyles fixture too once that method is implemented."
|
||||
|
||||
namespace InfrastructureEnrichment {
|
||||
class EnrichmentService {
|
||||
<<interface>>
|
||||
+GetLocationContext(loc, strategy) LocationContext
|
||||
}
|
||||
|
||||
class WikipediaService {
|
||||
-unique_ptr~WebClient~ client_
|
||||
-unordered_map~string, string~ extract_cache_
|
||||
+GetLocationContext(loc, strategy) LocationContext
|
||||
-FetchExtract(query) string
|
||||
}
|
||||
|
||||
class WebClient {
|
||||
<<interface>>
|
||||
+Get(url) string
|
||||
+UrlEncode(value) string
|
||||
}
|
||||
|
||||
class HttpWebClient {
|
||||
+Get(url) string
|
||||
+UrlEncode(value) string
|
||||
}
|
||||
}
|
||||
|
||||
namespace InfrastructureGeocoding {
|
||||
class AddressLookupResult {
|
||||
+string address_line1
|
||||
+string postal_code
|
||||
}
|
||||
|
||||
class AddressService {
|
||||
<<interface>>
|
||||
+ReverseGeocode(longitude, latitude) optional~AddressLookupResult~
|
||||
}
|
||||
|
||||
class NominatimAddressService {
|
||||
-unique_ptr~WebClient~ client_
|
||||
-Logger logger_
|
||||
+ReverseGeocode(longitude, latitude) optional~AddressLookupResult~
|
||||
}
|
||||
|
||||
class MockAddressService {
|
||||
+ReverseGeocode(longitude, latitude) optional~AddressLookupResult~
|
||||
}
|
||||
}
|
||||
note for NominatimAddressService "Already implemented (services/address/), not planned: queries the public Nominatim API, sleeping 1s per request per its usage policy; returns nullopt on failure or when no usable street address is found. Ships today, ahead of the rest of this diagram."
|
||||
note for MockAddressService "Already implemented, not planned: returns a fixed placeholder address ('123 Mock Street', '00000') for --mocked runs, no network call."
|
||||
AddressService <|.. NominatimAddressService
|
||||
AddressService <|.. MockAddressService
|
||||
AddressService ..> AddressLookupResult : returns
|
||||
NominatimAddressService *-- WebClient
|
||||
|
||||
namespace InfrastructurePrompting {
|
||||
class IPromptDirectory {
|
||||
<<interface>>
|
||||
+Load(key) string
|
||||
}
|
||||
|
||||
class PromptDirectory {
|
||||
-path prompt_dir_
|
||||
-unordered_map~string, string~ cache_
|
||||
+PromptDirectory(prompt_dir)
|
||||
+Load(key) string
|
||||
}
|
||||
}
|
||||
IPromptDirectory <|.. PromptDirectory
|
||||
|
||||
namespace InfrastructureDataGeneration {
|
||||
class DataGenerator {
|
||||
<<interface>>
|
||||
+GenerateBrewery(location, context) BreweryResult
|
||||
+GenerateBeer(brewery_id, location, context, style) BeerResult
|
||||
+GenerateUser(city, persona, name) UserResult
|
||||
+GenerateCheckin(user, brewery, timestamp) CheckinResult
|
||||
+GenerateRating(user, beer, checkin_id) RatingResult
|
||||
}
|
||||
|
||||
class MockGenerator {
|
||||
+GenerateBrewery(...) BreweryResult
|
||||
+GenerateBeer(...) BeerResult
|
||||
+GenerateUser(...) UserResult
|
||||
+GenerateCheckin(...) CheckinResult
|
||||
+GenerateRating(...) RatingResult
|
||||
-DeterministicHash(location) size_t
|
||||
}
|
||||
|
||||
class LlamaGenerator {
|
||||
-ModelHandle model_
|
||||
-ContextHandle context_
|
||||
-unique_ptr~PromptFormatter~ prompt_formatter_
|
||||
-unique_ptr~IPromptDirectory~ prompt_directory_
|
||||
-mt19937 rng_
|
||||
+GenerateBrewery(...) BreweryResult
|
||||
+GenerateBeer(...) BeerResult
|
||||
+GenerateUser(...) UserResult
|
||||
+GenerateCheckin(...) CheckinResult
|
||||
+GenerateRating(...) RatingResult
|
||||
-Load(opts) void
|
||||
-Infer(system_prompt, user_prompt, max_tokens, grammar) string
|
||||
-ValidateModelArchitecture() void
|
||||
}
|
||||
|
||||
class PromptFormatter {
|
||||
<<interface>>
|
||||
+Format(system_prompt, user_prompt) string
|
||||
+ExpectedArchitecture() string_view
|
||||
}
|
||||
|
||||
class Gemma4JinjaPromptFormatter {
|
||||
+Format(system_prompt, user_prompt) string
|
||||
+ExpectedArchitecture() string_view
|
||||
}
|
||||
|
||||
class OpenAIGenerator {
|
||||
-string api_key_
|
||||
-string model_
|
||||
-Logger logger_
|
||||
-unique_ptr~IPromptDirectory~ prompt_directory_
|
||||
-unique_ptr~WebClient~ web_client_
|
||||
+GenerateBrewery(location, context) BreweryResult
|
||||
+GenerateBeer(brewery_id, location, context, style) BeerResult
|
||||
+GenerateUser(city, persona, name) UserResult
|
||||
+GenerateCheckin(user, brewery, timestamp) CheckinResult
|
||||
+GenerateRating(user, beer, checkin_id) RatingResult
|
||||
-CallChatCompletionsApi(...) string
|
||||
}
|
||||
}
|
||||
note for OpenAIGenerator "Already implemented (data_generation/openai/), not planned: calls the OpenAI Chat Completions API with Structured Outputs (json_schema, strict) to guarantee schema-valid JSON, needing no IPromptFormatter or GBNF grammar. Ships today for GenerateBrewery/GenerateUser only, ahead of the rest of this diagram — GenerateBeer/GenerateCheckin/GenerateRating are shown here for interface completeness but not yet implemented on any backend."
|
||||
|
||||
namespace InfrastructureDataExport {
|
||||
class ExportService {
|
||||
<<interface>>
|
||||
+Initialize() void
|
||||
+ProcessBrewery(brewery) uint64_t
|
||||
+ProcessBeer(beer) uint64_t
|
||||
+ProcessUser(user) uint64_t
|
||||
+ProcessCheckin(checkin) uint64_t
|
||||
+ProcessRating(rating) void
|
||||
+ProcessFollow(follow) void
|
||||
+Finalize() void
|
||||
}
|
||||
|
||||
class SqliteExportService {
|
||||
-unique_ptr~DateTimeProvider~ date_time_provider_
|
||||
-SqliteDatabaseHandle db_handle_
|
||||
-SqliteStatementHandle insert_location_stmt_
|
||||
-SqliteStatementHandle insert_brewery_stmt_
|
||||
-SqliteStatementHandle insert_beer_stmt_
|
||||
-SqliteStatementHandle insert_user_stmt_
|
||||
-SqliteStatementHandle insert_checkin_stmt_
|
||||
-SqliteStatementHandle insert_rating_stmt_
|
||||
-SqliteStatementHandle insert_follow_stmt_
|
||||
-bool transaction_open_
|
||||
-unordered_map~string, uint64_t~ location_cache_
|
||||
-unordered_map~string, uint64_t~ brewery_cache_
|
||||
+Initialize() void
|
||||
+ProcessRecord(brewery) uint64_t
|
||||
+ProcessRecord(beer) uint64_t
|
||||
+ProcessRecord(user) uint64_t
|
||||
+ProcessRecord(checkin) uint64_t
|
||||
+ProcessRecord(rating) void
|
||||
+ProcessRecord(follow) void
|
||||
+Finalize() void
|
||||
-InitializeSchema() void
|
||||
-PrepareStatements() void
|
||||
-RollbackAndCloseNoThrow() void
|
||||
-FinalizeStatements() void
|
||||
}
|
||||
|
||||
class DateTimeProvider {
|
||||
<<interface>>
|
||||
+GetUtcTimestamp() string
|
||||
}
|
||||
|
||||
class SystemDateTimeProvider {
|
||||
+GetUtcTimestamp() string
|
||||
}
|
||||
}
|
||||
|
||||
class BiergartenPipelineOrchestrator {
|
||||
-unique_ptr~ICuratedDataService~ curated_data_service_
|
||||
-unique_ptr~EnrichmentService~ enrichment_service_
|
||||
-unique_ptr~DataGenerator~ generator_
|
||||
-unique_ptr~Logger~ logger_
|
||||
-unique_ptr~ExportService~ exporter_
|
||||
-unique_ptr~AddressService~ address_service_
|
||||
-unique_ptr~ContextStrategy~ brewery_context_strategy_
|
||||
-unique_ptr~SamplingStrategy~ sampling_strategy_
|
||||
-unique_ptr~BeerSelectionStrategy~ beer_selection_strategy_
|
||||
-unique_ptr~CheckinDistributionStrategy~ checkin_strategy_
|
||||
-unique_ptr~FollowGenerationStrategy~ follow_strategy_
|
||||
-vector~BeerStyle~ beer_style_palette_
|
||||
-ApplicationOptions options_
|
||||
-vector~GeneratedUser~ user_pool_
|
||||
-vector~GeneratedBrewery~ brewery_pool_
|
||||
-vector~GeneratedBeer~ beer_pool_
|
||||
-vector~GeneratedCheckin~ checkin_pool_
|
||||
-vector~GeneratedFollow~ follow_pool_
|
||||
+Run() bool
|
||||
-RunUserPhase(locations) void
|
||||
-RunBreweryAndBeerPhase(locations) void
|
||||
-RunCheckinPhase() void
|
||||
-RunRatingPhase() void
|
||||
-RunFollowPhase() void
|
||||
}
|
||||
|
||||
BiergartenPipelineOrchestrator *-- ICuratedDataService
|
||||
BiergartenPipelineOrchestrator *-- EnrichmentService
|
||||
BiergartenPipelineOrchestrator *-- DataGenerator
|
||||
BiergartenPipelineOrchestrator *-- ExportService
|
||||
BiergartenPipelineOrchestrator *-- AddressService
|
||||
BiergartenPipelineOrchestrator *-- CheckinDistributionStrategy
|
||||
BiergartenPipelineOrchestrator *-- FollowGenerationStrategy
|
||||
BiergartenPipelineOrchestrator *-- SamplingStrategy
|
||||
BiergartenPipelineOrchestrator *-- BeerSelectionStrategy
|
||||
BiergartenPipelineOrchestrator *-- ApplicationOptions
|
||||
BiergartenPipelineOrchestrator *-- Logger
|
||||
|
||||
BiergartenPipelineOrchestrator "1" *-- "0..*" GeneratedUser : user_pool_
|
||||
BiergartenPipelineOrchestrator "1" *-- "0..*" GeneratedBrewery : brewery_pool_
|
||||
BiergartenPipelineOrchestrator "1" *-- "0..*" GeneratedBeer : beer_pool_
|
||||
BiergartenPipelineOrchestrator "1" *-- "0..*" GeneratedCheckin : checkin_pool_
|
||||
BiergartenPipelineOrchestrator "1" *-- "0..*" GeneratedFollow : follow_pool_
|
||||
|
||||
ICuratedDataService <|.. JsonLoader
|
||||
ICuratedDataService <|.. MockCuratedDataService
|
||||
Logger <|.. PipelineLogger
|
||||
ContextStrategy <|.. BreweryContextStrategy
|
||||
ContextStrategy <|.. BeerContextStrategy
|
||||
SamplingStrategy <|.. UniformSamplingStrategy
|
||||
BeerSelectionStrategy <|.. RandomBeerSelectionStrategy
|
||||
CheckinDistributionStrategy <|.. JCurveCheckinStrategy
|
||||
CheckinDistributionStrategy <|.. RandomCheckinStrategy
|
||||
FollowGenerationStrategy <|.. RandomFollowStrategy
|
||||
FollowGenerationStrategy <|.. ActivityWeightedFollowStrategy
|
||||
EnrichmentService <|.. WikipediaService
|
||||
WebClient <|.. HttpWebClient
|
||||
DataGenerator <|.. MockGenerator
|
||||
DataGenerator <|.. LlamaGenerator
|
||||
DataGenerator <|.. OpenAIGenerator
|
||||
PromptFormatter <|.. Gemma4JinjaPromptFormatter
|
||||
ExportService <|.. SqliteExportService
|
||||
DateTimeProvider <|.. SystemDateTimeProvider
|
||||
|
||||
WikipediaService *-- WebClient
|
||||
WikipediaService ..> ContextStrategy
|
||||
LlamaGenerator *-- PromptFormatter
|
||||
LlamaGenerator *-- IPromptDirectory
|
||||
LlamaGenerator ..> GeneratorOptions
|
||||
OpenAIGenerator *-- IPromptDirectory
|
||||
OpenAIGenerator *-- WebClient
|
||||
SqliteExportService *-- DateTimeProvider
|
||||
|
||||
PipelineLogger o-- BoundedChannel : logs to
|
||||
LogWorker o-- BoundedChannel : drains from
|
||||
|
||||
EnrichedCity *-- Location
|
||||
EnrichedCity *-- LocationContext
|
||||
GeneratedBrewery *-- Location
|
||||
GeneratedBrewery *-- BreweryResult
|
||||
GeneratedBrewery *-- GenerationMetadata
|
||||
GeneratedBeer *-- Location
|
||||
GeneratedBeer *-- BeerStyle
|
||||
GeneratedBeer *-- BeerResult
|
||||
GeneratedBeer *-- GenerationMetadata
|
||||
GeneratedUser *-- Location
|
||||
GeneratedUser *-- UserResult
|
||||
GeneratedUser *-- GenerationMetadata
|
||||
GeneratedCheckin *-- CheckinResult
|
||||
GeneratedCheckin *-- GenerationMetadata
|
||||
GeneratedRating *-- RatingResult
|
||||
GeneratedRating *-- GenerationMetadata
|
||||
GeneratedFollow *-- GenerationMetadata
|
||||
@@ -1,646 +0,0 @@
|
||||
@startuml class_diagram
|
||||
|
||||
' ==========================================
|
||||
' CONFIGURATION & STYLING
|
||||
' ==========================================
|
||||
!include ../biergarten-weizen-theme.puml
|
||||
skinparam classAttributeFontSize 9
|
||||
skinparam defaultFontSize 25
|
||||
skinparam titleFontSize 30
|
||||
|
||||
title Biergarten Data Pipeline — Class Diagram
|
||||
|
||||
package "Domain: Models" {
|
||||
|
||||
class Location {
|
||||
+ city : std::string
|
||||
+ state_province : std::string
|
||||
+ iso3166_2 : std::string
|
||||
+ country : std::string
|
||||
+ iso3166_1 : std::string
|
||||
+ local_languages : std::vector<std::string>
|
||||
+ latitude : double
|
||||
+ longitude : double
|
||||
}
|
||||
|
||||
class LocationContext {
|
||||
+ text : std::string
|
||||
+ completeness : Completeness
|
||||
+ char_count : size_t
|
||||
}
|
||||
|
||||
enum Completeness {
|
||||
Full
|
||||
Partial
|
||||
Absent
|
||||
}
|
||||
|
||||
class EnrichedCity {
|
||||
+ location : Location
|
||||
+ context : LocationContext
|
||||
}
|
||||
|
||||
class BeerStyle {
|
||||
+ name : std::string
|
||||
+ description : std::string
|
||||
+ min_abv : float
|
||||
+ max_abv : float
|
||||
+ min_ibu : int
|
||||
+ max_ibu : int
|
||||
}
|
||||
|
||||
class BreweryResult {
|
||||
+ name_en : std::string
|
||||
+ description_en : std::string
|
||||
+ name_local : std::string
|
||||
+ description_local : std::string
|
||||
}
|
||||
|
||||
class BeerResult {
|
||||
+ name_en : std::string
|
||||
+ description_en : std::string
|
||||
+ name_local : std::string
|
||||
+ description_local : std::string
|
||||
+ style : std::string
|
||||
+ abv : float
|
||||
+ ibu : int
|
||||
}
|
||||
|
||||
class UserResult {
|
||||
+ first_name : std::string
|
||||
+ last_name : std::string
|
||||
+ gender : std::string
|
||||
+ username : std::string
|
||||
+ bio : std::string
|
||||
+ activity_weight : float
|
||||
}
|
||||
|
||||
class Name {
|
||||
+ first_name : std::string
|
||||
+ last_name : std::string
|
||||
+ gender : std::string
|
||||
}
|
||||
|
||||
class ForenameEntry {
|
||||
+ name : std::string
|
||||
+ gender : std::string
|
||||
}
|
||||
|
||||
note right of ForenameEntry
|
||||
forename_list = std::unordered_set<ForenameEntry>
|
||||
surname_list = std::unordered_set<std::string>
|
||||
(aliases declared in curated_data_service.h).
|
||||
ICuratedDataService::LoadForenamesByCountry() /
|
||||
LoadSurnamesByCountry() each return a flat
|
||||
std::unordered_map<std::string, forename_list>
|
||||
or <std::string, surname_list> keyed by ISO
|
||||
3166-1 country code -- no NamesByCountry
|
||||
wrapper class. SampleName(forenames_by_country,
|
||||
surnames_by_country, iso3166_1, rng) is a free
|
||||
function, not a method, so it can be called by
|
||||
any per-city worker without owning the maps.
|
||||
Sourced from popular-names-by-country-dataset
|
||||
(CC0) -- see ETHICS-AND-KNOWN-ISSUES.md. Gender
|
||||
is preserved per forename so it can be threaded
|
||||
through to persona/bio generation later; surnames
|
||||
carry no gender in the source data. Pairing
|
||||
happens at sample time, not at fixture-authoring
|
||||
time, so no information from the source dataset
|
||||
is discarded up front.
|
||||
end note
|
||||
|
||||
ForenameEntry ..> Name : SampleName() produces
|
||||
|
||||
class CheckinResult {
|
||||
+ checked_in_at : std::string
|
||||
+ note : std::string
|
||||
}
|
||||
|
||||
class RatingResult {
|
||||
+ score : float
|
||||
+ note : std::string
|
||||
}
|
||||
|
||||
class GenerationMetadata {
|
||||
+ generation_id : uint64_t
|
||||
+ generated_time : std::string
|
||||
+ context_provided : bool
|
||||
+ generated_with : std::string
|
||||
}
|
||||
|
||||
class GeneratedBrewery {
|
||||
+ brewery_id : uint64_t
|
||||
+ location : Location
|
||||
+ brewery : BreweryResult
|
||||
+ context_completeness : LocationContext::Completeness
|
||||
+ metadata : GenerationMetadata
|
||||
}
|
||||
|
||||
class GeneratedBeer {
|
||||
+ beer_id : uint64_t
|
||||
+ brewery_id : uint64_t
|
||||
+ location : Location
|
||||
+ style : BeerStyle
|
||||
+ beer : BeerResult
|
||||
+ metadata : GenerationMetadata
|
||||
}
|
||||
|
||||
class GeneratedUser {
|
||||
+ user_id : uint64_t
|
||||
+ location : Location
|
||||
+ user : UserResult
|
||||
+ email : std::string
|
||||
+ date_of_birth : std::string
|
||||
+ password : std::string
|
||||
+ metadata : GenerationMetadata
|
||||
}
|
||||
|
||||
note right of GeneratedUser
|
||||
email, date_of_birth, and password are
|
||||
programmatically generated by the orchestrator
|
||||
(not LLM-authored) so a downstream auth-account
|
||||
seeding consumer can register real accounts from
|
||||
this export. See ROADMAP.md §1.
|
||||
end note
|
||||
|
||||
class GeneratedCheckin {
|
||||
+ checkin_id : uint64_t
|
||||
+ user_id : uint64_t
|
||||
+ brewery_id : uint64_t
|
||||
+ checkin : CheckinResult
|
||||
+ metadata : GenerationMetadata
|
||||
}
|
||||
|
||||
class GeneratedRating {
|
||||
+ user_id : uint64_t
|
||||
+ beer_id : uint64_t
|
||||
+ checkin_id : uint64_t
|
||||
+ rating : RatingResult
|
||||
+ metadata : GenerationMetadata
|
||||
}
|
||||
|
||||
class GeneratedFollow {
|
||||
+ follower_id : uint64_t
|
||||
+ followed_id : uint64_t
|
||||
+ metadata : GenerationMetadata
|
||||
}
|
||||
|
||||
class UserPersona {
|
||||
+ name: std::string
|
||||
+ description: std::string
|
||||
+ style_affinities: std::vector<std::string>
|
||||
}
|
||||
|
||||
LocationContext *-- Completeness
|
||||
}
|
||||
|
||||
package "Domain: Application Configuration" {
|
||||
class SamplingOptions {
|
||||
+ temperature: float = 1.0F
|
||||
+ top_p: float = 0.95F
|
||||
+ top_k: uint32_t = 64
|
||||
+ n_ctx: uint32_t = 8192
|
||||
+ seed: int = -1
|
||||
}
|
||||
|
||||
class GeneratorOptions {
|
||||
+ model_path: std::filesystem::path
|
||||
+ use_mocked: bool = false
|
||||
+ sampling: std::optional<SamplingOptions>
|
||||
}
|
||||
|
||||
class PipelineOptions {
|
||||
+ output_path: std::filesystem::path
|
||||
+ log_path: std::filesystem::path
|
||||
}
|
||||
|
||||
class ApplicationOptions {
|
||||
+ generator: GeneratorOptions
|
||||
+ pipeline: PipelineOptions
|
||||
}
|
||||
|
||||
ApplicationOptions *-- GeneratorOptions
|
||||
ApplicationOptions *-- PipelineOptions
|
||||
GeneratorOptions o-- SamplingOptions
|
||||
}
|
||||
|
||||
package "Domain: Policy" {
|
||||
|
||||
interface ContextStrategy <<interface>> {
|
||||
+ QueriesFor(loc : const Location&) : std::vector<std::string>
|
||||
+ MaxContextChars() : size_t
|
||||
}
|
||||
|
||||
class BreweryContextStrategy {
|
||||
+ QueriesFor(loc : const Location&) : std::vector<std::string>
|
||||
+ MaxContextChars() : size_t
|
||||
}
|
||||
|
||||
class BeerContextStrategy {
|
||||
+ QueriesFor(loc : const Location&) : std::vector<std::string>
|
||||
+ MaxContextChars() : size_t
|
||||
}
|
||||
|
||||
interface SamplingStrategy <<interface>> {
|
||||
+ Sample(locations : const std::vector<Location>&) : std::vector<Location>
|
||||
}
|
||||
|
||||
class UniformSamplingStrategy {
|
||||
- sample_size_ : size_t
|
||||
+ Sample(locations : const std::vector<Location>&) : std::vector<Location>
|
||||
}
|
||||
|
||||
interface BeerSelectionStrategy <<interface>> {
|
||||
+ SelectStyles(brewery : const GeneratedBrewery&,\n palette : std::span<const BeerStyle>) : std::vector<BeerStyle>
|
||||
}
|
||||
|
||||
class RandomBeerSelectionStrategy {
|
||||
- rng_ : std::mt19937
|
||||
- min_beers_ : size_t
|
||||
- max_beers_ : size_t
|
||||
+ SelectStyles(brewery : const GeneratedBrewery&,\n palette : std::span<const BeerStyle>) : std::vector<BeerStyle>
|
||||
}
|
||||
|
||||
interface CheckinDistributionStrategy <<interface>> {
|
||||
+ AssignActivityWeights(users : std::vector<GeneratedUser>&) : void
|
||||
+ CheckinsForUser(user : const GeneratedUser&,\n brewery_count : size_t) : size_t
|
||||
+ TimestampFor(user : const GeneratedUser&,\n index : size_t) : std::string
|
||||
}
|
||||
|
||||
class JCurveCheckinStrategy {
|
||||
- rng_ : std::mt19937
|
||||
+ AssignActivityWeights(users : std::vector<GeneratedUser>&) : void
|
||||
+ CheckinsForUser(user : const GeneratedUser&,\n brewery_count : size_t) : size_t
|
||||
+ TimestampFor(user : const GeneratedUser&,\n index : size_t) : std::string
|
||||
}
|
||||
|
||||
class RandomCheckinStrategy {
|
||||
- rng_ : std::mt19937
|
||||
- min_checkins_ : size_t
|
||||
- max_checkins_ : size_t
|
||||
+ AssignActivityWeights(users : std::vector<GeneratedUser>&) : void
|
||||
+ CheckinsForUser(user : const GeneratedUser&,\n brewery_count : size_t) : size_t
|
||||
+ TimestampFor(user : const GeneratedUser&,\n index : size_t) : std::string
|
||||
}
|
||||
|
||||
interface FollowGenerationStrategy <<interface>> {
|
||||
+ GenerateFollows(users : const std::vector<GeneratedUser>&) : std::vector<GeneratedFollow>
|
||||
}
|
||||
|
||||
class RandomFollowStrategy {
|
||||
- rng_ : std::mt19937
|
||||
- min_follows_ : size_t
|
||||
- max_follows_ : size_t
|
||||
+ GenerateFollows(users : const std::vector<GeneratedUser>&) : std::vector<GeneratedFollow>
|
||||
}
|
||||
|
||||
class ActivityWeightedFollowStrategy {
|
||||
- rng_ : std::mt19937
|
||||
- min_follows_ : size_t
|
||||
- max_follows_ : size_t
|
||||
+ GenerateFollows(users : const std::vector<GeneratedUser>&) : std::vector<GeneratedFollow>
|
||||
}
|
||||
}
|
||||
|
||||
package "Infrastructure: Logging" {
|
||||
enum LogLevel {
|
||||
Debug
|
||||
Info
|
||||
Warn
|
||||
Error
|
||||
}
|
||||
|
||||
enum PipelinePhase {
|
||||
Startup
|
||||
UserGeneration
|
||||
BreweryAndBeerGeneration
|
||||
CheckinGeneration
|
||||
RatingGeneration
|
||||
FollowGeneration
|
||||
Teardown
|
||||
}
|
||||
|
||||
class LogEntry {
|
||||
+ timestamp : std::chrono::system_clock::time_point
|
||||
+ level : LogLevel
|
||||
+ phase : PipelinePhase
|
||||
+ message : std::string
|
||||
+ worker : std::optional<std::string>
|
||||
}
|
||||
|
||||
interface ILogger <<interface>> {
|
||||
+ Log(entry : const LogEntry&) : void
|
||||
}
|
||||
|
||||
class LogProducer {
|
||||
- channel_ : BoundedChannel<LogEntry>&
|
||||
+ Log(entry : const LogEntry&) : void
|
||||
}
|
||||
|
||||
class LogDispatcher {
|
||||
- channel_ : BoundedChannel<LogEntry>&
|
||||
+ Run() : void
|
||||
- ToSpdlogLevel(level) : spdlog::level::level_enum
|
||||
}
|
||||
|
||||
LogEntry *-- LogLevel
|
||||
LogEntry *-- PipelinePhase
|
||||
ILogger <|.. LogProducer
|
||||
LogProducer ..> LogEntry : emits
|
||||
LogDispatcher ..> LogEntry : consumes
|
||||
}
|
||||
|
||||
package "Infrastructure: Pipeline Channel" {
|
||||
|
||||
class "BoundedChannel<T>" as BoundedChannel {
|
||||
- queue_ : std::queue<T>
|
||||
- mutex_ : std::mutex
|
||||
- not_full_ : std::condition_variable
|
||||
- not_empty_ : std::condition_variable
|
||||
- capacity_ : size_t
|
||||
- closed_ : bool
|
||||
+ Send(item : T) : void
|
||||
+ Receive() : std::optional<T>
|
||||
+ Close() : void
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
package "Infrastructure: Data Preloading" {
|
||||
|
||||
interface ICuratedDataService <<interface>> {
|
||||
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
|
||||
+ LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&
|
||||
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
|
||||
+ LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
|
||||
+ LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
|
||||
}
|
||||
|
||||
class JsonLoader {
|
||||
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
|
||||
+ LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&
|
||||
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
|
||||
+ LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
|
||||
+ LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
|
||||
}
|
||||
|
||||
note right of JsonLoader
|
||||
Each Load* call is memoized after its first
|
||||
parse -- implemented today, ahead of the rest
|
||||
of this package (LoadBeerStyles is still planned).
|
||||
end note
|
||||
|
||||
class MockCuratedDataService {
|
||||
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
|
||||
+ LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&
|
||||
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
|
||||
+ LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
|
||||
+ LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
|
||||
}
|
||||
|
||||
note right of MockCuratedDataService
|
||||
Fixed in-memory dataset used in --mocked mode,
|
||||
implemented today for Locations/Personas/
|
||||
Forenames/Surnames; would need a LoadBeerStyles
|
||||
fixture too once that method is implemented.
|
||||
end note
|
||||
|
||||
}
|
||||
|
||||
package "Infrastructure: Enrichment" {
|
||||
|
||||
interface EnrichmentService <<interface>> {
|
||||
+ GetLocationContext(loc : const Location&,\n strategy : const ContextStrategy&) : LocationContext
|
||||
}
|
||||
|
||||
class WikipediaService {
|
||||
- client_ : std::unique_ptr<WebClient>
|
||||
- extract_cache_ : std::unordered_map<std::string, std::string>
|
||||
+ GetLocationContext(loc : const Location&,\n strategy : const ContextStrategy&) : LocationContext
|
||||
- FetchExtract(query : std::string_view) : std::string
|
||||
}
|
||||
|
||||
interface WebClient <<interface>> {
|
||||
+ Get(url : const std::string&) : std::string
|
||||
+ UrlEncode(value : const std::string&) : std::string
|
||||
}
|
||||
|
||||
class HttpWebClient {
|
||||
+ Get(url : const std::string&) : std::string
|
||||
+ UrlEncode(value : const std::string&) : std::string
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
package "Infrastructure: Prompting" {
|
||||
|
||||
interface IPromptDirectory <<interface>> {
|
||||
+ Load(key : std::string_view) : std::string
|
||||
}
|
||||
|
||||
class PromptDirectory {
|
||||
- prompt_dir_ : std::filesystem::path
|
||||
- cache_ : std::unordered_map<std::string, std::string>
|
||||
+ PromptDirectory(prompt_dir : const std::filesystem::path&)
|
||||
+ Load(key : std::string_view) : std::string
|
||||
}
|
||||
|
||||
IPromptDirectory <|.. PromptDirectory
|
||||
}
|
||||
|
||||
package "Infrastructure: Data Generation" {
|
||||
|
||||
interface DataGenerator <<interface>> {
|
||||
+ GenerateBrewery(location : const Location&,\n context : const LocationContext&) : BreweryResult
|
||||
+ GenerateBeer(brewery_id : uint64_t,\n location : const Location&,\n context : const LocationContext&,\n style : const BeerStyle&) : BeerResult
|
||||
+ GenerateUser(city : const EnrichedCity&,\n persona : const UserPersona&,\n name : const Name&) : UserResult
|
||||
+ GenerateCheckin(user : const GeneratedUser&,\n brewery : const GeneratedBrewery&,\n timestamp : const std::string&) : CheckinResult
|
||||
+ GenerateRating(user : const GeneratedUser&,\n beer : const GeneratedBeer&,\n checkin_id : uint64_t) : RatingResult
|
||||
}
|
||||
|
||||
class MockGenerator {
|
||||
+ GenerateBrewery(...) : BreweryResult
|
||||
+ GenerateBeer(...) : BeerResult
|
||||
+ GenerateUser(...) : UserResult
|
||||
+ GenerateCheckin(...) : CheckinResult
|
||||
+ GenerateRating(...) : RatingResult
|
||||
- DeterministicHash(location : const Location&) : size_t
|
||||
}
|
||||
|
||||
class LlamaGenerator {
|
||||
- model_ : ModelHandle
|
||||
- context_ : ContextHandle
|
||||
- prompt_formatter_ : std::unique_ptr<PromptFormatter>
|
||||
- prompt_directory_ : std::unique_ptr<IPromptDirectory>
|
||||
- rng_ : std::mt19937
|
||||
+ GenerateBrewery(...) : BreweryResult
|
||||
+ GenerateBeer(...) : BeerResult
|
||||
+ GenerateUser(...) : UserResult
|
||||
+ GenerateCheckin(...) : CheckinResult
|
||||
+ GenerateRating(...) : RatingResult
|
||||
- Load(opts : const GeneratorOptions&) : void
|
||||
- Infer(system_prompt, user_prompt,\n max_tokens, grammar) : std::string
|
||||
- ValidateModelArchitecture() : void
|
||||
}
|
||||
|
||||
interface PromptFormatter <<interface>> {
|
||||
+ Format(system_prompt : std::string_view,\n user_prompt : std::string_view) : std::string
|
||||
+ ExpectedArchitecture() : std::string_view
|
||||
}
|
||||
|
||||
class Gemma4JinjaPromptFormatter {
|
||||
+ Format(...) : std::string
|
||||
+ ExpectedArchitecture() : std::string_view
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
package "Infrastructure: Data Export" {
|
||||
|
||||
interface ExportService <<interface>> {
|
||||
+ Initialize() : void
|
||||
+ ProcessBrewery(brewery : const GeneratedBrewery&) : uint64_t
|
||||
+ ProcessBeer(beer : const GeneratedBeer&) : uint64_t
|
||||
+ ProcessUser(user : const GeneratedUser&) : uint64_t
|
||||
+ ProcessCheckin(checkin : const GeneratedCheckin&) : uint64_t
|
||||
+ ProcessRating(rating : const GeneratedRating&) : void
|
||||
+ ProcessFollow(follow : const GeneratedFollow&) : void
|
||||
+ Finalize() : void
|
||||
}
|
||||
|
||||
class SqliteExportService {
|
||||
- date_time_provider_ : std::unique_ptr<DateTimeProvider>
|
||||
- db_handle_ : SqliteDatabaseHandle
|
||||
- insert_location_stmt_ : SqliteStatementHandle
|
||||
- insert_brewery_stmt_ : SqliteStatementHandle
|
||||
- insert_beer_stmt_ : SqliteStatementHandle
|
||||
- insert_user_stmt_ : SqliteStatementHandle
|
||||
- insert_checkin_stmt_ : SqliteStatementHandle
|
||||
- insert_rating_stmt_ : SqliteStatementHandle
|
||||
- insert_follow_stmt_ : SqliteStatementHandle
|
||||
- transaction_open_ : bool
|
||||
- location_cache_ : std::unordered_map<std::string, uint64_t>
|
||||
- brewery_cache_ : std::unordered_map<std::string, uint64_t>
|
||||
+ Initialize() : void
|
||||
+ ProcessRecord(brewery : const GeneratedBrewery&) : uint64_t
|
||||
+ ProcessRecord(beer : const GeneratedBeer&) : uint64_t
|
||||
+ ProcessRecord(user : const GeneratedUser&) : uint64_t
|
||||
+ ProcessRecord(checkin : const GeneratedCheckin&) : uint64_t
|
||||
+ ProcessRecord(rating : const GeneratedRating&) : void
|
||||
+ ProcessRecord(follow : const GeneratedFollow&) : void
|
||||
+ Finalize() : void
|
||||
- InitializeSchema() : void
|
||||
- PrepareStatements() : void
|
||||
- RollbackAndCloseNoThrow() : void
|
||||
- FinalizeStatements() : void
|
||||
}
|
||||
|
||||
interface DateTimeProvider <<interface>> {
|
||||
+ GetUtcTimestamp() : std::string
|
||||
}
|
||||
|
||||
class SystemDateTimeProvider {
|
||||
+ GetUtcTimestamp() : std::string
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
class BiergartenPipelineOrchestrator {
|
||||
- curated_data_service_ : std::unique_ptr<ICuratedDataService>
|
||||
- enrichment_service_ : std::unique_ptr<EnrichmentService>
|
||||
- generator_ : std::unique_ptr<DataGenerator>
|
||||
- logger_ : std::unique_ptr<Logger>
|
||||
- exporter_ : std::unique_ptr<ExportService>
|
||||
- brewery_context_strategy_ : std::unique_ptr<ContextStrategy>
|
||||
- sampling_strategy_ : std::unique_ptr<SamplingStrategy>
|
||||
- beer_selection_strategy_ : std::unique_ptr<BeerSelectionStrategy>
|
||||
- checkin_strategy_ : std::unique_ptr<CheckinDistributionStrategy>
|
||||
- follow_strategy_ : std::unique_ptr<FollowGenerationStrategy>
|
||||
- beer_style_palette_ : std::vector<BeerStyle>
|
||||
- options_ : ApplicationOptions
|
||||
--
|
||||
- user_pool_ : std::vector<GeneratedUser>
|
||||
- brewery_pool_ : std::vector<GeneratedBrewery>
|
||||
- beer_pool_ : std::vector<GeneratedBeer>
|
||||
- checkin_pool_ : std::vector<GeneratedCheckin>
|
||||
- follow_pool_ : std::vector<GeneratedFollow>
|
||||
--
|
||||
+ Run() : bool
|
||||
- RunUserPhase(locations : const std::vector<Location>&) : void
|
||||
- RunBreweryAndBeerPhase(locations : const std::vector<Location>&) : void
|
||||
- RunCheckinPhase() : void
|
||||
- RunRatingPhase() : void
|
||||
- RunFollowPhase() : void
|
||||
}
|
||||
|
||||
' --- Orchestration Aggregations (Services & Strategies) ---
|
||||
BiergartenPipelineOrchestrator *-- ICuratedDataService
|
||||
BiergartenPipelineOrchestrator *-- EnrichmentService
|
||||
BiergartenPipelineOrchestrator *-- DataGenerator
|
||||
BiergartenPipelineOrchestrator *-- ExportService
|
||||
BiergartenPipelineOrchestrator *-- CheckinDistributionStrategy
|
||||
BiergartenPipelineOrchestrator *-- FollowGenerationStrategy
|
||||
BiergartenPipelineOrchestrator *-- SamplingStrategy
|
||||
BiergartenPipelineOrchestrator *-- BeerSelectionStrategy
|
||||
BiergartenPipelineOrchestrator *-- ApplicationOptions
|
||||
BiergartenPipelineOrchestrator *-- Logger
|
||||
|
||||
' --- Orchestration Aggregations (Data Pools) ---
|
||||
BiergartenPipelineOrchestrator *-- "0..*" GeneratedUser : user_pool_
|
||||
BiergartenPipelineOrchestrator *-- "0..*" GeneratedBrewery : brewery_pool_
|
||||
BiergartenPipelineOrchestrator *-- "0..*" GeneratedBeer : beer_pool_
|
||||
BiergartenPipelineOrchestrator *-- "0..*" GeneratedCheckin : checkin_pool_
|
||||
BiergartenPipelineOrchestrator *-- "0..*" GeneratedFollow : follow_pool_
|
||||
|
||||
' --- Interfaces & Implementations ---
|
||||
ICuratedDataService <|.. JsonLoader
|
||||
ICuratedDataService <|.. MockCuratedDataService
|
||||
Logger <|.. PipelineLogger
|
||||
ContextStrategy <|.. BreweryContextStrategy
|
||||
ContextStrategy <|.. BeerContextStrategy
|
||||
SamplingStrategy <|.. UniformSamplingStrategy
|
||||
BeerSelectionStrategy <|.. RandomBeerSelectionStrategy
|
||||
CheckinDistributionStrategy <|.. JCurveCheckinStrategy
|
||||
CheckinDistributionStrategy <|.. RandomCheckinStrategy
|
||||
FollowGenerationStrategy <|.. RandomFollowStrategy
|
||||
FollowGenerationStrategy <|.. ActivityWeightedFollowStrategy
|
||||
EnrichmentService <|.. WikipediaService
|
||||
WebClient <|.. HttpWebClient
|
||||
DataGenerator <|.. MockGenerator
|
||||
DataGenerator <|.. LlamaGenerator
|
||||
PromptFormatter <|.. Gemma4JinjaPromptFormatter
|
||||
ExportService <|.. SqliteExportService
|
||||
DateTimeProvider <|.. SystemDateTimeProvider
|
||||
|
||||
' --- Service Compositions & Dependencies ---
|
||||
WikipediaService *-- WebClient
|
||||
WikipediaService ..> ContextStrategy
|
||||
LlamaGenerator *-- PromptFormatter
|
||||
LlamaGenerator *-- IPromptDirectory
|
||||
LlamaGenerator ..> GeneratorOptions
|
||||
SqliteExportService *-- DateTimeProvider
|
||||
|
||||
' --- Cross-Component Aggregations (Held References) ---
|
||||
PipelineLogger o-- BoundedChannel : logs to
|
||||
LogWorker o-- BoundedChannel : drains from
|
||||
|
||||
' --- Domain Containment ---
|
||||
EnrichedCity *-- Location
|
||||
EnrichedCity *-- LocationContext
|
||||
GeneratedBrewery *-- Location
|
||||
GeneratedBrewery *-- BreweryResult
|
||||
GeneratedBrewery *-- GenerationMetadata
|
||||
GeneratedBeer *-- Location
|
||||
GeneratedBeer *-- BeerStyle
|
||||
GeneratedBeer *-- BeerResult
|
||||
GeneratedBeer *-- GenerationMetadata
|
||||
GeneratedUser *-- Location
|
||||
GeneratedUser *-- UserResult
|
||||
GeneratedUser *-- GenerationMetadata
|
||||
GeneratedCheckin *-- CheckinResult
|
||||
GeneratedCheckin *-- GenerationMetadata
|
||||
GeneratedRating *-- RatingResult
|
||||
GeneratedRating *-- GenerationMetadata
|
||||
GeneratedFollow *-- GenerationMetadata
|
||||
|
||||
@enduml
|
||||
File diff suppressed because one or more lines are too long
|
Before Width: | Height: | Size: 142 KiB |
File diff suppressed because one or more lines are too long
|
Before Width: | Height: | Size: 240 KiB |
Reference in New Issue
Block a user