restruc

docs/pipeline/diagrams/current/activity.md

@@ -3,5 +3,5 @@
 The only concurrency in the current pipeline is a dedicated log dispatcher thread that drains `log_channel` and forwards entries to spdlog for the entire run (joined during shutdown after `log_channel` closes). `BiergartenPipelineOrchestrator::Run()` itself is synchronous — sampling, enrichment, generation, and export all happen on the main thread.
 
 ```mermaid
---8<-- "diagrams/pipeline/current/activity.mmd"
+--8<-- "pipeline/diagrams/current/activity.mmd"
 ```

docs/pipeline/diagrams/current/activity.mmd

@@ -0,0 +1,74 @@
+flowchart TD
+    Start([start]) --> ParseArgs["main.cc: ParseArguments(argc, argv)"]
+    ParseArgs --> ArgsValid{Arguments valid?}
+    ArgsValid -- no --> LogErr[Log error / usage info] --> Stop1([stop])
+    ArgsValid -- yes --> Init["main.cc: Init OpenSSL global state and LlamaBackendState"]
+    Init --> DI["main.cc: di::make_injector(...)<br/>binds ILogger, IEnrichmentService, DataGenerator, IExportService, IPromptFormatter, WebClient based on --mocked vs. model-backed mode"]
+    DI --> LogThread["main.cc: create BoundedChannel log_channel, spawn log dispatcher thread (LogDispatcher::Run)"]
+    LogThread --> CreateOrch["main.cc: injector.create BiergartenPipelineOrchestrator"]
+    CreateOrch --> Run["BiergartenPipelineOrchestrator::Run()"]
+
+    Run --> SqliteInit["SqliteExportService.Initialize()<br/>builds biergarten_seed_&lt;UTC datetime&gt;.sqlite (numeric suffix if it exists), opens DB connection, executes schema DDL, begins transaction"]
+    SqliteInit --> QueryLoc[Orchestrator: QueryLocations]
+    QueryLoc --> LoadLoc["JsonLoader.LoadLocations(locations.json)"]
+    LoadLoc --> SampleLoc["std::ranges::sample(all_locations, --location-count) — defaults to 10"]
+
+    SampleLoc --> EnrichLoop{For each sampled Location}
+    EnrichLoop --> GetContext["IEnrichmentService.GetLocationContext(loc)<br/>WikipediaEnrichmentService fetches a generic 'brewing' extract and a 'beer in {country}' extract (not city/region-specific); MockEnrichmentService (--mocked) returns an empty string"]
+    GetContext --> LookupFailed{Lookup failed?}
+    LookupFailed -- yes --> SkipCity1[Log warning, skip city] --> EnrichLoop
+    LookupFailed -- no --> StoreEnriched["Store EnrichedCity{Location, region_context}"] --> EnrichLoop
+    EnrichLoop -- done --> GenUsers[Orchestrator: GenerateUsers enriched_cities]
+
+    GenUsers --> LoadUserData["JsonLoader: LoadPersonas / LoadForenamesByCountry / LoadSurnamesByCountry<br/>each memoized after first parse; MockCuratedDataService (--mocked) returns a fixed in-memory dataset instead, skipping these JSON files"]
+
+    LoadUserData --> UserLoop{For each EnrichedCity}
+    UserLoop --> SampleName["SampleName(forenames_by_country, surnames_by_country, location.iso3166_1)"]
+    SampleName --> NamesAvail{Names available for country?}
+    NamesAvail -- no --> SkipCity2[Log warning, skip city] --> UserLoop
+    NamesAvail -- yes --> SamplePersona["Sample UserPersona (uniform random)"]
+    SamplePersona --> UserGenMode{Generator mode}
+
+    UserGenMode -- MockGenerator --> UserMockGen[DeterministicHash and Format]
+    UserGenMode -- LlamaGenerator --> UserLlamaLoad["prompt_directory_.Load(USER_GENERATION) → USER_GENERATION.md in --prompt-dir"]
+    UserLlamaLoad --> UserLlamaRetry["Infer(...) with kUserJsonGrammar → ValidateUserJson<br/>retry up to 3 attempts, appending validation error to the retry prompt on failure"]
+    UserGenMode -- OpenAIGenerator --> UserOAILoad["prompt_directory_.Load(USER_GENERATION) — same prompt file as LlamaGenerator, content is backend-agnostic"]
+    UserOAILoad --> UserOAIRetry["CallChatCompletionsApi(...) with kUserJsonSchema → ValidateUserJson<br/>retry up to 2 attempts — Structured Outputs (json_schema, strict) already guarantees schema shape, so retries only cover transient failures / placeholder text"]
+
+    UserMockGen --> UserGenDone{Generation successful?}
+    UserLlamaRetry --> UserGenDone
+    UserOAIRetry --> UserGenDone
+
+    UserGenDone -- yes --> UserPostProc["GetRandomCoordsWithinRange (5km jitter), BuildEmail, GenerateDateOfBirth<br/>email and date_of_birth are generated programmatically, never LLM-authored; generation + coordinate jitter run inside one try/catch per record"]
+    UserGenDone -- no --> UserGenFail["Log warning: generation failed, skipping"] --> UserLoop
+
+    UserPostProc --> UserExport["SqliteExportService.ProcessRecord(UserRecord)<br/>reuse cached location_id or insert Location and cache its id; insert User (FK location_id)"]
+    UserExport --> UserExportFail{Exception during insert?}
+    UserExportFail -- yes --> UserExportWarn["Log warning: SQLite export failed"] --> UserLoop
+    UserExportFail -- no --> UserLoop
+    UserLoop -- done --> GenBrew[Orchestrator: GenerateBreweries enriched_cities]
+
+    GenBrew --> BrewLoop{For each EnrichedCity}
+    BrewLoop --> BrewGenMode{Generator mode}
+
+    BrewGenMode -- MockGenerator --> BrewMockGen[DeterministicHash and Format]
+    BrewGenMode -- LlamaGenerator --> BrewLlamaLoad["PrepareRegionContext, prompt_directory_.Load(BREWERY_GENERATION) → BREWERY_GENERATION.md"]
+    BrewLlamaLoad --> BrewLlamaRetry["Infer(...) with kBreweryJsonGrammar → ValidateBreweryJson<br/>retry up to 3 attempts; max_tokens is fixed across retries, not raised on truncation"]
+    BrewGenMode -- OpenAIGenerator --> BrewOAILoad["PrepareRegionContext, prompt_directory_.Load(BREWERY_GENERATION) — same prompt file as LlamaGenerator"]
+    BrewOAILoad --> BrewOAIRetry["CallChatCompletionsApi(...) with kBreweryJsonSchema → ValidateBreweryJson<br/>retry up to 2 attempts — Structured Outputs already guarantees schema shape"]
+
+    BrewMockGen --> BrewGenDone{Generation successful?}
+    BrewLlamaRetry --> BrewGenDone
+    BrewOAIRetry --> BrewGenDone
+
+    BrewGenDone -- yes --> BrewPostProc["GetRandomCoordsWithinRange (5km jitter)<br/>generation + jitter run inside one try/catch per record"]
+    BrewGenDone -- no --> BrewGenFail["Log warning: generation failed, skipping"] --> BrewLoop
+
+    BrewPostProc --> BrewExport["SqliteExportService.ProcessRecord(GeneratedBrewery)<br/>reuse cached location_id or insert Location and cache its id; insert Brewery (FK location_id)"]
+    BrewExport --> BrewExportFail{Exception during insert?}
+    BrewExportFail -- yes --> BrewExportWarn["Log warning: SQLite export failed (data loss prevented per-record; pipeline continues)"] --> BrewLoop
+    BrewExportFail -- no --> BrewLoop
+    BrewLoop -- done --> Finalize["SqliteExportService.Finalize()<br/>commits transaction, closes database connection"]
+
+    Finalize --> CloseLog["main.cc: close log_channel, join log dispatcher thread"]
+    CloseLog --> Stop2(["return 0 / stop"])

docs/pipeline/diagrams/current/class.md

@@ -1,5 +1,5 @@
 # The Biergarten Data Pipeline — Class Diagram (Current)
 
 ```mermaid
---8<-- "diagrams/pipeline/current/class.mmd"
+--8<-- "pipeline/diagrams/current/class.mmd"
 ```

docs/pipeline/diagrams/current/class.mmd

@@ -0,0 +1,260 @@
+classDiagram
+    class BiergartenPipelineOrchestrator {
+        -shared_ptr~ILogger~ logger_
+        -unique_ptr~IEnrichmentService~ context_service_
+        -unique_ptr~DataGenerator~ generator_
+        -unique_ptr~IExportService~ exporter_
+        -unique_ptr~ICuratedDataService~ curated_data_service_
+        -ApplicationOptions application_options_
+        -vector~BreweryRecord~ generated_breweries_
+        -vector~UserRecord~ generated_users_
+        +Run() bool
+        -QueryLocations() vector~City~
+        -GenerateBreweries(cities) void
+        -GenerateUsers(cities) void
+        -LogResults() void
+    }
+    note for BiergartenPipelineOrchestrator "GenerateBreweries/GenerateUsers jitter each address via GetRandomCoordsWithinRange(), uniformly within 5km of the city centre"
+
+    class LogLevel {
+        <<enumeration>>
+        Debug
+        Info
+        Warn
+        Error
+    }
+
+    class PipelinePhase {
+        <<enumeration>>
+        Startup
+        Enrichment
+        UserGeneration
+        BreweryAndBeerGeneration
+        CheckinGeneration
+        RatingGeneration
+        FollowGeneration
+        Teardown
+    }
+
+    class LogDTO {
+        <<struct>>
+        +LogLevel level
+        +PipelinePhase phase
+        +string message
+    }
+
+    class LogEntry {
+        <<struct>>
+        +time_point timestamp
+        +source_location origin
+        +thread::id thread_id
+        +LogLevel level
+        +PipelinePhase phase
+        +string message
+    }
+
+    class ILogger {
+        <<interface>>
+        +Log(payload) void
+        -DoLog(entry)* void
+    }
+
+    class LogProducer {
+        -BoundedChannel~LogEntry~ channel_
+        -DoLog(entry) void
+    }
+
+    class LogDispatcher {
+        -BoundedChannel~LogEntry~ channel_
+        +Run() void
+        -ToSpdlogLevel(level) level_enum
+    }
+
+    class IEnrichmentService {
+        <<interface>>
+        +GetLocationContext(loc) string
+    }
+
+    class MockEnrichmentService {
+        +GetLocationContext(loc) string
+    }
+
+    class WikipediaEnrichmentService {
+        -unique_ptr~WebClient~ client_
+        -unordered_map~string, string~ extract_cache_
+        +GetLocationContext(loc) string
+        -FetchExtract(query) string
+    }
+
+    class WebClient {
+        <<interface>>
+        +Get(url) string
+        +UrlEncode(value) string
+    }
+
+    class HttpWebClient {
+        +Get(url) string
+        +UrlEncode(value) string
+    }
+
+    class DataGenerator {
+        <<interface>>
+        +GenerateBrewery(city) BreweryResult
+        +GenerateUser(city, persona, name) UserResult
+    }
+
+    class MockGenerator {
+        +GenerateBrewery(city) BreweryResult
+        +GenerateUser(city, persona, name) UserResult
+        -DeterministicHash(location) size_t
+        -DeterministicHash(location, persona, name) size_t
+    }
+
+    class LlamaGenerator {
+        -ModelHandle model_
+        -ContextHandle context_
+        -unique_ptr~IPromptFormatter~ prompt_formatter_
+        -unique_ptr~IPromptDirectory~ prompt_directory_
+        -mt19937 rng_
+        +GenerateBrewery(...) BreweryResult
+        +GenerateUser(...) UserResult
+        -Load(model_path) void
+        -Infer(...) string
+        -InferFormatted(...) string
+    }
+
+    class OpenAIGenerator {
+        -string api_key_
+        -string model_
+        -shared_ptr~ILogger~ logger_
+        -unique_ptr~IPromptDirectory~ prompt_directory_
+        -unique_ptr~WebClient~ web_client_
+        +GenerateBrewery(city) BreweryResult
+        +GenerateUser(city, persona, name) UserResult
+        -CallChatCompletionsApi(...) string
+    }
+    note for OpenAIGenerator "Chat Completions with Structured Outputs (json_schema, strict) guarantees schema-valid JSON, replacing llama.cpp's GBNF grammar — no IPromptFormatter needed"
+
+    class IPromptFormatter {
+        <<interface>>
+        +Format(system_prompt, user_prompt) string
+    }
+
+    class Gemma4JinjaPromptFormatter {
+        +Format(system_prompt, user_prompt) string
+    }
+
+    class IPromptDirectory {
+        <<interface>>
+        +Load(key) string
+    }
+
+    class PromptDirectory {
+        -path prompt_dir_
+        -unordered_map~string, string~ cache_
+        +Load(key) string
+    }
+
+    class ICuratedDataService {
+        <<interface>>
+        +LoadLocations(filepath) vector~Location~
+        +LoadPersonas(filepath) vector~UserPersona~
+        +LoadForenamesByCountry(filepath) unordered_map~string, forename_list~
+        +LoadSurnamesByCountry(filepath) unordered_map~string, surname_list~
+    }
+
+    class JsonLoader {
+        -cache cache_
+        +LoadLocations(filepath) vector~Location~
+        +LoadPersonas(filepath) vector~UserPersona~
+        +LoadForenamesByCountry(filepath) unordered_map~string, forename_list~
+        +LoadSurnamesByCountry(filepath) unordered_map~string, surname_list~
+    }
+    note for JsonLoader "Each Load* method memoizes its result in cache_ after its first call"
+
+    class MockCuratedDataService {
+        -vector~Location~ locations_
+        -vector~UserPersona~ personas_
+        -unordered_map~string, forename_list~ forenames_by_country_
+        -unordered_map~string, surname_list~ surnames_by_country_
+        +LoadLocations(filepath) vector~Location~
+        +LoadPersonas(filepath) vector~UserPersona~
+        +LoadForenamesByCountry(filepath) unordered_map~string, forename_list~
+        +LoadSurnamesByCountry(filepath) unordered_map~string, surname_list~
+    }
+    note for MockCuratedDataService "Fixed 4-location/3-persona dataset (US, DE, FR, BE) for --mocked runs; filepath arguments are ignored"
+
+    class IExportService {
+        <<interface>>
+        +Initialize() void
+        +ProcessRecord(brewery) uint64_t
+        +ProcessRecord(user) uint64_t
+        +Finalize() void
+    }
+
+    class SqliteExportService {
+        -unique_ptr~IDateTimeProvider~ date_time_provider_
+        -string run_timestamp_utc_
+        -path database_path_
+        -SqliteDatabaseHandle db_handle_
+        -SqliteStatementHandle insert_location_stmt_
+        -SqliteStatementHandle insert_brewery_stmt_
+        -SqliteStatementHandle insert_user_stmt_
+        -bool transaction_open_
+        -unordered_map~string, sqlite3_int64~ location_cache_
+        +Initialize() void
+        +ProcessRecord(brewery) uint64_t
+        +ProcessRecord(user) uint64_t
+        +Finalize() void
+        -InitializeSchema() void
+        -ResolveLocationId(location) sqlite3_int64
+    }
+
+    class IDateTimeProvider {
+        <<interface>>
+        +GetUtcTimestamp() string
+    }
+
+    class SystemDateTimeProvider {
+        +GetUtcTimestamp() string
+    }
+
+    BiergartenPipelineOrchestrator *-- ILogger : owns
+    BiergartenPipelineOrchestrator *-- IEnrichmentService : owns
+    BiergartenPipelineOrchestrator *-- DataGenerator : owns
+    BiergartenPipelineOrchestrator *-- IExportService : owns
+    BiergartenPipelineOrchestrator *-- ICuratedDataService : owns
+
+    LogEntry *-- LogLevel
+    LogEntry *-- PipelinePhase
+    LogDTO *-- LogLevel
+    LogDTO *-- PipelinePhase
+    ILogger <|.. LogProducer : implements
+    LogProducer ..> LogEntry : emits
+    LogDispatcher ..> LogEntry : consumes
+
+    IEnrichmentService <|.. WikipediaEnrichmentService : implements
+    IEnrichmentService <|.. MockEnrichmentService : implements
+    WikipediaEnrichmentService *-- WebClient : owns
+
+    WebClient <|.. HttpWebClient : implements
+
+    DataGenerator <|.. MockGenerator : implements
+    DataGenerator <|.. LlamaGenerator : implements
+    DataGenerator <|.. OpenAIGenerator : implements
+
+    LlamaGenerator *-- IPromptFormatter : uses
+    LlamaGenerator *-- IPromptDirectory : uses
+
+    OpenAIGenerator *-- IPromptDirectory : uses
+    OpenAIGenerator *-- WebClient : uses
+
+    IPromptFormatter <|.. Gemma4JinjaPromptFormatter : implements
+    IPromptDirectory <|.. PromptDirectory : implements
+
+    ICuratedDataService <|.. JsonLoader : implements
+    ICuratedDataService <|.. MockCuratedDataService : implements
+
+    IExportService <|.. SqliteExportService : implements
+    SqliteExportService *-- IDateTimeProvider : owns
+    IDateTimeProvider <|.. SystemDateTimeProvider : implements