3 Commits

Author SHA1 Message Date
Aaron Po
0473f48fb7 fix mock-only builds 2026-07-03 02:27:00 -04:00
2b8a900d12 Pipeline: Add LLM and mocked user generation to the pipeline (#230) 2026-07-01 07:47:24 -04:00
Aaron Po
880f73e004 update pipeline docs 2026-06-21 21:58:03 -04:00
82 changed files with 3621 additions and 961 deletions

View File

@@ -111,11 +111,11 @@ persona/bio generation later.
The full multinational dataset is kept as-is (106 countries for forenames, The full multinational dataset is kept as-is (106 countries for forenames,
75 for surnames) rather than trimmed to `locations.json`'s current country 75 for surnames) rather than trimmed to `locations.json`'s current country
list, so it doesn't need re-sourcing if more countries are added later. list, so it doesn't need re-sourcing if more countries are added later.
`NamesByCountry::SampleName()` returns no result for a country present in `SampleName()` (a free helper in `generate_users.cc`) returns no result for
neither file; of the countries in `locations.json`, that's currently `KE`, a country present in neither file; of the countries in `locations.json`,
`SE`, `SG`, `TH`, `VN`, and `ZA``GenerateUsers` skips cities in those that's currently `KE`, `SE`, `SG`, `TH`, `VN`, and `ZA``GenerateUsers`
countries the same way brewery generation skips cities whose enrichment skips cities in those countries the same way brewery generation skips
lookup fails. cities whose enrichment lookup fails.
--- ---

View File

@@ -1,8 +1,9 @@
# Biergarten Pipeline # Biergarten Pipeline
A C++20 command-line pipeline that samples city records from local JSON, A C++20 command-line pipeline that samples city records from local JSON,
enriches each with Wikipedia context, and generates bilingual brewery names and enriches each with Wikipedia context, and generates bilingual brewery names
descriptions via a local GGUF model or a deterministic mock. and descriptions plus locale-grounded user profiles via a local GGUF model or
a deterministic mock.
> **This pipeline produces AI-generated data.** It is not a source of truth for > **This pipeline produces AI-generated data.** It is not a source of truth for
> brewing techniques, cultural representation, or local-language accuracy. See > brewing techniques, cultural representation, or local-language accuracy. See
@@ -45,6 +46,7 @@ step.
| Beer reviews and ratings | Stable brewery fixtures with enough context to anchor review pages | | Beer reviews and ratings | Stable brewery fixtures with enough context to anchor review pages |
| Social follow relationships | Repeatable brewery entities for feeds, follows, and saved lists | | Social follow relationships | Repeatable brewery entities for feeds, follows, and saved lists |
| Geospatial brewery experiences | Latitude, longitude, and country-level metadata | | Geospatial brewery experiences | Latitude, longitude, and country-level metadata |
| User accounts and profiles | Locale-grounded names, bios, and an auth-ready email/date-of-birth pair for seeding real accounts |
--- ---
@@ -219,34 +221,45 @@ environment variables listed above to match your run.
### Pipeline Stages ### Pipeline Stages
| Stage | Implementation | | Stage | Implementation |
| -------- | --------------------------------------------------------------------------------------------------------------------------------------- | | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| Load | `JsonLoader::LoadLocations()` reads `locations.json` into typed `Location` records. | | Load | `ICuratedDataService` (`CuratedJsonDataService`) reads `locations.json`, `personas.json`, `forenames-by-country.json`, and `surnames-by-country.json` (paths supplied via a `CuratedDataFilePaths` DTO at construction) into typed records, caching each after its first load. `--mocked` runs use `MockCuratedDataService`'s fixed in-memory dataset instead. |
| Sample | `BiergartenPipelineOrchestrator::QueryCitiesWithCountries()` samples `--location-count` locations per run (default `10`). | | Sample | `BiergartenPipelineOrchestrator::QueryCitiesWithCountries()` samples `--location-count` locations per run (default `10`). |
| Enrich | `WikipediaEnrichmentService` fetches brewing and beer-related context. Keeps going when a lookup fails. `--mocked` runs use `MockEnrichmentService` instead and skip Wikipedia entirely. | | Enrich | `WikipediaEnrichmentService` fetches brewing and beer-related context. Keeps going when a lookup fails. `--mocked` runs use `MockEnrichmentService` instead and skip Wikipedia entirely. |
| Generate | `MockGenerator` or `LlamaGenerator` produces brewery names and descriptions in English and the local language. | | Generate Users | `GenerateUsers()` samples a persona and a forename/surname pair per enriched city (skipping countries with no name data), then `MockGenerator` or `LlamaGenerator` produces a username, bio, and activity weight around the sampled name. |
| Store | `SqliteExportService` writes each successful brewery into a fresh dated `.sqlite` database with normalized location and brewery tables. | | Generate Breweries | `MockGenerator` or `LlamaGenerator` produces brewery names and descriptions in English and the local language. |
| Store | `SqliteExportService` writes each successful user and brewery into a fresh dated `.sqlite` database with normalized `locations`, `users`, and `breweries` tables. |
| Log | `spdlog` writes results and warnings to the console. | | Log | `spdlog` writes results and warnings to the console. |
If enrichment or generation fails for a city, that city is skipped and the If name sampling, enrichment, or generation fails for a city, that city is
pipeline continues. skipped and the pipeline continues. `GenerateUsers()` runs before
`GenerateBreweries()` in `BiergartenPipelineOrchestrator::Run()`.
### Key Components ### Key Components
- `src/main.cc` — argument parsing and Boost.DI composition root. - `src/main.cc` — argument parsing and Boost.DI composition root.
- `JsonLoader` — validates curated location input. - `CuratedJsonDataService` — implements `ICuratedDataService`; takes a
`CuratedDataFilePaths` DTO (locations/personas/forenames/surnames paths) in
its constructor, then parses and validates curated location, persona, and
forename/surname JSON, memoizing each result after its first load on a
given instance. `MockCuratedDataService` is the in-memory substitute (4
fixed locations, 3 personas, and name data for `US`/`DE`/`FR`/`BE`) used in
`--mocked` runs.
- `WikipediaEnrichmentService` — queries Wikipedia extracts, caches results, - `WikipediaEnrichmentService` — queries Wikipedia extracts, caches results,
returns empty context on failure. `MockEnrichmentService` is the no-op returns empty context on failure. `MockEnrichmentService` is the no-op
substitute used in `--mocked` runs. substitute used in `--mocked` runs.
- `LlamaGenerator` — formats prompts for Gemma 4, validates JSON output, retries - `LlamaGenerator` — formats prompts for Gemma 4, validates JSON output for
malformed responses up to three times with corrective feedback in the both `GenerateBrewery` and `GenerateUser`, retries malformed responses up
retry prompt. The token budget is fixed across attempts; it is not raised to three times with corrective feedback in the retry prompt. The token
automatically on truncation. budget is fixed across attempts; it is not raised automatically on
- `MockGenerator` — stable hash-based output so the same city input always truncation.
produces the same brewery. - `MockGenerator` — stable hash-based output so the same city/persona/name
- `SqliteExportService` — creates a dated SQLite file per run and persists each input always produces the same brewery or user.
successful brewery into normalized tables. - `SqliteExportService` — creates a dated SQLite file per run and persists
each successful user and brewery into normalized tables.
- Brewery payloads include English and local-language name and description - Brewery payloads include English and local-language name and description
fields. fields. User payloads carry a sampled first/last name and gender, an
LLM-generated username/bio/activity weight, and a programmatically
generated (not LLM-authored) unique email and date of birth.
### Runtime Behaviour ### Runtime Behaviour
@@ -266,6 +279,18 @@ valid output on the first pass; the retry path is kept for resilience.
`MockGenerator` uses stable hashes for repeatable output in demos and Storybook `MockGenerator` uses stable hashes for repeatable output in demos and Storybook
runs. runs.
`CuratedJsonDataService` memoizes each of `LoadLocations()`, `LoadPersonas()`,
`LoadForenamesByCountry()`, and `LoadSurnamesByCountry()` independently the
first time each is called, since `BiergartenPipelineOrchestrator` owns a
single `ICuratedDataService` instance for the whole run — later calls return
the cached result instead of re-parsing.
`GenerateUsers()` samples a forename/surname pair per city via `SampleName()`,
keyed by the city's ISO 3166-1 code. Countries present in `locations.json`
but absent from either name fixture (currently `KE`, `SE`, `SG`, `TH`, `VN`,
`ZA`) are skipped, the same way a failed enrichment or generation call skips
a city — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section.
### Process Flow - Activity Diagram ### Process Flow - Activity Diagram
![An activity diagram](./diagrams/current/output/activity.svg) ![An activity diagram](./diagrams/current/output/activity.svg)
@@ -278,9 +303,10 @@ runs.
## Generated Output ## Generated Output
Each successful run stores a `GeneratedBrewery` pair with the source location Each successful run stores a `BreweryRecord` pair with the source location
and a `BreweryResult` payload. The same generated records are also written to a and a `BreweryResult` payload, and a `UserRecord` pair with the source
fresh SQLite export file named with the current UTC timestamp. location and a `UserResult` payload. The same generated records are also
written to a fresh SQLite export file named with the current UTC timestamp.
| Field | Meaning | | Field | Meaning |
| ------------------- | ------------------------------------------ | | ------------------- | ------------------------------------------ |
@@ -289,6 +315,17 @@ fresh SQLite export file named with the current UTC timestamp.
| `name_local` | Brewery name in the local language. | | `name_local` | Brewery name in the local language. |
| `description_local` | Brewery description in the local language. | | `description_local` | Brewery description in the local language. |
| Field | Meaning |
| ----------------- | ----------------------------------------------------------------- |
| `first_name` | Sampled forename, copied from the curated name data (not LLM-invented). |
| `last_name` | Sampled surname, copied from the curated name data (not LLM-invented). |
| `gender` | Gender associated with the sampled forename in the source dataset. |
| `username` | LLM-generated handle. |
| `bio` | LLM-generated short biography. |
| `activity_weight` | Relative check-in/activity weight, reserved for a future J-curve activity profile. |
| `email` | Unique `@thebiergarten.app` address, generated programmatically from the sampled name. |
| `date_of_birth` | Randomized date of birth (age 19-48), generated programmatically. |
The log dump also includes city, country, state or province, ISO subdivision The log dump also includes city, country, state or province, ISO subdivision
code, latitude, and longitude for each entry. code, latitude, and longitude for each entry.
@@ -367,11 +404,17 @@ Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present.
## Fixture Strategy ## Fixture Strategy
- `--mocked` for stable fixtures, repeatable screenshots, and Storybook runs. - `--mocked` for stable fixtures, repeatable screenshots, and Storybook runs.
`MockCuratedDataService` swaps in for `CuratedJsonDataService`, so no
fixture files need to be present on disk.
- `--model` when geographically grounded content matters for demos. - `--model` when geographically grounded content matters for demos.
- Keep `locations.json` structured enough to support discovery and future - Keep `locations.json` structured enough to support discovery and future
filtering. filtering.
- Treat SQLite output as seed material for the app's brewery domain, not - `personas.json`, `forenames-by-country.json`, and
production data. `surnames-by-country.json` are curated/vendored fixture data, not
LLM-generated — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset
section for provenance.
- Treat SQLite output as seed material for the app's brewery and user
domains, not production data.
--- ---
@@ -382,6 +425,9 @@ Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present.
| `tooling/pipeline/includes/` | Public headers and shared models. | | `tooling/pipeline/includes/` | Public headers and shared models. |
| `tooling/pipeline/src/` | Implementation files. | | `tooling/pipeline/src/` | Implementation files. |
| `tooling/pipeline/locations.json` | Curated city input copied into the build tree. | | `tooling/pipeline/locations.json` | Curated city input copied into the build tree. |
| `tooling/pipeline/personas.json` | Curated user persona archetypes copied into the build tree. |
| `tooling/pipeline/forenames-by-country.json` | Vendored (CC0) forename data by ISO 3166-1 country code. |
| `tooling/pipeline/surnames-by-country.json` | Vendored (CC0) surname data by ISO 3166-1 country code. |
| `tooling/pipeline/prompts/` | System prompts used by the model-backed path. | | `tooling/pipeline/prompts/` | System prompts used by the model-backed path. |
| `tooling/pipeline/runpod/` | Dockerfile, launcher, and RunPod pod template. | | `tooling/pipeline/runpod/` | Dockerfile, launcher, and RunPod pod template. |
| `docs/pipeline/diagrams/` | Architecture and pipeline diagrams. | | `docs/pipeline/diagrams/` | Architecture and pipeline diagrams. |
@@ -396,6 +442,9 @@ Paths below are relative to `tooling/pipeline/`.
- `src/main.cc` — argument parsing and DI composition root. - `src/main.cc` — argument parsing and DI composition root.
- `src/biergarten_pipeline_orchestrator/` — orchestration, sampling, logging, - `src/biergarten_pipeline_orchestrator/` — orchestration, sampling, logging,
and export. and export.
- `src/services/curated_data/``CuratedJsonDataService`, the file-backed
`ICuratedDataService`, and `MockCuratedDataService`, the in-memory
`ICuratedDataService` used in `--mocked` runs.
- `src/services/enrichment/wikipedia/` — enrichment service and cache. - `src/services/enrichment/wikipedia/` — enrichment service and cache.
- `src/services/sqlite/` — SQLite export implementation. - `src/services/sqlite/` — SQLite export implementation.
- `src/data_generation/llama/` — local inference, prompt loading, output - `src/data_generation/llama/` — local inference, prompt loading, output
@@ -407,11 +456,12 @@ Paths below are relative to `tooling/pipeline/`.
## Next Steps ## Next Steps
The pipeline currently produces city-aware brewery records and dated SQLite The pipeline currently produces city-aware brewery and user records and
exports. The next passes add additional fixture types so the app can exercise dated SQLite exports. The next passes add additional fixture types so the
the full brewery domain without live data. For the detailed engineering app can exercise the full brewery and social domains without live data. For
breakdown of what's needed to reach the architecture in the detailed engineering breakdown of what's needed to reach the
[`diagrams/planned/`](./diagrams/planned/), see [ROADMAP.md](./ROADMAP.md). architecture in [`diagrams/planned/`](./diagrams/planned/), see
[ROADMAP.md](./ROADMAP.md).
### Testing — Very High Priority ### Testing — Very High Priority
@@ -433,12 +483,6 @@ Generate catalog entries with style, ABV, IBU, color, aroma notes, and food
pairing hints. Link beers back to breweries and cities. Keep style coverage wide pairing hints. Link beers back to breweries and cities. Keep style coverage wide
enough to exercise search, sort, and category filters. enough to exercise search, sort, and category filters.
### User Generation
Generate user profiles with stable names, bios, locale hints, and preference
signals. Include stable IDs for downstream fixture joins. Keep output
deterministic for screenshots while allowing larger randomized batches.
### Check-In System ### Check-In System
Produce timestamped check-in events between users and breweries. Use a J-curve Produce timestamped check-in events between users and breweries. Use a J-curve

View File

@@ -39,47 +39,80 @@ architecture needs.
not LLM-invented. not LLM-invented.
- [x] Add `Name` (`first_name`, `last_name`, `gender`) — the sampled result - [x] Add `Name` (`first_name`, `last_name`, `gender`) — the sampled result
handed to `DataGenerator::GenerateUser`. Add `ForenameEntry` (`name`, handed to `DataGenerator::GenerateUser`. Add `ForenameEntry` (`name`,
`gender`) and a `NamesByCountry` class (not a bare typedef) that owns `gender`). Landed as a flatter shape than originally planned here: no
two maps — `std::unordered_map<std::string, std::vector<ForenameEntry>>` `NamesByCountry` wrapper class. `curated_data_service.h` instead
and `std::unordered_map<std::string, std::vector<std::string>>`, both declares `ForenameList = std::vector<ForenameEntry>` and
keyed by ISO 3166-1 code — and exposes `SampleName(iso3166_1, rng)`. `SurnameList = std::vector<std::string>`, and
`ICuratedDataService` exposes two maps keyed by ISO 3166-1 code —
`ForenamesByCountryMap = unordered_map<string, ForenameList>` and
`SurnamesByCountryMap = unordered_map<string, SurnameList>`
directly (see §2). Sampling is
a free function, `SampleName(forenames_by_country,
surnames_by_country, iso3166_1, rng)`, in
`generate_users.cc`'s anonymous namespace, not a method on a class.
Pairing a forename with a surname happens at sample time so gender Pairing a forename with a surname happens at sample time so gender
(present per-forename in the source data) is never discarded the way a (present per-forename in the source data) is never discarded the way a
pre-flattened `{first_name, last_name}` list would lose it; see pre-flattened `{first_name, last_name}` list would lose it; see §2.
`LoadNamesByCountry()` below.
- [ ] Extend `GeneratedBrewery` with `brewery_id`, `context_completeness`, - [ ] Extend `GeneratedBrewery` with `brewery_id`, `context_completeness`,
`metadata` (currently just `{location, brewery}`). `metadata` (currently just `{location, brewery}`).
- [ ] Add `GeneratedBeer`, `GeneratedCheckin`, `GeneratedRating`, - [ ] Add `GeneratedBeer`, `GeneratedCheckin`, `GeneratedRating`,
`GeneratedFollow` aggregate structs. `GeneratedFollow` aggregate structs.
- [x] Add `GeneratedUser` aggregate struct. Carries `email`, `date_of_birth`, - [x] Add `UserRecord` (`location`, `user : UserResult`, `email`,
and `password` — programmatically generated by the orchestrator, never `date_of_birth`) as the current stand-in for the planned
LLM-authored — so a downstream auth-account seeding consumer (outside `GeneratedUser``email` and `date_of_birth` are programmatically
this repo tree's docs) can register real accounts from the pipeline's generated by the orchestrator, never LLM-authored, so a downstream
SQLite export. Skips `user_id`/`metadata` for now, matching auth-account seeding consumer can register real accounts from the
`GeneratedBrewery`'s current (pre-`brewery_id`/`metadata`) shape. Not pipeline's SQLite export. No `password`, `user_id`, or `metadata`
yet wired into `IExportService`/`SqliteExportService``GenerateUsers` field yet, matching `BreweryRecord`'s equally pre-`metadata` shape.
currently only logs and holds `generated_users_` in memory; no `users` Already wired into `IExportService`/`SqliteExportService`: a `users`
table exists in the SQLite export yet (see §5). table exists and `GenerateUsers()` exports each successful record via
`ProcessRecord(const UserRecord&)` (see §5) in addition to logging and
holding `generated_users_` in memory. Still missing vs. the planned
`GeneratedUser`: `user_id`, `password`, and `metadata`.
- [x] Add `UserPersona` (`name`, `description`, `style_affinities`). - [x] Add `UserPersona` (`name`, `description`, `style_affinities`).
## 2. Data Preloading ## 2. Data Preloading
`tooling/pipeline/includes/json_handling/`, fixture files. `tooling/pipeline/includes/services/curated_data/`, fixture files.
- [ ] Extract a `DataPreloader` interface; have `JsonLoader` implement it. - [x] Extract an interface; have `CuratedJsonDataService` implement it.
`JsonLoader` now also exposes static `LoadPersonas()` and Landed as `ICuratedDataService`
`LoadNamesByCountry()` (see below), but as additional static methods, (`services/curated_data/curated_data_service.h`), not `DataPreloader`
not yet behind a formal interface. `LoadLocations()`, `LoadPersonas()`, `LoadForenamesByCountry()`, and
`LoadSurnamesByCountry()` are all virtual methods returning `const&`
and take no arguments. `CuratedJsonDataService`
(`services/curated_data/curated_json_data_service.h`, originally
landed as `JsonLoader` under `json_handling/` before being renamed and
moved) takes a `CuratedDataFilePaths` DTO (`locations_path`,
`personas_path`, `forenames_path`, `surnames_path`) in its
constructor rather than a path per `Load*()` call, and memoizes each
result in a private `cache` struct (`locations`, `personas`,
`forenames_by_country`, `surnames_by_country`), so a second call on
the same instance returns the cached result instead of re-parsing —
safe because `CuratedJsonDataService` outlives every call site (owned
by `BiergartenPipelineOrchestrator` via
`unique_ptr<ICuratedDataService>` for the whole run). `main.cc`'s DI
injector also gained a `MockCuratedDataService` (fixed in-memory
data: 4 locations across `US`/`DE`/`FR`/`BE`, 3 personas, and
matching forename/surname sets for those four countries), bound in
place of `CuratedJsonDataService` under `--mocked`, mirroring the
existing `MockEnrichmentService`/`MockGenerator` pattern.
- [ ] Add `LoadBeerStyles()`. `beer-styles.json` already exists in the repo - [ ] Add `LoadBeerStyles()`. `beer-styles.json` already exists in the repo
and is already copied into the Docker image and is already copied into the Docker image
(`runpod/Dockerfile`), but no loader reads it yet, and the native (`runpod/Dockerfile`), but no loader reads it yet, and the native
CMake build doesn't copy it into `build/` at all — `CMakeLists.txt`'s CMake build doesn't copy it into `build/` at all — `CMakeLists.txt`'s
"Runtime Assets" step only copies `locations.json` and `prompts/`. "Runtime Assets" step only copies `locations.json` and `prompts/`.
- [x] Add `LoadPersonas()` and author `personas.json` (doesn't exist yet). - [x] Add `LoadPersonas()` and author `personas.json` (doesn't exist yet).
- [x] Add `LoadNamesByCountry(forenames_filepath, surnames_filepath)`, - [x] Add name-by-country loading, parsing
parsing `tooling/pipeline/forenames-by-country.json` and `tooling/pipeline/forenames-by-country.json` and
`surnames-by-country.json` into a `NamesByCountry`. Both files are `surnames-by-country.json`. Landed as two separate methods —
vendored verbatim (unmodified, full multinational coverage) from `LoadForenamesByCountry()` and `LoadSurnamesByCountry()` — each
returning a flat `ForenamesByCountryMap` / `SurnamesByCountryMap`
(aliases for `unordered_map<string, ForenameList>` /
`unordered_map<string, SurnameList>`) keyed by ISO 3166-1 code, rather
than one combined `LoadNamesByCountry() : NamesByCountry` call. Both
files are vendored verbatim (unmodified,
full multinational coverage) from
`sigpwned/popular-names-by-country-dataset` (CC0) — see `sigpwned/popular-names-by-country-dataset` (CC0) — see
ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for
provenance. Deliberately not pre-paired or filtered down to just the provenance. Deliberately not pre-paired or filtered down to just the
@@ -136,12 +169,19 @@ Entirely new — no `includes/policy/` (or equivalent) directory exists today.
`tooling/pipeline/includes/services/database/`, `src/services/sqlite/`. `tooling/pipeline/includes/services/database/`, `src/services/sqlite/`.
- [ ] Extend `IExportService` with `ProcessBeer`, `ProcessUser`, - [x] `IExportService::ProcessRecord(const UserRecord&)` and
`ProcessCheckin`, `ProcessRating`, `ProcessFollow` (today only `SqliteExportService`'s `users` table (see
`ProcessRecord(const GeneratedBrewery&)` exists). `kCreateUsersTableSql` / `kInsertUserSql` in
- [ ] Add `beers`, `users`, `checkins`, `ratings`, `follows` tables to `includes/services/database/sqlite_statement_helpers.h`) are
`SqliteExportService::InitializeSchema()` — see implemented — `GenerateUsers()` exports every successful user
`kCreateLocationsTableSql` / `kCreateBreweriesTableSql` in alongside its resolved `location_id`.
- [ ] Extend `IExportService` with `ProcessBeer`, `ProcessCheckin`,
`ProcessRating`, `ProcessFollow` (today `ProcessRecord(const
BreweryRecord&)` and `ProcessRecord(const UserRecord&)` exist).
- [ ] Add `beers`, `checkins`, `ratings`, `follows` tables to
`SqliteExportService::InitializeSchema()` (`locations`, `breweries`,
and `users` already exist) — see `kCreateLocationsTableSql` /
`kCreateBreweriesTableSql` / `kCreateUsersTableSql` in
`includes/services/database/sqlite_statement_helpers.h` for the `includes/services/database/sqlite_statement_helpers.h` for the
existing pattern to follow. existing pattern to follow.
- [ ] Add a `brewery_cache_` alongside the existing `location_cache_`, and - [ ] Add a `brewery_cache_` alongside the existing `location_cache_`, and
@@ -187,7 +227,8 @@ Entirely new — no `includes/policy/` (or equivalent) directory exists today.
## 8. Fixtures / Build ## 8. Fixtures / Build
- [ ] Author `personas.json` and `names-by-country.json` (see §2). - [x] Author `personas.json`, `forenames-by-country.json`, and
`surnames-by-country.json` (see §2).
- [ ] Fix the `beer-styles.json` build-tree gap: add it to the "Runtime - [ ] Fix the `beer-styles.json` build-tree gap: add it to the "Runtime
Assets" `configure_file` step in `CMakeLists.txt` so native builds and Assets" `configure_file` step in `CMakeLists.txt` so native builds and
the Docker image agree on what's available at runtime. the Docker image agree on what's available at runtime.

View File

@@ -93,6 +93,77 @@ while (For each sampled Location?) is (Remaining cities)
endif endif
endwhile (Done) endwhile (Done)
|#EAF0E8|BiergartenPipelineOrchestrator|
:GenerateUsers(enriched_cities);
|#E2EBDC|JsonLoader|
:LoadPersonas("personas.json");
:LoadForenamesByCountry("forenames-by-country.json");
:LoadSurnamesByCountry("surnames-by-country.json");
note right
Each call is cached after its first parse.
MockCuratedDataService (--mocked) returns a
fixed in-memory dataset instead and skips
these three JSON files entirely.
end note
|#EAF0E8|BiergartenPipelineOrchestrator|
while (For each EnrichedCity?) is (Remaining cities)
:SampleName(forenames_by_country,\n surnames_by_country, location.iso3166_1);
if (Names available for country?) then (no)
:Log warning, skip city;
else (yes)
:Sample UserPersona (uniform random);
|#E5EDE1|DataGenerator|
if (Generator Mode) then (MockGenerator)
:DeterministicHash & Format;
else (LlamaGenerator)
:prompt_directory_->Load("USER_GENERATION");
note right
Resolves to USER_GENERATION.md inside --prompt-dir.
end note
repeat
:Infer(system_prompt, user_prompt, max_tokens, kUserJsonGrammar);
:ValidateUserJson(raw, user);
if (Is JSON Valid?) then (yes)
break
else (no)
:Attempt++;
:Append validation error to retry prompt;
endif
repeat while (Attempt < 3?) is (yes)
endif
|#EAF0E8|BiergartenPipelineOrchestrator|
if (Generation successful?) then (yes)
:BuildEmail(name, used_email_local_parts);
:GenerateDateOfBirth(rng);
note right
email and date_of_birth are generated
programmatically, never LLM-authored.
end note
|#E0EAE0|SqliteExportService|
:ProcessRecord(UserRecord);
if (Location in cache?) then (yes)
:Reuse location_id;
else (no)
:Insert Location & Cache ID;
endif
:Insert User (FK: location_id);
if (Exception caught during insert?) then (yes)
|#EAF0E8|BiergartenPipelineOrchestrator|
:Log warning "SQLite export failed";
else (no)
endif
else (no)
:Log warning "Generation failed, skipping...";
endif
endif
endwhile (Done)
|#EAF0E8|BiergartenPipelineOrchestrator| |#EAF0E8|BiergartenPipelineOrchestrator|
:GenerateBreweries(enriched_cities); :GenerateBreweries(enriched_cities);

View File

@@ -30,11 +30,14 @@ class BiergartenPipelineOrchestrator {
- context_service_ : std::unique_ptr<IEnrichmentService> - context_service_ : std::unique_ptr<IEnrichmentService>
- generator_ : std::unique_ptr<DataGenerator> - generator_ : std::unique_ptr<DataGenerator>
- exporter_ : std::unique_ptr<IExportService> - exporter_ : std::unique_ptr<IExportService>
- curated_data_service_ : std::unique_ptr<ICuratedDataService>
- application_options_ : ApplicationOptions - application_options_ : ApplicationOptions
- generated_breweries_ : std::vector<GeneratedBrewery> - generated_breweries_ : std::vector<BreweryRecord>
- generated_users_ : std::vector<UserRecord>
+ Run() : bool + Run() : bool
- QueryCitiesWithCountries() : std::vector<Location> - QueryCitiesWithCountries() : std::vector<Location>
- GenerateBreweries(cities : std::span<const EnrichedCity>) : void - GenerateBreweries(cities : std::span<const EnrichedCity>) : void
- GenerateUsers(cities : std::span<const EnrichedCity>) : void
- LogResults() : void - LogResults() : void
} }
@@ -113,13 +116,14 @@ class HttpWebClient {
interface DataGenerator <<interface>> { interface DataGenerator <<interface>> {
+ GenerateBrewery(location : const Location&, region_context : const std::string&) : BreweryResult + GenerateBrewery(location : const Location&, region_context : const std::string&) : BreweryResult
+ GenerateUser(locale : const std::string&) : UserResult + GenerateUser(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult
} }
class MockGenerator { class MockGenerator {
+ GenerateBrewery(...) : BreweryResult + GenerateBrewery(location : const Location&, region_context : const std::string&) : BreweryResult
+ GenerateUser(...) : UserResult + GenerateUser(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult
- DeterministicHash(location : const Location&) : size_t - DeterministicHash(location : const Location&) : size_t
- DeterministicHash(location : const Location&, persona : const UserPersona&, name : const Name&) : size_t
} }
class LlamaGenerator { class LlamaGenerator {
@@ -153,13 +157,49 @@ class PromptDirectory {
+ Load(key : std::string_view) : std::string + Load(key : std::string_view) : std::string
} }
class JsonLoader { interface ICuratedDataService <<interface>> {
+ {static} LoadLocations(filepath : const std::filesystem::path&) : std::vector<Location> + LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
} }
class JsonLoader {
- cache_ : cache
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
}
note right of JsonLoader
Each Load* method memoizes its result in
cache_ on first call; later calls on the
same instance return the cached value
without re-parsing.
end note
class MockCuratedDataService {
- locations_ : std::vector<Location>
- personas_ : std::vector<UserPersona>
- forenames_by_country_ : std::unordered_map<std::string, forename_list>
- surnames_by_country_ : std::unordered_map<std::string, surname_list>
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
}
note right of MockCuratedDataService
Fixed 4-location / 3-persona dataset
(US, DE, FR, BE) used in --mocked runs;
filepath arguments are ignored.
end note
interface IExportService <<interface>> { interface IExportService <<interface>> {
+ Initialize() : void + Initialize() : void
+ ProcessRecord(brewery : const GeneratedBrewery&) : void + ProcessRecord(brewery : const BreweryRecord&) : uint64_t
+ ProcessRecord(user : const UserRecord&) : uint64_t
+ Finalize() : void + Finalize() : void
} }
@@ -167,15 +207,18 @@ class SqliteExportService {
- date_time_provider_ : std::unique_ptr<IDateTimeProvider> - date_time_provider_ : std::unique_ptr<IDateTimeProvider>
- run_timestamp_utc_ : std::string - run_timestamp_utc_ : std::string
- database_path_ : std::filesystem::path - database_path_ : std::filesystem::path
- db_handle_ : sqlite3* - db_handle_ : SqliteDatabaseHandle
- insert_location_stmt_ : sqlite3_stmt* - insert_location_stmt_ : SqliteStatementHandle
- insert_brewery_stmt_ : sqlite3_stmt* - insert_brewery_stmt_ : SqliteStatementHandle
- insert_user_stmt_ : SqliteStatementHandle
- transaction_open_ : bool - transaction_open_ : bool
- location_cache_ : std::unordered_map<std::string, sqlite3_int64> - location_cache_ : std::unordered_map<std::string, sqlite3_int64>
+ Initialize() : void + Initialize() : void
+ ProcessRecord(brewery : const GeneratedBrewery&) : void + ProcessRecord(brewery : const BreweryRecord&) : uint64_t
+ ProcessRecord(user : const UserRecord&) : uint64_t
+ Finalize() : void + Finalize() : void
- InitializeSchema() : void - InitializeSchema() : void
- ResolveLocationId(location : const Location&) : sqlite3_int64
} }
interface IDateTimeProvider <<interface>> { interface IDateTimeProvider <<interface>> {
@@ -191,6 +234,7 @@ BiergartenPipelineOrchestrator *-- ILogger : owns
BiergartenPipelineOrchestrator *-- IEnrichmentService : owns BiergartenPipelineOrchestrator *-- IEnrichmentService : owns
BiergartenPipelineOrchestrator *-- DataGenerator : owns BiergartenPipelineOrchestrator *-- DataGenerator : owns
BiergartenPipelineOrchestrator *-- IExportService : owns BiergartenPipelineOrchestrator *-- IExportService : owns
BiergartenPipelineOrchestrator *-- ICuratedDataService : owns
LogEntry *-- LogLevel LogEntry *-- LogLevel
LogEntry *-- PipelinePhase LogEntry *-- PipelinePhase
@@ -215,7 +259,8 @@ LlamaGenerator *-- IPromptDirectory : uses
IPromptFormatter <|.. Gemma4JinjaPromptFormatter : implements IPromptFormatter <|.. Gemma4JinjaPromptFormatter : implements
IPromptDirectory <|.. PromptDirectory : implements IPromptDirectory <|.. PromptDirectory : implements
BiergartenPipelineOrchestrator ..> JsonLoader : uses ICuratedDataService <|.. JsonLoader : implements
ICuratedDataService <|.. MockCuratedDataService : implements
IExportService <|.. SqliteExportService : implements IExportService <|.. SqliteExportService : implements
SqliteExportService *-- IDateTimeProvider : owns SqliteExportService *-- IDateTimeProvider : owns

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 55 KiB

After

Width:  |  Height:  |  Size: 86 KiB

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 71 KiB

After

Width:  |  Height:  |  Size: 87 KiB

View File

@@ -43,10 +43,18 @@ fork again
:EnrichmentService::PreWarmLocationCache(sampled_locations); :EnrichmentService::PreWarmLocationCache(sampled_locations);
end fork end fork
fork fork
:JsonLoader::LoadNamesByCountry(\n "forenames-by-country.json",\n "surnames-by-country.json"); :JsonLoader::LoadForenamesByCountry(\n "forenames-by-country.json");
fork again
:JsonLoader::LoadSurnamesByCountry(\n "surnames-by-country.json");
fork again fork again
:JsonLoader::LoadPersonas("personas.json"); :JsonLoader::LoadPersonas("personas.json");
end fork end fork
note right
Each call is memoized after its first parse
(implemented today). MockCuratedDataService
returns a fixed in-memory dataset instead
under --mocked, skipping these JSON files.
end note
' ═══════════════════════════════════════════ ' ═══════════════════════════════════════════
' PHASE 0 — USER GENERATION ' PHASE 0 — USER GENERATION
@@ -82,11 +90,15 @@ fork again
ibu_preference, checkin_weight. ibu_preference, checkin_weight.
end note end note
:NamesByCountry::SampleName(\n location.iso3166_1); :SampleName(forenames_by_country,\n surnames_by_country, location.iso3166_1, rng);
note right note right
Deterministic lookup -- no LLM involved. Deterministic lookup -- no LLM involved.
Name selected from pre-keyed table Free function (implemented today in
generate_users.cc), not a class method.
Name selected from pre-keyed maps
and passed into the generation prompt. and passed into the generation prompt.
Skips the city if either map has no
entry for iso3166_1.
end note end note
:GenerateUser(enriched_city, persona, sampled_name)\nvia DataGenerator; :GenerateUser(enriched_city, persona, sampled_name)\nvia DataGenerator;

View File

@@ -86,27 +86,30 @@ package "Domain: Models" {
+ gender : std::string + gender : std::string
} }
class NamesByCountry { note right of ForenameEntry
- forenames_by_country_ : std::unordered_map<std::string, std::vector<ForenameEntry>> forename_list = std::unordered_set<ForenameEntry>
- surnames_by_country_ : std::unordered_map<std::string, std::vector<std::string>> surname_list = std::unordered_set<std::string>
+ SampleName(iso3166_1 : const std::string&,\n rng : std::mt19937&) : std::optional<Name> (aliases declared in curated_data_service.h).
} ICuratedDataService::LoadForenamesByCountry() /
LoadSurnamesByCountry() each return a flat
note right of NamesByCountry std::unordered_map<std::string, forename_list>
Backed by two source-shaped maps, or <std::string, surname_list> keyed by ISO
keyed by ISO 3166-1 country code, 3166-1 country code -- no NamesByCountry
sourced from popular-names-by-country-dataset wrapper class. SampleName(forenames_by_country,
(CC0) -- see ETHICS-AND-KNOWN-ISSUES.md. surnames_by_country, iso3166_1, rng) is a free
Gender is preserved per forename so it can function, not a method, so it can be called by
be threaded through to persona/bio generation any per-city worker without owning the maps.
later; surnames carry no gender in the source Sourced from popular-names-by-country-dataset
data. Pairing happens at sample time, not at (CC0) -- see ETHICS-AND-KNOWN-ISSUES.md. Gender
fixture-authoring time, so no information from is preserved per forename so it can be threaded
the source dataset is discarded up front. through to persona/bio generation later; surnames
carry no gender in the source data. Pairing
happens at sample time, not at fixture-authoring
time, so no information from the source dataset
is discarded up front.
end note end note
NamesByCountry *-- "0..*" ForenameEntry ForenameEntry ..> Name : SampleName() produces
NamesByCountry ..> Name : produces
class CheckinResult { class CheckinResult {
+ checked_in_at : std::string + checked_in_at : std::string
@@ -365,20 +368,43 @@ package "Infrastructure: Pipeline Channel" {
package "Infrastructure: Data Preloading" { package "Infrastructure: Data Preloading" {
interface DataPreloader <<interface>> { interface ICuratedDataService <<interface>> {
+ LoadLocations(filepath : const std::filesystem::path&) : std::vector<Location> + LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
+ LoadBeerStyles(filepath : const std::filesystem::path&) : std::vector<BeerStyle> + LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&
+ LoadPersonas(filepath : const std::filesystem::path&) : std::vector<UserPersona> + LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadNamesByCountry(forenames_filepath : const std::filesystem::path&,\n surnames_filepath : const std::filesystem::path&) : NamesByCountry + LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
} }
class JsonLoader { class JsonLoader {
+ LoadLocations(filepath : const std::filesystem::path&) : std::vector<Location> + LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
+ LoadBeerStyles(filepath : const std::filesystem::path&) : std::vector<BeerStyle> + LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&
+ LoadPersonas(filepath : const std::filesystem::path&) : std::vector<UserPersona> + LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadNamesByCountry(forenames_filepath : const std::filesystem::path&,\n surnames_filepath : const std::filesystem::path&) : NamesByCountry + LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
} }
note right of JsonLoader
Each Load* call is memoized after its first
parse -- implemented today, ahead of the rest
of this package (LoadBeerStyles is still planned).
end note
class MockCuratedDataService {
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
+ LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
}
note right of MockCuratedDataService
Fixed in-memory dataset used in --mocked mode,
implemented today for Locations/Personas/
Forenames/Surnames; would need a LoadBeerStyles
fixture too once that method is implemented.
end note
} }
package "Infrastructure: Enrichment" { package "Infrastructure: Enrichment" {
@@ -520,7 +546,7 @@ package "Infrastructure: Data Export" {
} }
class BiergartenPipelineOrchestrator { class BiergartenPipelineOrchestrator {
- preloader_ : std::unique_ptr<DataPreloader> - curated_data_service_ : std::unique_ptr<ICuratedDataService>
- enrichment_service_ : std::unique_ptr<EnrichmentService> - enrichment_service_ : std::unique_ptr<EnrichmentService>
- generator_ : std::unique_ptr<DataGenerator> - generator_ : std::unique_ptr<DataGenerator>
- logger_ : std::unique_ptr<Logger> - logger_ : std::unique_ptr<Logger>
@@ -548,7 +574,7 @@ class BiergartenPipelineOrchestrator {
} }
' --- Orchestration Aggregations (Services & Strategies) --- ' --- Orchestration Aggregations (Services & Strategies) ---
BiergartenPipelineOrchestrator *-- DataPreloader BiergartenPipelineOrchestrator *-- ICuratedDataService
BiergartenPipelineOrchestrator *-- EnrichmentService BiergartenPipelineOrchestrator *-- EnrichmentService
BiergartenPipelineOrchestrator *-- DataGenerator BiergartenPipelineOrchestrator *-- DataGenerator
BiergartenPipelineOrchestrator *-- ExportService BiergartenPipelineOrchestrator *-- ExportService
@@ -567,7 +593,8 @@ BiergartenPipelineOrchestrator *-- "0..*" GeneratedCheckin : checkin_pool_
BiergartenPipelineOrchestrator *-- "0..*" GeneratedFollow : follow_pool_ BiergartenPipelineOrchestrator *-- "0..*" GeneratedFollow : follow_pool_
' --- Interfaces & Implementations --- ' --- Interfaces & Implementations ---
DataPreloader <|.. JsonLoader ICuratedDataService <|.. JsonLoader
ICuratedDataService <|.. MockCuratedDataService
Logger <|.. PipelineLogger Logger <|.. PipelineLogger
ContextStrategy <|.. BreweryContextStrategy ContextStrategy <|.. BreweryContextStrategy
ContextStrategy <|.. BeerContextStrategy ContextStrategy <|.. BeerContextStrategy

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 139 KiB

After

Width:  |  Height:  |  Size: 142 KiB

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 234 KiB

After

Width:  |  Height:  |  Size: 240 KiB

View File

@@ -0,0 +1,24 @@
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
lerna-debug.log*
node_modules
dist
dist-ssr
*.local
# Editor directories and files
.vscode/*
!.vscode/extensions.json
.idea
.DS_Store
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?

View File

@@ -0,0 +1,74 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<link rel="icon" type="image/svg+xml" href="/favicon.svg" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Pipeline Results Viewer</title>
</head>
<body>
<div class="dialog" id="window">
<div class="dialog-header">
<p>Pipeline Results Viewer</p>
</div>
<div class="tab-strip" role="tablist">
<button
type="button"
class="tab is-active"
id="tab-tables"
role="tab"
aria-selected="true"
aria-controls="panel-tables"
>
Tables
</button>
<button
type="button"
class="tab"
id="tab-query"
role="tab"
aria-selected="false"
aria-controls="panel-query"
>
Query
</button>
</div>
<div class="dialog-body">
<div
class="tab-panel"
id="panel-tables"
role="tabpanel"
aria-labelledby="tab-tables"
>
<div class="field-group">
<label for="table-select">Table</label>
<select id="table-select"></select>
</div>
<p id="status"></p>
<div id="table-container"></div>
</div>
<div
class="tab-panel"
id="panel-query"
role="tabpanel"
aria-labelledby="tab-query"
hidden
>
<div class="field-group field-group--stacked">
<label for="query-input">SQL Query</label>
<textarea
id="query-input"
rows="4"
spellcheck="false"
placeholder="SELECT * FROM ..."
></textarea>
</div>
<button type="button" id="query-run" class="btn">Run Query</button>
<p id="query-status"></p>
<div id="query-container"></div>
</div>
</div>
</div>
<script type="module" src="/src/main.ts"></script>
</body>
</html>

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,20 @@
{
"name": "pipeline-results-viewer",
"version": "0.0.0",
"private": true,
"type": "module",
"scripts": {
"dev": "vite",
"build": "tsc && vite build",
"preview": "vite preview"
},
"devDependencies": {
"@types/sql.js": "^1.4.11",
"sass": "^1.101.0",
"typescript": "^6.0.3",
"vite": "^8.0.12"
},
"dependencies": {
"sql.js": "^1.14.1"
}
}

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 9.3 KiB

View File

@@ -0,0 +1,147 @@
import initSqlJs, { type Database, type SqlValue } from "sql.js";
import sqlWasmUrl from "sql.js/dist/sql-wasm.wasm?url";
import "./style.scss";
const DB_URL = "/biergarten.sqlite";
const tabs = document.querySelectorAll<HTMLButtonElement>("[role='tab']");
const panels = document.querySelectorAll<HTMLDivElement>("[role='tabpanel']");
const tableSelect = document.querySelector<HTMLSelectElement>("#table-select")!;
const tableContainer =
document.querySelector<HTMLDivElement>("#table-container")!;
const status = document.querySelector<HTMLParagraphElement>("#status")!;
const queryInput = document.querySelector<HTMLTextAreaElement>("#query-input")!;
const queryRun = document.querySelector<HTMLButtonElement>("#query-run")!;
const queryStatus = document.querySelector<HTMLParagraphElement>("#query-status")!;
const queryContainer =
document.querySelector<HTMLDivElement>("#query-container")!;
for (const tab of tabs) {
tab.addEventListener("click", () => {
for (const otherTab of tabs) {
const isActive = otherTab === tab;
otherTab.classList.toggle("is-active", isActive);
otherTab.setAttribute("aria-selected", String(isActive));
}
for (const panel of panels) {
panel.hidden = panel.id !== tab.getAttribute("aria-controls");
}
});
}
const renderResultTable = (
container: HTMLDivElement,
columns: string[],
values: SqlValue[][],
) => {
const table = document.createElement("table");
const head = table.createTHead().insertRow();
for (const column of columns) {
const th = document.createElement("th");
th.textContent = column;
head.append(th);
}
const body = table.createTBody();
for (const row of values) {
const tr = body.insertRow();
for (const cell of row) {
const td = document.createElement("td");
td.textContent = String(cell ?? "");
tr.append(td);
}
}
container.append(table);
};
const renderTable = (db: Database, name: string) => {
const result = db.exec(`SELECT * FROM "${name}"`);
tableContainer.replaceChildren();
if (result.length === 0) {
tableContainer.append(
Object.assign(document.createElement("p"), { textContent: "No rows." }),
);
return;
}
const { columns, values } = result[0];
renderResultTable(tableContainer, columns, values);
};
const runQuery = (db: Database) => {
const sql = queryInput.value.trim();
queryContainer.replaceChildren();
if (!sql) {
queryStatus.textContent = "Enter a query to run.";
return;
}
try {
const results = db.exec(sql);
if (results.length === 0) {
queryStatus.textContent = "Query executed. No rows returned.";
return;
}
const { columns, values } = results[results.length - 1];
renderResultTable(queryContainer, columns, values);
queryStatus.textContent = `${values.length} row${values.length === 1 ? "" : "s"} returned.`;
} catch (error) {
queryStatus.textContent = `Error: ${error instanceof Error ? error.message : String(error)}`;
}
};
async function main(): Promise<void> {
status.textContent = "Loading database...";
const sqlJs = await initSqlJs({ locateFile: () => sqlWasmUrl });
const response = await fetch(DB_URL);
const buffer = await response.arrayBuffer();
const db = new sqlJs.Database(new Uint8Array(buffer));
const tables =
db
.exec(
`SELECT name
FROM sqlite_master
WHERE
type = 'table'
AND
name NOT LIKE 'sqlite_%'
ORDER BY name`,
)[0]
?.values.map((row) => String(row[0])) ?? [];
for (const name of tables) {
tableSelect.append(new Option(name, name));
}
tableSelect.addEventListener("change", () =>
renderTable(db, tableSelect.value),
);
if (tables.length > 0) {
renderTable(db, tables[0]);
}
status.textContent = "";
queryRun.addEventListener("click", () => runQuery(db));
queryInput.addEventListener("keydown", (event) => {
if ((event.metaKey || event.ctrlKey) && event.key === "Enter") {
event.preventDefault();
runQuery(db);
}
});
}
main().catch((error: unknown) => {
status.textContent = `Failed to load database: ${String(error)}`;
});

View File

@@ -0,0 +1,267 @@
$gray-100: #dedede;
$gray-300: #c3c3c3;
$gray-500: #808080;
$white: #ffffff;
$black: #000000;
$navy: #000080;
$blue: #1084d0;
$teal: #008080;
$color-surface: $gray-300;
$color-surface-alt: $gray-100;
$color-surface-sunken: $white;
$color-border-highlight: $gray-100;
$color-border-lowlight: $gray-500;
$color-border-light: $white;
$color-border-dark: $black;
$color-text: $black;
$color-text-on-accent: $white;
$color-text-status: $navy;
$color-accent-start: $navy;
$color-accent-end: $blue;
$color-desktop-bg: $teal;
$color-row-stripe: $gray-100;
$color-shadow: rgba($black, 0.4);
$color-focus-ring: $black;
$bevel-size: 2px;
$font-stack: Tahoma, "MS Sans Serif", Verdana, sans-serif;
@mixin bevel-raised($size: $bevel-size) {
border-style: solid;
border-width: $size;
border-top-color: $color-border-highlight;
border-left-color: $color-border-highlight;
border-right-color: $color-border-dark;
border-bottom-color: $color-border-dark;
box-shadow:
inset $size $size 0 0 $color-border-light,
inset (-$size) (-$size) 0 0 $color-border-lowlight;
}
@mixin bevel-sunken($size: $bevel-size) {
border-style: solid;
border-width: $size;
border-top-color: $color-border-lowlight;
border-left-color: $color-border-lowlight;
border-right-color: $color-border-light;
border-bottom-color: $color-border-light;
box-shadow:
inset $size $size 0 0 $color-border-dark,
inset (-$size) (-$size) 0 0 $color-surface-alt;
}
* {
box-sizing: border-box;
}
body {
margin: 0;
padding: 24px;
background: $color-desktop-bg;
font-family: $font-stack;
color: $color-text;
min-height: 100vh;
}
#window {
width: 100%;
max-width: 1600px;
margin: 0 auto;
background: $color-surface;
padding: 6px;
display: flex;
flex-direction: column;
gap: 6px;
@include bevel-raised(3px);
box-shadow: 8px 8px 0 0 $color-shadow;
}
.dialog-header {
background: linear-gradient(90deg, $color-accent-start, $color-accent-end);
color: $color-text-on-accent;
padding: 6px 8px;
display: flex;
align-items: center;
p {
margin: 0;
font-size: 14px;
font-weight: bold;
letter-spacing: 0.5px;
text-transform: uppercase;
}
}
.tab-strip {
display: flex;
gap: 4px;
padding: 8px 8px 0;
}
.tab {
position: relative;
top: 3px;
margin-bottom: -2px;
font-family: $font-stack;
font-size: 12px;
font-weight: bold;
text-transform: uppercase;
padding: 6px 14px;
background: $color-surface;
color: $color-text;
border-style: solid;
border-width: 2px;
border-top-color: $color-border-light;
border-left-color: $color-border-light;
border-right-color: $color-border-dark;
border-bottom-color: $color-border-dark;
border-radius: 0;
cursor: pointer;
&.is-active {
top: 0;
z-index: 1;
border-bottom-color: $color-surface;
}
&:focus {
outline: 1px dotted $color-focus-ring;
outline-offset: -4px;
}
}
.dialog-body {
border-top: 2px solid $color-border-dark;
padding: 12px;
}
.tab-panel {
display: flex;
flex-direction: column;
gap: 12px;
&[hidden] {
display: none;
}
}
.field-group {
display: flex;
align-items: center;
gap: 8px;
label {
font-size: 12px;
font-weight: bold;
text-transform: uppercase;
}
&.field-group--stacked {
flex-direction: column;
align-items: stretch;
}
}
select#table-select {
font-family: $font-stack;
font-size: 13px;
padding: 3px 6px;
background: $color-surface-sunken;
color: $color-text;
border-radius: 0;
@include bevel-sunken();
&:focus {
outline: 1px dotted $color-focus-ring;
outline-offset: 2px;
}
}
textarea#query-input {
font-family: $font-stack;
font-size: 13px;
padding: 6px;
background: $color-surface-sunken;
color: $color-text;
border-radius: 0;
resize: vertical;
@include bevel-sunken();
&:focus {
outline: 1px dotted $color-focus-ring;
outline-offset: 2px;
}
}
.btn {
align-self: flex-start;
font-family: $font-stack;
font-size: 12px;
font-weight: bold;
text-transform: uppercase;
padding: 6px 16px;
background: $color-surface;
color: $color-text;
cursor: pointer;
@include bevel-raised();
&:active {
@include bevel-sunken();
}
&:focus {
outline: 1px dotted $color-focus-ring;
outline-offset: -4px;
}
}
#status,
#query-status {
margin: 0;
font-size: 12px;
font-style: italic;
color: $color-text-status;
}
#table-container,
#query-container {
background: $color-surface-sunken;
padding: 4px;
overflow-x: auto;
@include bevel-sunken();
}
table {
border-collapse: collapse;
margin: 0 auto;
font-family: $font-stack;
}
th,
td {
border: 1px solid $color-border-lowlight;
padding: 4px 8px;
text-align: left;
font-size: 13px;
}
th {
background: $color-surface;
white-space: nowrap;
font-weight: bold;
text-transform: uppercase;
@include bevel-raised(1px);
}
td {
min-width: 120px;
max-width: 800px;
white-space: normal;
word-break: break-word;
background: $color-surface-sunken;
}
tbody tr:nth-child(even) td {
background: $color-row-stripe;
}

View File

@@ -0,0 +1,24 @@
{
"compilerOptions": {
"target": "es2023",
"module": "esnext",
"lib": ["ES2023", "DOM"],
"types": ["vite/client"],
"skipLibCheck": true,
/* Bundler mode */
"moduleResolution": "bundler",
"esModuleInterop": true,
"allowImportingTsExtensions": true,
"verbatimModuleSyntax": true,
"moduleDetection": "force",
"noEmit": true,
/* Linting */
"noUnusedLocals": true,
"noUnusedParameters": true,
"erasableSyntaxOnly": true,
"noFallthroughCasesInSwitch": true
},
"include": ["src"]
}

View File

@@ -139,6 +139,7 @@ FetchContent_MakeAvailable(cpp-httplib)
# 5. Executable & Sources # 5. Executable & Sources
add_executable(${PROJECT_NAME} add_executable(${PROJECT_NAME}
includes/services/enrichment/mock_enrichment.h includes/services/enrichment/mock_enrichment.h
includes/services/curated_data/mock_curated_data_service.h
includes/json_handling/pretty_print.h) includes/json_handling/pretty_print.h)
# --- Entry point --- # --- Entry point ---
@@ -148,12 +149,12 @@ target_sources(${PROJECT_NAME} PRIVATE
# --- json_handling --- # --- json_handling ---
target_sources(${PROJECT_NAME} PRIVATE target_sources(${PROJECT_NAME} PRIVATE
src/json_handling/json_loader.cc src/services/curated_data/curated_json_data_service.cc
) )
# --- data_model --- # --- services: curated_data ---
target_sources(${PROJECT_NAME} PRIVATE target_sources(${PROJECT_NAME} PRIVATE
src/data_model/names_by_country.cc src/services/curated_data/mock_curated_data_service.cc
) )
# --- application_options --- # --- application_options ---

View File

@@ -0,0 +1,72 @@
#!/bin/bash
#
# Walks every directory under src/ and includes/ (including nested
# subdirectories), runs clang-format on the C/C++ files in that directory
# only (non-recursive), and commits the result with its own commit.
#
# Usage: ./format-per-directory.sh [-y]
# -y skip the confirmation prompt
set -euo pipefail
SKIP_CONFIRM=false
if [[ "${1:-}" == "-y" ]]; then
SKIP_CONFIRM=true
fi
cd "$(dirname "$0")"
if [ ! -f .clang-format ]; then
echo "ERROR: .clang-format file not found."
exit 1
fi
if ! command -v clang-format &>/dev/null; then
echo "ERROR: clang-format not found."
exit 1
fi
if ! command -v git &>/dev/null; then
echo "ERROR: git not found."
exit 1
fi
echo "WARNING: This script will format .cpp, .h, .cxx, .cc, .c, .hpp files"
echo "directory-by-directory under src/ and includes/, committing after each"
echo "directory is formatted."
if [[ "$SKIP_CONFIRM" == false ]]; then
read -p "Do you want to continue? (y/n) " -n 1 -r
echo
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
echo "Aborted."
exit 1
fi
fi
if [[ -n "$(git status --porcelain)" ]]; then
echo "ERROR: working tree is not clean. Commit or stash your changes first."
exit 1
fi
dirs=$(find src includes -type d | sort)
for dir in $dirs; do
files=$(find "$dir" -maxdepth 1 -type f \( -name "*.cpp" -o -name "*.hpp" -o -name "*.h" -o -name "*.c" -o -name "*.cc" -o -name "*.cxx" \))
if [[ -z "$files" ]]; then
continue
fi
echo "Formatting $dir..."
echo "$files" | xargs clang-format -i
if [[ -n "$(git status --porcelain -- "$dir")" ]]; then
git add -- "$dir"
git commit -m "Formatted $dir"
else
echo "No changes in $dir, skipping commit."
fi
done
echo "Done."

View File

@@ -0,0 +1,74 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_H_
#define BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_H_
/**
* @file biergarten_pipeline.h
* @brief Master umbrella header — includes every public header in the
* Biergarten pipeline.
*/
// --- orchestrator ---
#include "biergarten_pipeline_orchestrator.h"
// --- concurrency ---
#include "concurrency/bounded_channel.h"
// --- data_generation ---
#include "data_generation/data_generator.h"
#include "data_generation/json_grammars.h"
#ifndef BIERGARTEN_MOCK_ONLY
#include "data_generation/llama_generator.h"
#include "data_generation/llama_generator_helpers.h"
#endif
#include "data_generation/mock_generator.h"
#include "data_generation/prompt_formatting/gemma4_jinja_prompt_formatter.h"
#include "data_generation/prompt_formatting/prompt_formatter.h"
// --- data_model ---
#include "data_model/generated_models.h"
#include "data_model/models.h"
// --- json_handling ---
#include "json_handling/pretty_print.h"
#include "services/curated_data/curated_json_data_service.h"
// --- llama backend ---
#ifndef BIERGARTEN_MOCK_ONLY
#include "llama_backend_state.h"
#endif
// --- services: curated_data ---
#include "services/curated_data/curated_data_service.h"
#include "services/curated_data/mock_curated_data_service.h"
// --- services: database ---
#include "services/database/export_service.h"
#include "services/database/sqlite_connection_helpers.h"
#include "services/database/sqlite_export_service.h"
#include "services/database/sqlite_export_service_helpers.h"
#include "services/database/sqlite_handle_types.h"
#include "services/database/sqlite_statement_helpers.h"
// --- services: datetime ---
#include "services/datetime/date_time_provider.h"
#include "services/datetime/timer.h"
// --- services: enrichment ---
#include "services/enrichment/enrichment_service.h"
#include "services/enrichment/mock_enrichment.h"
#include "services/enrichment/wikipedia_service.h"
// --- services: logging ---
#include "services/logging/log_dispatcher.h"
#include "services/logging/log_entry.h"
#include "services/logging/log_producer.h"
#include "services/logging/logger.h"
// --- services: prompting ---
#include "services/prompting/prompt_directory.h"
// --- web_client ---
#include "web_client/http_web_client.h"
#include "web_client/web_client.h"
#endif // BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_H_

View File

@@ -1,12 +1,9 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_DATA_GENERATOR_H_ #ifndef BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_ORCHESTRATOR_H_
#define BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_DATA_GENERATOR_H_ #define BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_ORCHESTRATOR_H_
/** /**
* @file biergarten_data_generator.h * @file biergarten_pipeline_orchestrator.h
* @brief Orchestration for end-to-end brewery data generation pipeline. * @brief Orchestration for end-to-end brewery and user data generation.
*
* Intent: Coordinates location loading, enrichment, and generation phases
* to produce a complete dataset. Coordinates dependencies via composition root.
*/ */
#include <memory> #include <memory>
@@ -15,25 +12,28 @@
#include "data_generation/data_generator.h" #include "data_generation/data_generator.h"
#include "data_model/generated_models.h" #include "data_model/generated_models.h"
#include "services/curated_data/curated_data_service.h"
#include "services/database/export_service.h" #include "services/database/export_service.h"
#include "services/enrichment/enrichment_service.h" #include "services/enrichment/enrichment_service.h"
#include "services/logging/logger.h" #include "services/logging/logger.h"
/** /**
* @brief Main data generator class for the Biergarten pipeline. * @brief Main orchestrator for the Biergarten data generation pipeline.
* *
* This class encapsulates the core logic for generating brewery data. * Handles location loading, city enrichment, and brewery/user generation.
* It handles location loading, city enrichment, and brewery generation.
*/ */
class BiergartenPipelineOrchestrator { class BiergartenPipelineOrchestrator {
public: public:
/** /**
* @brief Constructs the orchestrator with injected pipeline dependencies. * @brief Constructs the orchestrator with injected pipeline dependencies.
* *
* @param logger Sink for pipeline diagnostics.
* @param context_service Provides regional context for locations. * @param context_service Provides regional context for locations.
* @param generator Implementation (Llama or Mock) for brewery/user generation. * @param generator Implementation (Llama or Mock) for brewery/user
* generation.
* @param exporter Database backend for persisting generated records. * @param exporter Database backend for persisting generated records.
* @param curated_data_service Loads curated location, persona, and name
* data used to seed generation.
* @param application_options CLI configuration and paths. * @param application_options CLI configuration and paths.
*/ */
BiergartenPipelineOrchestrator( BiergartenPipelineOrchestrator(
@@ -41,6 +41,7 @@ class BiergartenPipelineOrchestrator {
std::unique_ptr<IEnrichmentService> context_service, std::unique_ptr<IEnrichmentService> context_service,
std::unique_ptr<DataGenerator> generator, std::unique_ptr<DataGenerator> generator,
std::unique_ptr<IExportService> exporter, std::unique_ptr<IExportService> exporter,
std::unique_ptr<ICuratedDataService> curated_data_service,
const ApplicationOptions& application_options); const ApplicationOptions& application_options);
/** /**
@@ -49,38 +50,42 @@ class BiergartenPipelineOrchestrator {
* Performs the following steps: * Performs the following steps:
* 1. Load curated locations from JSON * 1. Load curated locations from JSON
* 2. Resolve context for each city using the injected context service * 2. Resolve context for each city using the injected context service
* 3. Generate brewery data for sampled cities * 3. Generate brewery and user data for sampled cities
* *
* @note STRUCTURAL CONCURRENCY REQUIREMENT: * @note STRUCTURAL CONCURRENCY REQUIREMENT:
* When transitioned to a multithreaded design, this method MUST structurally * When transitioned to a multithreaded design, this method MUST
* enforce that all deployed worker threads are joined before returning (e.g. * structurally enforce that all deployed worker threads are joined before
* by using std::jthread or a structured concurrency primitive). This ensures * returning (e.g. by using std::jthread or a structured concurrency
* workers do not attempt to log to a closed channel during application teardown. * primitive). This ensures workers do not attempt to log to a closed
* channel during application teardown.
* *
* @return true if successful, false if not * @return true if successful, false if not
*/ */
bool Run(); bool Run();
private: private:
/// @brief Logger instance for emitting pipeline messages.
std::shared_ptr<ILogger> logger_; std::shared_ptr<ILogger> logger_;
/// @brief Owning context provider dependency.
std::unique_ptr<IEnrichmentService> context_service_; std::unique_ptr<IEnrichmentService> context_service_;
/// @brief Generator dependency selected in the composition root. /**
* @brief Generator implementation selected at the composition root (Llama
* or Mock).
*/
std::unique_ptr<DataGenerator> generator_; std::unique_ptr<DataGenerator> generator_;
/// @brief Storage backend for generated brewery records. /**
* @brief Storage backend for generated brewery and user records.
*/
std::unique_ptr<IExportService> exporter_; std::unique_ptr<IExportService> exporter_;
std::unique_ptr<ICuratedDataService> curated_data_service_;
/// @brief CLI configuration: paths, model settings, generation parameters.
ApplicationOptions application_options_; ApplicationOptions application_options_;
/** /**
* @brief Load locations from JSON and sample cities. * @brief Load locations from JSON and sample cities.
* *
* @return Vector of sampled locations capped at 50 entries. * @return Vector of locations randomly sampled per
* PipelineOptions::location_count.
*/ */
std::vector<Location> QueryCitiesWithCountries(); std::vector<Location> QueryCitiesWithCountries();
@@ -95,23 +100,17 @@ class BiergartenPipelineOrchestrator {
* @brief Generate users grounded in sampled names and personas for * @brief Generate users grounded in sampled names and personas for
* enriched cities. * enriched cities.
* *
* Loads personas.json / forenames-by-country.json / surnames-by-country.json
* itself, mirroring how QueryCitiesWithCountries() owns its own
* locations.json load.
*
* @param cities Span of enriched city data. * @param cities Span of enriched city data.
*/ */
void GenerateUsers(std::span<const EnrichedCity> cities); void GenerateUsers(std::span<const EnrichedCity> cities);
/** /**
* @brief Log the generated brewery results. * @brief Log the generated brewery and user results.
*/ */
void LogResults() const; void LogResults() const;
/// @brief Stores generated brewery data. std::vector<BreweryRecord> generated_breweries_;
std::vector<GeneratedBrewery> generated_breweries_; std::vector<UserRecord> generated_users_;
/// @brief Stores generated user data.
std::vector<GeneratedUser> generated_users_;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_DATA_GENERATOR_H_
#endif // BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_ORCHESTRATOR_H_

View File

@@ -9,10 +9,12 @@
/** /**
* @file bounded_channel.h * @file bounded_channel.h
* @brief Thread-safe, bounded multi-producer/multi-consumer synchronous channel. * @brief Thread-safe, bounded multi-producer/multi-consumer synchronous
* channel.
* *
* Intent: Enables asynchronous inter-thread communication with backpressure. * Intent: Enables asynchronous inter-thread communication with backpressure.
* Models a synchronous channel where producers/consumers block on capacity limits. * Models a synchronous channel where producers/consumers block on capacity
* limits.
*/ */
/** /**

View File

@@ -1,4 +1,4 @@
#include "bounded_channel.h" #include "concurrency/bounded_channel.h"
template <typename T> template <typename T>
void BoundedChannel<T>::Send(T item) { void BoundedChannel<T>::Send(T item) {

View File

@@ -32,7 +32,7 @@ class DataGenerator {
* *
* @param city Enriched city the user is associated with. * @param city Enriched city the user is associated with.
* @param persona Persona archetype grounding the generated bio. * @param persona Persona archetype grounding the generated bio.
* @param name Sampled first/last name (and gender) -- not LLM-invented. * @param name Sampled first/last name (and gender)
* @return User generation result. * @return User generation result.
*/ */
virtual UserResult GenerateUser(const EnrichedCity& city, virtual UserResult GenerateUser(const EnrichedCity& city,

View File

@@ -0,0 +1,52 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_JSON_GRAMMARS_H_
#define BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_JSON_GRAMMARS_H_
/**
* @file data_generation/json_grammars.h
* @brief GBNF grammars constraining structured JSON output from
* LlamaGenerator inference calls.
*/
#include <string_view>
// GBNF grammar for structured user JSON output.
// thought-block permits the model to emit free-form reasoning before the
// JSON object (the prompts explicitly invite this); only the "{...}" tail is
// constrained to the expected shape.
inline constexpr std::string_view kUserJsonGrammar = R"json_user(
root ::= thought-block (
"{" ws
"\"username\"" ws ":" ws string ws "," ws
"\"bio\"" ws ":" ws string ws "," ws
"\"activity_weight\"" ws ":" ws number ws
"}" ws
)
thought-block ::= [^{]*
ws ::= [ \t\n\r]*
string ::= "\"" char+ "\""
char ::= [^"\\\x7F\x00-\x1F] | [\\] escape
escape ::= ["\\/bfnrt] | "u" hex hex hex hex
hex ::= [0-9a-fA-F]
number ::= "-"? ("0" | [1-9] [0-9]*) ("." [0-9]+)?
)json_user";
// GBNF grammar for structured brewery JSON output (see thought-block note
// above).
inline constexpr std::string_view kBreweryJsonGrammar = R"json_brewery(
root ::= thought-block (
"{" ws
"\"name_en\"" ws ":" ws string ws "," ws
"\"description_en\"" ws ":" ws string ws "," ws
"\"name_local\"" ws ":" ws string ws "," ws
"\"description_local\"" ws ":" ws string ws
"}" ws
)
thought-block ::= [^{]*
ws ::= [ \t\n\r]*
string ::= "\"" char+ "\""
char ::= [^"\\\x7F\x00-\x1F] | [\\] escape
escape ::= ["\\/bfnrt] | "u" hex hex hex hex
hex ::= [0-9a-fA-F]
)json_brewery";
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_JSON_GRAMMARS_H_

View File

@@ -1,24 +1,23 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_LLAMA_GENERATOR_H_ #ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_LLAMA_GENERATOR_H_
#define BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_LLAMA_GENERATOR_H_ #define BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_LLAMA_GENERATOR_H_
#include <filesystem>
/** /**
* @file data_generation/llama_generator.h * @file data_generation/llama_generator.h
* @brief llama.cpp-backed implementation of DataGenerator. * @brief llama.cpp-backed implementation of DataGenerator.
*/ */
#include <cstdint> #include <cstdint>
#include <filesystem>
#include <memory> #include <memory>
#include <random> #include <random>
#include <string> #include <string>
#include <string_view> #include <string_view>
#include "../services/prompting/prompt_directory.h"
#include "data_generation/data_generator.h" #include "data_generation/data_generator.h"
#include "data_generation/prompt_formatting/prompt_formatter.h" #include "data_generation/prompt_formatting/prompt_formatter.h"
#include "data_model/models.h" #include "data_model/models.h"
#include "services/logging/logger.h" #include "services/logging/logger.h"
#include "services/prompting/prompt_directory.h"
struct llama_model; struct llama_model;
struct llama_context; struct llama_context;
@@ -44,14 +43,9 @@ class LlamaGenerator final : public DataGenerator {
~LlamaGenerator() override; ~LlamaGenerator() override;
// disable copy constructor
LlamaGenerator(const LlamaGenerator&) = delete; LlamaGenerator(const LlamaGenerator&) = delete;
// disable copy assignment operator
LlamaGenerator& operator=(const LlamaGenerator&) = delete; LlamaGenerator& operator=(const LlamaGenerator&) = delete;
// disable move constructor
LlamaGenerator(LlamaGenerator&&) = delete; LlamaGenerator(LlamaGenerator&&) = delete;
// disable move assignment operator
LlamaGenerator& operator=(LlamaGenerator&&) = delete; LlamaGenerator& operator=(LlamaGenerator&&) = delete;
/** /**

View File

@@ -57,8 +57,7 @@ class MockGenerator final : public DataGenerator {
* @return Deterministic hash value. * @return Deterministic hash value.
*/ */
static size_t DeterministicHash(const Location& location, static size_t DeterministicHash(const Location& location,
const UserPersona& persona, const UserPersona& persona, const Name& name);
const Name& name);
// Hash stride constants for deterministic distribution across fixed-size // Hash stride constants for deterministic distribution across fixed-size
// arrays. These coprime strides spread hash values uniformly without // arrays. These coprime strides spread hash values uniformly without
@@ -142,7 +141,8 @@ class MockGenerator final : public DataGenerator {
"Visits breweries for the stories, stays for the flagship pours.", "Visits breweries for the stories, stays for the flagship pours.",
"Craft beer fan mapping tasting notes and favorite brew routes.", "Craft beer fan mapping tasting notes and favorite brew routes.",
"Always ready to trade recommendations for underrated local breweries.", "Always ready to trade recommendations for underrated local breweries.",
"Keeping a running list of must-try collab releases and tap takeovers."}; "Keeping a running list of must-try collab releases and tap "
"takeovers."};
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_MOCK_GENERATOR_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_MOCK_GENERATOR_H_

View File

@@ -3,8 +3,8 @@
/** /**
* @file data_model/generated_models.h * @file data_model/generated_models.h
* @brief Generated output models from the pipeline: brewery/user results, enriched data, * @brief Generated output models from the pipeline: brewery/user results,
* and complete generation results. * enriched data, and complete generation results.
*/ */
#include <string> #include <string>
@@ -19,16 +19,24 @@
* @brief Generated brewery payload. * @brief Generated brewery payload.
*/ */
struct BreweryResult { struct BreweryResult {
/// @brief Brewery display name in English. /**
* @brief Brewery display name in English.
*/
std::string name_en; std::string name_en;
/// @brief Brewery description text in English. /**
* @brief Brewery description text in English.
*/
std::string description_en; std::string description_en;
/// @brief Brewery display name in the local language. /**
* @brief Brewery display name in the local language.
*/
std::string name_local; std::string name_local;
/// @brief Brewery description text in the local language. /**
* @brief Brewery description text in the local language.
*/
std::string description_local; std::string description_local;
}; };
@@ -36,26 +44,38 @@ struct BreweryResult {
* @brief Generated user profile payload. * @brief Generated user profile payload.
*/ */
struct UserResult { struct UserResult {
/// @brief First (given) name, copied from the sampled Name -- not /**
/// LLM-invented. * @brief First (given) name, copied from the sampled Name -- not
* LLM-invented.
*/
std::string first_name{}; std::string first_name{};
/// @brief Last (family) name, copied from the sampled Name -- not /**
/// LLM-invented. * @brief Last (family) name, copied from the sampled Name -- not
* LLM-invented.
*/
std::string last_name{}; std::string last_name{};
/// @brief Gender associated with the sampled first name, copied from the /**
/// sampled Name -- not LLM-invented. * @brief Gender associated with the sampled first name, copied from the
* sampled Name -- not LLM-invented.
*/
std::string gender{}; std::string gender{};
/// @brief Username handle. /**
* @brief Username handle.
*/
std::string username{}; std::string username{};
/// @brief Short user biography. /**
* @brief Short user biography.
*/
std::string bio{}; std::string bio{};
/// @brief Relative check-in/activity weight for this user, used to drive /**
/// a J-curve activity profile in later pipeline phases. * @brief Relative check-in/activity weight for this user, used to drive
* a J-curve activity profile in later pipeline phases.
*/
float activity_weight{}; float activity_weight{};
}; };
@@ -74,7 +94,7 @@ struct EnrichedCity {
/** /**
* @brief Helper struct to store generated brewery data. * @brief Helper struct to store generated brewery data.
*/ */
struct GeneratedBrewery { struct BreweryRecord {
Location location; Location location;
BreweryResult brewery; BreweryResult brewery;
}; };
@@ -82,16 +102,15 @@ struct GeneratedBrewery {
/** /**
* @brief Helper struct to store generated user data. * @brief Helper struct to store generated user data.
* *
* `email`, `date_of_birth`, and `password` are programmatically generated by * `email` and `date_of_birth` are programmatically generated by the
* the orchestrator (never LLM-authored) so a downstream auth-account seeding * orchestrator (never LLM-authored) so a downstream auth-account seeding
* consumer can register real accounts from the pipeline's SQLite export. * consumer can register real accounts from the pipeline's SQLite export.
*/ */
struct GeneratedUser { struct UserRecord {
Location location; Location location;
UserResult user; UserResult user;
std::string email{}; std::string email{};
std::string date_of_birth{}; std::string date_of_birth{};
std::string password{};
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_GENERATED_MODELS_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_GENERATED_MODELS_H_

View File

@@ -10,6 +10,7 @@
#include <boost/program_options.hpp> #include <boost/program_options.hpp>
#include <cstdint> #include <cstdint>
#include <filesystem> #include <filesystem>
#include <functional>
#include <memory> #include <memory>
#include <optional> #include <optional>
#include <string> #include <string>
@@ -28,40 +29,35 @@ namespace prog_opts = boost::program_options;
* @brief Canonical location record for city-level generation. * @brief Canonical location record for city-level generation.
*/ */
struct Location { struct Location {
/// @brief City name.
std::string city{}; std::string city{};
/// @brief State or province name.
std::string state_province{}; std::string state_province{};
/// @brief ISO 3166-2 subdivision code. /**
* @brief ISO 3166-2 subdivision code.
*/
std::string iso3166_2{}; std::string iso3166_2{};
/// @brief Country name.
std::string country{}; std::string country{};
/// @brief ISO 3166-1 country code. /**
* @brief ISO 3166-1 country code.
*/
std::string iso3166_1{}; std::string iso3166_1{};
/// @brief Local language codes in priority order. /**
* @brief Local language codes in priority order.
*/
std::vector<std::string> local_languages{}; std::vector<std::string> local_languages{};
/// @brief Latitude in decimal degrees. /**
* @brief Latitude in decimal degrees.
*/
double latitude{}; double latitude{};
/// @brief Longitude in decimal degrees. /**
double longitude{}; * @brief Longitude in decimal degrees.
};
/**
* @brief Non-owning brewery location input.
*/ */
struct BreweryLocation { double longitude{};
/// @brief City name.
std::string_view city_name;
/// @brief Country name.
std::string_view country_name;
}; };
// ============================================================================ // ============================================================================
@@ -71,17 +67,16 @@ struct BreweryLocation {
/** /**
* @brief A sampled first/last name pair, with the source forename's gender. * @brief A sampled first/last name pair, with the source forename's gender.
* *
* Produced by NamesByCountry::SampleName(); * Produced by the SampleName() helper in generate_users.cc.
*/ */
struct Name { struct Name {
/// @brief First (given) name.
std::string first_name{}; std::string first_name{};
/// @brief Last (family) name.
std::string last_name{}; std::string last_name{};
/// @brief Gender associated with the sampled forename (e.g. "M", "F"), as /**
/// reported by the source dataset. * @brief Gender associated with the sampled forename (e.g. "M", "F"), as
* reported by the source dataset.
*/
std::string gender{}; std::string gender{};
}; };
@@ -89,25 +84,50 @@ struct Name {
* @brief A single forename entry from the names-by-country fixture data. * @brief A single forename entry from the names-by-country fixture data.
*/ */
struct ForenameEntry { struct ForenameEntry {
/// @brief Romanized forename. /**
* @brief Romanized forename.
*/
std::string name{}; std::string name{};
/// @brief Gender associated with this forename, as reported by the source /**
/// dataset (e.g. "M", "F"). * @brief Gender associated with this forename, as reported by the source
* dataset (e.g. "M", "F").
*/
std::string gender{}; std::string gender{};
bool operator==(const ForenameEntry& other) const {
return name == other.name && gender == other.gender;
}
}; };
namespace std {
template <>
struct hash<ForenameEntry> {
size_t operator()(const ForenameEntry& entry) const noexcept {
const size_t name_hash = std::hash<std::string>{}(entry.name);
const size_t gender_hash = std::hash<std::string>{}(entry.gender);
return name_hash ^ (gender_hash << 1);
}
};
} // namespace std
/** /**
* @brief A persona archetype used to ground LLM-generated user bios. * @brief A persona archetype used to ground LLM-generated user bios.
*/ */
struct UserPersona { struct UserPersona {
/// @brief Persona display name (e.g. "Hophead Explorer"). /**
* @brief Persona display name (e.g. "Hophead Explorer").
*/
std::string name{}; std::string name{};
/// @brief Short description of the persona's interests and voice. /**
* @brief Short description of the persona's interests and voice.
*/
std::string description{}; std::string description{};
/// @brief Beer styles this persona gravitates toward. /**
* @brief Beer styles this persona gravitates toward.
*/
std::vector<std::string> style_affinities{}; std::vector<std::string> style_affinities{};
}; };
@@ -119,22 +139,34 @@ struct UserPersona {
* @brief LLM sampling parameters. * @brief LLM sampling parameters.
*/ */
struct SamplingOptions { struct SamplingOptions {
/// @brief LLM sampling temperature (higher = more random). /**
* @brief LLM sampling temperature (higher = more random).
*/
float temperature = 1.0F; float temperature = 1.0F;
/// @brief LLM nucleus sampling top-p parameter. /**
* @brief LLM nucleus sampling top-p parameter.
*/
float top_p = 0.95F; float top_p = 0.95F;
/// @brief LLM top-k sampling parameter. /**
* @brief LLM top-k sampling parameter.
*/
uint32_t top_k = 64; uint32_t top_k = 64;
/// @brief Context window size (tokens). /**
* @brief Context window size (tokens).
*/
uint32_t n_ctx = 8192; uint32_t n_ctx = 8192;
/// @brief Random seed (-1 for random, otherwise non-negative). /**
* @brief Random seed (-1 for random, otherwise non-negative).
*/
int seed = -1; int seed = -1;
/// @brief Number of layers to offload to GPU. /**
* @brief Number of layers to offload to GPU.
*/
int n_gpu_layers = 0; int n_gpu_layers = 0;
}; };
@@ -142,16 +174,20 @@ struct SamplingOptions {
* @brief Configuration for the LLM generator component. * @brief Configuration for the LLM generator component.
*/ */
struct GeneratorOptions { struct GeneratorOptions {
/// @brief Path to the LLM model file (gguf format). /**
* @brief Path to the LLM model file (gguf format).
*/
std::filesystem::path model_path; std::filesystem::path model_path;
/// @brief Use mocked generator instead of actual LLM inference. /**
* @brief Use mocked generator instead of actual LLM inference.
*/
bool use_mocked = false; bool use_mocked = false;
/**
* @brief Specific sampling parameters for this generator.
/// @brief Specific sampling parameters for this generator. * If nullopt, the application should use global defaults.
/// If nullopt, the application should use global defaults. */
std::optional<SamplingOptions> sampling; std::optional<SamplingOptions> sampling;
}; };
@@ -159,18 +195,23 @@ struct GeneratorOptions {
* @brief Configuration for the pipeline execution and output. * @brief Configuration for the pipeline execution and output.
*/ */
struct PipelineOptions { struct PipelineOptions {
/// @brief Directory for generated artifacts. /**
* @brief Directory for generated artifacts.
*/
std::filesystem::path output_path; std::filesystem::path output_path;
/// @brief Directory that contains named prompt files (e.g. /**
/// BREWERY_GENERATION.md). * @brief Directory that contains named prompt files (e.g.
* BREWERY_GENERATION.md).
*/
std::filesystem::path prompt_dir; std::filesystem::path prompt_dir;
/// @brief Path for application logs.
std::filesystem::path log_path; std::filesystem::path log_path;
/// @brief Number of locations to sample from the dataset /**
/// More locations -> more users/more breweries * @brief Number of locations to sample from the dataset
* More locations -> more users/more breweries
*/
uint32_t location_count; uint32_t location_count;
}; };
@@ -186,7 +227,7 @@ struct ApplicationOptions {
// Function Declarations // Function Declarations
// ============================================================================ // ============================================================================
std::optional<ApplicationOptions> ParseArguments(const int argc, char** argv, std::optional<ApplicationOptions> ParseArguments(
std::shared_ptr<ILogger> logger = nullptr); const int argc, char** argv, std::shared_ptr<ILogger> logger = nullptr);
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_MODELS_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_MODELS_H_

View File

@@ -1,52 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_NAMES_BY_COUNTRY_H_
#define BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_NAMES_BY_COUNTRY_H_
/**
* @file data_model/names_by_country.h
* @brief Sampling over the names-by-country fixture data.
*/
#include <optional>
#include <random>
#include <string>
#include <unordered_map>
#include <vector>
#include "data_model/models.h"
/**
* @brief Samples locale-appropriate names from curated forename/surname
* fixture data, keyed by ISO 3166-1 country code.
*
* Forenames and surnames are kept in separate maps (mirroring the source
* dataset's own shape) so per-forename gender is preserved; pairing happens
* at sample time rather than being precomputed, so no information from the
* source dataset is discarded up front.
*/
class NamesByCountry {
public:
NamesByCountry(
std::unordered_map<std::string, std::vector<ForenameEntry>>
forenames_by_country,
std::unordered_map<std::string, std::vector<std::string>>
surnames_by_country);
/**
* @brief Samples a first/last name pair for the given country.
*
* @param iso3166_1 ISO 3166-1 country code to sample for.
* @param rng Random source used for sampling.
* @return A sampled Name, or std::nullopt if the country has no forename
* or no surname entries.
*/
[[nodiscard]] std::optional<Name> SampleName(const std::string& iso3166_1,
std::mt19937& rng) const;
private:
std::unordered_map<std::string, std::vector<ForenameEntry>>
forenames_by_country_;
std::unordered_map<std::string, std::vector<std::string>>
surnames_by_country_;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_NAMES_BY_COUNTRY_H_

View File

@@ -1,43 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_
#define BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_
/**
* @file json_handling/json_loader.h
* @brief Loader API for curated location data.
*/
#include <filesystem>
#include <memory>
#include <vector>
#include "data_model/models.h"
#include "data_model/names_by_country.h"
#include "services/logging/logger.h"
/// @brief Loads curated world locations from a JSON file into memory.
class JsonLoader {
public:
/// @brief Parses a JSON array file and returns all location records.
static std::vector<Location> LoadLocations(
const std::filesystem::path& filepath,
std::shared_ptr<ILogger> logger = nullptr);
/// @brief Parses a JSON array file and returns all persona records.
static std::vector<UserPersona> LoadPersonas(
const std::filesystem::path& filepath,
std::shared_ptr<ILogger> logger = nullptr);
/**
* @brief Parses the names-by-country fixture pair into a sampling-capable
* NamesByCountry.
*
* @param forenames_filepath Path to forenames-by-country.json.
* @param surnames_filepath Path to surnames-by-country.json.
*/
static NamesByCountry LoadNamesByCountry(
const std::filesystem::path& forenames_filepath,
const std::filesystem::path& surnames_filepath,
std::shared_ptr<ILogger> logger = nullptr);
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_

View File

@@ -106,4 +106,4 @@ inline void PrettyPrint(std::ostream& outstream,
} }
} }
#endif #endif // BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_PRETTY_PRINT_H_

View File

@@ -16,16 +16,10 @@
*/ */
class LlamaBackendState { class LlamaBackendState {
public: public:
/// @brief Initializes global llama backend state.
LlamaBackendState() { llama_backend_init(); } LlamaBackendState() { llama_backend_init(); }
/// @brief Cleans up global llama backend state.
~LlamaBackendState() { llama_backend_free(); } ~LlamaBackendState() { llama_backend_free(); }
/// @brief Non-copyable type.
LlamaBackendState(const LlamaBackendState&) = delete; LlamaBackendState(const LlamaBackendState&) = delete;
/// @brief Non-copyable type.
LlamaBackendState& operator=(const LlamaBackendState&) = delete; LlamaBackendState& operator=(const LlamaBackendState&) = delete;
}; };

View File

@@ -0,0 +1,50 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_CURATED_DATA_SERVICE_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_CURATED_DATA_SERVICE_H_
/**
* @file services/curated_data/curated_data_service.h
* @brief Abstraction for loading curated location, persona, and name data.
*/
#include <filesystem>
#include <unordered_map>
#include <unordered_set>
#include <vector>
#include "data_model/models.h"
using ForenameList = std::vector<ForenameEntry>;
using SurnameList = std::vector<std::string>;
using LocationsList = std::vector<Location>;
using PersonasList = std::vector<UserPersona>;
using ForenamesByCountryMap = std::unordered_map<std::string, ForenameList>;
using SurnamesByCountryMap = std::unordered_map<std::string, SurnameList>;
/**
* @brief Interface for services that load curated data used to seed
* brewery/user generation.
*/
class ICuratedDataService {
public:
ICuratedDataService() = default;
virtual ~ICuratedDataService() = default;
ICuratedDataService(const ICuratedDataService&) = delete;
ICuratedDataService& operator=(const ICuratedDataService&) = delete;
ICuratedDataService(ICuratedDataService&&) = delete;
ICuratedDataService& operator=(ICuratedDataService&&) = delete;
/**
* @brief Loads all curated location records.
*/
virtual const LocationsList& LoadLocations() = 0;
/**
* @brief Loads all curated persona records.
*/
virtual const PersonasList& LoadPersonas() = 0;
virtual const ForenamesByCountryMap& LoadForenamesByCountry() = 0;
virtual const SurnamesByCountryMap& LoadSurnamesByCountry() = 0;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_CURATED_DATA_SERVICE_H_

View File

@@ -0,0 +1,63 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_
#define BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_
/**
* @file json_handling/json_loader.h
* @brief JSON-backed implementation of ICuratedDataService.
*/
#include <filesystem>
#include "data_model/models.h"
#include "services/curated_data/curated_data_service.h"
/**
* @brief File locations for the curated JSON fixtures consumed by
* CuratedJsonDataService.
*/
struct CuratedDataFilePaths {
std::filesystem::path locations_path;
std::filesystem::path personas_path;
std::filesystem::path forenames_path;
std::filesystem::path surnames_path;
};
/**
* @brief Loads curated location, persona, and name data from JSON files.
*/
class CuratedJsonDataService final : public ICuratedDataService {
struct cache {
LocationsList locations;
PersonasList personas;
ForenamesByCountryMap forenames_by_country;
SurnamesByCountryMap surnames_by_country;
cache() = default;
~cache() = default;
};
CuratedDataFilePaths filepaths_;
cache cache_;
public:
explicit CuratedJsonDataService(CuratedDataFilePaths filepaths);
/**
* @brief Parses a JSON array file and returns all location records.
*/
const LocationsList& LoadLocations() override;
/**
* @brief Parses a JSON array file and returns all persona records.
*/
const PersonasList& LoadPersonas() override;
const ForenamesByCountryMap& LoadForenamesByCountry() override;
/**
* @brief Parses a JSON file and returns all the forenames per country.
*/
const SurnamesByCountryMap& LoadSurnamesByCountry() override;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_

View File

@@ -0,0 +1,38 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_MOCK_CURATED_DATA_SERVICE_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_MOCK_CURATED_DATA_SERVICE_H_
/**
* @file services/curated_data/mock_curated_data_service.h
* @brief In-memory ICuratedDataService backed by a small fixed dataset, used
* when file-backed curated data is disabled (mock mode).
*/
#include <filesystem>
#include "data_model/models.h"
#include "services/curated_data/curated_data_service.h"
/**
* @brief Curated data service returning a small fixed in-memory dataset in
* place of the JSON fixture files used by JsonLoader.
*/
class MockCuratedDataService final : public ICuratedDataService {
public:
MockCuratedDataService();
const LocationsList& LoadLocations() override;
const PersonasList& LoadPersonas() override;
const ForenamesByCountryMap& LoadForenamesByCountry() override;
const SurnamesByCountryMap& LoadSurnamesByCountry() override;
private:
LocationsList locations_;
PersonasList personas_;
ForenamesByCountryMap forenames_by_country_;
SurnamesByCountryMap surnames_by_country_;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_MOCK_CURATED_DATA_SERVICE_H_

View File

@@ -16,8 +16,6 @@
class IExportService { class IExportService {
public: public:
IExportService() = default; IExportService() = default;
/// @brief Virtual destructor for polymorphic cleanup.
virtual ~IExportService() = default; virtual ~IExportService() = default;
IExportService(const IExportService&) = delete; IExportService(const IExportService&) = delete;
@@ -25,7 +23,9 @@ class IExportService {
IExportService(IExportService&&) = delete; IExportService(IExportService&&) = delete;
IExportService& operator=(IExportService&&) = delete; IExportService& operator=(IExportService&&) = delete;
/// @brief Prepares the export destination for a new run. /**
* @brief Prepares the export destination for a new run.
*/
virtual void Initialize() = 0; virtual void Initialize() = 0;
/** /**
@@ -33,16 +33,18 @@ class IExportService {
* *
* @param brewery Generated brewery payload to store. * @param brewery Generated brewery payload to store.
*/ */
virtual uint64_t ProcessRecord(const GeneratedBrewery& brewery) = 0; virtual uint64_t ProcessRecord(const BreweryRecord& brewery) = 0;
/** /**
* @brief Persists one generated user record. * @brief Persists one generated user record.
* *
* @param user Generated user payload to store. * @param user Generated user payload to store.
*/ */
virtual uint64_t ProcessRecord(const GeneratedUser& user) = 0; virtual uint64_t ProcessRecord(const UserRecord& user) = 0;
/// @brief Finalizes the export destination. /**
* @brief Finalizes the export destination.
*/
virtual void Finalize() = 0; virtual void Finalize() = 0;
}; };

View File

@@ -12,7 +12,7 @@
#include <string> #include <string>
#include <string_view> #include <string_view>
#include "sqlite_handle_types.h" #include "services/database/sqlite_handle_types.h"
namespace sqlite_export_service_internal { namespace sqlite_export_service_internal {

View File

@@ -12,9 +12,9 @@
#include <unordered_map> #include <unordered_map>
#include "data_model/models.h" #include "data_model/models.h"
#include "../datetime/date_time_provider.h" #include "services/database/export_service.h"
#include "export_service.h" #include "services/database/sqlite_export_service_helpers.h"
#include "sqlite_export_service_helpers.h" #include "services/datetime/date_time_provider.h"
/** /**
* @brief Persists generated brewery records into a fresh SQLite database. * @brief Persists generated brewery records into a fresh SQLite database.
@@ -30,8 +30,8 @@ class SqliteExportService final : public IExportService {
SqliteExportService& operator=(SqliteExportService&&) = delete; SqliteExportService& operator=(SqliteExportService&&) = delete;
void Initialize() override; void Initialize() override;
uint64_t ProcessRecord(const GeneratedBrewery& brewery) override; uint64_t ProcessRecord(const BreweryRecord& brewery) override;
uint64_t ProcessRecord(const GeneratedUser& user) override; uint64_t ProcessRecord(const UserRecord& user) override;
void Finalize() override; void Finalize() override;
private: private:

View File

@@ -3,8 +3,8 @@
/* Umbrella header for backward compatibility. */ /* Umbrella header for backward compatibility. */
#include "sqlite_connection_helpers.h" #include "services/database/sqlite_connection_helpers.h"
#include "sqlite_handle_types.h" #include "services/database/sqlite_handle_types.h"
#include "sqlite_statement_helpers.h" #include "services/database/sqlite_statement_helpers.h"
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_SQLITE_EXPORT_SERVICE_HELPERS_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_SQLITE_EXPORT_SERVICE_HELPERS_H_

View File

@@ -24,8 +24,10 @@ using SqliteDatabaseHandle = std::unique_ptr<sqlite3, SqliteDatabaseDeleter>;
using SqliteStatementHandle = using SqliteStatementHandle =
std::unique_ptr<sqlite3_stmt, SqliteStatementDeleter>; std::unique_ptr<sqlite3_stmt, SqliteStatementDeleter>;
// Represents a parameter that is bound to a prepared SQLite statement.
// N.B. indices are 1 based.
template <typename T> template <typename T>
struct BindParam { struct BoundParam {
int index; int index;
T value; T value;
std::string_view action; std::string_view action;

View File

@@ -13,7 +13,7 @@
#include <string_view> #include <string_view>
#include <vector> #include <vector>
#include "sqlite_handle_types.h" #include "services/database/sqlite_handle_types.h"
namespace sqlite_export_service_internal { namespace sqlite_export_service_internal {
@@ -63,7 +63,6 @@ CREATE TABLE IF NOT EXISTS users (
activity_weight REAL NOT NULL, activity_weight REAL NOT NULL,
email TEXT NOT NULL UNIQUE, email TEXT NOT NULL UNIQUE,
date_of_birth TEXT NOT NULL, date_of_birth TEXT NOT NULL,
password TEXT NOT NULL,
FOREIGN KEY(location_id) REFERENCES locations(id) ON DELETE CASCADE FOREIGN KEY(location_id) REFERENCES locations(id) ON DELETE CASCADE
); );
@@ -104,36 +103,42 @@ INSERT INTO users (
bio, bio,
activity_weight, activity_weight,
email, email,
date_of_birth, date_of_birth
password ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?);
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?);
)sql"; )sql";
inline constexpr int kLocationCityBindIndex = 1; // sqlite3_bind_*() parameter indices are 1-based, matching the "?"
inline constexpr int kLocationStateProvinceBindIndex = 2; // placeholder order in the SQL above.
inline constexpr int kLocationIso31662BindIndex = 3; enum LocationBindIndex {
inline constexpr int kLocationCountryBindIndex = 4; kLocationCityBindIndex = 1,
inline constexpr int kLocationIso31661BindIndex = 5; kLocationStateProvinceBindIndex,
inline constexpr int kLocationLanguagesBindIndex = 6; kLocationIso31662BindIndex,
inline constexpr int kLocationLatitudeBindIndex = 7; kLocationCountryBindIndex,
inline constexpr int kLocationLongitudeBindIndex = 8; kLocationIso31661BindIndex,
kLocationLanguagesBindIndex,
kLocationLatitudeBindIndex,
kLocationLongitudeBindIndex,
};
inline constexpr int kBreweryLocationIdBindIndex = 1; enum BreweryBindIndex {
inline constexpr int kBreweryEnglishNameBindIndex = 2; kBreweryLocationIdBindIndex = 1,
inline constexpr int kBreweryEnglishDescriptionBindIndex = 3; kBreweryEnglishNameBindIndex,
inline constexpr int kBreweryLocalNameBindIndex = 4; kBreweryEnglishDescriptionBindIndex,
inline constexpr int kBreweryLocalDescriptionBindIndex = 5; kBreweryLocalNameBindIndex,
kBreweryLocalDescriptionBindIndex,
};
inline constexpr int kUserLocationIdBindIndex = 1; enum UserBindIndex {
inline constexpr int kUserFirstNameBindIndex = 2; kUserLocationIdBindIndex = 1,
inline constexpr int kUserLastNameBindIndex = 3; kUserFirstNameBindIndex,
inline constexpr int kUserGenderBindIndex = 4; kUserLastNameBindIndex,
inline constexpr int kUserUsernameBindIndex = 5; kUserGenderBindIndex,
inline constexpr int kUserBioBindIndex = 6; kUserUsernameBindIndex,
inline constexpr int kUserActivityWeightBindIndex = 7; kUserBioBindIndex,
inline constexpr int kUserEmailBindIndex = 8; kUserActivityWeightBindIndex,
inline constexpr int kUserDateOfBirthBindIndex = 9; kUserEmailBindIndex,
inline constexpr int kUserPasswordBindIndex = 10; kUserDateOfBirthBindIndex,
};
SqliteStatementHandle PrepareStatement(const SqliteDatabaseHandle& db_handle, SqliteStatementHandle PrepareStatement(const SqliteDatabaseHandle& db_handle,
std::string_view sql, std::string_view sql,
@@ -142,13 +147,13 @@ SqliteStatementHandle PrepareStatement(const SqliteDatabaseHandle& db_handle,
void ResetStatement(SqliteStatementHandle& statement); void ResetStatement(SqliteStatementHandle& statement);
void Bind(const SqliteStatementHandle& statement, void Bind(const SqliteStatementHandle& statement,
const BindParam<std::string_view>& param); const BoundParam<std::string_view>& param);
void Bind(const SqliteStatementHandle& statement, void Bind(const SqliteStatementHandle& statement,
const BindParam<double>& param); const BoundParam<double>& param);
void Bind(const SqliteStatementHandle& statement, void Bind(const SqliteStatementHandle& statement,
const BindParam<sqlite3_int64>& param); const BoundParam<sqlite3_int64>& param);
void StepStatement(const SqliteDatabaseHandle& db_handle, void StepStatement(const SqliteDatabaseHandle& db_handle,
const SqliteStatementHandle& statement, const SqliteStatementHandle& statement,

View File

@@ -18,7 +18,6 @@
*/ */
class IDateTimeProvider { class IDateTimeProvider {
public: public:
/// @brief Virtual destructor for polymorphic cleanup.
virtual ~IDateTimeProvider() = default; virtual ~IDateTimeProvider() = default;
IDateTimeProvider() = default; IDateTimeProvider() = default;

View File

@@ -15,7 +15,6 @@
*/ */
class IEnrichmentService { class IEnrichmentService {
public: public:
/// @brief Virtual destructor for polymorphic cleanup.
virtual ~IEnrichmentService() = default; virtual ~IEnrichmentService() = default;
/** /**

View File

@@ -1,13 +1,18 @@
//
// Created by aaronpo on 13/05/2026.
//
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_MOCK_ENRICHMENT_H_ #ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_MOCK_ENRICHMENT_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_MOCK_ENRICHMENT_H_ #define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_MOCK_ENRICHMENT_H_
/**
* @file services/enrichment/mock_enrichment.h
* @brief No-op IEnrichmentService used when network enrichment is disabled.
*/
#include <string> #include <string>
#include "enrichment_service.h" #include "services/enrichment/enrichment_service.h"
/**
* @brief Enrichment service that returns no context for any location.
*/
class MockEnrichmentService final : public IEnrichmentService { class MockEnrichmentService final : public IEnrichmentService {
public: public:
std::string GetLocationContext(const Location& /*loc*/) override { std::string GetLocationContext(const Location& /*loc*/) override {

View File

@@ -11,25 +11,37 @@
#include <string_view> #include <string_view>
#include <unordered_map> #include <unordered_map>
#include "enrichment_service.h" #include "services/enrichment/enrichment_service.h"
#include "services/logging/logger.h" #include "services/logging/logger.h"
#include "web_client/web_client.h" #include "web_client/web_client.h"
/// @brief Provides Wikipedia summary lookups backed by cached raw extracts. /**
* @brief Provides Wikipedia summary lookups backed by cached raw extracts.
*/
class WikipediaEnrichmentService final : public IEnrichmentService { class WikipediaEnrichmentService final : public IEnrichmentService {
public: public:
/// @brief Creates a new Wikipedia service with the provided web client. /**
* @brief Creates a new Wikipedia service with the provided web client.
*/
explicit WikipediaEnrichmentService(std::unique_ptr<WebClient> client, explicit WikipediaEnrichmentService(std::unique_ptr<WebClient> client,
std::shared_ptr<ILogger> logger); std::shared_ptr<ILogger> logger);
/// @brief Returns the Wikipedia-derived context for a location. /**
* @brief Returns the Wikipedia-derived context for a location.
*/
[[nodiscard]] std::string GetLocationContext(const Location& loc) override; [[nodiscard]] std::string GetLocationContext(const Location& loc) override;
private: private:
std::string FetchExtract(std::string_view query); std::string FetchExtract(std::string_view query);
std::unique_ptr<WebClient> client_; std::unique_ptr<WebClient> client_;
std::shared_ptr<ILogger> logger_; std::shared_ptr<ILogger> logger_;
/// @brief Canonical cache for raw Wikipedia query extracts. /**
* @brief Cache for raw Wikipedia query extracts, keyed by query string.
*
* GetLocationContext() always queries "brewing" and reuses "beer in
* {country}" for every city in that country, so caching avoids refetching
* the same extract across locations in a run.
*/
std::unordered_map<std::string, std::string> extract_cache_; std::unordered_map<std::string, std::string> extract_cache_;
}; };

View File

@@ -20,10 +20,22 @@
* @brief Severity levels supported by the logging infra. * @brief Severity levels supported by the logging infra.
*/ */
enum class LogLevel { enum class LogLevel {
Debug, ///< Development/debugging information. /**
Info, ///< General informational messages. * @brief Development/debugging information.
Warn, ///< Warning conditions. */
Error, ///< Error conditions. Debug,
/**
* @brief General informational messages.
*/
Info,
/**
* @brief Warning conditions.
*/
Warn,
/**
* @brief Error conditions.
*/
Error,
}; };
/** /**
@@ -34,19 +46,44 @@ enum class LogLevel {
* pipeline that emitted it. * pipeline that emitted it.
*/ */
enum class PipelinePhase { enum class PipelinePhase {
Startup, ///< Initialization and validation. /**
Enrichment, ///< Location/context enrichment (e.g. Wikipedia lookups). * @brief Initialization and validation.
UserGeneration, ///< User profile generation. */
BreweryAndBeerGeneration, ///< Brewery and beer data generation. Startup,
CheckinGeneration, ///< Checkin (visit) record generation. /**
RatingGeneration, ///< Rating and review generation. * @brief Location/context enrichment (e.g. Wikipedia lookups).
FollowGeneration, ///< Follow relationship generation. */
Teardown, ///< Finalization and cleanup. Enrichment,
/**
* @brief User profile generation.
*/
UserGeneration,
/**
* @brief Brewery and beer data generation.
*/
BreweryAndBeerGeneration,
/**
* @brief Checkin (visit) record generation.
*/
CheckinGeneration,
/**
* @brief Rating and review generation.
*/
RatingGeneration,
/**
* @brief Follow relationship generation.
*/
FollowGeneration,
/**
* @brief Finalization and cleanup.
*/
Teardown,
}; };
/** /**
* @struct LogDTO * @struct LogDTO
* @brief User-provided subset of log fields. Used to capture call-site info transparently. * @brief User-provided subset of log fields. Used to capture call-site info
* transparently.
*/ */
struct LogDTO { struct LogDTO {
LogLevel level; LogLevel level;
@@ -65,25 +102,35 @@ struct LogDTO {
* before the entry is dispatched. * before the entry is dispatched.
*/ */
struct LogEntry { struct LogEntry {
/// @brief Timestamp when the entry was created. /**
* @brief Timestamp when the entry was created.
*/
std::chrono::system_clock::time_point timestamp{}; std::chrono::system_clock::time_point timestamp{};
/// @brief Source location where the log call was made. /**
* @brief Source location where the log call was made.
*/
std::source_location origin{}; std::source_location origin{};
/// @brief Thread responsible for emitting the log. /**
* @brief Thread responsible for emitting the log.
*/
std::thread::id thread_id{}; std::thread::id thread_id{};
/**
/// @brief Severity level of this entry. * @brief Severity level of this entry.
*/
LogLevel level; LogLevel level;
/// @brief Pipeline phase associated with the entry. /**
* @brief Pipeline phase associated with the entry.
*/
PipelinePhase phase; PipelinePhase phase;
/// @brief Log message text. /**
* @brief Log message text.
*/
std::string message; std::string message;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_ENTRY_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_ENTRY_H_

View File

@@ -6,8 +6,8 @@
* a bounded channel for later processing by the dispatcher. * a bounded channel for later processing by the dispatcher.
*/ */
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_CHANNEL_LOGGER_H_ #ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_PRODUCER_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_CHANNEL_LOGGER_H_ #define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_PRODUCER_H_
#include <string_view> #include <string_view>
@@ -50,4 +50,4 @@ class LogProducer final : public ILogger {
BoundedChannel<LogEntry>& channel_; BoundedChannel<LogEntry>& channel_;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_CHANNEL_LOGGER_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_PRODUCER_H_

View File

@@ -39,7 +39,8 @@ class ILogger {
*/ */
void Log(LogDTO payload, void Log(LogDTO payload,
std::source_location origin = std::source_location::current(), std::source_location origin = std::source_location::current(),
std::chrono::system_clock::time_point timestamp = std::chrono::system_clock::now(), std::chrono::system_clock::time_point timestamp =
std::chrono::system_clock::now(),
std::thread::id thread_id = std::this_thread::get_id()) { std::thread::id thread_id = std::this_thread::get_id()) {
LogEntry entry; LogEntry entry;
entry.timestamp = timestamp; entry.timestamp = timestamp;

View File

@@ -1,19 +1,18 @@
/** /**
* @file web_client/http_web_client.h * @file web_client/http_web_client.h
* @brief cpp-httplib implementation of the WebClient interface. * @brief cpp-httplib implementation of the WebClient interface.
*/ */
#ifndef BIERGARTEN_PIPELINE_INCLUDES_WEB_CLIENT_HTTP_WEB_CLIENT_H_ #ifndef BIERGARTEN_PIPELINE_INCLUDES_WEB_CLIENT_HTTP_WEB_CLIENT_H_
#define BIERGARTEN_PIPELINE_INCLUDES_WEB_CLIENT_HTTP_WEB_CLIENT_H_ #define BIERGARTEN_PIPELINE_INCLUDES_WEB_CLIENT_HTTP_WEB_CLIENT_H_
#include "web_client/web_client.h"
#include "services/logging/logger.h"
#include <memory> #include <memory>
#include <string> #include <string>
#include <utility> #include <utility>
#include "services/logging/logger.h"
#include "web_client/web_client.h"
/** /**
* @brief WebClient implementation backed by cpp-httplib. * @brief WebClient implementation backed by cpp-httplib.
* *
@@ -26,7 +25,7 @@
* bound to a single origin at construction time. * bound to a single origin at construction time.
*/ */
class HttpWebClient final : public WebClient { class HttpWebClient final : public WebClient {
public: public:
explicit HttpWebClient(std::shared_ptr<ILogger> logger) explicit HttpWebClient(std::shared_ptr<ILogger> logger)
: logger_(std::move(logger)) {} : logger_(std::move(logger)) {}
~HttpWebClient() override = default; ~HttpWebClient() override = default;
@@ -34,7 +33,8 @@ public:
/** /**
* @brief Executes a blocking HTTP/HTTPS GET request against a full URL. * @brief Executes a blocking HTTP/HTTPS GET request against a full URL.
* *
* @param url Fully-qualified URL, e.g. "https://en.wikipedia.org/api/rest_v1/page/summary/Berlin" * @param url Fully-qualified URL, e.g.
* "https://en.wikipedia.org/api/rest_v1/page/summary/Berlin"
* @return Response body on HTTP 2xx; throws std::runtime_error otherwise. * @return Response body on HTTP 2xx; throws std::runtime_error otherwise.
*/ */
std::string Get(const std::string& url) override; std::string Get(const std::string& url) override;
@@ -52,5 +52,4 @@ public:
std::shared_ptr<ILogger> logger_; std::shared_ptr<ILogger> logger_;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_WEB_CLIENT_HTTP_WEB_CLIENT_H_
#endif

View File

@@ -13,7 +13,6 @@
*/ */
class WebClient { class WebClient {
public: public:
/// @brief Virtual destructor for polymorphic cleanup.
virtual ~WebClient() = default; virtual ~WebClient() = default;
/** /**

8
tooling/pipeline/run.sh Normal file
View File

@@ -0,0 +1,8 @@
./biergarten-pipeline \
--model ../models/google_gemma-4-E4B-it-Q6_K.gguf \
--temperature 1.0 \
--top-p 0.95 \
--top-k 64 \
--n-ctx 8192 \
--location-count 15 \
--prompt-dir ../prompts

View File

@@ -51,7 +51,8 @@ std::optional<ApplicationOptions> ParseArguments(
prog_opts::value<std::string>()->default_value("pipeline.log"), prog_opts::value<std::string>()->default_value("pipeline.log"),
"Path for application logs"); "Path for application logs");
opt("prompt-dir", prog_opts::value<std::string>()->default_value(""), opt("prompt-dir", prog_opts::value<std::string>()->default_value(""),
"Directory containing named prompt files (e.g. BREWERY_GENERATION.md)." "Directory containing named prompt files (e.g. "
"BREWERY_GENERATION.md)."
" Required when not using --mocked."); " Required when not using --mocked.");
opt("location-count", prog_opts::value<uint32_t>()->default_value(10)); opt("location-count", prog_opts::value<uint32_t>()->default_value(10));
}; };
@@ -106,10 +107,12 @@ std::optional<ApplicationOptions> ParseArguments(
const std::string model_path = var_map["model"].as<std::string>(); const std::string model_path = var_map["model"].as<std::string>();
const int n_gpu_layers = var_map["n-gpu-layers"].as<int>(); const int n_gpu_layers = var_map["n-gpu-layers"].as<int>();
// Enforce mutual exclusivity before any further configuration is applied. // Enforce mutual exclusivity before any further configuration is
// applied.
if (use_mocked && !model_path.empty()) { if (use_mocked && !model_path.empty()) {
const std::string msg = const std::string msg =
"Invalid arguments: --mocked and --model are mutually exclusive"; "Invalid arguments: --mocked and --model are mutually "
"exclusive";
if (logger) { if (logger) {
logger->Log(LogDTO{.level = LogLevel::Error, logger->Log(LogDTO{.level = LogLevel::Error,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
@@ -122,7 +125,8 @@ std::optional<ApplicationOptions> ParseArguments(
if (!use_mocked && model_path.empty()) { if (!use_mocked && model_path.empty()) {
const std::string msg = const std::string msg =
"Invalid arguments: either --mocked or --model must be specified"; "Invalid arguments: either --mocked or --model must be "
"specified";
if (logger) { if (logger) {
logger->Log(LogDTO{.level = LogLevel::Error, logger->Log(LogDTO{.level = LogLevel::Error,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
@@ -137,7 +141,8 @@ std::optional<ApplicationOptions> ParseArguments(
// generator has no use for it and should not require it to be present. // generator has no use for it and should not require it to be present.
if (!use_mocked && options.pipeline.prompt_dir.empty()) { if (!use_mocked && options.pipeline.prompt_dir.empty()) {
const std::string msg = const std::string msg =
"Invalid arguments: --prompt-dir is required when not using --mocked"; "Invalid arguments: --prompt-dir is required when not using "
"--mocked";
if (logger) { if (logger) {
logger->Log({.level = LogLevel::Error, logger->Log({.level = LogLevel::Error,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,

View File

@@ -1,6 +1,6 @@
/** /**
* @file biergarten_pipeline_orchestrator/biergarten_pipeline_orchestrator.cc * @file biergarten_pipeline_orchestrator/biergarten_pipeline_orchestrator.cc
* @brief BiergartenDataGenerator constructor implementation. * @brief BiergartenPipelineOrchestrator constructor implementation.
*/ */
#include "biergarten_pipeline_orchestrator.h" #include "biergarten_pipeline_orchestrator.h"
@@ -12,9 +12,11 @@ BiergartenPipelineOrchestrator::BiergartenPipelineOrchestrator(
std::unique_ptr<IEnrichmentService> context_service, std::unique_ptr<IEnrichmentService> context_service,
std::unique_ptr<DataGenerator> generator, std::unique_ptr<DataGenerator> generator,
std::unique_ptr<IExportService> exporter, std::unique_ptr<IExportService> exporter,
const ApplicationOptions &app_options) std::unique_ptr<ICuratedDataService> curated_data_service,
const ApplicationOptions& application_options)
: logger_(std::move(logger)), : logger_(std::move(logger)),
context_service_(std::move(context_service)), context_service_(std::move(context_service)),
generator_(std::move(generator)), generator_(std::move(generator)),
exporter_(std::move(exporter)), exporter_(std::move(exporter)),
application_options_(app_options) {} curated_data_service_(std::move(curated_data_service)),
application_options_(application_options) {}

View File

@@ -1,10 +1,11 @@
/** /**
* @file biergarten_pipeline_orchestrator/generate_breweries.cc * @file biergarten_pipeline_orchestrator/generate_breweries.cc
* @brief BiergartenDataGenerator::GenerateBreweries() implementation. * @brief BiergartenPipelineOrchestrator::GenerateBreweries() implementation.
*/ */
#include <chrono> #include <chrono>
#include <format> #include <format>
#include <optional>
#include "biergarten_pipeline_orchestrator.h" #include "biergarten_pipeline_orchestrator.h"
#include "services/logging/logger.h" #include "services/logging/logger.h"
@@ -19,39 +20,57 @@ void BiergartenPipelineOrchestrator::GenerateBreweries(
size_t skipped_count = 0; size_t skipped_count = 0;
size_t export_failed_count = 0; size_t export_failed_count = 0;
for (const auto& [location, region_context] : cities) { const auto generate_record =
[this, &skipped_count](
const Location& location,
const std::string& region_context) -> std::optional<BreweryRecord> {
try { try {
const BreweryResult brewery = const BreweryResult brewery =
generator_->GenerateBrewery(location, region_context); generator_->GenerateBrewery(location, region_context);
return BreweryRecord{.location = location, .brewery = brewery};
const GeneratedBrewery gen{.location = location, .brewery = brewery}; } catch (const std::exception& e) {
++skipped_count;
generated_breweries_.push_back(gen);
try {
exporter_->ProcessRecord(gen);
} catch (const std::exception& export_exception) {
++export_failed_count;
logger_->Log( logger_->Log(
{.level = LogLevel::Warn, {.level = LogLevel::Warn,
.phase = PipelinePhase::BreweryAndBeerGeneration, .phase = PipelinePhase::BreweryAndBeerGeneration,
.message = .message = std::format("[Pipeline] Skipping city '{}' ({}): brewery "
std::format("[Pipeline] Generated brewery for '{}' ({}) but SQLite export failed: {}", "generation failed: {}",
location.city, location.country, export_exception.what())});
}
} catch (const std::exception& e) {
++skipped_count;
logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::BreweryAndBeerGeneration,
.message = std::format("[Pipeline] Skipping city '{}' ({}): brewery generation failed: {}",
location.city, location.country, e.what())}); location.city, location.country, e.what())});
return std::nullopt;
} }
};
const auto export_record = [this, &export_failed_count](
const BreweryRecord& record) {
try {
exporter_->ProcessRecord(record);
} catch (const std::exception& export_exception) {
++export_failed_count;
logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::BreweryAndBeerGeneration,
.message = std::format("[Pipeline] Generated brewery for '{}' ({}) "
"but SQLite export failed: {}",
record.location.city, record.location.country,
export_exception.what())});
}
};
for (const auto& [location, region_context] : cities) {
const std::optional<BreweryRecord> record =
generate_record(location, region_context);
if (!record.has_value()) {
continue;
}
generated_breweries_.push_back(*record);
export_record(*record);
} }
if (skipped_count > 0) { if (skipped_count > 0) {
logger_->Log({.level = LogLevel::Warn, logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::BreweryAndBeerGeneration, .phase = PipelinePhase::BreweryAndBeerGeneration,
.message = std::format( .message = std::format(
"[Pipeline] Skipped {} city/cities due to generation errors", "[Pipeline] Skipped {} city/cities due to generation errors",
@@ -59,10 +78,11 @@ void BiergartenPipelineOrchestrator::GenerateBreweries(
} }
if (export_failed_count > 0) { if (export_failed_count > 0) {
logger_->Log({.level = LogLevel::Warn, logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::Teardown, .phase = PipelinePhase::Teardown,
.message = std::format( .message = std::format("[Pipeline] Failed to export {} generated "
"[Pipeline] Failed to export {} generated brewery/breweries to SQLite", "brewery/breweries to SQLite",
export_failed_count)}); export_failed_count)});
} }
} }

View File

@@ -1,11 +1,12 @@
/** /**
* @file biergarten_pipeline_orchestrator/generate_users.cc * @file biergarten_pipeline_orchestrator/generate_users.cc
* @brief BiergartenDataGenerator::GenerateUsers() implementation. * @brief BiergartenPipelineOrchestrator::GenerateUsers() implementation.
*/ */
#include <cctype> #include <cctype>
#include <chrono> #include <chrono>
#include <format> #include <format>
#include <iterator>
#include <optional> #include <optional>
#include <random> #include <random>
#include <stdexcept> #include <stdexcept>
@@ -14,18 +15,17 @@
#include <unordered_set> #include <unordered_set>
#include "biergarten_pipeline_orchestrator.h" #include "biergarten_pipeline_orchestrator.h"
#include "json_handling/json_loader.h" #include "services/curated_data/curated_json_data_service.h"
#include "services/logging/logger.h" #include "services/logging/logger.h"
namespace { namespace {
std::string Sanitize(std::string_view value) { std::string Sanitize(std::string_view value) {
std::string out; std::string out;
out.reserve(value.size()); out.reserve(value.size());
for (const char character : value) { for (const char character : value) {
if (std::isalnum(static_cast<unsigned char>(character)) != 0) { if (std::isalnum(static_cast<unsigned char>(character)) != 0) {
out.push_back( out.push_back(static_cast<char>(
static_cast<char>(std::tolower(static_cast<unsigned char>(character)))); std::tolower(static_cast<unsigned char>(character))));
} }
} }
return out; return out;
@@ -37,9 +37,13 @@ std::string BuildEmail(const Name& name,
std::format("{}.{}", Sanitize(name.first_name), Sanitize(name.last_name)); std::format("{}.{}", Sanitize(name.first_name), Sanitize(name.last_name));
std::string local_part = base; std::string local_part = base;
for (int suffix = 1; used_local_parts.contains(local_part); ++suffix) { uint32_t suffix = 1;
while (used_local_parts.contains(local_part)) {
local_part = std::format("{}{}", base, suffix); local_part = std::format("{}{}", base, suffix);
++suffix;
} }
used_local_parts.insert(local_part); used_local_parts.insert(local_part);
return std::format("{}@thebiergarten.app", local_part); return std::format("{}@thebiergarten.app", local_part);
@@ -62,27 +66,40 @@ std::string GenerateDateOfBirth(std::mt19937& rng) {
sys_days{birth_year_anchor} - days{day_offset_dist(rng)}; sys_days{birth_year_anchor} - days{day_offset_dist(rng)};
const year_month_day birth_ymd{birth_date}; const year_month_day birth_ymd{birth_date};
return std::format("{:04}-{:02}-{:02}", return std::format("{:04}-{:02}-{:02}", static_cast<int>(birth_ymd.year()),
static_cast<int>(birth_ymd.year()),
static_cast<unsigned>(birth_ymd.month()), static_cast<unsigned>(birth_ymd.month()),
static_cast<unsigned>(birth_ymd.day())); static_cast<unsigned>(birth_ymd.day()));
} }
std::string GenerateRandomPassword(std::mt19937& rng) { std::optional<Name> SampleName(
constexpr size_t kPasswordLength = 32; const ForenamesByCountryMap& forenames_by_country,
constexpr std::string_view kCharset = const SurnamesByCountryMap& surnames_by_country,
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*"; const std::string& iso3166_1, std::mt19937& rng) {
const auto forenames_it = forenames_by_country.find(iso3166_1);
const auto surnames_it = surnames_by_country.find(iso3166_1);
std::uniform_int_distribution<size_t> char_dist(0, kCharset.size() - 1); if (forenames_it == forenames_by_country.end() ||
surnames_it == surnames_by_country.end() ||
std::string password; forenames_it->second.empty() || surnames_it->second.empty()) {
password.reserve(kPasswordLength); return std::nullopt;
for (size_t i = 0; i < kPasswordLength; ++i) {
password.push_back(kCharset[char_dist(rng)]);
} }
return password;
}
const ForenameList& forenames = forenames_it->second;
const SurnameList& surnames = surnames_it->second;
std::uniform_int_distribution<size_t> forename_dist(0, forenames.size() - 1);
std::uniform_int_distribution<size_t> surname_dist(0, surnames.size() - 1);
auto forename_it = forenames.begin();
std::advance(forename_it, forename_dist(rng));
auto surname_it = surnames.begin();
std::advance(surname_it, surname_dist(rng));
return Name{.first_name = forename_it->name,
.last_name = *surname_it,
.gender = forename_it->gender};
}
} // namespace } // namespace
void BiergartenPipelineOrchestrator::GenerateUsers( void BiergartenPipelineOrchestrator::GenerateUsers(
@@ -91,36 +108,81 @@ void BiergartenPipelineOrchestrator::GenerateUsers(
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::UserGeneration,
.message = "=== SAMPLE USER GENERATION ==="}); .message = "=== SAMPLE USER GENERATION ==="});
const std::vector<UserPersona> personas = const PersonasList& personas = curated_data_service_->LoadPersonas();
JsonLoader::LoadPersonas("personas.json", logger_);
if (personas.empty()) { if (personas.empty()) {
throw std::runtime_error( throw std::runtime_error(
"No personas available in personas.json for user generation"); "No personas available in personas.json for user generation");
} }
const NamesByCountry names_by_country = JsonLoader::LoadNamesByCountry( const ForenamesByCountryMap& forenames_by_country =
"forenames-by-country.json", "surnames-by-country.json", logger_); curated_data_service_->LoadForenamesByCountry();
const SurnamesByCountryMap& surnames_by_country =
curated_data_service_->LoadSurnamesByCountry();
std::mt19937 rng(std::random_device{}()); std::mt19937 rng(std::random_device{}());
std::uniform_int_distribution<size_t> persona_dist(0, personas.size() - 1); std::uniform_int_distribution<size_t> persona_dist(0, personas.size() - 1);
generated_users_.clear(); generated_users_.clear();
std::unordered_set<std::string> used_email_local_parts;
size_t skipped_count = 0; size_t skipped_count = 0;
size_t export_failed_count = 0; size_t export_failed_count = 0;
std::unordered_set<std::string> used_email_local_parts;
for (const auto& city : cities) { const auto generate_record =
const std::optional<Name> sampled_name = [this, &rng, &skipped_count, &used_email_local_parts](
names_by_country.SampleName(city.location.iso3166_1, rng); const EnrichedCity& city, const UserPersona& persona,
const Name& sampled_name) -> std::optional<UserRecord> {
try {
const UserResult user =
generator_->GenerateUser(city, persona, sampled_name);
if (!sampled_name.has_value()) { return UserRecord{
.location = city.location,
.user = user,
.email = BuildEmail(sampled_name, used_email_local_parts),
.date_of_birth = GenerateDateOfBirth(rng),
};
} catch (const std::exception& e) {
++skipped_count; ++skipped_count;
logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration,
.message = std::format(
"[Pipeline] Skipping city '{}' ({}): "
"user generation failed: {}",
city.location.city, city.location.country, e.what())});
return std::nullopt;
}
};
const auto export_record = [this,
&export_failed_count](const UserRecord& record) {
try {
exporter_->ProcessRecord(record);
} catch (const std::exception& export_exception) {
++export_failed_count;
logger_->Log( logger_->Log(
{.level = LogLevel::Warn, {.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration,
.message = std::format("[Pipeline] Generated user for '{}' ({}) but "
"SQLite export failed: {}",
record.location.city, record.location.country,
export_exception.what())});
}
};
for (const auto& city : cities) {
const std::optional<Name> sampled_name =
SampleName(forenames_by_country, surnames_by_country,
city.location.iso3166_1, rng);
if (!sampled_name.has_value()) {
++skipped_count;
logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::UserGeneration,
.message = std::format( .message = std::format(
"[Pipeline] Skipping city '{}' ({}): no names available for " "[Pipeline] Skipping city '{}' ({}): no names "
"country '{}'", "available for country '{}'",
city.location.city, city.location.country, city.location.city, city.location.country,
city.location.iso3166_1)}); city.location.iso3166_1)});
continue; continue;
@@ -128,42 +190,14 @@ void BiergartenPipelineOrchestrator::GenerateUsers(
const UserPersona& persona = personas[persona_dist(rng)]; const UserPersona& persona = personas[persona_dist(rng)];
try { const std::optional<UserRecord> record =
const UserResult user = generate_record(city, persona, *sampled_name);
generator_->GenerateUser(city, persona, *sampled_name); if (!record.has_value()) {
continue;
const GeneratedUser generated_user{
.location = city.location,
.user = user,
.email = BuildEmail(*sampled_name, used_email_local_parts),
.date_of_birth = GenerateDateOfBirth(rng),
.password = GenerateRandomPassword(rng),
};
generated_users_.push_back(generated_user);
try {
exporter_->ProcessRecord(generated_user);
} catch (const std::exception& export_exception) {
++export_failed_count;
logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration,
.message = std::format(
"[Pipeline] Generated user for '{}' ({}) but SQLite export failed: {}",
city.location.city, city.location.country,
export_exception.what())});
}
} catch (const std::exception& e) {
++skipped_count;
logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration,
.message = std::format(
"[Pipeline] Skipping city '{}' ({}): user generation failed: {}",
city.location.city, city.location.country, e.what())});
} }
generated_users_.push_back(*record);
export_record(*record);
} }
if (skipped_count > 0) { if (skipped_count > 0) {
@@ -176,11 +210,10 @@ void BiergartenPipelineOrchestrator::GenerateUsers(
} }
if (export_failed_count > 0) { if (export_failed_count > 0) {
logger_->Log( logger_->Log({.level = LogLevel::Warn,
{.level = LogLevel::Warn,
.phase = PipelinePhase::Teardown, .phase = PipelinePhase::Teardown,
.message = std::format( .message = std::format("[Pipeline] Failed to export {} "
"[Pipeline] Failed to export {} generated user/users to SQLite", "generated user/users to SQLite",
export_failed_count)}); export_failed_count)});
} }
} }

View File

@@ -1,14 +1,14 @@
/** /**
* @file biergarten_pipeline_orchestrator/log_results.cc * @file biergarten_pipeline_orchestrator/log_results.cc
* @brief BiergartenDataGenerator::LogResults() implementation. * @brief BiergartenPipelineOrchestrator::LogResults() implementation.
*/ */
#include <boost/json/array.hpp> #include <boost/json/array.hpp>
#include <chrono> #include <chrono>
#include <format> #include <format>
#include "../../includes/json_handling/pretty_print.h"
#include "biergarten_pipeline_orchestrator.h" #include "biergarten_pipeline_orchestrator.h"
#include "json_handling/pretty_print.h"
#include "services/logging/logger.h" #include "services/logging/logger.h"
void BiergartenPipelineOrchestrator::LogResults() const { void BiergartenPipelineOrchestrator::LogResults() const {
@@ -38,7 +38,6 @@ void BiergartenPipelineOrchestrator::LogResults() const {
boost::json::array user_output; boost::json::array user_output;
for (const auto& generated_user : generated_users_) { for (const auto& generated_user : generated_users_) {
user_output.push_back(boost::json::object{ user_output.push_back(boost::json::object{
{"first_name", generated_user.user.first_name}, {"first_name", generated_user.user.first_name},

View File

@@ -1,17 +1,17 @@
/** /**
* @file biergarten_pipeline_orchestrator/query_cities_with_countries.cc * @file biergarten_pipeline_orchestrator/query_cities_with_countries.cc
* @brief BiergartenDataGenerator::QueryCitiesWithCountries() implementation. * @brief BiergartenPipelineOrchestrator::QueryCitiesWithCountries()
* implementation.
*/ */
#include <algorithm> #include <algorithm>
#include <chrono> #include <chrono>
#include <filesystem>
#include <format> #include <format>
#include <iterator> #include <iterator>
#include <random> #include <random>
#include "biergarten_pipeline_orchestrator.h" #include "biergarten_pipeline_orchestrator.h"
#include "json_handling/json_loader.h" #include "services/curated_data/curated_json_data_service.h"
#include "services/logging/logger.h" #include "services/logging/logger.h"
std::vector<Location> std::vector<Location>
@@ -20,9 +20,8 @@ BiergartenPipelineOrchestrator::QueryCitiesWithCountries() {
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = "=== GEOGRAPHIC DATA OVERVIEW ==="}); .message = "=== GEOGRAPHIC DATA OVERVIEW ==="});
const std::filesystem::path locations_path = "locations.json"; const std::vector<Location>& all_locations =
curated_data_service_->LoadLocations();
auto all_locations = JsonLoader::LoadLocations(locations_path, logger_);
const size_t sample_count = std::min( const size_t sample_count = std::min(
static_cast<size_t>(application_options_.pipeline.location_count), static_cast<size_t>(application_options_.pipeline.location_count),

View File

@@ -1,6 +1,6 @@
/** /**
* @file biergarten_pipeline_orchestrator/run.cc * @file biergarten_pipeline_orchestrator/run.cc
* @brief BiergartenDataGenerator::Run() implementation. * @brief BiergartenPipelineOrchestrator::Run() implementation.
*/ */
#include <chrono> #include <chrono>
@@ -28,20 +28,21 @@ bool BiergartenPipelineOrchestrator::Run() {
.region_context = std::move(region_context)}); .region_context = std::move(region_context)});
} catch (const std::exception& exception) { } catch (const std::exception& exception) {
++skipped_count; ++skipped_count;
logger_->Log( logger_->Log({.level = LogLevel::Warn,
{.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment, .phase = PipelinePhase::Enrichment,
.message = std::format( .message = std::format(
"[Pipeline] Skipping city '{}' ({}): context lookup failed: {}", "[Pipeline] Skipping city '{}' ({}): context "
"lookup failed: {}",
city.city, city.country, exception.what())}); city.city, city.country, exception.what())});
} }
} }
if (skipped_count > 0) { if (skipped_count > 0) {
logger_->Log({.level = LogLevel::Warn, logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment, .phase = PipelinePhase::Enrichment,
.message = std::format( .message = std::format("[Pipeline] Skipped {} city/cities due "
"[Pipeline] Skipped {} city/cities due to context lookup errors", "to context lookup errors",
skipped_count)}); skipped_count)});
} }
@@ -51,11 +52,10 @@ bool BiergartenPipelineOrchestrator::Run() {
this->LogResults(); this->LogResults();
return true; return true;
} catch (const std::exception& e) { } catch (const std::exception& e) {
logger_->Log( logger_->Log({.level = LogLevel::Error,
{.level = LogLevel::Error,
.phase = PipelinePhase::Teardown, .phase = PipelinePhase::Teardown,
.message = .message = std::format(
std::format("Pipeline execution failed with error: {}", e.what())}); "Pipeline execution failed with error: {}", e.what())});
return false; return false;
} }
} }

View File

@@ -12,6 +12,7 @@
#include <string_view> #include <string_view>
#include <vector> #include <vector>
#include "data_generation/json_grammars.h"
#include "data_generation/llama_generator.h" #include "data_generation/llama_generator.h"
#include "data_generation/llama_generator_helpers.h" #include "data_generation/llama_generator_helpers.h"
@@ -32,19 +33,6 @@ static std::string FormatLocalLanguageCodes(
return formatted; return formatted;
} }
// GBNF grammar for structured brewery JSON output.
// @TODO move to a separate gbnf file if it grows in complexity or is shared
// across modules.
static constexpr std::string_view kBreweryJsonGrammar = R"json_brewery(
root ::= thought-block "{" ws "\"name_en\"" ws ":" ws string ws "," ws "\"description_en\"" ws ":" ws string ws "," ws "\"name_local\"" ws ":" ws string ws "," ws "\"description_local\"" ws ":" ws string ws "}" ws
thought-block ::= [^{]*
ws ::= [ \t\n\r]*
string ::= "\"" char+ "\""
char ::= [^"\\\x7F\x00-\x1F] | [\\] escape
escape ::= ["\\/bfnrt] | "u" hex hex hex hex
hex ::= [0-9a-fA-F]
)json_brewery";
static constexpr int kBreweryInitialMaxTokens = 2800; static constexpr int kBreweryInitialMaxTokens = 2800;
BreweryResult LlamaGenerator::GenerateBrewery( BreweryResult LlamaGenerator::GenerateBrewery(
@@ -83,8 +71,8 @@ BreweryResult LlamaGenerator::GenerateBrewery(
/** /**
* RETRY LOOP with validation and error correction * RETRY LOOP with validation and error correction
* Attempts to generate valid brewery data up to 3 times, with feedback-based * Attempts to generate valid brewery data up to 3 times, with
* refinement * feedback-based refinement
*/ */
constexpr int max_attempts = 3; constexpr int max_attempts = 3;
std::string raw; std::string raw;
@@ -119,7 +107,8 @@ BreweryResult LlamaGenerator::GenerateBrewery(
logger_->Log( logger_->Log(
{.level = LogLevel::Info, {.level = LogLevel::Info,
.phase = PipelinePhase::BreweryAndBeerGeneration, .phase = PipelinePhase::BreweryAndBeerGeneration,
.message = std::format("LlamaGenerator: successfully generated brewery data on attempt {}", .message = std::format("LlamaGenerator: successfully generated "
"brewery data on attempt {}",
attempt + 1)}); attempt + 1)});
} }
@@ -133,32 +122,37 @@ BreweryResult LlamaGenerator::GenerateBrewery(
logger_->Log( logger_->Log(
{.level = LogLevel::Warn, {.level = LogLevel::Warn,
.phase = PipelinePhase::BreweryAndBeerGeneration, .phase = PipelinePhase::BreweryAndBeerGeneration,
.message = .message = std::format(
std::format("LlamaGenerator: malformed brewery JSON (attempt {}): {}", "LlamaGenerator: malformed brewery JSON (attempt {}): {}",
attempt + 1, *validation_error)}); attempt + 1, *validation_error)});
} }
// Update prompt with error details to guide LLM toward correct output. // Update prompt with error details to guide LLM toward correct output.
user_prompt = std::format( user_prompt = std::format(
"Your previous response was invalid. Error: {}\nReturn the thought " "Your previous response was invalid. Error: {}\nReturn the thought "
"process before the JSON if needed, then return ONLY valid JSON with " "process before the JSON if needed, then return ONLY valid JSON "
"exactly these keys, in this exact order: {{\"name_en\": \"<English " "with "
"exactly these keys, in this exact order: {{\"name_en\": "
"\"<English "
"brewery name>\", \"description_en\": \"<English single-paragraph " "brewery name>\", \"description_en\": \"<English single-paragraph "
"description>\", \"name_local\": \"<local-language brewery name>\", " "description>\", \"name_local\": \"<local-language brewery "
"name>\", "
"\"description_local\": \"<local-language single-paragraph " "\"description_local\": \"<local-language single-paragraph "
"description>\"}}.\nDo not include markdown, comments, extra keys, or " "description>\"}}.\nDo not include markdown, comments, extra keys, "
"literal placeholder values.\n\nKeep the JSON strings concise enough " "or "
"literal placeholder values.\n\nKeep the JSON strings concise "
"enough "
"to fit within the token budget.\n\n{}", "to fit within the token budget.\n\n{}",
*validation_error, retry_location); *validation_error, retry_location);
} }
// All retry attempts exhausted: log failure and throw exception // All retry attempts exhausted: log failure and throw exception
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Error,
{.level = LogLevel::Error,
.phase = PipelinePhase::BreweryAndBeerGeneration, .phase = PipelinePhase::BreweryAndBeerGeneration,
.message = std::format( .message = std::format(
"LlamaGenerator: malformed brewery response after {} attempts: {}", "LlamaGenerator: malformed brewery "
"response after {} attempts: {}",
max_attempts, last_error.empty() ? raw : last_error)}); max_attempts, last_error.empty() ? raw : last_error)});
} }
throw std::runtime_error("LlamaGenerator: malformed brewery response"); throw std::runtime_error("LlamaGenerator: malformed brewery response");

View File

@@ -10,21 +10,10 @@
#include <string> #include <string>
#include <string_view> #include <string_view>
#include "data_generation/json_grammars.h"
#include "data_generation/llama_generator.h" #include "data_generation/llama_generator.h"
#include "data_generation/llama_generator_helpers.h" #include "data_generation/llama_generator_helpers.h"
// GBNF grammar for structured user JSON output.
static constexpr std::string_view kUserJsonGrammar = R"json_user(
root ::= thought-block "{" ws "\"username\"" ws ":" ws string ws "," ws "\"bio\"" ws ":" ws string ws "," ws "\"activity_weight\"" ws ":" ws number ws "}" ws
thought-block ::= [^{]*
ws ::= [ \t\n\r]*
string ::= "\"" char+ "\""
char ::= [^"\\\x7F\x00-\x1F] | [\\] escape
escape ::= ["\\/bfnrt] | "u" hex hex hex hex
hex ::= [0-9a-fA-F]
number ::= "-"? ("0" | [1-9] [0-9]*) ("." [0-9]+)?
)json_user";
static constexpr int kUserInitialMaxTokens = 1200; static constexpr int kUserInitialMaxTokens = 1200;
UserResult LlamaGenerator::GenerateUser(const EnrichedCity& city, UserResult LlamaGenerator::GenerateUser(const EnrichedCity& city,
@@ -41,17 +30,20 @@ UserResult LlamaGenerator::GenerateUser(const EnrichedCity& city,
const std::string system_prompt = prompt_directory_->Load("USER_GENERATION"); const std::string system_prompt = prompt_directory_->Load("USER_GENERATION");
std::string user_prompt = std::format( std::string user_prompt = std::format(
"## NAME:\n{} {}\n\n## GENDER:\n{}\n\n## CITY:\n{}\n\n## COUNTRY:\n{}\n\n" "## NAME:\n\n{} {}\n\n"
"## PERSONA:\n{}\n\n## PERSONA DESCRIPTION:\n{}\n\n## STYLE " "## GENDER:\n\n{}\n\n"
"AFFINITIES:\n{}", "## CITY:\n\n{}\n\n"
"## COUNTRY:\n\n{}\n\n"
"## PERSONA:\n\n{}\n\n"
"## PERSONA DESCRIPTION:\n\n{}\n\n"
"## STYLE AFFINITIES:\n\n{}",
name.first_name, name.last_name, name.gender, city.location.city, name.first_name, name.last_name, name.gender, city.location.city,
city.location.country, persona.name, persona.description, city.location.country, persona.name, persona.description,
style_affinities); style_affinities);
const std::string retry_context = const std::string retry_context = std::format(
std::format("Name: {} {}\nCity: {}, {}\nPersona: {}", name.first_name, "Name: {} {}\nCity: {}, {}\nPersona: {}", name.first_name, name.last_name,
name.last_name, city.location.city, city.location.country, city.location.city, city.location.country, persona.name);
persona.name);
constexpr int max_attempts = 3; constexpr int max_attempts = 3;
std::string raw; std::string raw;
@@ -59,8 +51,7 @@ UserResult LlamaGenerator::GenerateUser(const EnrichedCity& city,
int max_tokens = kUserInitialMaxTokens; int max_tokens = kUserInitialMaxTokens;
for (int attempt = 0; attempt < max_attempts; ++attempt) { for (int attempt = 0; attempt < max_attempts; ++attempt) {
raw = this->Infer(system_prompt, user_prompt, max_tokens, raw = this->Infer(system_prompt, user_prompt, max_tokens, kUserJsonGrammar);
kUserJsonGrammar);
if (logger_) { if (logger_) {
logger_->Log( logger_->Log(
{.level = LogLevel::Debug, {.level = LogLevel::Debug,
@@ -78,7 +69,8 @@ UserResult LlamaGenerator::GenerateUser(const EnrichedCity& city,
logger_->Log( logger_->Log(
{.level = LogLevel::Info, {.level = LogLevel::Info,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::UserGeneration,
.message = std::format("LlamaGenerator: successfully generated user data on attempt {}", .message = std::format("LlamaGenerator: successfully "
"generated user data on attempt {}",
attempt + 1)}); attempt + 1)});
} }
@@ -90,31 +82,33 @@ UserResult LlamaGenerator::GenerateUser(const EnrichedCity& city,
last_error = *validation_error; last_error = *validation_error;
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Warn,
{.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::UserGeneration,
.message = .message = std::format(
std::format("LlamaGenerator: malformed user JSON (attempt {}): {}", "LlamaGenerator: malformed user JSON (attempt {}): {}",
attempt + 1, *validation_error)}); attempt + 1, *validation_error)});
} }
user_prompt = std::format( user_prompt = std::format(
"Your previous response was invalid. Error: {}\nReturn the thought " "Your previous response was invalid. Error: {}\nReturn the thought "
"process before the JSON if needed, then return ONLY valid JSON with " "process before the JSON if needed, then return ONLY valid JSON "
"exactly these keys, in this exact order: {{\"username\": \"<handle " "with "
"exactly these keys, in this exact order: {{\"username\": "
"\"<handle "
"derived from the given name>\", \"bio\": \"<first-person bio " "derived from the given name>\", \"bio\": \"<first-person bio "
"grounded in the persona>\", \"activity_weight\": <number between 0 " "grounded in the persona>\", \"activity_weight\": <number between "
"0 "
"and 1>}}.\nDo not include markdown, comments, extra keys, or " "and 1>}}.\nDo not include markdown, comments, extra keys, or "
"literal placeholder values.\n\n{}", "literal placeholder values.\n\n{}",
*validation_error, retry_context); *validation_error, retry_context);
} }
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Error,
{.level = LogLevel::Error,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::UserGeneration,
.message = std::format( .message = std::format(
"LlamaGenerator: malformed user response after {} attempts: {}", "LlamaGenerator: malformed user response "
"after {} attempts: {}",
max_attempts, last_error.empty() ? raw : last_error)}); max_attempts, last_error.empty() ? raw : last_error)});
} }
throw std::runtime_error("LlamaGenerator: malformed user response"); throw std::runtime_error("LlamaGenerator: malformed user response");

View File

@@ -15,8 +15,10 @@
#include <string_view> #include <string_view>
#include <vector> #include <vector>
#include <llama.h>
#include "data_generation/llama_generator_helpers.h" #include "data_generation/llama_generator_helpers.h"
#include "llama.h"
namespace { namespace {
/** /**
* String trimming: removes leading and trailing whitespace * String trimming: removes leading and trailing whitespace

View File

@@ -14,9 +14,10 @@
#include <string_view> #include <string_view>
#include <vector> #include <vector>
#include <llama.h>
#include "data_generation/llama_generator.h" #include "data_generation/llama_generator.h"
#include "data_generation/llama_generator_helpers.h" #include "data_generation/llama_generator_helpers.h"
#include "llama.h"
static constexpr size_t kPromptTokenSlack = 8; static constexpr size_t kPromptTokenSlack = 8;
// Minimum tokens to keep when using top-p sampling. Ensures at least one // Minimum tokens to keep when using top-p sampling. Ensures at least one

View File

@@ -11,8 +11,9 @@
#include <stdexcept> #include <stdexcept>
#include <string> #include <string>
#include <llama.h>
#include "data_model/models.h" #include "data_model/models.h"
#include "llama.h"
static constexpr uint32_t kMaxContextSize = 32768U; static constexpr uint32_t kMaxContextSize = 32768U;

View File

@@ -10,9 +10,10 @@
#include <string> #include <string>
#include <utility> #include <utility>
#include <ggml-backend.h>
#include <llama.h>
#include "data_generation/llama_generator.h" #include "data_generation/llama_generator.h"
#include "ggml-backend.h"
#include "llama.h"
// Maximum batch size for decode operations. Capping the batch prevents // Maximum batch size for decode operations. Capping the batch prevents
// excessive memory allocation while maintaining inference performance. // excessive memory allocation while maintaining inference performance.

View File

@@ -1,41 +0,0 @@
/**
* @file data_model/names_by_country.cc
* @brief NamesByCountry::SampleName() implementation.
*/
#include "data_model/names_by_country.h"
#include <utility>
NamesByCountry::NamesByCountry(
std::unordered_map<std::string, std::vector<ForenameEntry>>
forenames_by_country,
std::unordered_map<std::string, std::vector<std::string>>
surnames_by_country)
: forenames_by_country_(std::move(forenames_by_country)),
surnames_by_country_(std::move(surnames_by_country)) {}
std::optional<Name> NamesByCountry::SampleName(const std::string& iso3166_1,
std::mt19937& rng) const {
const auto forenames_it = forenames_by_country_.find(iso3166_1);
const auto surnames_it = surnames_by_country_.find(iso3166_1);
if (forenames_it == forenames_by_country_.end() ||
surnames_it == surnames_by_country_.end() ||
forenames_it->second.empty() || surnames_it->second.empty()) {
return std::nullopt;
}
const std::vector<ForenameEntry>& forenames = forenames_it->second;
const std::vector<std::string>& surnames = surnames_it->second;
std::uniform_int_distribution<size_t> forename_dist(0,
forenames.size() - 1);
std::uniform_int_distribution<size_t> surname_dist(0, surnames.size() - 1);
const ForenameEntry& forename = forenames[forename_dist(rng)];
return Name{.first_name = forename.name,
.last_name = surnames[surname_dist(rng)],
.gender = forename.gender};
}

View File

@@ -18,25 +18,7 @@
#include <string> #include <string>
#include <thread> #include <thread>
#include "biergarten_pipeline_orchestrator.h" #include "biergarten_pipeline.h"
#include "concurrency/bounded_channel.h"
#include "data_generation/llama_generator.h"
#include "data_generation/mock_generator.h"
#include "data_generation/prompt_formatting/gemma4_jinja_prompt_formatter.h"
#include "data_model/models.h"
#include "llama_backend_state.h"
#include "services/database/export_service.h"
#include "services/database/sqlite_export_service.h"
#include "services/datetime/timer.h"
#include "services/enrichment/enrichment_service.h"
#include "services/enrichment/mock_enrichment.h"
#include "services/enrichment/wikipedia_service.h"
#include "services/logging/log_dispatcher.h"
#include "services/logging/log_entry.h"
#include "services/logging/log_producer.h"
#include "services/logging/logger.h"
#include "services/prompting/prompt_directory.h"
#include "web_client/http_web_client.h"
namespace di = boost::di; namespace di = boost::di;
@@ -104,6 +86,28 @@ int main(const int argc, char** argv) {
di::bind<ApplicationOptions>().to(options), di::bind<ApplicationOptions>().to(options),
di::bind<std::string>().to(model_path), di::bind<std::string>().to(model_path),
di::bind<IExportService>().to<SqliteExportService>(), di::bind<IExportService>().to<SqliteExportService>(),
di::bind<ICuratedDataService>().to(
[options, &log_producer]() -> std::unique_ptr<ICuratedDataService> {
if (options.generator.use_mocked) {
log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "Curated data: mock"});
return std::make_unique<MockCuratedDataService>();
}
log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "Curated data: JsonLoader"});
return std::make_unique<CuratedJsonDataService>(
CuratedDataFilePaths{
.locations_path = "locations.json",
.personas_path = "personas.json",
.forenames_path = "forenames-by-country.json",
.surnames_path = "surnames-by-country.json",
});
}),
di::bind<IPromptFormatter>().to([options, log_producer] { di::bind<IPromptFormatter>().to([options, log_producer] {
if (options.generator.use_mocked) { if (options.generator.use_mocked) {
{ {
@@ -132,11 +136,11 @@ int main(const int argc, char** argv) {
} }
return std::unique_ptr<WebClient>(nullptr); return std::unique_ptr<WebClient>(nullptr);
} }
{
log_producer->Log({.level = LogLevel::Info, log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = "Web client: HttpWebClient"}); .message = "Web client: HttpWebClient"});
}
return std::unique_ptr<WebClient>( return std::unique_ptr<WebClient>(
std::make_unique<HttpWebClient>(log_producer)); std::make_unique<HttpWebClient>(log_producer));
}), }),
@@ -144,18 +148,17 @@ int main(const int argc, char** argv) {
[options, &log_producer]( [options, &log_producer](
const auto& inj) -> std::unique_ptr<IEnrichmentService> { const auto& inj) -> std::unique_ptr<IEnrichmentService> {
if (options.generator.use_mocked) { if (options.generator.use_mocked) {
{
log_producer->Log({.level = LogLevel::Info, log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = "Enrichment: mock"}); .message = "Enrichment: mock"});
}
return std::make_unique<MockEnrichmentService>(); return std::make_unique<MockEnrichmentService>();
} }
{
log_producer->Log({.level = LogLevel::Info, log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = "Enrichment: Wikipedia"}); .message = "Enrichment: Wikipedia"});
}
return std::make_unique<WikipediaEnrichmentService>( return std::make_unique<WikipediaEnrichmentService>(
inj.template create<std::unique_ptr<WebClient>>(), inj.template create<std::unique_ptr<WebClient>>(),
log_producer); log_producer);
@@ -164,27 +167,35 @@ int main(const int argc, char** argv) {
[&options, &model_path, &sampling, &prompt_directory, [&options, &model_path, &sampling, &prompt_directory,
&log_producer](const auto& inj) -> std::unique_ptr<DataGenerator> { &log_producer](const auto& inj) -> std::unique_ptr<DataGenerator> {
if (options.generator.use_mocked) { if (options.generator.use_mocked) {
{
log_producer->Log({.level = LogLevel::Info, log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = "Generator: mock"}); .message = "Generator: mock"});
}
return std::make_unique<MockGenerator>(); return std::make_unique<MockGenerator>();
} }
{
#ifdef BIERGARTEN_MOCK_ONLY
// Unreachable: ParseArguments requires --mocked when
// BIERGARTEN_MOCK_ONLY is compiled in, so LlamaGenerator is
// never constructed and is not linked into this binary.
throw std::runtime_error(
"LlamaGenerator is unavailable in a BIERGARTEN_MOCK_ONLY "
"build");
#else
log_producer->Log( log_producer->Log(
{.level = LogLevel::Info, {.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = std::format( .message = std::format(
"Generator: LlamaGenerator | model={} | temp={:.2f} " "Generator: LlamaGenerator | model={} | "
"top_p={:.2f} top_k={} n_ctx={} seed={}", "temp={:.2f} top_p={:.2f} top_k={} n_ctx={} seed={}",
model_path, sampling.temperature, sampling.top_p, model_path, sampling.temperature, sampling.top_p,
sampling.top_k, sampling.n_ctx, sampling.seed)}); sampling.top_k, sampling.n_ctx, sampling.seed)});
}
return std::make_unique<LlamaGenerator>( return std::make_unique<LlamaGenerator>(
options, model_path, log_producer, options, model_path, log_producer,
inj.template create<std::unique_ptr<IPromptFormatter>>(), inj.template create<std::unique_ptr<IPromptFormatter>>(),
std::move(prompt_directory)); std::move(prompt_directory));
#endif
})); }));
const auto orchestrator = const auto orchestrator =
@@ -203,7 +214,6 @@ int main(const int argc, char** argv) {
timer.Elapsed())}); timer.Elapsed())});
return shutdown(EXIT_SUCCESS); return shutdown(EXIT_SUCCESS);
} catch (const std::exception& exception) { } catch (const std::exception& exception) {
const LogDTO log_entry{.level = LogLevel::Error, const LogDTO log_entry{.level = LogLevel::Error,
.phase = PipelinePhase::Teardown, .phase = PipelinePhase::Teardown,

View File

@@ -4,20 +4,20 @@
* records with strict field validation and descriptive error reporting. * records with strict field validation and descriptive error reporting.
*/ */
#include "json_handling/json_loader.h" #include "services/curated_data/curated_json_data_service.h"
#include <format>
#include "services/logging/logger.h"
#include <iostream>
#include <boost/json.hpp> #include <boost/json.hpp>
#include <format>
#include <fstream> #include <fstream>
#include <iostream>
#include <sstream> #include <sstream>
#include <stdexcept> #include <stdexcept>
#include <string> #include <string>
#include <string_view> #include <string_view>
#include <unordered_map>
#include <utility> #include <utility>
#include <vector>
#include "services/logging/logger.h"
static std::string ReadRequiredString(const boost::json::object& object, static std::string ReadRequiredString(const boost::json::object& object,
const char* key) { const char* key) {
@@ -67,8 +67,8 @@ boost::json::value ParseJsonFile(const std::filesystem::path& filepath,
const char* what) { const char* what) {
std::ifstream input(filepath); std::ifstream input(filepath);
if (!input.is_open()) { if (!input.is_open()) {
throw std::runtime_error(std::format("Failed to open {} file: {}", what, throw std::runtime_error(
filepath.string())); std::format("Failed to open {} file: {}", what, filepath.string()));
} }
std::stringstream buffer; std::stringstream buffer;
@@ -83,12 +83,13 @@ boost::json::value ParseJsonFile(const std::filesystem::path& filepath,
return root; return root;
} }
/// @brief Returns the first element of a string array field, falling back to /**
/// `fallback_key` if `key` is missing/empty (some source entries only have a * @brief Returns the first element of a string array field, falling back to
/// "localized" form and no "romanized" form). * `fallback_key` if `key` is missing/empty (some source entries only have a
* "localized" form and no "romanized" form).
*/
std::string ReadFirstOfStringArray(const boost::json::object& object, std::string ReadFirstOfStringArray(const boost::json::object& object,
const char* key, const char* key, const char* fallback_key) {
const char* fallback_key) {
for (const char* candidate_key : {key, fallback_key}) { for (const char* candidate_key : {key, fallback_key}) {
const boost::json::value* value = object.if_contains(candidate_key); const boost::json::value* value = object.if_contains(candidate_key);
if (value == nullptr || !value->is_array()) { if (value == nullptr || !value->is_array()) {
@@ -105,16 +106,23 @@ std::string ReadFirstOfStringArray(const boost::json::object& object,
} // namespace } // namespace
std::vector<Location> JsonLoader::LoadLocations( CuratedJsonDataService::CuratedJsonDataService(CuratedDataFilePaths filepaths)
const std::filesystem::path& filepath, std::shared_ptr<ILogger> logger) { : filepaths_(std::move(filepaths)) {}
const boost::json::value root = ParseJsonFile(filepath, "locations");
const LocationsList& CuratedJsonDataService::LoadLocations() {
if (!cache_.locations.empty()) {
return cache_.locations;
}
const boost::json::value root =
ParseJsonFile(filepaths_.locations_path, "locations");
if (!root.is_array()) { if (!root.is_array()) {
throw std::runtime_error( throw std::runtime_error(
"Invalid locations JSON: root element must be an array"); "Invalid locations JSON: root element must be an array");
} }
std::vector<Location> locations; LocationsList locations;
const auto& items = root.as_array(); const auto& items = root.as_array();
locations.reserve(items.size()); locations.reserve(items.size());
@@ -136,20 +144,24 @@ std::vector<Location> JsonLoader::LoadLocations(
.longitude = ReadRequiredNumber(object, "longitude"), .longitude = ReadRequiredNumber(object, "longitude"),
}); });
} }
cache_.locations = std::move(locations);
return locations; return cache_.locations;
} }
std::vector<UserPersona> JsonLoader::LoadPersonas( const PersonasList& CuratedJsonDataService::LoadPersonas() {
const std::filesystem::path& filepath, std::shared_ptr<ILogger> logger) { if (!cache_.personas.empty()) {
const boost::json::value root = ParseJsonFile(filepath, "personas"); return cache_.personas;
}
const boost::json::value root =
ParseJsonFile(filepaths_.personas_path, "personas");
if (!root.is_array()) { if (!root.is_array()) {
throw std::runtime_error( throw std::runtime_error(
"Invalid personas JSON: root element must be an array"); "Invalid personas JSON: root element must be an array");
} }
std::vector<UserPersona> personas; PersonasList personas;
const auto& items = root.as_array(); const auto& items = root.as_array();
personas.reserve(items.size()); personas.reserve(items.size());
@@ -163,36 +175,34 @@ std::vector<UserPersona> JsonLoader::LoadPersonas(
personas.push_back(UserPersona{ personas.push_back(UserPersona{
.name = ReadRequiredString(object, "name"), .name = ReadRequiredString(object, "name"),
.description = ReadRequiredString(object, "description"), .description = ReadRequiredString(object, "description"),
.style_affinities = .style_affinities = ReadRequiredStringArray(object, "style_affinities"),
ReadRequiredStringArray(object, "style_affinities"),
}); });
} }
return personas; cache_.personas = std::move(personas);
return cache_.personas;
} }
NamesByCountry JsonLoader::LoadNamesByCountry( const ForenamesByCountryMap& CuratedJsonDataService::LoadForenamesByCountry() {
const std::filesystem::path& forenames_filepath, if (!cache_.forenames_by_country.empty()) {
const std::filesystem::path& surnames_filepath, return cache_.forenames_by_country;
std::shared_ptr<ILogger> logger) { }
const boost::json::value forenames_root =
ParseJsonFile(forenames_filepath, "forenames-by-country");
const boost::json::value surnames_root =
ParseJsonFile(surnames_filepath, "surnames-by-country");
if (!forenames_root.is_object() || !surnames_root.is_object()) { const boost::json::value root =
ParseJsonFile(filepaths_.forenames_path, "forenames-by-country");
if (!root.is_object()) {
throw std::runtime_error( throw std::runtime_error(
"Invalid names-by-country JSON: root element must be an object " "Invalid forenames-by-country JSON: root element must be an object "
"keyed by ISO 3166-1 country code"); "keyed by ISO 3166-1 country code");
} }
std::unordered_map<std::string, std::vector<ForenameEntry>> ForenamesByCountryMap forenames_by_country;
forenames_by_country; for (const auto& [country_code, regions] : root.as_object()) {
for (const auto& [country_code, regions] : forenames_root.as_object()) {
if (!regions.is_array()) { if (!regions.is_array()) {
continue; continue;
} }
std::vector<ForenameEntry> entries; ForenameList entries;
for (const auto& region : regions.as_array()) { for (const auto& region : regions.as_array()) {
if (!region.is_object()) { if (!region.is_object()) {
continue; continue;
@@ -206,9 +216,9 @@ NamesByCountry JsonLoader::LoadNamesByCountry(
continue; continue;
} }
const auto& name_object = name_value.as_object(); const auto& name_object = name_value.as_object();
entries.push_back(ForenameEntry{ entries.emplace_back(ForenameEntry{
.name = ReadFirstOfStringArray(name_object, "romanized", .name =
"localized"), ReadFirstOfStringArray(name_object, "romanized", "localized"),
.gender = ReadRequiredString(name_object, "gender"), .gender = ReadRequiredString(name_object, "gender"),
}); });
} }
@@ -216,23 +226,39 @@ NamesByCountry JsonLoader::LoadNamesByCountry(
forenames_by_country.emplace(country_code, std::move(entries)); forenames_by_country.emplace(country_code, std::move(entries));
} }
std::unordered_map<std::string, std::vector<std::string>> cache_.forenames_by_country = std::move(forenames_by_country);
surnames_by_country; return cache_.forenames_by_country;
for (const auto& [country_code, name_entries] : surnames_root.as_object()) { }
if (!name_entries.is_array()) {
continue; const SurnamesByCountryMap& CuratedJsonDataService::LoadSurnamesByCountry() {
if (!cache_.surnames_by_country.empty()) {
return cache_.surnames_by_country;
} }
std::vector<std::string> surnames;
const boost::json::value root =
ParseJsonFile(filepaths_.surnames_path, "surnames-by-country");
if (!root.is_object()) {
throw std::runtime_error(
"Invalid surnames-by-country JSON: root element must be an object "
"keyed by ISO 3166-1 country code");
}
SurnamesByCountryMap surnames_by_country;
for (const auto& [country_code, name_entries] : root.as_object()) {
if (!name_entries.is_array()) continue;
SurnameList surnames;
for (const auto& name_value : name_entries.as_array()) { for (const auto& name_value : name_entries.as_array()) {
if (!name_value.is_object()) { if (!name_value.is_object()) {
continue; continue;
} }
surnames.push_back(ReadFirstOfStringArray(name_value.as_object(), surnames.emplace_back(ReadFirstOfStringArray(name_value.as_object(),
"romanized", "localized")); "romanized", "localized"));
} }
surnames_by_country.emplace(country_code, std::move(surnames)); surnames_by_country.emplace(country_code, std::move(surnames));
} }
return NamesByCountry(std::move(forenames_by_country), cache_.surnames_by_country = std::move(surnames_by_country);
std::move(surnames_by_country)); return cache_.surnames_by_country;
} }

View File

@@ -0,0 +1,98 @@
/**
* @file services/curated_data/mock_curated_data_service.cc
* @brief Fixed in-memory location, persona, and name dataset for mock mode.
*/
#include "services/curated_data/mock_curated_data_service.h"
MockCuratedDataService::MockCuratedDataService()
: locations_{
Location{.city = "Portland",
.state_province = "Oregon",
.iso3166_2 = "US-OR",
.country = "United States",
.iso3166_1 = "US",
.local_languages = {"en"},
.latitude = 45.5152,
.longitude = -122.6784},
Location{.city = "Munich",
.state_province = "Bavaria",
.iso3166_2 = "DE-BY",
.country = "Germany",
.iso3166_1 = "DE",
.local_languages = {"de"},
.latitude = 48.1351,
.longitude = 11.5820},
Location{.city = "Lyon",
.state_province = "Auvergne-Rhone-Alpes",
.iso3166_2 = "FR-ARA",
.country = "France",
.iso3166_1 = "FR",
.local_languages = {"fr"},
.latitude = 45.7640,
.longitude = 4.8357},
Location{.city = "Brussels",
.state_province = "Brussels-Capital",
.iso3166_2 = "BE-BRU",
.country = "Belgium",
.iso3166_1 = "BE",
.local_languages = {"nl", "fr"},
.latitude = 50.8503,
.longitude = 4.3517},
},
personas_{
UserPersona{.name = "Hophead Explorer",
.description = "Chases hop-forward IPAs and seeks out "
"taprooms with rotating drafts.",
.style_affinities = {"IPA", "Pale Ale"}},
UserPersona{.name = "Lager Traditionalist",
.description = "Prefers clean, balanced lagers and "
"classic pub styles.",
.style_affinities = {"Lager", "Pilsner"}},
UserPersona{.name = "Sour Curious",
.description = "Seeks out wild ales, sours, and "
"barrel-aged experiments.",
.style_affinities = {"Sour", "Wild Ale"}},
},
forenames_by_country_{
{"US",
ForenameList{
ForenameEntry{.name = "James", .gender = "M"},
ForenameEntry{.name = "Mary", .gender = "F"},
}},
{"DE",
ForenameList{
ForenameEntry{.name = "Lukas", .gender = "M"},
ForenameEntry{.name = "Anna", .gender = "F"},
}},
{"FR",
ForenameList{
ForenameEntry{.name = "Lucas", .gender = "M"},
ForenameEntry{.name = "Camille", .gender = "F"},
}},
{"BE",
ForenameList{
ForenameEntry{.name = "Noah", .gender = "M"},
ForenameEntry{.name = "Emma", .gender = "F"},
}},
},
surnames_by_country_{
{"US", SurnameList{"Smith", "Johnson"}},
{"DE", SurnameList{"Muller", "Schmidt"}},
{"FR", SurnameList{"Martin", "Bernard"}},
{"BE", SurnameList{"Peeters", "Janssens"}},
} {}
const LocationsList& MockCuratedDataService::LoadLocations() {
return locations_;
}
const PersonasList& MockCuratedDataService::LoadPersonas() { return personas_; }
const ForenamesByCountryMap& MockCuratedDataService::LoadForenamesByCountry() {
return forenames_by_country_;
}
const SurnamesByCountryMap& MockCuratedDataService::LoadSurnamesByCountry() {
return surnames_by_country_;
}

View File

@@ -1,5 +1,6 @@
/** /**
* @file wikipedia/fetch_extract.cc * @file wikipedia/fetch_extract.cc
* @brief WikipediaEnrichmentService::FetchExtract() implementation.
*/ */
#include <boost/json.hpp> #include <boost/json.hpp>
@@ -20,7 +21,8 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
if (const auto cache_it = this->extract_cache_.find(cache_key); if (const auto cache_it = this->extract_cache_.find(cache_key);
cache_it != this->extract_cache_.end()) { cache_it != this->extract_cache_.end()) {
if (logger_) { if (logger_) {
logger_->Log({.level = LogLevel::Debug, logger_->Log(
{.level = LogLevel::Debug,
.phase = PipelinePhase::Enrichment, .phase = PipelinePhase::Enrichment,
.message = std::format("Wikipedia: Cache hit for {}!", cache_key)}); .message = std::format("Wikipedia: Cache hit for {}!", cache_key)});
} }
@@ -30,7 +32,8 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
const std::string encoded = this->client_->EncodeURL(cache_key); const std::string encoded = this->client_->EncodeURL(cache_key);
const std::string url = std::format( const std::string url = std::format(
"https://en.wikipedia.org/w/" "https://en.wikipedia.org/w/"
"api.php?action=query&titles={}&prop=extracts&explaintext=1&format=json", "api.php?action=query&titles={}&prop=extracts&explaintext=1&format="
"json",
encoded); encoded);
const std::string body = this->client_->Get(url); const std::string body = this->client_->Get(url);
@@ -45,10 +48,10 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
if (ec) { if (ec) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Warn,
{.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment, .phase = PipelinePhase::Enrichment,
.message = std::format("WikipediaService: JSON parse error for '{}': {}", .message = std::format(
"WikipediaService: JSON parse error for '{}': {}",
std::string(query), ec.message())}); std::string(query), ec.message())});
} }
return {}; return {};
@@ -58,11 +61,10 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
const json::object* obj = doc.if_object(); const json::object* obj = doc.if_object();
if (obj == nullptr) { if (obj == nullptr) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Warn,
{.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment, .phase = PipelinePhase::Enrichment,
.message = .message = std::format(
std::format("WikipediaService: Expected root object for '{}'", "WikipediaService: Expected root object for '{}'",
std::string(query))}); std::string(query))});
} }
return {}; return {};
@@ -76,11 +78,10 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
if ((pages_ptr == nullptr) || !pages_ptr->is_object()) { if ((pages_ptr == nullptr) || !pages_ptr->is_object()) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Warn,
{.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment, .phase = PipelinePhase::Enrichment,
.message = .message = std::format(
std::format("WikipediaService: Missing query.pages for '{}'", "WikipediaService: Missing query.pages for '{}'",
std::string(query))}); std::string(query))});
} }
return {}; return {};
@@ -90,10 +91,10 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
if (pages.empty()) { if (pages.empty()) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Warn,
{.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment, .phase = PipelinePhase::Enrichment,
.message = std::format("WikipediaService: No pages returned for '{}'", .message = std::format(
"WikipediaService: No pages returned for '{}'",
std::string(query))}); std::string(query))});
} }
this->extract_cache_.emplace(cache_key, ""); this->extract_cache_.emplace(cache_key, "");
@@ -106,11 +107,10 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
if (!page_val.is_object()) { if (!page_val.is_object()) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Warn,
{.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment, .phase = PipelinePhase::Enrichment,
.message = .message = std::format(
std::format("WikipediaService: Unexpected page format for '{}'", "WikipediaService: Unexpected page format for '{}'",
std::string(query))}); std::string(query))});
} }
return {}; return {};
@@ -121,7 +121,8 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
// Handle 404/Missing status // Handle 404/Missing status
if (page.contains("missing")) { if (page.contains("missing")) {
if (logger_) { if (logger_) {
logger_->Log({.level = LogLevel::Warn, logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment, .phase = PipelinePhase::Enrichment,
.message = std::format("WikipediaService: Page '{}' does not exist", .message = std::format("WikipediaService: Page '{}' does not exist",
std::string(query))}); std::string(query))});
@@ -134,11 +135,10 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
if ((extract_ptr == nullptr) || !extract_ptr->is_string()) { if ((extract_ptr == nullptr) || !extract_ptr->is_string()) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Warn,
{.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment, .phase = PipelinePhase::Enrichment,
.message = .message = std::format(
std::format("WikipediaService: No extract string found for '{}'", "WikipediaService: No extract string found for '{}'",
std::string(query))}); std::string(query))});
} }
this->extract_cache_.emplace(cache_key, ""); this->extract_cache_.emplace(cache_key, "");
@@ -148,7 +148,8 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
// 4. Success // 4. Success
std::string extract(extract_ptr->as_string()); std::string extract(extract_ptr->as_string());
if (logger_) { if (logger_) {
logger_->Log({.level = LogLevel::Info, logger_->Log(
{.level = LogLevel::Info,
.phase = PipelinePhase::Enrichment, .phase = PipelinePhase::Enrichment,
.message = std::format("WikipediaService: Fetched {} chars for '{}'", .message = std::format("WikipediaService: Fetched {} chars for '{}'",
extract.size(), std::string(query))}); extract.size(), std::string(query))});

View File

@@ -1,6 +1,6 @@
/** /**
* @file wikipedia/get_summary.cc * @file wikipedia/get_summary.cc
* @brief WikipediaService::GetLocationContext() implementation. * @brief WikipediaEnrichmentService::GetLocationContext() implementation.
*/ */
#include <chrono> #include <chrono>
@@ -45,7 +45,8 @@ std::string WikipediaEnrichmentService::GetLocationContext(
if (logger_) { if (logger_) {
logger_->Log({.level = LogLevel::Info, logger_->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Enrichment, .phase = PipelinePhase::Enrichment,
.message = std::format("Done fetching for {}. Sleeping for 10 seconds.", .message = std::format(
"Done fetching for {}. Sleeping for 10 seconds.",
location_query)}); location_query)});
} }
std::this_thread::sleep_for(10s); std::this_thread::sleep_for(10s);

View File

@@ -1,6 +1,6 @@
/** /**
* @file services/wikipedia/wikipedia_service.cc * @file services/wikipedia/wikipedia_service.cc
* @brief WikipediaService constructor implementation. * @brief WikipediaEnrichmentService constructor implementation.
*/ */
#include "services/enrichment/wikipedia_service.h" #include "services/enrichment/wikipedia_service.h"

View File

@@ -44,8 +44,8 @@ PromptDirectory::PromptDirectory(const std::filesystem::path& prompt_dir,
// Scenario 4: directory must be readable (probe with directory_iterator). // Scenario 4: directory must be readable (probe with directory_iterator).
std::filesystem::directory_iterator probe(prompt_dir_, ec); std::filesystem::directory_iterator probe(prompt_dir_, ec);
if (ec) { if (ec) {
throw std::runtime_error( throw std::runtime_error(std::format(
std::format("PromptDirectory: prompt directory is not readable: {} ({})", "PromptDirectory: prompt directory is not readable: {} ({})",
prompt_dir_.string(), ec.message())); prompt_dir_.string(), ec.message()));
} }
@@ -84,14 +84,17 @@ std::string PromptDirectory::Load(std::string_view key) {
file.close(); file.close();
if (content.empty()) { if (content.empty()) {
throw std::runtime_error(std::format("PromptDirectory: prompt file for key '{}' is empty: {}", throw std::runtime_error(
std::format("PromptDirectory: prompt file for key '{}' is empty: {}",
key_str, file_path.string())); key_str, file_path.string()));
} }
if (logger_) { if (logger_) {
logger_->Log({.level = LogLevel::Info, logger_->Log(
{.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = std::format("[PromptDirectory] Loaded prompt '{}' from '{}' ({} chars)", .message = std::format(
"[PromptDirectory] Loaded prompt '{}' from '{}' ({} chars)",
key_str, file_path.string(), content.size())}); key_str, file_path.string(), content.size())});
} }

View File

@@ -33,7 +33,7 @@ void ResetStatement(SqliteStatementHandle& statement) {
} }
void Bind(const SqliteStatementHandle& statement, void Bind(const SqliteStatementHandle& statement,
const BindParam<std::string_view>& param) { const BoundParam<std::string_view>& param) {
const auto byte_count = param.value.size(); const auto byte_count = param.value.size();
if (byte_count > static_cast<std::size_t>(std::numeric_limits<int>::max())) { if (byte_count > static_cast<std::size_t>(std::numeric_limits<int>::max())) {
ThrowSqliteError(sqlite3_db_handle(statement.get()), param.action); ThrowSqliteError(sqlite3_db_handle(statement.get()), param.action);
@@ -60,7 +60,7 @@ void Bind(const SqliteStatementHandle& statement,
} }
void Bind(const SqliteStatementHandle& statement, void Bind(const SqliteStatementHandle& statement,
const BindParam<double>& param) { const BoundParam<double>& param) {
if (sqlite3_bind_double(statement.get(), param.index, param.value) != if (sqlite3_bind_double(statement.get(), param.index, param.value) !=
SQLITE_OK) { SQLITE_OK) {
ThrowSqliteError(sqlite3_db_handle(statement.get()), param.action); ThrowSqliteError(sqlite3_db_handle(statement.get()), param.action);
@@ -68,7 +68,7 @@ void Bind(const SqliteStatementHandle& statement,
} }
void Bind(const SqliteStatementHandle& statement, void Bind(const SqliteStatementHandle& statement,
const BindParam<sqlite3_int64>& param) { const BoundParam<sqlite3_int64>& param) {
if (sqlite3_bind_int64(statement.get(), param.index, param.value) != if (sqlite3_bind_int64(statement.get(), param.index, param.value) !=
SQLITE_OK) { SQLITE_OK) {
ThrowSqliteError(sqlite3_db_handle(statement.get()), param.action); ThrowSqliteError(sqlite3_db_handle(statement.get()), param.action);

View File

@@ -18,8 +18,8 @@ std::filesystem::path SqliteExportService::BuildDatabasePath() const {
std::filesystem::path candidate = output_path_ / base_filename; std::filesystem::path candidate = output_path_ / base_filename;
for (int suffix = 1; std::filesystem::exists(candidate); ++suffix) { for (int suffix = 1; std::filesystem::exists(candidate); ++suffix) {
candidate = output_path_ / candidate = output_path_ / std::filesystem::path(
std::filesystem::path(std::format("biergarten_seed_{}-{}.sqlite", std::format("biergarten_seed_{}-{}.sqlite",
run_timestamp_utc_, suffix)); run_timestamp_utc_, suffix));
} }

View File

@@ -1,6 +1,6 @@
/** /**
* @file services/sqlite/process_record.cc * @file services/sqlite/process_record.cc
* @brief SqliteExportService::ProcessRecord(GeneratedBrewery) implementation * @brief SqliteExportService::ProcessRecord() implementation
* and the shared location-resolution helper. * and the shared location-resolution helper.
*/ */
@@ -38,61 +38,61 @@ sqlite3_int64 SqliteExportService::ResolveLocationId(const Location& location) {
} }
const std::string local_languages_json = const std::string local_languages_json =
sqlite_export_service_internal::SerializeVector( sqlite_export_service_internal::SerializeVector(location.local_languages);
location.local_languages);
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_location_stmt_, insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationCityBindIndex, .index = sqlite_export_service_internal::kLocationCityBindIndex,
.value = location.city, .value = location.city,
.action = "Failed to bind SQLite location city"}); .action = "Failed to bind SQLite location city"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_location_stmt_, insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = .index =
sqlite_export_service_internal::kLocationStateProvinceBindIndex, sqlite_export_service_internal::kLocationStateProvinceBindIndex,
.value = location.state_province, .value = location.state_province,
.action = "Failed to bind SQLite location state/province"}); .action = "Failed to bind SQLite location state/province"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_location_stmt_, insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationIso31662BindIndex, .index = sqlite_export_service_internal::kLocationIso31662BindIndex,
.value = location.iso3166_2, .value = location.iso3166_2,
.action = "Failed to bind SQLite location ISO 3166-2 code"}); .action = "Failed to bind SQLite location ISO 3166-2 code"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_location_stmt_, insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationCountryBindIndex, .index = sqlite_export_service_internal::kLocationCountryBindIndex,
.value = location.country, .value = location.country,
.action = "Failed to bind SQLite location country"}); .action = "Failed to bind SQLite location country"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_location_stmt_, insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationIso31661BindIndex, .index = sqlite_export_service_internal::kLocationIso31661BindIndex,
.value = location.iso3166_1, .value = location.iso3166_1,
.action = "Failed to bind SQLite location ISO 3166-1 code"}); .action = "Failed to bind SQLite location ISO 3166-1 code"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_location_stmt_, insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationLanguagesBindIndex, .index = sqlite_export_service_internal::kLocationLanguagesBindIndex,
.value = local_languages_json, .value = local_languages_json,
.action = "Failed to bind SQLite location languages"}); .action = "Failed to bind SQLite location languages"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_location_stmt_, insert_location_stmt_,
sqlite_export_service_internal::BindParam{ sqlite_export_service_internal::BoundParam{
.index = sqlite_export_service_internal::kLocationLatitudeBindIndex, .index = sqlite_export_service_internal::kLocationLatitudeBindIndex,
.value = location.latitude, .value = location.latitude,
.action = "Failed to bind SQLite location latitude"}); .action = "Failed to bind SQLite location latitude"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_location_stmt_, insert_location_stmt_,
sqlite_export_service_internal::BindParam{ sqlite_export_service_internal::BoundParam{
.index = sqlite_export_service_internal::kLocationLongitudeBindIndex, .index = sqlite_export_service_internal::kLocationLongitudeBindIndex,
.value = location.longitude, .value = location.longitude,
.action = "Failed to bind SQLite location longitude"}); .action = "Failed to bind SQLite location longitude"});
sqlite_export_service_internal::StepStatement( sqlite_export_service_internal::StepStatement(
db_handle_, insert_location_stmt_, "Failed to insert SQLite location row"); db_handle_, insert_location_stmt_,
"Failed to insert SQLite location row");
const sqlite3_int64 location_id = const sqlite3_int64 location_id =
sqlite_export_service_internal::LastInsertRowId(db_handle_); sqlite_export_service_internal::LastInsertRowId(db_handle_);
@@ -102,7 +102,7 @@ sqlite3_int64 SqliteExportService::ResolveLocationId(const Location& location) {
return location_id; return location_id;
} }
uint64_t SqliteExportService::ProcessRecord(const GeneratedBrewery& brewery) { uint64_t SqliteExportService::ProcessRecord(const BreweryRecord& brewery) {
if (db_handle_ == nullptr || !transaction_open_) { if (db_handle_ == nullptr || !transaction_open_) {
throw std::runtime_error("SQLite export service is not initialized"); throw std::runtime_error("SQLite export service is not initialized");
} }
@@ -111,32 +111,36 @@ uint64_t SqliteExportService::ProcessRecord(const GeneratedBrewery& brewery) {
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_stmt_, insert_brewery_stmt_,
sqlite_export_service_internal::BindParam<sqlite3_int64>{ sqlite_export_service_internal::BoundParam<sqlite3_int64>{
.index = sqlite_export_service_internal::kBreweryLocationIdBindIndex, .index = sqlite_export_service_internal::kBreweryLocationIdBindIndex,
.value = location_id, .value = location_id,
.action = "Failed to bind SQLite brewery location id"}); .action = "Failed to bind SQLite brewery location id"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_stmt_, insert_brewery_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kBreweryEnglishNameBindIndex, .index = sqlite_export_service_internal::kBreweryEnglishNameBindIndex,
.value = brewery.brewery.name_en, .value = brewery.brewery.name_en,
.action = "Failed to bind SQLite brewery English name"}); .action = "Failed to bind SQLite brewery English name"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_stmt_, insert_brewery_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal:: .index = sqlite_export_service_internal::
kBreweryEnglishDescriptionBindIndex, kBreweryEnglishDescriptionBindIndex,
.value = brewery.brewery.description_en, .value = brewery.brewery.description_en,
.action = "Failed to bind SQLite brewery English description"}); .action = "Failed to bind SQLite brewery English description"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_stmt_, insert_brewery_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kBreweryLocalNameBindIndex, .index = sqlite_export_service_internal::kBreweryLocalNameBindIndex,
.value = brewery.brewery.name_local, .value = brewery.brewery.name_local,
.action = "Failed to bind SQLite brewery local name"}); .action = "Failed to bind SQLite brewery local name"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_stmt_, insert_brewery_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = .index =
sqlite_export_service_internal::kBreweryLocalDescriptionBindIndex, sqlite_export_service_internal::kBreweryLocalDescriptionBindIndex,
.value = brewery.brewery.description_local, .value = brewery.brewery.description_local,

View File

@@ -1,6 +1,6 @@
/** /**
* @file services/sqlite/process_user_record.cc * @file services/sqlite/process_user_record.cc
* @brief SqliteExportService::ProcessRecord(GeneratedUser) implementation. * @brief SqliteExportService::ProcessRecord(UserRecord) implementation.
*/ */
#include <stdexcept> #include <stdexcept>
@@ -8,7 +8,7 @@
#include "services/database/sqlite_export_service.h" #include "services/database/sqlite_export_service.h"
#include "services/database/sqlite_export_service_helpers.h" #include "services/database/sqlite_export_service_helpers.h"
uint64_t SqliteExportService::ProcessRecord(const GeneratedUser& user) { uint64_t SqliteExportService::ProcessRecord(const UserRecord& user) {
if (db_handle_ == nullptr || !transaction_open_) { if (db_handle_ == nullptr || !transaction_open_) {
throw std::runtime_error("SQLite export service is not initialized"); throw std::runtime_error("SQLite export service is not initialized");
} }
@@ -17,64 +17,58 @@ uint64_t SqliteExportService::ProcessRecord(const GeneratedUser& user) {
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_stmt_, insert_user_stmt_,
sqlite_export_service_internal::BindParam<sqlite3_int64>{ sqlite_export_service_internal::BoundParam<sqlite3_int64>{
.index = sqlite_export_service_internal::kUserLocationIdBindIndex, .index = sqlite_export_service_internal::kUserLocationIdBindIndex,
.value = location_id, .value = location_id,
.action = "Failed to bind SQLite user location id"}); .action = "Failed to bind SQLite user location id"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_stmt_, insert_user_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kUserFirstNameBindIndex, .index = sqlite_export_service_internal::kUserFirstNameBindIndex,
.value = user.user.first_name, .value = user.user.first_name,
.action = "Failed to bind SQLite user first name"}); .action = "Failed to bind SQLite user first name"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_stmt_, insert_user_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kUserLastNameBindIndex, .index = sqlite_export_service_internal::kUserLastNameBindIndex,
.value = user.user.last_name, .value = user.user.last_name,
.action = "Failed to bind SQLite user last name"}); .action = "Failed to bind SQLite user last name"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_stmt_, insert_user_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kUserGenderBindIndex, .index = sqlite_export_service_internal::kUserGenderBindIndex,
.value = user.user.gender, .value = user.user.gender,
.action = "Failed to bind SQLite user gender"}); .action = "Failed to bind SQLite user gender"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_stmt_, insert_user_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kUserUsernameBindIndex, .index = sqlite_export_service_internal::kUserUsernameBindIndex,
.value = user.user.username, .value = user.user.username,
.action = "Failed to bind SQLite user username"}); .action = "Failed to bind SQLite user username"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_stmt_, insert_user_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kUserBioBindIndex, .index = sqlite_export_service_internal::kUserBioBindIndex,
.value = user.user.bio, .value = user.user.bio,
.action = "Failed to bind SQLite user bio"}); .action = "Failed to bind SQLite user bio"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_stmt_, insert_user_stmt_,
sqlite_export_service_internal::BindParam<double>{ sqlite_export_service_internal::BoundParam<double>{
.index = sqlite_export_service_internal::kUserActivityWeightBindIndex, .index = sqlite_export_service_internal::kUserActivityWeightBindIndex,
.value = static_cast<double>(user.user.activity_weight), .value = static_cast<double>(user.user.activity_weight),
.action = "Failed to bind SQLite user activity weight"}); .action = "Failed to bind SQLite user activity weight"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_stmt_, insert_user_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kUserEmailBindIndex, .index = sqlite_export_service_internal::kUserEmailBindIndex,
.value = user.email, .value = user.email,
.action = "Failed to bind SQLite user email"}); .action = "Failed to bind SQLite user email"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_stmt_, insert_user_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kUserDateOfBirthBindIndex, .index = sqlite_export_service_internal::kUserDateOfBirthBindIndex,
.value = user.date_of_birth, .value = user.date_of_birth,
.action = "Failed to bind SQLite user date of birth"}); .action = "Failed to bind SQLite user date of birth"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserPasswordBindIndex,
.value = user.password,
.action = "Failed to bind SQLite user password"});
sqlite_export_service_internal::StepStatement( sqlite_export_service_internal::StepStatement(
db_handle_, insert_user_stmt_, "Failed to insert SQLite user row"); db_handle_, insert_user_stmt_, "Failed to insert SQLite user row");

View File

@@ -48,18 +48,17 @@ std::string HttpWebClient::Get(const std::string& url) {
const httplib::Result result = client.Get(path); const httplib::Result result = client.Get(path);
if (!result) { if (!result) {
throw std::runtime_error(std::format( throw std::runtime_error(
"[HttpWebClient] Request failed for URL: {} — {}", url, std::format("[HttpWebClient] Request failed for URL: {} — {}", url,
httplib::to_string(result.error()))); httplib::to_string(result.error())));
} }
if (result->status < kSuccessMin || result->status >= kSuccessMax) { if (result->status < kSuccessMin || result->status >= kSuccessMax) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Error,
{.level = LogLevel::Error,
.phase = PipelinePhase::Enrichment, .phase = PipelinePhase::Enrichment,
.message = .message = std::format(
std::format("[HttpWebClient] Request failed for URL: {}", url)}); "[HttpWebClient] Request failed for URL: {}", url)});
} }
throw std::runtime_error(std::format("[HttpWebClient] HTTP {} for URL: {}", throw std::runtime_error(std::format("[HttpWebClient] HTTP {} for URL: {}",
result->status, url)); result->status, url));

1
web/docs Symbolic link
View File

@@ -0,0 +1 @@
/Users/aaronpo/projects/the-biergarten-app/docs/website