18 Commits

Author SHA1 Message Date
Aaron Po
98f97b8e2c add symlink to web docs 2026-06-27 23:34:30 -04:00
Aaron Po
2adc1eae38 update pipeline results viewer 2026-06-27 23:32:08 -04:00
Aaron Po
9d2ce67a78 Run pipeling, update sqllite file 2026-06-27 22:40:22 -04:00
Aaron Po
d1ad1eb397 rename json loader 2026-06-27 15:16:34 -04:00
Aaron Po
9137f4bddb Do full Gemma-4 run of pipeline, transfer sqllite file to public dir of preview 2026-06-23 22:02:17 -04:00
Aaron Po
6883b05824 remove password field 2026-06-23 01:39:57 -04:00
Aaron Po
b631c0f5db add results viewer 2026-06-23 01:27:37 -04:00
Aaron Po
9a9ffd59e6 Formatted src/biergarten_pipeline_orchestrator 2026-06-23 01:04:52 -04:00
Aaron Po
35a5c0785c documentation updates, remove superfluous comments 2026-06-23 01:04:46 -04:00
Aaron Po
e590cf2543 edit .clang-format 2026-06-23 00:50:27 -04:00
Aaron Po
3b32766a83 edits 2026-06-23 00:48:20 -04:00
Aaron Po
8d6b83673b Update doxygen comments 2026-06-23 00:45:32 -04:00
Aaron Po
22b3f2b3f9 cleanup gbnf grammars 2026-06-23 00:38:53 -04:00
Aaron Po
ac701d661a Remove logger from JSON export 2026-06-23 00:19:28 -04:00
Aaron Po
3f978f4de4 Cleanup user and brewery generation exception logic 2026-06-21 23:09:29 -04:00
Aaron Po
ad97b0ea6c format 2026-06-21 22:14:42 -04:00
Aaron Po
4de0ea6638 Persist generated users to SQLite and code cleanup 2026-06-21 22:14:42 -04:00
Aaron Po
51b40a39c9 Add user generation feature 2026-06-21 22:14:42 -04:00
48 changed files with 486 additions and 948 deletions

View File

@@ -111,11 +111,11 @@ persona/bio generation later.
The full multinational dataset is kept as-is (106 countries for forenames, The full multinational dataset is kept as-is (106 countries for forenames,
75 for surnames) rather than trimmed to `locations.json`'s current country 75 for surnames) rather than trimmed to `locations.json`'s current country
list, so it doesn't need re-sourcing if more countries are added later. list, so it doesn't need re-sourcing if more countries are added later.
`SampleName()` (a free helper in `generate_users.cc`) returns no result for `NamesByCountry::SampleName()` returns no result for a country present in
a country present in neither file; of the countries in `locations.json`, neither file; of the countries in `locations.json`, that's currently `KE`,
that's currently `KE`, `SE`, `SG`, `TH`, `VN`, and `ZA``GenerateUsers` `SE`, `SG`, `TH`, `VN`, and `ZA``GenerateUsers` skips cities in those
skips cities in those countries the same way brewery generation skips countries the same way brewery generation skips cities whose enrichment
cities whose enrichment lookup fails. lookup fails.
--- ---

View File

@@ -1,9 +1,8 @@
# Biergarten Pipeline # Biergarten Pipeline
A C++20 command-line pipeline that samples city records from local JSON, A C++20 command-line pipeline that samples city records from local JSON,
enriches each with Wikipedia context, and generates bilingual brewery names enriches each with Wikipedia context, and generates bilingual brewery names and
and descriptions plus locale-grounded user profiles via a local GGUF model or descriptions via a local GGUF model or a deterministic mock.
a deterministic mock.
> **This pipeline produces AI-generated data.** It is not a source of truth for > **This pipeline produces AI-generated data.** It is not a source of truth for
> brewing techniques, cultural representation, or local-language accuracy. See > brewing techniques, cultural representation, or local-language accuracy. See
@@ -46,7 +45,6 @@ step.
| Beer reviews and ratings | Stable brewery fixtures with enough context to anchor review pages | | Beer reviews and ratings | Stable brewery fixtures with enough context to anchor review pages |
| Social follow relationships | Repeatable brewery entities for feeds, follows, and saved lists | | Social follow relationships | Repeatable brewery entities for feeds, follows, and saved lists |
| Geospatial brewery experiences | Latitude, longitude, and country-level metadata | | Geospatial brewery experiences | Latitude, longitude, and country-level metadata |
| User accounts and profiles | Locale-grounded names, bios, and an auth-ready email/date-of-birth pair for seeding real accounts |
--- ---
@@ -220,46 +218,35 @@ environment variables listed above to match your run.
### Pipeline Stages ### Pipeline Stages
| Stage | Implementation | | Stage | Implementation |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | -------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| Load | `ICuratedDataService` (`CuratedJsonDataService`) reads `locations.json`, `personas.json`, `forenames-by-country.json`, and `surnames-by-country.json` (paths supplied via a `CuratedDataFilePaths` DTO at construction) into typed records, caching each after its first load. `--mocked` runs use `MockCuratedDataService`'s fixed in-memory dataset instead. | | Load | `JsonLoader::LoadLocations()` reads `locations.json` into typed `Location` records. |
| Sample | `BiergartenPipelineOrchestrator::QueryCitiesWithCountries()` samples `--location-count` locations per run (default `10`). | | Sample | `BiergartenPipelineOrchestrator::QueryCitiesWithCountries()` samples `--location-count` locations per run (default `10`). |
| Enrich | `WikipediaEnrichmentService` fetches brewing and beer-related context. Keeps going when a lookup fails. `--mocked` runs use `MockEnrichmentService` instead and skip Wikipedia entirely. | | Enrich | `WikipediaEnrichmentService` fetches brewing and beer-related context. Keeps going when a lookup fails. `--mocked` runs use `MockEnrichmentService` instead and skip Wikipedia entirely. |
| Generate Users | `GenerateUsers()` samples a persona and a forename/surname pair per enriched city (skipping countries with no name data), then `MockGenerator` or `LlamaGenerator` produces a username, bio, and activity weight around the sampled name. | | Generate | `MockGenerator` or `LlamaGenerator` produces brewery names and descriptions in English and the local language. |
| Generate Breweries | `MockGenerator` or `LlamaGenerator` produces brewery names and descriptions in English and the local language. | | Store | `SqliteExportService` writes each successful brewery into a fresh dated `.sqlite` database with normalized location and brewery tables. |
| Store | `SqliteExportService` writes each successful user and brewery into a fresh dated `.sqlite` database with normalized `locations`, `users`, and `breweries` tables. | | Log | `spdlog` writes results and warnings to the console. |
| Log | `spdlog` writes results and warnings to the console. |
If name sampling, enrichment, or generation fails for a city, that city is If enrichment or generation fails for a city, that city is skipped and the
skipped and the pipeline continues. `GenerateUsers()` runs before pipeline continues.
`GenerateBreweries()` in `BiergartenPipelineOrchestrator::Run()`.
### Key Components ### Key Components
- `src/main.cc` — argument parsing and Boost.DI composition root. - `src/main.cc` — argument parsing and Boost.DI composition root.
- `CuratedJsonDataService` — implements `ICuratedDataService`; takes a - `JsonLoader` — validates curated location input.
`CuratedDataFilePaths` DTO (locations/personas/forenames/surnames paths) in
its constructor, then parses and validates curated location, persona, and
forename/surname JSON, memoizing each result after its first load on a
given instance. `MockCuratedDataService` is the in-memory substitute (4
fixed locations, 3 personas, and name data for `US`/`DE`/`FR`/`BE`) used in
`--mocked` runs.
- `WikipediaEnrichmentService` — queries Wikipedia extracts, caches results, - `WikipediaEnrichmentService` — queries Wikipedia extracts, caches results,
returns empty context on failure. `MockEnrichmentService` is the no-op returns empty context on failure. `MockEnrichmentService` is the no-op
substitute used in `--mocked` runs. substitute used in `--mocked` runs.
- `LlamaGenerator` — formats prompts for Gemma 4, validates JSON output for - `LlamaGenerator` — formats prompts for Gemma 4, validates JSON output, retries
both `GenerateBrewery` and `GenerateUser`, retries malformed responses up malformed responses up to three times with corrective feedback in the
to three times with corrective feedback in the retry prompt. The token retry prompt. The token budget is fixed across attempts; it is not raised
budget is fixed across attempts; it is not raised automatically on automatically on truncation.
truncation. - `MockGenerator` — stable hash-based output so the same city input always
- `MockGenerator` — stable hash-based output so the same city/persona/name produces the same brewery.
input always produces the same brewery or user. - `SqliteExportService` — creates a dated SQLite file per run and persists each
- `SqliteExportService` — creates a dated SQLite file per run and persists successful brewery into normalized tables.
each successful user and brewery into normalized tables.
- Brewery payloads include English and local-language name and description - Brewery payloads include English and local-language name and description
fields. User payloads carry a sampled first/last name and gender, an fields.
LLM-generated username/bio/activity weight, and a programmatically
generated (not LLM-authored) unique email and date of birth.
### Runtime Behaviour ### Runtime Behaviour
@@ -279,18 +266,6 @@ valid output on the first pass; the retry path is kept for resilience.
`MockGenerator` uses stable hashes for repeatable output in demos and Storybook `MockGenerator` uses stable hashes for repeatable output in demos and Storybook
runs. runs.
`CuratedJsonDataService` memoizes each of `LoadLocations()`, `LoadPersonas()`,
`LoadForenamesByCountry()`, and `LoadSurnamesByCountry()` independently the
first time each is called, since `BiergartenPipelineOrchestrator` owns a
single `ICuratedDataService` instance for the whole run — later calls return
the cached result instead of re-parsing.
`GenerateUsers()` samples a forename/surname pair per city via `SampleName()`,
keyed by the city's ISO 3166-1 code. Countries present in `locations.json`
but absent from either name fixture (currently `KE`, `SE`, `SG`, `TH`, `VN`,
`ZA`) are skipped, the same way a failed enrichment or generation call skips
a city — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section.
### Process Flow - Activity Diagram ### Process Flow - Activity Diagram
![An activity diagram](./diagrams/current/output/activity.svg) ![An activity diagram](./diagrams/current/output/activity.svg)
@@ -303,10 +278,9 @@ a city — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section.
## Generated Output ## Generated Output
Each successful run stores a `BreweryRecord` pair with the source location Each successful run stores a `GeneratedBrewery` pair with the source location
and a `BreweryResult` payload, and a `UserRecord` pair with the source and a `BreweryResult` payload. The same generated records are also written to a
location and a `UserResult` payload. The same generated records are also fresh SQLite export file named with the current UTC timestamp.
written to a fresh SQLite export file named with the current UTC timestamp.
| Field | Meaning | | Field | Meaning |
| ------------------- | ------------------------------------------ | | ------------------- | ------------------------------------------ |
@@ -315,17 +289,6 @@ written to a fresh SQLite export file named with the current UTC timestamp.
| `name_local` | Brewery name in the local language. | | `name_local` | Brewery name in the local language. |
| `description_local` | Brewery description in the local language. | | `description_local` | Brewery description in the local language. |
| Field | Meaning |
| ----------------- | ----------------------------------------------------------------- |
| `first_name` | Sampled forename, copied from the curated name data (not LLM-invented). |
| `last_name` | Sampled surname, copied from the curated name data (not LLM-invented). |
| `gender` | Gender associated with the sampled forename in the source dataset. |
| `username` | LLM-generated handle. |
| `bio` | LLM-generated short biography. |
| `activity_weight` | Relative check-in/activity weight, reserved for a future J-curve activity profile. |
| `email` | Unique `@thebiergarten.app` address, generated programmatically from the sampled name. |
| `date_of_birth` | Randomized date of birth (age 19-48), generated programmatically. |
The log dump also includes city, country, state or province, ISO subdivision The log dump also includes city, country, state or province, ISO subdivision
code, latitude, and longitude for each entry. code, latitude, and longitude for each entry.
@@ -404,17 +367,11 @@ Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present.
## Fixture Strategy ## Fixture Strategy
- `--mocked` for stable fixtures, repeatable screenshots, and Storybook runs. - `--mocked` for stable fixtures, repeatable screenshots, and Storybook runs.
`MockCuratedDataService` swaps in for `CuratedJsonDataService`, so no
fixture files need to be present on disk.
- `--model` when geographically grounded content matters for demos. - `--model` when geographically grounded content matters for demos.
- Keep `locations.json` structured enough to support discovery and future - Keep `locations.json` structured enough to support discovery and future
filtering. filtering.
- `personas.json`, `forenames-by-country.json`, and - Treat SQLite output as seed material for the app's brewery domain, not
`surnames-by-country.json` are curated/vendored fixture data, not production data.
LLM-generated — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset
section for provenance.
- Treat SQLite output as seed material for the app's brewery and user
domains, not production data.
--- ---
@@ -425,9 +382,6 @@ Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present.
| `tooling/pipeline/includes/` | Public headers and shared models. | | `tooling/pipeline/includes/` | Public headers and shared models. |
| `tooling/pipeline/src/` | Implementation files. | | `tooling/pipeline/src/` | Implementation files. |
| `tooling/pipeline/locations.json` | Curated city input copied into the build tree. | | `tooling/pipeline/locations.json` | Curated city input copied into the build tree. |
| `tooling/pipeline/personas.json` | Curated user persona archetypes copied into the build tree. |
| `tooling/pipeline/forenames-by-country.json` | Vendored (CC0) forename data by ISO 3166-1 country code. |
| `tooling/pipeline/surnames-by-country.json` | Vendored (CC0) surname data by ISO 3166-1 country code. |
| `tooling/pipeline/prompts/` | System prompts used by the model-backed path. | | `tooling/pipeline/prompts/` | System prompts used by the model-backed path. |
| `tooling/pipeline/runpod/` | Dockerfile, launcher, and RunPod pod template. | | `tooling/pipeline/runpod/` | Dockerfile, launcher, and RunPod pod template. |
| `docs/pipeline/diagrams/` | Architecture and pipeline diagrams. | | `docs/pipeline/diagrams/` | Architecture and pipeline diagrams. |
@@ -442,9 +396,6 @@ Paths below are relative to `tooling/pipeline/`.
- `src/main.cc` — argument parsing and DI composition root. - `src/main.cc` — argument parsing and DI composition root.
- `src/biergarten_pipeline_orchestrator/` — orchestration, sampling, logging, - `src/biergarten_pipeline_orchestrator/` — orchestration, sampling, logging,
and export. and export.
- `src/services/curated_data/``CuratedJsonDataService`, the file-backed
`ICuratedDataService`, and `MockCuratedDataService`, the in-memory
`ICuratedDataService` used in `--mocked` runs.
- `src/services/enrichment/wikipedia/` — enrichment service and cache. - `src/services/enrichment/wikipedia/` — enrichment service and cache.
- `src/services/sqlite/` — SQLite export implementation. - `src/services/sqlite/` — SQLite export implementation.
- `src/data_generation/llama/` — local inference, prompt loading, output - `src/data_generation/llama/` — local inference, prompt loading, output
@@ -456,12 +407,11 @@ Paths below are relative to `tooling/pipeline/`.
## Next Steps ## Next Steps
The pipeline currently produces city-aware brewery and user records and The pipeline currently produces city-aware brewery records and dated SQLite
dated SQLite exports. The next passes add additional fixture types so the exports. The next passes add additional fixture types so the app can exercise
app can exercise the full brewery and social domains without live data. For the full brewery domain without live data. For the detailed engineering
the detailed engineering breakdown of what's needed to reach the breakdown of what's needed to reach the architecture in
architecture in [`diagrams/planned/`](./diagrams/planned/), see [`diagrams/planned/`](./diagrams/planned/), see [ROADMAP.md](./ROADMAP.md).
[ROADMAP.md](./ROADMAP.md).
### Testing — Very High Priority ### Testing — Very High Priority
@@ -483,6 +433,12 @@ Generate catalog entries with style, ABV, IBU, color, aroma notes, and food
pairing hints. Link beers back to breweries and cities. Keep style coverage wide pairing hints. Link beers back to breweries and cities. Keep style coverage wide
enough to exercise search, sort, and category filters. enough to exercise search, sort, and category filters.
### User Generation
Generate user profiles with stable names, bios, locale hints, and preference
signals. Include stable IDs for downstream fixture joins. Keep output
deterministic for screenshots while allowing larger randomized batches.
### Check-In System ### Check-In System
Produce timestamped check-in events between users and breweries. Use a J-curve Produce timestamped check-in events between users and breweries. Use a J-curve

View File

@@ -39,80 +39,47 @@ architecture needs.
not LLM-invented. not LLM-invented.
- [x] Add `Name` (`first_name`, `last_name`, `gender`) — the sampled result - [x] Add `Name` (`first_name`, `last_name`, `gender`) — the sampled result
handed to `DataGenerator::GenerateUser`. Add `ForenameEntry` (`name`, handed to `DataGenerator::GenerateUser`. Add `ForenameEntry` (`name`,
`gender`). Landed as a flatter shape than originally planned here: no `gender`) and a `NamesByCountry` class (not a bare typedef) that owns
`NamesByCountry` wrapper class. `curated_data_service.h` instead two maps — `std::unordered_map<std::string, std::vector<ForenameEntry>>`
declares `ForenameList = std::vector<ForenameEntry>` and and `std::unordered_map<std::string, std::vector<std::string>>`, both
`SurnameList = std::vector<std::string>`, and keyed by ISO 3166-1 code — and exposes `SampleName(iso3166_1, rng)`.
`ICuratedDataService` exposes two maps keyed by ISO 3166-1 code —
`ForenamesByCountryMap = unordered_map<string, ForenameList>` and
`SurnamesByCountryMap = unordered_map<string, SurnameList>`
directly (see §2). Sampling is
a free function, `SampleName(forenames_by_country,
surnames_by_country, iso3166_1, rng)`, in
`generate_users.cc`'s anonymous namespace, not a method on a class.
Pairing a forename with a surname happens at sample time so gender Pairing a forename with a surname happens at sample time so gender
(present per-forename in the source data) is never discarded the way a (present per-forename in the source data) is never discarded the way a
pre-flattened `{first_name, last_name}` list would lose it; see §2. pre-flattened `{first_name, last_name}` list would lose it; see
`LoadNamesByCountry()` below.
- [ ] Extend `GeneratedBrewery` with `brewery_id`, `context_completeness`, - [ ] Extend `GeneratedBrewery` with `brewery_id`, `context_completeness`,
`metadata` (currently just `{location, brewery}`). `metadata` (currently just `{location, brewery}`).
- [ ] Add `GeneratedBeer`, `GeneratedCheckin`, `GeneratedRating`, - [ ] Add `GeneratedBeer`, `GeneratedCheckin`, `GeneratedRating`,
`GeneratedFollow` aggregate structs. `GeneratedFollow` aggregate structs.
- [x] Add `UserRecord` (`location`, `user : UserResult`, `email`, - [x] Add `GeneratedUser` aggregate struct. Carries `email`, `date_of_birth`,
`date_of_birth`) as the current stand-in for the planned and `password` — programmatically generated by the orchestrator, never
`GeneratedUser``email` and `date_of_birth` are programmatically LLM-authored — so a downstream auth-account seeding consumer (outside
generated by the orchestrator, never LLM-authored, so a downstream this repo tree's docs) can register real accounts from the pipeline's
auth-account seeding consumer can register real accounts from the SQLite export. Skips `user_id`/`metadata` for now, matching
pipeline's SQLite export. No `password`, `user_id`, or `metadata` `GeneratedBrewery`'s current (pre-`brewery_id`/`metadata`) shape. Not
field yet, matching `BreweryRecord`'s equally pre-`metadata` shape. yet wired into `IExportService`/`SqliteExportService``GenerateUsers`
Already wired into `IExportService`/`SqliteExportService`: a `users` currently only logs and holds `generated_users_` in memory; no `users`
table exists and `GenerateUsers()` exports each successful record via table exists in the SQLite export yet (see §5).
`ProcessRecord(const UserRecord&)` (see §5) in addition to logging and
holding `generated_users_` in memory. Still missing vs. the planned
`GeneratedUser`: `user_id`, `password`, and `metadata`.
- [x] Add `UserPersona` (`name`, `description`, `style_affinities`). - [x] Add `UserPersona` (`name`, `description`, `style_affinities`).
## 2. Data Preloading ## 2. Data Preloading
`tooling/pipeline/includes/services/curated_data/`, fixture files. `tooling/pipeline/includes/json_handling/`, fixture files.
- [x] Extract an interface; have `CuratedJsonDataService` implement it. - [ ] Extract a `DataPreloader` interface; have `JsonLoader` implement it.
Landed as `ICuratedDataService` `JsonLoader` now also exposes static `LoadPersonas()` and
(`services/curated_data/curated_data_service.h`), not `DataPreloader` `LoadNamesByCountry()` (see below), but as additional static methods,
`LoadLocations()`, `LoadPersonas()`, `LoadForenamesByCountry()`, and not yet behind a formal interface.
`LoadSurnamesByCountry()` are all virtual methods returning `const&`
and take no arguments. `CuratedJsonDataService`
(`services/curated_data/curated_json_data_service.h`, originally
landed as `JsonLoader` under `json_handling/` before being renamed and
moved) takes a `CuratedDataFilePaths` DTO (`locations_path`,
`personas_path`, `forenames_path`, `surnames_path`) in its
constructor rather than a path per `Load*()` call, and memoizes each
result in a private `cache` struct (`locations`, `personas`,
`forenames_by_country`, `surnames_by_country`), so a second call on
the same instance returns the cached result instead of re-parsing —
safe because `CuratedJsonDataService` outlives every call site (owned
by `BiergartenPipelineOrchestrator` via
`unique_ptr<ICuratedDataService>` for the whole run). `main.cc`'s DI
injector also gained a `MockCuratedDataService` (fixed in-memory
data: 4 locations across `US`/`DE`/`FR`/`BE`, 3 personas, and
matching forename/surname sets for those four countries), bound in
place of `CuratedJsonDataService` under `--mocked`, mirroring the
existing `MockEnrichmentService`/`MockGenerator` pattern.
- [ ] Add `LoadBeerStyles()`. `beer-styles.json` already exists in the repo - [ ] Add `LoadBeerStyles()`. `beer-styles.json` already exists in the repo
and is already copied into the Docker image and is already copied into the Docker image
(`runpod/Dockerfile`), but no loader reads it yet, and the native (`runpod/Dockerfile`), but no loader reads it yet, and the native
CMake build doesn't copy it into `build/` at all — `CMakeLists.txt`'s CMake build doesn't copy it into `build/` at all — `CMakeLists.txt`'s
"Runtime Assets" step only copies `locations.json` and `prompts/`. "Runtime Assets" step only copies `locations.json` and `prompts/`.
- [x] Add `LoadPersonas()` and author `personas.json` (doesn't exist yet). - [x] Add `LoadPersonas()` and author `personas.json` (doesn't exist yet).
- [x] Add name-by-country loading, parsing - [x] Add `LoadNamesByCountry(forenames_filepath, surnames_filepath)`,
`tooling/pipeline/forenames-by-country.json` and parsing `tooling/pipeline/forenames-by-country.json` and
`surnames-by-country.json`. Landed as two separate methods — `surnames-by-country.json` into a `NamesByCountry`. Both files are
`LoadForenamesByCountry()` and `LoadSurnamesByCountry()` — each vendored verbatim (unmodified, full multinational coverage) from
returning a flat `ForenamesByCountryMap` / `SurnamesByCountryMap`
(aliases for `unordered_map<string, ForenameList>` /
`unordered_map<string, SurnameList>`) keyed by ISO 3166-1 code, rather
than one combined `LoadNamesByCountry() : NamesByCountry` call. Both
files are vendored verbatim (unmodified,
full multinational coverage) from
`sigpwned/popular-names-by-country-dataset` (CC0) — see `sigpwned/popular-names-by-country-dataset` (CC0) — see
ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for
provenance. Deliberately not pre-paired or filtered down to just the provenance. Deliberately not pre-paired or filtered down to just the
@@ -169,19 +136,12 @@ Entirely new — no `includes/policy/` (or equivalent) directory exists today.
`tooling/pipeline/includes/services/database/`, `src/services/sqlite/`. `tooling/pipeline/includes/services/database/`, `src/services/sqlite/`.
- [x] `IExportService::ProcessRecord(const UserRecord&)` and - [ ] Extend `IExportService` with `ProcessBeer`, `ProcessUser`,
`SqliteExportService`'s `users` table (see `ProcessCheckin`, `ProcessRating`, `ProcessFollow` (today only
`kCreateUsersTableSql` / `kInsertUserSql` in `ProcessRecord(const GeneratedBrewery&)` exists).
`includes/services/database/sqlite_statement_helpers.h`) are - [ ] Add `beers`, `users`, `checkins`, `ratings`, `follows` tables to
implemented — `GenerateUsers()` exports every successful user `SqliteExportService::InitializeSchema()` — see
alongside its resolved `location_id`. `kCreateLocationsTableSql` / `kCreateBreweriesTableSql` in
- [ ] Extend `IExportService` with `ProcessBeer`, `ProcessCheckin`,
`ProcessRating`, `ProcessFollow` (today `ProcessRecord(const
BreweryRecord&)` and `ProcessRecord(const UserRecord&)` exist).
- [ ] Add `beers`, `checkins`, `ratings`, `follows` tables to
`SqliteExportService::InitializeSchema()` (`locations`, `breweries`,
and `users` already exist) — see `kCreateLocationsTableSql` /
`kCreateBreweriesTableSql` / `kCreateUsersTableSql` in
`includes/services/database/sqlite_statement_helpers.h` for the `includes/services/database/sqlite_statement_helpers.h` for the
existing pattern to follow. existing pattern to follow.
- [ ] Add a `brewery_cache_` alongside the existing `location_cache_`, and - [ ] Add a `brewery_cache_` alongside the existing `location_cache_`, and
@@ -227,8 +187,7 @@ Entirely new — no `includes/policy/` (or equivalent) directory exists today.
## 8. Fixtures / Build ## 8. Fixtures / Build
- [x] Author `personas.json`, `forenames-by-country.json`, and - [ ] Author `personas.json` and `names-by-country.json` (see §2).
`surnames-by-country.json` (see §2).
- [ ] Fix the `beer-styles.json` build-tree gap: add it to the "Runtime - [ ] Fix the `beer-styles.json` build-tree gap: add it to the "Runtime
Assets" `configure_file` step in `CMakeLists.txt` so native builds and Assets" `configure_file` step in `CMakeLists.txt` so native builds and
the Docker image agree on what's available at runtime. the Docker image agree on what's available at runtime.

View File

@@ -93,77 +93,6 @@ while (For each sampled Location?) is (Remaining cities)
endif endif
endwhile (Done) endwhile (Done)
|#EAF0E8|BiergartenPipelineOrchestrator|
:GenerateUsers(enriched_cities);
|#E2EBDC|JsonLoader|
:LoadPersonas("personas.json");
:LoadForenamesByCountry("forenames-by-country.json");
:LoadSurnamesByCountry("surnames-by-country.json");
note right
Each call is cached after its first parse.
MockCuratedDataService (--mocked) returns a
fixed in-memory dataset instead and skips
these three JSON files entirely.
end note
|#EAF0E8|BiergartenPipelineOrchestrator|
while (For each EnrichedCity?) is (Remaining cities)
:SampleName(forenames_by_country,\n surnames_by_country, location.iso3166_1);
if (Names available for country?) then (no)
:Log warning, skip city;
else (yes)
:Sample UserPersona (uniform random);
|#E5EDE1|DataGenerator|
if (Generator Mode) then (MockGenerator)
:DeterministicHash & Format;
else (LlamaGenerator)
:prompt_directory_->Load("USER_GENERATION");
note right
Resolves to USER_GENERATION.md inside --prompt-dir.
end note
repeat
:Infer(system_prompt, user_prompt, max_tokens, kUserJsonGrammar);
:ValidateUserJson(raw, user);
if (Is JSON Valid?) then (yes)
break
else (no)
:Attempt++;
:Append validation error to retry prompt;
endif
repeat while (Attempt < 3?) is (yes)
endif
|#EAF0E8|BiergartenPipelineOrchestrator|
if (Generation successful?) then (yes)
:BuildEmail(name, used_email_local_parts);
:GenerateDateOfBirth(rng);
note right
email and date_of_birth are generated
programmatically, never LLM-authored.
end note
|#E0EAE0|SqliteExportService|
:ProcessRecord(UserRecord);
if (Location in cache?) then (yes)
:Reuse location_id;
else (no)
:Insert Location & Cache ID;
endif
:Insert User (FK: location_id);
if (Exception caught during insert?) then (yes)
|#EAF0E8|BiergartenPipelineOrchestrator|
:Log warning "SQLite export failed";
else (no)
endif
else (no)
:Log warning "Generation failed, skipping...";
endif
endif
endwhile (Done)
|#EAF0E8|BiergartenPipelineOrchestrator| |#EAF0E8|BiergartenPipelineOrchestrator|
:GenerateBreweries(enriched_cities); :GenerateBreweries(enriched_cities);

View File

@@ -30,14 +30,11 @@ class BiergartenPipelineOrchestrator {
- context_service_ : std::unique_ptr<IEnrichmentService> - context_service_ : std::unique_ptr<IEnrichmentService>
- generator_ : std::unique_ptr<DataGenerator> - generator_ : std::unique_ptr<DataGenerator>
- exporter_ : std::unique_ptr<IExportService> - exporter_ : std::unique_ptr<IExportService>
- curated_data_service_ : std::unique_ptr<ICuratedDataService>
- application_options_ : ApplicationOptions - application_options_ : ApplicationOptions
- generated_breweries_ : std::vector<BreweryRecord> - generated_breweries_ : std::vector<GeneratedBrewery>
- generated_users_ : std::vector<UserRecord>
+ Run() : bool + Run() : bool
- QueryCitiesWithCountries() : std::vector<Location> - QueryCitiesWithCountries() : std::vector<Location>
- GenerateBreweries(cities : std::span<const EnrichedCity>) : void - GenerateBreweries(cities : std::span<const EnrichedCity>) : void
- GenerateUsers(cities : std::span<const EnrichedCity>) : void
- LogResults() : void - LogResults() : void
} }
@@ -116,14 +113,13 @@ class HttpWebClient {
interface DataGenerator <<interface>> { interface DataGenerator <<interface>> {
+ GenerateBrewery(location : const Location&, region_context : const std::string&) : BreweryResult + GenerateBrewery(location : const Location&, region_context : const std::string&) : BreweryResult
+ GenerateUser(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult + GenerateUser(locale : const std::string&) : UserResult
} }
class MockGenerator { class MockGenerator {
+ GenerateBrewery(location : const Location&, region_context : const std::string&) : BreweryResult + GenerateBrewery(...) : BreweryResult
+ GenerateUser(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult + GenerateUser(...) : UserResult
- DeterministicHash(location : const Location&) : size_t - DeterministicHash(location : const Location&) : size_t
- DeterministicHash(location : const Location&, persona : const UserPersona&, name : const Name&) : size_t
} }
class LlamaGenerator { class LlamaGenerator {
@@ -157,49 +153,13 @@ class PromptDirectory {
+ Load(key : std::string_view) : std::string + Load(key : std::string_view) : std::string
} }
interface ICuratedDataService <<interface>> {
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
}
class JsonLoader { class JsonLoader {
- cache_ : cache + {static} LoadLocations(filepath : const std::filesystem::path&) : std::vector<Location>
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
} }
note right of JsonLoader
Each Load* method memoizes its result in
cache_ on first call; later calls on the
same instance return the cached value
without re-parsing.
end note
class MockCuratedDataService {
- locations_ : std::vector<Location>
- personas_ : std::vector<UserPersona>
- forenames_by_country_ : std::unordered_map<std::string, forename_list>
- surnames_by_country_ : std::unordered_map<std::string, surname_list>
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
}
note right of MockCuratedDataService
Fixed 4-location / 3-persona dataset
(US, DE, FR, BE) used in --mocked runs;
filepath arguments are ignored.
end note
interface IExportService <<interface>> { interface IExportService <<interface>> {
+ Initialize() : void + Initialize() : void
+ ProcessRecord(brewery : const BreweryRecord&) : uint64_t + ProcessRecord(brewery : const GeneratedBrewery&) : void
+ ProcessRecord(user : const UserRecord&) : uint64_t
+ Finalize() : void + Finalize() : void
} }
@@ -207,18 +167,15 @@ class SqliteExportService {
- date_time_provider_ : std::unique_ptr<IDateTimeProvider> - date_time_provider_ : std::unique_ptr<IDateTimeProvider>
- run_timestamp_utc_ : std::string - run_timestamp_utc_ : std::string
- database_path_ : std::filesystem::path - database_path_ : std::filesystem::path
- db_handle_ : SqliteDatabaseHandle - db_handle_ : sqlite3*
- insert_location_stmt_ : SqliteStatementHandle - insert_location_stmt_ : sqlite3_stmt*
- insert_brewery_stmt_ : SqliteStatementHandle - insert_brewery_stmt_ : sqlite3_stmt*
- insert_user_stmt_ : SqliteStatementHandle
- transaction_open_ : bool - transaction_open_ : bool
- location_cache_ : std::unordered_map<std::string, sqlite3_int64> - location_cache_ : std::unordered_map<std::string, sqlite3_int64>
+ Initialize() : void + Initialize() : void
+ ProcessRecord(brewery : const BreweryRecord&) : uint64_t + ProcessRecord(brewery : const GeneratedBrewery&) : void
+ ProcessRecord(user : const UserRecord&) : uint64_t
+ Finalize() : void + Finalize() : void
- InitializeSchema() : void - InitializeSchema() : void
- ResolveLocationId(location : const Location&) : sqlite3_int64
} }
interface IDateTimeProvider <<interface>> { interface IDateTimeProvider <<interface>> {
@@ -234,7 +191,6 @@ BiergartenPipelineOrchestrator *-- ILogger : owns
BiergartenPipelineOrchestrator *-- IEnrichmentService : owns BiergartenPipelineOrchestrator *-- IEnrichmentService : owns
BiergartenPipelineOrchestrator *-- DataGenerator : owns BiergartenPipelineOrchestrator *-- DataGenerator : owns
BiergartenPipelineOrchestrator *-- IExportService : owns BiergartenPipelineOrchestrator *-- IExportService : owns
BiergartenPipelineOrchestrator *-- ICuratedDataService : owns
LogEntry *-- LogLevel LogEntry *-- LogLevel
LogEntry *-- PipelinePhase LogEntry *-- PipelinePhase
@@ -259,8 +215,7 @@ LlamaGenerator *-- IPromptDirectory : uses
IPromptFormatter <|.. Gemma4JinjaPromptFormatter : implements IPromptFormatter <|.. Gemma4JinjaPromptFormatter : implements
IPromptDirectory <|.. PromptDirectory : implements IPromptDirectory <|.. PromptDirectory : implements
ICuratedDataService <|.. JsonLoader : implements BiergartenPipelineOrchestrator ..> JsonLoader : uses
ICuratedDataService <|.. MockCuratedDataService : implements
IExportService <|.. SqliteExportService : implements IExportService <|.. SqliteExportService : implements
SqliteExportService *-- IDateTimeProvider : owns SqliteExportService *-- IDateTimeProvider : owns

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 86 KiB

After

Width:  |  Height:  |  Size: 55 KiB

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 87 KiB

After

Width:  |  Height:  |  Size: 71 KiB

View File

@@ -43,18 +43,10 @@ fork again
:EnrichmentService::PreWarmLocationCache(sampled_locations); :EnrichmentService::PreWarmLocationCache(sampled_locations);
end fork end fork
fork fork
:JsonLoader::LoadForenamesByCountry(\n "forenames-by-country.json"); :JsonLoader::LoadNamesByCountry(\n "forenames-by-country.json",\n "surnames-by-country.json");
fork again
:JsonLoader::LoadSurnamesByCountry(\n "surnames-by-country.json");
fork again fork again
:JsonLoader::LoadPersonas("personas.json"); :JsonLoader::LoadPersonas("personas.json");
end fork end fork
note right
Each call is memoized after its first parse
(implemented today). MockCuratedDataService
returns a fixed in-memory dataset instead
under --mocked, skipping these JSON files.
end note
' ═══════════════════════════════════════════ ' ═══════════════════════════════════════════
' PHASE 0 — USER GENERATION ' PHASE 0 — USER GENERATION
@@ -90,15 +82,11 @@ fork again
ibu_preference, checkin_weight. ibu_preference, checkin_weight.
end note end note
:SampleName(forenames_by_country,\n surnames_by_country, location.iso3166_1, rng); :NamesByCountry::SampleName(\n location.iso3166_1);
note right note right
Deterministic lookup -- no LLM involved. Deterministic lookup -- no LLM involved.
Free function (implemented today in Name selected from pre-keyed table
generate_users.cc), not a class method.
Name selected from pre-keyed maps
and passed into the generation prompt. and passed into the generation prompt.
Skips the city if either map has no
entry for iso3166_1.
end note end note
:GenerateUser(enriched_city, persona, sampled_name)\nvia DataGenerator; :GenerateUser(enriched_city, persona, sampled_name)\nvia DataGenerator;

View File

@@ -86,30 +86,27 @@ package "Domain: Models" {
+ gender : std::string + gender : std::string
} }
note right of ForenameEntry class NamesByCountry {
forename_list = std::unordered_set<ForenameEntry> - forenames_by_country_ : std::unordered_map<std::string, std::vector<ForenameEntry>>
surname_list = std::unordered_set<std::string> - surnames_by_country_ : std::unordered_map<std::string, std::vector<std::string>>
(aliases declared in curated_data_service.h). + SampleName(iso3166_1 : const std::string&,\n rng : std::mt19937&) : std::optional<Name>
ICuratedDataService::LoadForenamesByCountry() / }
LoadSurnamesByCountry() each return a flat
std::unordered_map<std::string, forename_list> note right of NamesByCountry
or <std::string, surname_list> keyed by ISO Backed by two source-shaped maps,
3166-1 country code -- no NamesByCountry keyed by ISO 3166-1 country code,
wrapper class. SampleName(forenames_by_country, sourced from popular-names-by-country-dataset
surnames_by_country, iso3166_1, rng) is a free (CC0) -- see ETHICS-AND-KNOWN-ISSUES.md.
function, not a method, so it can be called by Gender is preserved per forename so it can
any per-city worker without owning the maps. be threaded through to persona/bio generation
Sourced from popular-names-by-country-dataset later; surnames carry no gender in the source
(CC0) -- see ETHICS-AND-KNOWN-ISSUES.md. Gender data. Pairing happens at sample time, not at
is preserved per forename so it can be threaded fixture-authoring time, so no information from
through to persona/bio generation later; surnames the source dataset is discarded up front.
carry no gender in the source data. Pairing
happens at sample time, not at fixture-authoring
time, so no information from the source dataset
is discarded up front.
end note end note
ForenameEntry ..> Name : SampleName() produces NamesByCountry *-- "0..*" ForenameEntry
NamesByCountry ..> Name : produces
class CheckinResult { class CheckinResult {
+ checked_in_at : std::string + checked_in_at : std::string
@@ -368,43 +365,20 @@ package "Infrastructure: Pipeline Channel" {
package "Infrastructure: Data Preloading" { package "Infrastructure: Data Preloading" {
interface ICuratedDataService <<interface>> { interface DataPreloader <<interface>> {
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>& + LoadLocations(filepath : const std::filesystem::path&) : std::vector<Location>
+ LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>& + LoadBeerStyles(filepath : const std::filesystem::path&) : std::vector<BeerStyle>
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>& + LoadPersonas(filepath : const std::filesystem::path&) : std::vector<UserPersona>
+ LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>& + LoadNamesByCountry(forenames_filepath : const std::filesystem::path&,\n surnames_filepath : const std::filesystem::path&) : NamesByCountry
+ LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
} }
class JsonLoader { class JsonLoader {
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>& + LoadLocations(filepath : const std::filesystem::path&) : std::vector<Location>
+ LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>& + LoadBeerStyles(filepath : const std::filesystem::path&) : std::vector<BeerStyle>
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>& + LoadPersonas(filepath : const std::filesystem::path&) : std::vector<UserPersona>
+ LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>& + LoadNamesByCountry(forenames_filepath : const std::filesystem::path&,\n surnames_filepath : const std::filesystem::path&) : NamesByCountry
+ LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
} }
note right of JsonLoader
Each Load* call is memoized after its first
parse -- implemented today, ahead of the rest
of this package (LoadBeerStyles is still planned).
end note
class MockCuratedDataService {
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
+ LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
}
note right of MockCuratedDataService
Fixed in-memory dataset used in --mocked mode,
implemented today for Locations/Personas/
Forenames/Surnames; would need a LoadBeerStyles
fixture too once that method is implemented.
end note
} }
package "Infrastructure: Enrichment" { package "Infrastructure: Enrichment" {
@@ -546,7 +520,7 @@ package "Infrastructure: Data Export" {
} }
class BiergartenPipelineOrchestrator { class BiergartenPipelineOrchestrator {
- curated_data_service_ : std::unique_ptr<ICuratedDataService> - preloader_ : std::unique_ptr<DataPreloader>
- enrichment_service_ : std::unique_ptr<EnrichmentService> - enrichment_service_ : std::unique_ptr<EnrichmentService>
- generator_ : std::unique_ptr<DataGenerator> - generator_ : std::unique_ptr<DataGenerator>
- logger_ : std::unique_ptr<Logger> - logger_ : std::unique_ptr<Logger>
@@ -574,7 +548,7 @@ class BiergartenPipelineOrchestrator {
} }
' --- Orchestration Aggregations (Services & Strategies) --- ' --- Orchestration Aggregations (Services & Strategies) ---
BiergartenPipelineOrchestrator *-- ICuratedDataService BiergartenPipelineOrchestrator *-- DataPreloader
BiergartenPipelineOrchestrator *-- EnrichmentService BiergartenPipelineOrchestrator *-- EnrichmentService
BiergartenPipelineOrchestrator *-- DataGenerator BiergartenPipelineOrchestrator *-- DataGenerator
BiergartenPipelineOrchestrator *-- ExportService BiergartenPipelineOrchestrator *-- ExportService
@@ -593,8 +567,7 @@ BiergartenPipelineOrchestrator *-- "0..*" GeneratedCheckin : checkin_pool_
BiergartenPipelineOrchestrator *-- "0..*" GeneratedFollow : follow_pool_ BiergartenPipelineOrchestrator *-- "0..*" GeneratedFollow : follow_pool_
' --- Interfaces & Implementations --- ' --- Interfaces & Implementations ---
ICuratedDataService <|.. JsonLoader DataPreloader <|.. JsonLoader
ICuratedDataService <|.. MockCuratedDataService
Logger <|.. PipelineLogger Logger <|.. PipelineLogger
ContextStrategy <|.. BreweryContextStrategy ContextStrategy <|.. BreweryContextStrategy
ContextStrategy <|.. BeerContextStrategy ContextStrategy <|.. BeerContextStrategy

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 142 KiB

After

Width:  |  Height:  |  Size: 139 KiB

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 240 KiB

After

Width:  |  Height:  |  Size: 234 KiB

View File

@@ -139,7 +139,6 @@ FetchContent_MakeAvailable(cpp-httplib)
# 5. Executable & Sources # 5. Executable & Sources
add_executable(${PROJECT_NAME} add_executable(${PROJECT_NAME}
includes/services/enrichment/mock_enrichment.h includes/services/enrichment/mock_enrichment.h
includes/services/curated_data/mock_curated_data_service.h
includes/json_handling/pretty_print.h) includes/json_handling/pretty_print.h)
# --- Entry point --- # --- Entry point ---
@@ -149,12 +148,12 @@ target_sources(${PROJECT_NAME} PRIVATE
# --- json_handling --- # --- json_handling ---
target_sources(${PROJECT_NAME} PRIVATE target_sources(${PROJECT_NAME} PRIVATE
src/services/curated_data/curated_json_data_service.cc src/json_handling/json_loader.cc
) )
# --- services: curated_data --- # --- data_model ---
target_sources(${PROJECT_NAME} PRIVATE target_sources(${PROJECT_NAME} PRIVATE
src/services/curated_data/mock_curated_data_service.cc src/data_model/names_by_country.cc
) )
# --- application_options --- # --- application_options ---

View File

@@ -1,74 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_H_
#define BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_H_
/**
* @file biergarten_pipeline.h
* @brief Master umbrella header — includes every public header in the
* Biergarten pipeline.
*/
// --- orchestrator ---
#include "biergarten_pipeline_orchestrator.h"
// --- concurrency ---
#include "concurrency/bounded_channel.h"
// --- data_generation ---
#include "data_generation/data_generator.h"
#include "data_generation/json_grammars.h"
#ifndef BIERGARTEN_MOCK_ONLY
#include "data_generation/llama_generator.h"
#include "data_generation/llama_generator_helpers.h"
#endif
#include "data_generation/mock_generator.h"
#include "data_generation/prompt_formatting/gemma4_jinja_prompt_formatter.h"
#include "data_generation/prompt_formatting/prompt_formatter.h"
// --- data_model ---
#include "data_model/generated_models.h"
#include "data_model/models.h"
// --- json_handling ---
#include "json_handling/pretty_print.h"
#include "services/curated_data/curated_json_data_service.h"
// --- llama backend ---
#ifndef BIERGARTEN_MOCK_ONLY
#include "llama_backend_state.h"
#endif
// --- services: curated_data ---
#include "services/curated_data/curated_data_service.h"
#include "services/curated_data/mock_curated_data_service.h"
// --- services: database ---
#include "services/database/export_service.h"
#include "services/database/sqlite_connection_helpers.h"
#include "services/database/sqlite_export_service.h"
#include "services/database/sqlite_export_service_helpers.h"
#include "services/database/sqlite_handle_types.h"
#include "services/database/sqlite_statement_helpers.h"
// --- services: datetime ---
#include "services/datetime/date_time_provider.h"
#include "services/datetime/timer.h"
// --- services: enrichment ---
#include "services/enrichment/enrichment_service.h"
#include "services/enrichment/mock_enrichment.h"
#include "services/enrichment/wikipedia_service.h"
// --- services: logging ---
#include "services/logging/log_dispatcher.h"
#include "services/logging/log_entry.h"
#include "services/logging/log_producer.h"
#include "services/logging/logger.h"
// --- services: prompting ---
#include "services/prompting/prompt_directory.h"
// --- web_client ---
#include "web_client/http_web_client.h"
#include "web_client/web_client.h"
#endif // BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_H_

View File

@@ -1,9 +1,12 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_ORCHESTRATOR_H_ #ifndef BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_DATA_GENERATOR_H_
#define BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_ORCHESTRATOR_H_ #define BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_DATA_GENERATOR_H_
/** /**
* @file biergarten_pipeline_orchestrator.h * @file biergarten_pipeline_orchestrator.h
* @brief Orchestration for end-to-end brewery and user data generation. * @brief Orchestration for end-to-end brewery and user data generation.
*
* Coordinates location loading, enrichment, and generation phases to produce
* a complete dataset, wiring dependencies selected at the composition root.
*/ */
#include <memory> #include <memory>
@@ -100,6 +103,10 @@ class BiergartenPipelineOrchestrator {
* @brief Generate users grounded in sampled names and personas for * @brief Generate users grounded in sampled names and personas for
* enriched cities. * enriched cities.
* *
* Loads personas.json / forenames-by-country.json /
* surnames-by-country.json itself, mirroring how QueryCitiesWithCountries()
* owns its own locations.json load.
*
* @param cities Span of enriched city data. * @param cities Span of enriched city data.
*/ */
void GenerateUsers(std::span<const EnrichedCity> cities); void GenerateUsers(std::span<const EnrichedCity> cities);
@@ -113,4 +120,4 @@ class BiergartenPipelineOrchestrator {
std::vector<UserRecord> generated_users_; std::vector<UserRecord> generated_users_;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_ORCHESTRATOR_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_DATA_GENERATOR_H_

View File

@@ -1,4 +1,4 @@
#include "concurrency/bounded_channel.h" #include "bounded_channel.h"
template <typename T> template <typename T>
void BoundedChannel<T>::Send(T item) { void BoundedChannel<T>::Send(T item) {

View File

@@ -1,23 +1,24 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_LLAMA_GENERATOR_H_ #ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_LLAMA_GENERATOR_H_
#define BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_LLAMA_GENERATOR_H_ #define BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_LLAMA_GENERATOR_H_
#include <filesystem>
/** /**
* @file data_generation/llama_generator.h * @file data_generation/llama_generator.h
* @brief llama.cpp-backed implementation of DataGenerator. * @brief llama.cpp-backed implementation of DataGenerator.
*/ */
#include <cstdint> #include <cstdint>
#include <filesystem>
#include <memory> #include <memory>
#include <random> #include <random>
#include <string> #include <string>
#include <string_view> #include <string_view>
#include "../services/prompting/prompt_directory.h"
#include "data_generation/data_generator.h" #include "data_generation/data_generator.h"
#include "data_generation/prompt_formatting/prompt_formatter.h" #include "data_generation/prompt_formatting/prompt_formatter.h"
#include "data_model/models.h" #include "data_model/models.h"
#include "services/logging/logger.h" #include "services/logging/logger.h"
#include "services/prompting/prompt_directory.h"
struct llama_model; struct llama_model;
struct llama_context; struct llama_context;

View File

@@ -10,7 +10,6 @@
#include <boost/program_options.hpp> #include <boost/program_options.hpp>
#include <cstdint> #include <cstdint>
#include <filesystem> #include <filesystem>
#include <functional>
#include <memory> #include <memory>
#include <optional> #include <optional>
#include <string> #include <string>
@@ -67,7 +66,7 @@ struct Location {
/** /**
* @brief A sampled first/last name pair, with the source forename's gender. * @brief A sampled first/last name pair, with the source forename's gender.
* *
* Produced by the SampleName() helper in generate_users.cc. * Produced by NamesByCountry::SampleName();
*/ */
struct Name { struct Name {
std::string first_name{}; std::string first_name{};
@@ -94,23 +93,8 @@ struct ForenameEntry {
* dataset (e.g. "M", "F"). * dataset (e.g. "M", "F").
*/ */
std::string gender{}; std::string gender{};
bool operator==(const ForenameEntry& other) const {
return name == other.name && gender == other.gender;
}
}; };
namespace std {
template <>
struct hash<ForenameEntry> {
size_t operator()(const ForenameEntry& entry) const noexcept {
const size_t name_hash = std::hash<std::string>{}(entry.name);
const size_t gender_hash = std::hash<std::string>{}(entry.gender);
return name_hash ^ (gender_hash << 1);
}
};
} // namespace std
/** /**
* @brief A persona archetype used to ground LLM-generated user bios. * @brief A persona archetype used to ground LLM-generated user bios.
*/ */

View File

@@ -0,0 +1,51 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_NAMES_BY_COUNTRY_H_
#define BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_NAMES_BY_COUNTRY_H_
/**
* @file data_model/names_by_country.h
* @brief Sampling over the names-by-country fixture data.
*/
#include <optional>
#include <random>
#include <string>
#include <unordered_map>
#include <vector>
#include "data_model/models.h"
/**
* @brief Samples locale-appropriate names from curated forename/surname
* fixture data, keyed by ISO 3166-1 country code.
*
* Forenames and surnames are kept in separate maps (mirroring the source
* dataset's own shape) so per-forename gender is preserved; pairing happens
* at sample time rather than being precomputed, so no information from the
* source dataset is discarded up front.
*/
class NamesByCountry {
public:
NamesByCountry(std::unordered_map<std::string, std::vector<ForenameEntry>>
forenames_by_country,
std::unordered_map<std::string, std::vector<std::string>>
surnames_by_country);
/**
* @brief Samples a first/last name pair for the given country.
*
* @param iso3166_1 ISO 3166-1 country code to sample for.
* @param rng Random source used for sampling.
* @return A sampled Name, or std::nullopt if the country has no forename
* or no surname entries.
*/
[[nodiscard]] std::optional<Name> SampleName(const std::string& iso3166_1,
std::mt19937& rng) const;
private:
std::unordered_map<std::string, std::vector<ForenameEntry>>
forenames_by_country_;
std::unordered_map<std::string, std::vector<std::string>>
surnames_by_country_;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_NAMES_BY_COUNTRY_H_

View File

@@ -0,0 +1,47 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_
#define BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_
/**
* @file json_handling/json_loader.h
* @brief JSON-backed implementation of ICuratedDataService.
*/
#include <filesystem>
#include <vector>
#include "data_model/models.h"
#include "data_model/names_by_country.h"
#include "services/curated_data/curated_data_service.h"
/**
* @brief Loads curated location, persona, and name data from JSON files.
*/
class JsonLoader final : public ICuratedDataService {
public:
JsonLoader() = default;
/**
* @brief Parses a JSON array file and returns all location records.
*/
std::vector<Location> LoadLocations(
const std::filesystem::path& filepath) override;
/**
* @brief Parses a JSON array file and returns all persona records.
*/
std::vector<UserPersona> LoadPersonas(
const std::filesystem::path& filepath) override;
/**
* @brief Parses the names-by-country fixture pair into a sampling-capable
* NamesByCountry.
*
* @param forenames_filepath Path to forenames-by-country.json.
* @param surnames_filepath Path to surnames-by-country.json.
*/
NamesByCountry LoadNamesByCountry(
const std::filesystem::path& forenames_filepath,
const std::filesystem::path& surnames_filepath) override;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_

View File

@@ -106,4 +106,4 @@ inline void PrettyPrint(std::ostream& outstream,
} }
} }
#endif // BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_PRETTY_PRINT_H_ #endif

View File

@@ -7,18 +7,10 @@
*/ */
#include <filesystem> #include <filesystem>
#include <unordered_map>
#include <unordered_set>
#include <vector> #include <vector>
#include "data_model/models.h" #include "data_model/models.h"
#include "data_model/names_by_country.h"
using ForenameList = std::vector<ForenameEntry>;
using SurnameList = std::vector<std::string>;
using LocationsList = std::vector<Location>;
using PersonasList = std::vector<UserPersona>;
using ForenamesByCountryMap = std::unordered_map<std::string, ForenameList>;
using SurnamesByCountryMap = std::unordered_map<std::string, SurnameList>;
/** /**
* @brief Interface for services that load curated data used to seed * @brief Interface for services that load curated data used to seed
@@ -37,14 +29,25 @@ class ICuratedDataService {
/** /**
* @brief Loads all curated location records. * @brief Loads all curated location records.
*/ */
virtual const LocationsList& LoadLocations() = 0; virtual std::vector<Location> LoadLocations(
const std::filesystem::path& filepath) = 0;
/** /**
* @brief Loads all curated persona records. * @brief Loads all curated persona records.
*/ */
virtual const PersonasList& LoadPersonas() = 0; virtual std::vector<UserPersona> LoadPersonas(
virtual const ForenamesByCountryMap& LoadForenamesByCountry() = 0; const std::filesystem::path& filepath) = 0;
virtual const SurnamesByCountryMap& LoadSurnamesByCountry() = 0;
/**
* @brief Loads the names-by-country fixture pair into a sampling-capable
* NamesByCountry.
*
* @param forenames_filepath Path to forenames-by-country.json.
* @param surnames_filepath Path to surnames-by-country.json.
*/
virtual NamesByCountry LoadNamesByCountry(
const std::filesystem::path& forenames_filepath,
const std::filesystem::path& surnames_filepath) = 0;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_CURATED_DATA_SERVICE_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_CURATED_DATA_SERVICE_H_

View File

@@ -1,63 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_
#define BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_
/**
* @file json_handling/json_loader.h
* @brief JSON-backed implementation of ICuratedDataService.
*/
#include <filesystem>
#include "data_model/models.h"
#include "services/curated_data/curated_data_service.h"
/**
* @brief File locations for the curated JSON fixtures consumed by
* CuratedJsonDataService.
*/
struct CuratedDataFilePaths {
std::filesystem::path locations_path;
std::filesystem::path personas_path;
std::filesystem::path forenames_path;
std::filesystem::path surnames_path;
};
/**
* @brief Loads curated location, persona, and name data from JSON files.
*/
class CuratedJsonDataService final : public ICuratedDataService {
struct cache {
LocationsList locations;
PersonasList personas;
ForenamesByCountryMap forenames_by_country;
SurnamesByCountryMap surnames_by_country;
cache() = default;
~cache() = default;
};
CuratedDataFilePaths filepaths_;
cache cache_;
public:
explicit CuratedJsonDataService(CuratedDataFilePaths filepaths);
/**
* @brief Parses a JSON array file and returns all location records.
*/
const LocationsList& LoadLocations() override;
/**
* @brief Parses a JSON array file and returns all persona records.
*/
const PersonasList& LoadPersonas() override;
const ForenamesByCountryMap& LoadForenamesByCountry() override;
/**
* @brief Parses a JSON file and returns all the forenames per country.
*/
const SurnamesByCountryMap& LoadSurnamesByCountry() override;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_

View File

@@ -1,38 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_MOCK_CURATED_DATA_SERVICE_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_MOCK_CURATED_DATA_SERVICE_H_
/**
* @file services/curated_data/mock_curated_data_service.h
* @brief In-memory ICuratedDataService backed by a small fixed dataset, used
* when file-backed curated data is disabled (mock mode).
*/
#include <filesystem>
#include "data_model/models.h"
#include "services/curated_data/curated_data_service.h"
/**
* @brief Curated data service returning a small fixed in-memory dataset in
* place of the JSON fixture files used by JsonLoader.
*/
class MockCuratedDataService final : public ICuratedDataService {
public:
MockCuratedDataService();
const LocationsList& LoadLocations() override;
const PersonasList& LoadPersonas() override;
const ForenamesByCountryMap& LoadForenamesByCountry() override;
const SurnamesByCountryMap& LoadSurnamesByCountry() override;
private:
LocationsList locations_;
PersonasList personas_;
ForenamesByCountryMap forenames_by_country_;
SurnamesByCountryMap surnames_by_country_;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_MOCK_CURATED_DATA_SERVICE_H_

View File

@@ -12,7 +12,7 @@
#include <string> #include <string>
#include <string_view> #include <string_view>
#include "services/database/sqlite_handle_types.h" #include "sqlite_handle_types.h"
namespace sqlite_export_service_internal { namespace sqlite_export_service_internal {

View File

@@ -11,10 +11,10 @@
#include <string> #include <string>
#include <unordered_map> #include <unordered_map>
#include "../datetime/date_time_provider.h"
#include "data_model/models.h" #include "data_model/models.h"
#include "services/database/export_service.h" #include "export_service.h"
#include "services/database/sqlite_export_service_helpers.h" #include "sqlite_export_service_helpers.h"
#include "services/datetime/date_time_provider.h"
/** /**
* @brief Persists generated brewery records into a fresh SQLite database. * @brief Persists generated brewery records into a fresh SQLite database.

View File

@@ -3,8 +3,8 @@
/* Umbrella header for backward compatibility. */ /* Umbrella header for backward compatibility. */
#include "services/database/sqlite_connection_helpers.h" #include "sqlite_connection_helpers.h"
#include "services/database/sqlite_handle_types.h" #include "sqlite_handle_types.h"
#include "services/database/sqlite_statement_helpers.h" #include "sqlite_statement_helpers.h"
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_SQLITE_EXPORT_SERVICE_HELPERS_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_SQLITE_EXPORT_SERVICE_HELPERS_H_

View File

@@ -24,10 +24,8 @@ using SqliteDatabaseHandle = std::unique_ptr<sqlite3, SqliteDatabaseDeleter>;
using SqliteStatementHandle = using SqliteStatementHandle =
std::unique_ptr<sqlite3_stmt, SqliteStatementDeleter>; std::unique_ptr<sqlite3_stmt, SqliteStatementDeleter>;
// Represents a parameter that is bound to a prepared SQLite statement.
// N.B. indices are 1 based.
template <typename T> template <typename T>
struct BoundParam { struct BindParam {
int index; int index;
T value; T value;
std::string_view action; std::string_view action;

View File

@@ -13,7 +13,7 @@
#include <string_view> #include <string_view>
#include <vector> #include <vector>
#include "services/database/sqlite_handle_types.h" #include "sqlite_handle_types.h"
namespace sqlite_export_service_internal { namespace sqlite_export_service_internal {
@@ -109,36 +109,30 @@ INSERT INTO users (
// sqlite3_bind_*() parameter indices are 1-based, matching the "?" // sqlite3_bind_*() parameter indices are 1-based, matching the "?"
// placeholder order in the SQL above. // placeholder order in the SQL above.
enum LocationBindIndex { inline constexpr int kLocationCityBindIndex = 1;
kLocationCityBindIndex = 1, inline constexpr int kLocationStateProvinceBindIndex = 2;
kLocationStateProvinceBindIndex, inline constexpr int kLocationIso31662BindIndex = 3;
kLocationIso31662BindIndex, inline constexpr int kLocationCountryBindIndex = 4;
kLocationCountryBindIndex, inline constexpr int kLocationIso31661BindIndex = 5;
kLocationIso31661BindIndex, inline constexpr int kLocationLanguagesBindIndex = 6;
kLocationLanguagesBindIndex, inline constexpr int kLocationLatitudeBindIndex = 7;
kLocationLatitudeBindIndex, inline constexpr int kLocationLongitudeBindIndex = 8;
kLocationLongitudeBindIndex,
};
enum BreweryBindIndex { inline constexpr int kBreweryLocationIdBindIndex = 1;
kBreweryLocationIdBindIndex = 1, inline constexpr int kBreweryEnglishNameBindIndex = 2;
kBreweryEnglishNameBindIndex, inline constexpr int kBreweryEnglishDescriptionBindIndex = 3;
kBreweryEnglishDescriptionBindIndex, inline constexpr int kBreweryLocalNameBindIndex = 4;
kBreweryLocalNameBindIndex, inline constexpr int kBreweryLocalDescriptionBindIndex = 5;
kBreweryLocalDescriptionBindIndex,
};
enum UserBindIndex { inline constexpr int kUserLocationIdBindIndex = 1;
kUserLocationIdBindIndex = 1, inline constexpr int kUserFirstNameBindIndex = 2;
kUserFirstNameBindIndex, inline constexpr int kUserLastNameBindIndex = 3;
kUserLastNameBindIndex, inline constexpr int kUserGenderBindIndex = 4;
kUserGenderBindIndex, inline constexpr int kUserUsernameBindIndex = 5;
kUserUsernameBindIndex, inline constexpr int kUserBioBindIndex = 6;
kUserBioBindIndex, inline constexpr int kUserActivityWeightBindIndex = 7;
kUserActivityWeightBindIndex, inline constexpr int kUserEmailBindIndex = 8;
kUserEmailBindIndex, inline constexpr int kUserDateOfBirthBindIndex = 9;
kUserDateOfBirthBindIndex,
};
SqliteStatementHandle PrepareStatement(const SqliteDatabaseHandle& db_handle, SqliteStatementHandle PrepareStatement(const SqliteDatabaseHandle& db_handle,
std::string_view sql, std::string_view sql,
@@ -147,13 +141,13 @@ SqliteStatementHandle PrepareStatement(const SqliteDatabaseHandle& db_handle,
void ResetStatement(SqliteStatementHandle& statement); void ResetStatement(SqliteStatementHandle& statement);
void Bind(const SqliteStatementHandle& statement, void Bind(const SqliteStatementHandle& statement,
const BoundParam<std::string_view>& param); const BindParam<std::string_view>& param);
void Bind(const SqliteStatementHandle& statement, void Bind(const SqliteStatementHandle& statement,
const BoundParam<double>& param); const BindParam<double>& param);
void Bind(const SqliteStatementHandle& statement, void Bind(const SqliteStatementHandle& statement,
const BoundParam<sqlite3_int64>& param); const BindParam<sqlite3_int64>& param);
void StepStatement(const SqliteDatabaseHandle& db_handle, void StepStatement(const SqliteDatabaseHandle& db_handle,
const SqliteStatementHandle& statement, const SqliteStatementHandle& statement,

View File

@@ -8,7 +8,7 @@
#include <string> #include <string>
#include "services/enrichment/enrichment_service.h" #include "enrichment_service.h"
/** /**
* @brief Enrichment service that returns no context for any location. * @brief Enrichment service that returns no context for any location.

View File

@@ -11,7 +11,7 @@
#include <string_view> #include <string_view>
#include <unordered_map> #include <unordered_map>
#include "services/enrichment/enrichment_service.h" #include "enrichment_service.h"
#include "services/logging/logger.h" #include "services/logging/logger.h"
#include "web_client/web_client.h" #include "web_client/web_client.h"

View File

@@ -6,8 +6,8 @@
* a bounded channel for later processing by the dispatcher. * a bounded channel for later processing by the dispatcher.
*/ */
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_PRODUCER_H_ #ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_CHANNEL_LOGGER_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_PRODUCER_H_ #define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_CHANNEL_LOGGER_H_
#include <string_view> #include <string_view>
@@ -50,4 +50,4 @@ class LogProducer final : public ILogger {
BoundedChannel<LogEntry>& channel_; BoundedChannel<LogEntry>& channel_;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_PRODUCER_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_CHANNEL_LOGGER_H_

View File

@@ -52,4 +52,4 @@ class HttpWebClient final : public WebClient {
std::shared_ptr<ILogger> logger_; std::shared_ptr<ILogger> logger_;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_WEB_CLIENT_HTTP_WEB_CLIENT_H_ #endif

View File

@@ -14,9 +14,10 @@ BiergartenPipelineOrchestrator::BiergartenPipelineOrchestrator(
std::unique_ptr<IExportService> exporter, std::unique_ptr<IExportService> exporter,
std::unique_ptr<ICuratedDataService> curated_data_service, std::unique_ptr<ICuratedDataService> curated_data_service,
const ApplicationOptions& application_options) const ApplicationOptions& application_options)
: logger_(std::move(logger)), : logger_(std::move(logger)),
context_service_(std::move(context_service)), context_service_(std::move(context_service)),
generator_(std::move(generator)), generator_(std::move(generator)),
exporter_(std::move(exporter)), exporter_(std::move(exporter)),
curated_data_service_(std::move(curated_data_service)), curated_data_service_(std::move(curated_data_service)),
application_options_(application_options) {} application_options_(application_options) {
}

View File

@@ -6,7 +6,6 @@
#include <cctype> #include <cctype>
#include <chrono> #include <chrono>
#include <format> #include <format>
#include <iterator>
#include <optional> #include <optional>
#include <random> #include <random>
#include <stdexcept> #include <stdexcept>
@@ -15,7 +14,7 @@
#include <unordered_set> #include <unordered_set>
#include "biergarten_pipeline_orchestrator.h" #include "biergarten_pipeline_orchestrator.h"
#include "services/curated_data/curated_json_data_service.h" #include "json_handling/json_loader.h"
#include "services/logging/logger.h" #include "services/logging/logger.h"
namespace { namespace {
@@ -25,7 +24,7 @@ std::string Sanitize(std::string_view value) {
for (const char character : value) { for (const char character : value) {
if (std::isalnum(static_cast<unsigned char>(character)) != 0) { if (std::isalnum(static_cast<unsigned char>(character)) != 0) {
out.push_back(static_cast<char>( out.push_back(static_cast<char>(
std::tolower(static_cast<unsigned char>(character)))); std::tolower(static_cast<unsigned char>(character))));
} }
} }
return out; return out;
@@ -70,37 +69,7 @@ std::string GenerateDateOfBirth(std::mt19937& rng) {
static_cast<unsigned>(birth_ymd.month()), static_cast<unsigned>(birth_ymd.month()),
static_cast<unsigned>(birth_ymd.day())); static_cast<unsigned>(birth_ymd.day()));
} }
} // namespace
std::optional<Name> SampleName(
const ForenamesByCountryMap& forenames_by_country,
const SurnamesByCountryMap& surnames_by_country,
const std::string& iso3166_1, std::mt19937& rng) {
const auto forenames_it = forenames_by_country.find(iso3166_1);
const auto surnames_it = surnames_by_country.find(iso3166_1);
if (forenames_it == forenames_by_country.end() ||
surnames_it == surnames_by_country.end() ||
forenames_it->second.empty() || surnames_it->second.empty()) {
return std::nullopt;
}
const ForenameList& forenames = forenames_it->second;
const SurnameList& surnames = surnames_it->second;
std::uniform_int_distribution<size_t> forename_dist(0, forenames.size() - 1);
std::uniform_int_distribution<size_t> surname_dist(0, surnames.size() - 1);
auto forename_it = forenames.begin();
std::advance(forename_it, forename_dist(rng));
auto surname_it = surnames.begin();
std::advance(surname_it, surname_dist(rng));
return Name{.first_name = forename_it->name,
.last_name = *surname_it,
.gender = forename_it->gender};
}
} // namespace
void BiergartenPipelineOrchestrator::GenerateUsers( void BiergartenPipelineOrchestrator::GenerateUsers(
std::span<const EnrichedCity> cities) { std::span<const EnrichedCity> cities) {
@@ -108,17 +77,17 @@ void BiergartenPipelineOrchestrator::GenerateUsers(
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::UserGeneration,
.message = "=== SAMPLE USER GENERATION ==="}); .message = "=== SAMPLE USER GENERATION ==="});
const PersonasList& personas = curated_data_service_->LoadPersonas(); const std::vector<UserPersona> personas =
curated_data_service_->LoadPersonas("personas.json");
if (personas.empty()) { if (personas.empty()) {
throw std::runtime_error( throw std::runtime_error(
"No personas available in personas.json for user generation"); "No personas available in personas.json for user generation");
} }
const ForenamesByCountryMap& forenames_by_country = const NamesByCountry names_by_country =
curated_data_service_->LoadForenamesByCountry(); curated_data_service_->LoadNamesByCountry(
const SurnamesByCountryMap& surnames_by_country = "forenames-by-country.json", "surnames-by-country.json");
curated_data_service_->LoadSurnamesByCountry();
std::mt19937 rng(std::random_device{}()); std::mt19937 rng(std::random_device{}());
std::uniform_int_distribution<size_t> persona_dist(0, personas.size() - 1); std::uniform_int_distribution<size_t> persona_dist(0, personas.size() - 1);
@@ -126,13 +95,14 @@ void BiergartenPipelineOrchestrator::GenerateUsers(
generated_users_.clear(); generated_users_.clear();
size_t skipped_count = 0; size_t skipped_count = 0;
size_t export_failed_count = 0; size_t export_failed_count = 0;
std::unordered_set<std::string> used_email_local_parts;
const auto generate_record = const auto generate_record =
[this, &rng, &skipped_count, &used_email_local_parts]( [this, &rng, &skipped_count](
const EnrichedCity& city, const UserPersona& persona, const EnrichedCity& city, const UserPersona& persona,
const Name& sampled_name) -> std::optional<UserRecord> { const Name& sampled_name) -> std::optional<UserRecord> {
try { try {
std::unordered_set<std::string> used_email_local_parts;
const UserResult user = const UserResult user =
generator_->GenerateUser(city, persona, sampled_name); generator_->GenerateUser(city, persona, sampled_name);
@@ -155,26 +125,25 @@ void BiergartenPipelineOrchestrator::GenerateUsers(
}; };
const auto export_record = [this, const auto export_record = [this,
&export_failed_count](const UserRecord& record) { &export_failed_count](const UserRecord& record) {
try { try {
exporter_->ProcessRecord(record); exporter_->ProcessRecord(record);
} catch (const std::exception& export_exception) { } catch (const std::exception& export_exception) {
++export_failed_count; ++export_failed_count;
logger_->Log( logger_->Log(
{.level = LogLevel::Warn, {.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::UserGeneration,
.message = std::format("[Pipeline] Generated user for '{}' ({}) but " .message = std::format("[Pipeline] Generated user for '{}' ({}) but "
"SQLite export failed: {}", "SQLite export failed: {}",
record.location.city, record.location.country, record.location.city, record.location.country,
export_exception.what())}); export_exception.what())});
} }
}; };
for (const auto& city : cities) { for (const auto& city : cities) {
const std::optional<Name> sampled_name = const std::optional<Name> sampled_name =
SampleName(forenames_by_country, surnames_by_country, names_by_country.SampleName(city.location.iso3166_1, rng);
city.location.iso3166_1, rng);
if (!sampled_name.has_value()) { if (!sampled_name.has_value()) {
++skipped_count; ++skipped_count;
@@ -202,11 +171,11 @@ void BiergartenPipelineOrchestrator::GenerateUsers(
if (skipped_count > 0) { if (skipped_count > 0) {
logger_->Log( logger_->Log(
{.level = LogLevel::Warn, {.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::UserGeneration,
.message = std::format( .message = std::format(
"[Pipeline] Skipped {} city/cities during user generation", "[Pipeline] Skipped {} city/cities during user generation",
skipped_count)}); skipped_count)});
} }
if (export_failed_count > 0) { if (export_failed_count > 0) {

View File

@@ -7,8 +7,8 @@
#include <chrono> #include <chrono>
#include <format> #include <format>
#include "../../includes/json_handling/pretty_print.h"
#include "biergarten_pipeline_orchestrator.h" #include "biergarten_pipeline_orchestrator.h"
#include "json_handling/pretty_print.h"
#include "services/logging/logger.h" #include "services/logging/logger.h"
void BiergartenPipelineOrchestrator::LogResults() const { void BiergartenPipelineOrchestrator::LogResults() const {

View File

@@ -6,12 +6,13 @@
#include <algorithm> #include <algorithm>
#include <chrono> #include <chrono>
#include <filesystem>
#include <format> #include <format>
#include <iterator> #include <iterator>
#include <random> #include <random>
#include "biergarten_pipeline_orchestrator.h" #include "biergarten_pipeline_orchestrator.h"
#include "services/curated_data/curated_json_data_service.h" #include "json_handling/json_loader.h"
#include "services/logging/logger.h" #include "services/logging/logger.h"
std::vector<Location> std::vector<Location>
@@ -20,8 +21,9 @@ BiergartenPipelineOrchestrator::QueryCitiesWithCountries() {
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = "=== GEOGRAPHIC DATA OVERVIEW ==="}); .message = "=== GEOGRAPHIC DATA OVERVIEW ==="});
const std::vector<Location>& all_locations = const std::filesystem::path locations_path = "locations.json";
curated_data_service_->LoadLocations();
auto all_locations = curated_data_service_->LoadLocations(locations_path);
const size_t sample_count = std::min( const size_t sample_count = std::min(
static_cast<size_t>(application_options_.pipeline.location_count), static_cast<size_t>(application_options_.pipeline.location_count),

View File

@@ -30,13 +30,10 @@ UserResult LlamaGenerator::GenerateUser(const EnrichedCity& city,
const std::string system_prompt = prompt_directory_->Load("USER_GENERATION"); const std::string system_prompt = prompt_directory_->Load("USER_GENERATION");
std::string user_prompt = std::format( std::string user_prompt = std::format(
"## NAME:\n\n{} {}\n\n" "## NAME:\n{} {}\n\n## GENDER:\n{}\n\n## CITY:\n{}\n\n## "
"## GENDER:\n\n{}\n\n" "COUNTRY:\n{}\n\n"
"## CITY:\n\n{}\n\n" "## PERSONA:\n{}\n\n## PERSONA DESCRIPTION:\n{}\n\n## STYLE "
"## COUNTRY:\n\n{}\n\n" "AFFINITIES:\n{}",
"## PERSONA:\n\n{}\n\n"
"## PERSONA DESCRIPTION:\n\n{}\n\n"
"## STYLE AFFINITIES:\n\n{}",
name.first_name, name.last_name, name.gender, city.location.city, name.first_name, name.last_name, name.gender, city.location.city,
city.location.country, persona.name, persona.description, city.location.country, persona.name, persona.description,
style_affinities); style_affinities);

View File

@@ -15,10 +15,8 @@
#include <string_view> #include <string_view>
#include <vector> #include <vector>
#include <llama.h>
#include "data_generation/llama_generator_helpers.h" #include "data_generation/llama_generator_helpers.h"
#include "llama.h"
namespace { namespace {
/** /**
* String trimming: removes leading and trailing whitespace * String trimming: removes leading and trailing whitespace

View File

@@ -14,10 +14,9 @@
#include <string_view> #include <string_view>
#include <vector> #include <vector>
#include <llama.h>
#include "data_generation/llama_generator.h" #include "data_generation/llama_generator.h"
#include "data_generation/llama_generator_helpers.h" #include "data_generation/llama_generator_helpers.h"
#include "llama.h"
static constexpr size_t kPromptTokenSlack = 8; static constexpr size_t kPromptTokenSlack = 8;
// Minimum tokens to keep when using top-p sampling. Ensures at least one // Minimum tokens to keep when using top-p sampling. Ensures at least one

View File

@@ -11,9 +11,8 @@
#include <stdexcept> #include <stdexcept>
#include <string> #include <string>
#include <llama.h>
#include "data_model/models.h" #include "data_model/models.h"
#include "llama.h"
static constexpr uint32_t kMaxContextSize = 32768U; static constexpr uint32_t kMaxContextSize = 32768U;

View File

@@ -10,10 +10,9 @@
#include <string> #include <string>
#include <utility> #include <utility>
#include <ggml-backend.h>
#include <llama.h>
#include "data_generation/llama_generator.h" #include "data_generation/llama_generator.h"
#include "ggml-backend.h"
#include "llama.h"
// Maximum batch size for decode operations. Capping the batch prevents // Maximum batch size for decode operations. Capping the batch prevents
// excessive memory allocation while maintaining inference performance. // excessive memory allocation while maintaining inference performance.

View File

@@ -0,0 +1,40 @@
/**
* @file data_model/names_by_country.cc
* @brief NamesByCountry::SampleName() implementation.
*/
#include "data_model/names_by_country.h"
#include <utility>
NamesByCountry::NamesByCountry(
std::unordered_map<std::string, std::vector<ForenameEntry>>
forenames_by_country,
std::unordered_map<std::string, std::vector<std::string>>
surnames_by_country)
: forenames_by_country_(std::move(forenames_by_country)),
surnames_by_country_(std::move(surnames_by_country)) {}
std::optional<Name> NamesByCountry::SampleName(const std::string& iso3166_1,
std::mt19937& rng) const {
const auto forenames_it = forenames_by_country_.find(iso3166_1);
const auto surnames_it = surnames_by_country_.find(iso3166_1);
if (forenames_it == forenames_by_country_.end() ||
surnames_it == surnames_by_country_.end() ||
forenames_it->second.empty() || surnames_it->second.empty()) {
return std::nullopt;
}
const std::vector<ForenameEntry>& forenames = forenames_it->second;
const std::vector<std::string>& surnames = surnames_it->second;
std::uniform_int_distribution<size_t> forename_dist(0, forenames.size() - 1);
std::uniform_int_distribution<size_t> surname_dist(0, surnames.size() - 1);
const ForenameEntry& forename = forenames[forename_dist(rng)];
return Name{.first_name = forename.name,
.last_name = surnames[surname_dist(rng)],
.gender = forename.gender};
}

View File

@@ -4,7 +4,7 @@
* records with strict field validation and descriptive error reporting. * records with strict field validation and descriptive error reporting.
*/ */
#include "services/curated_data/curated_json_data_service.h" #include "json_handling/json_loader.h"
#include <boost/json.hpp> #include <boost/json.hpp>
#include <format> #include <format>
@@ -14,8 +14,8 @@
#include <stdexcept> #include <stdexcept>
#include <string> #include <string>
#include <string_view> #include <string_view>
#include <unordered_map>
#include <utility> #include <utility>
#include <vector>
#include "services/logging/logger.h" #include "services/logging/logger.h"
@@ -106,23 +106,16 @@ std::string ReadFirstOfStringArray(const boost::json::object& object,
} // namespace } // namespace
CuratedJsonDataService::CuratedJsonDataService(CuratedDataFilePaths filepaths) std::vector<Location> JsonLoader::LoadLocations(
: filepaths_(std::move(filepaths)) {} const std::filesystem::path& filepath) {
const boost::json::value root = ParseJsonFile(filepath, "locations");
const LocationsList& CuratedJsonDataService::LoadLocations() {
if (!cache_.locations.empty()) {
return cache_.locations;
}
const boost::json::value root =
ParseJsonFile(filepaths_.locations_path, "locations");
if (!root.is_array()) { if (!root.is_array()) {
throw std::runtime_error( throw std::runtime_error(
"Invalid locations JSON: root element must be an array"); "Invalid locations JSON: root element must be an array");
} }
LocationsList locations; std::vector<Location> locations;
const auto& items = root.as_array(); const auto& items = root.as_array();
locations.reserve(items.size()); locations.reserve(items.size());
@@ -144,24 +137,20 @@ const LocationsList& CuratedJsonDataService::LoadLocations() {
.longitude = ReadRequiredNumber(object, "longitude"), .longitude = ReadRequiredNumber(object, "longitude"),
}); });
} }
cache_.locations = std::move(locations);
return cache_.locations; return locations;
} }
const PersonasList& CuratedJsonDataService::LoadPersonas() { std::vector<UserPersona> JsonLoader::LoadPersonas(
if (!cache_.personas.empty()) { const std::filesystem::path& filepath) {
return cache_.personas; const boost::json::value root = ParseJsonFile(filepath, "personas");
}
const boost::json::value root =
ParseJsonFile(filepaths_.personas_path, "personas");
if (!root.is_array()) { if (!root.is_array()) {
throw std::runtime_error( throw std::runtime_error(
"Invalid personas JSON: root element must be an array"); "Invalid personas JSON: root element must be an array");
} }
PersonasList personas; std::vector<UserPersona> personas;
const auto& items = root.as_array(); const auto& items = root.as_array();
personas.reserve(items.size()); personas.reserve(items.size());
@@ -179,30 +168,30 @@ const PersonasList& CuratedJsonDataService::LoadPersonas() {
}); });
} }
cache_.personas = std::move(personas); return personas;
return cache_.personas;
} }
const ForenamesByCountryMap& CuratedJsonDataService::LoadForenamesByCountry() { NamesByCountry JsonLoader::LoadNamesByCountry(
if (!cache_.forenames_by_country.empty()) { const std::filesystem::path& forenames_filepath,
return cache_.forenames_by_country; const std::filesystem::path& surnames_filepath) {
} const boost::json::value forenames_root =
ParseJsonFile(forenames_filepath, "forenames-by-country");
const boost::json::value surnames_root =
ParseJsonFile(surnames_filepath, "surnames-by-country");
const boost::json::value root = if (!forenames_root.is_object() || !surnames_root.is_object()) {
ParseJsonFile(filepaths_.forenames_path, "forenames-by-country");
if (!root.is_object()) {
throw std::runtime_error( throw std::runtime_error(
"Invalid forenames-by-country JSON: root element must be an object " "Invalid names-by-country JSON: root element must be an object "
"keyed by ISO 3166-1 country code"); "keyed by ISO 3166-1 country code");
} }
ForenamesByCountryMap forenames_by_country; std::unordered_map<std::string, std::vector<ForenameEntry>>
for (const auto& [country_code, regions] : root.as_object()) { forenames_by_country;
for (const auto& [country_code, regions] : forenames_root.as_object()) {
if (!regions.is_array()) { if (!regions.is_array()) {
continue; continue;
} }
ForenameList entries; std::vector<ForenameEntry> entries;
for (const auto& region : regions.as_array()) { for (const auto& region : regions.as_array()) {
if (!region.is_object()) { if (!region.is_object()) {
continue; continue;
@@ -216,7 +205,7 @@ const ForenamesByCountryMap& CuratedJsonDataService::LoadForenamesByCountry() {
continue; continue;
} }
const auto& name_object = name_value.as_object(); const auto& name_object = name_value.as_object();
entries.emplace_back(ForenameEntry{ entries.push_back(ForenameEntry{
.name = .name =
ReadFirstOfStringArray(name_object, "romanized", "localized"), ReadFirstOfStringArray(name_object, "romanized", "localized"),
.gender = ReadRequiredString(name_object, "gender"), .gender = ReadRequiredString(name_object, "gender"),
@@ -226,39 +215,22 @@ const ForenamesByCountryMap& CuratedJsonDataService::LoadForenamesByCountry() {
forenames_by_country.emplace(country_code, std::move(entries)); forenames_by_country.emplace(country_code, std::move(entries));
} }
cache_.forenames_by_country = std::move(forenames_by_country); std::unordered_map<std::string, std::vector<std::string>> surnames_by_country;
return cache_.forenames_by_country; for (const auto& [country_code, name_entries] : surnames_root.as_object()) {
} if (!name_entries.is_array()) {
continue;
const SurnamesByCountryMap& CuratedJsonDataService::LoadSurnamesByCountry() { }
if (!cache_.surnames_by_country.empty()) { std::vector<std::string> surnames;
return cache_.surnames_by_country;
}
const boost::json::value root =
ParseJsonFile(filepaths_.surnames_path, "surnames-by-country");
if (!root.is_object()) {
throw std::runtime_error(
"Invalid surnames-by-country JSON: root element must be an object "
"keyed by ISO 3166-1 country code");
}
SurnamesByCountryMap surnames_by_country;
for (const auto& [country_code, name_entries] : root.as_object()) {
if (!name_entries.is_array()) continue;
SurnameList surnames;
for (const auto& name_value : name_entries.as_array()) { for (const auto& name_value : name_entries.as_array()) {
if (!name_value.is_object()) { if (!name_value.is_object()) {
continue; continue;
} }
surnames.emplace_back(ReadFirstOfStringArray(name_value.as_object(), surnames.push_back(ReadFirstOfStringArray(name_value.as_object(),
"romanized", "localized")); "romanized", "localized"));
} }
surnames_by_country.emplace(country_code, std::move(surnames)); surnames_by_country.emplace(country_code, std::move(surnames));
} }
cache_.surnames_by_country = std::move(surnames_by_country); return NamesByCountry(std::move(forenames_by_country),
return cache_.surnames_by_country; std::move(surnames_by_country));
} }

View File

@@ -18,7 +18,27 @@
#include <string> #include <string>
#include <thread> #include <thread>
#include "biergarten_pipeline.h" #include "biergarten_pipeline_orchestrator.h"
#include "concurrency/bounded_channel.h"
#include "data_generation/llama_generator.h"
#include "data_generation/mock_generator.h"
#include "data_generation/prompt_formatting/gemma4_jinja_prompt_formatter.h"
#include "data_model/models.h"
#include "json_handling/json_loader.h"
#include "llama_backend_state.h"
#include "services/curated_data/curated_data_service.h"
#include "services/database/export_service.h"
#include "services/database/sqlite_export_service.h"
#include "services/datetime/timer.h"
#include "services/enrichment/enrichment_service.h"
#include "services/enrichment/mock_enrichment.h"
#include "services/enrichment/wikipedia_service.h"
#include "services/logging/log_dispatcher.h"
#include "services/logging/log_entry.h"
#include "services/logging/log_producer.h"
#include "services/logging/logger.h"
#include "services/prompting/prompt_directory.h"
#include "web_client/http_web_client.h"
namespace di = boost::di; namespace di = boost::di;
@@ -29,7 +49,7 @@ int main(const int argc, char** argv) {
spdlog::set_pattern("│ %Y-%m-%d %H:%M:%S.%e │ %^%-7l%$ │ %v"); spdlog::set_pattern("│ %Y-%m-%d %H:%M:%S.%e │ %^%-7l%$ │ %v");
BoundedChannel<LogEntry> log_channel(kLogMaxCount); BoundedChannel<LogEntry> log_channel(kLogMaxCount);
auto log_dispatcher = // auto log_dispatcher = //
std::make_unique<LogDispatcher>(log_channel); std::make_unique<LogDispatcher>(log_channel);
std::shared_ptr<ILogger> log_producer = std::shared_ptr<ILogger> log_producer =
std::make_shared<LogProducer>(log_channel); std::make_shared<LogProducer>(log_channel);
@@ -86,43 +106,22 @@ int main(const int argc, char** argv) {
di::bind<ApplicationOptions>().to(options), di::bind<ApplicationOptions>().to(options),
di::bind<std::string>().to(model_path), di::bind<std::string>().to(model_path),
di::bind<IExportService>().to<SqliteExportService>(), di::bind<IExportService>().to<SqliteExportService>(),
di::bind<ICuratedDataService>().to( di::bind<ICuratedDataService>().to<JsonLoader>(),
[options, &log_producer]() -> std::unique_ptr<ICuratedDataService> {
if (options.generator.use_mocked) {
log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "Curated data: mock"});
return std::make_unique<MockCuratedDataService>();
}
log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "Curated data: JsonLoader"});
return std::make_unique<CuratedJsonDataService>(
CuratedDataFilePaths{
.locations_path = "locations.json",
.personas_path = "personas.json",
.forenames_path = "forenames-by-country.json",
.surnames_path = "surnames-by-country.json",
});
}),
di::bind<IPromptFormatter>().to([options, log_producer] { di::bind<IPromptFormatter>().to([options, log_producer] {
if (options.generator.use_mocked) { if (options.generator.use_mocked) {
{ {
log_producer->Log( log_producer->Log(
{.level = LogLevel::Info, {.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = "Prompt formatter: none (mock mode)"}); .message = "Prompt formatter: none (mock mode)"});
} }
return std::unique_ptr<IPromptFormatter>(nullptr); return std::unique_ptr<IPromptFormatter>(nullptr);
} }
{ {
log_producer->Log( log_producer->Log(
{.level = LogLevel::Info, {.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = "Prompt formatter: Gemma4JinjaPromptFormatter"}); .message = "Prompt formatter: Gemma4JinjaPromptFormatter"});
} }
return std::unique_ptr<IPromptFormatter>( return std::unique_ptr<IPromptFormatter>(
std::make_unique<Gemma4JinjaPromptFormatter>()); std::make_unique<Gemma4JinjaPromptFormatter>());
@@ -146,7 +145,7 @@ int main(const int argc, char** argv) {
}), }),
di::bind<IEnrichmentService>().to( di::bind<IEnrichmentService>().to(
[options, &log_producer]( [options, &log_producer](
const auto& inj) -> std::unique_ptr<IEnrichmentService> { const auto& inj) -> std::unique_ptr<IEnrichmentService> {
if (options.generator.use_mocked) { if (options.generator.use_mocked) {
log_producer->Log({.level = LogLevel::Info, log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
@@ -165,7 +164,8 @@ int main(const int argc, char** argv) {
}), }),
di::bind<DataGenerator>().to( di::bind<DataGenerator>().to(
[&options, &model_path, &sampling, &prompt_directory, [&options, &model_path, &sampling, &prompt_directory,
&log_producer](const auto& inj) -> std::unique_ptr<DataGenerator> { &log_producer
](const auto& inj) -> std::unique_ptr<DataGenerator> {
if (options.generator.use_mocked) { if (options.generator.use_mocked) {
log_producer->Log({.level = LogLevel::Info, log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
@@ -174,28 +174,19 @@ int main(const int argc, char** argv) {
return std::make_unique<MockGenerator>(); return std::make_unique<MockGenerator>();
} }
#ifdef BIERGARTEN_MOCK_ONLY
// Unreachable: ParseArguments requires --mocked when
// BIERGARTEN_MOCK_ONLY is compiled in, so LlamaGenerator is
// never constructed and is not linked into this binary.
throw std::runtime_error(
"LlamaGenerator is unavailable in a BIERGARTEN_MOCK_ONLY "
"build");
#else
log_producer->Log( log_producer->Log(
{.level = LogLevel::Info, {.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = std::format( .message = std::format(
"Generator: LlamaGenerator | model={} | " "Generator: LlamaGenerator | model={} | "
"temp={:.2f} top_p={:.2f} top_k={} n_ctx={} seed={}", "temp={:.2f} top_p={:.2f} top_k={} n_ctx={} seed={}",
model_path, sampling.temperature, sampling.top_p, model_path, sampling.temperature, sampling.top_p,
sampling.top_k, sampling.n_ctx, sampling.seed)}); sampling.top_k, sampling.n_ctx, sampling.seed)});
return std::make_unique<LlamaGenerator>( return std::make_unique<LlamaGenerator>(
options, model_path, log_producer, options, model_path, log_producer,
inj.template create<std::unique_ptr<IPromptFormatter>>(), inj.template create<std::unique_ptr<IPromptFormatter>>(),
std::move(prompt_directory)); std::move(prompt_directory));
#endif
})); }));
const auto orchestrator = const auto orchestrator =

View File

@@ -1,98 +0,0 @@
/**
* @file services/curated_data/mock_curated_data_service.cc
* @brief Fixed in-memory location, persona, and name dataset for mock mode.
*/
#include "services/curated_data/mock_curated_data_service.h"
MockCuratedDataService::MockCuratedDataService()
: locations_{
Location{.city = "Portland",
.state_province = "Oregon",
.iso3166_2 = "US-OR",
.country = "United States",
.iso3166_1 = "US",
.local_languages = {"en"},
.latitude = 45.5152,
.longitude = -122.6784},
Location{.city = "Munich",
.state_province = "Bavaria",
.iso3166_2 = "DE-BY",
.country = "Germany",
.iso3166_1 = "DE",
.local_languages = {"de"},
.latitude = 48.1351,
.longitude = 11.5820},
Location{.city = "Lyon",
.state_province = "Auvergne-Rhone-Alpes",
.iso3166_2 = "FR-ARA",
.country = "France",
.iso3166_1 = "FR",
.local_languages = {"fr"},
.latitude = 45.7640,
.longitude = 4.8357},
Location{.city = "Brussels",
.state_province = "Brussels-Capital",
.iso3166_2 = "BE-BRU",
.country = "Belgium",
.iso3166_1 = "BE",
.local_languages = {"nl", "fr"},
.latitude = 50.8503,
.longitude = 4.3517},
},
personas_{
UserPersona{.name = "Hophead Explorer",
.description = "Chases hop-forward IPAs and seeks out "
"taprooms with rotating drafts.",
.style_affinities = {"IPA", "Pale Ale"}},
UserPersona{.name = "Lager Traditionalist",
.description = "Prefers clean, balanced lagers and "
"classic pub styles.",
.style_affinities = {"Lager", "Pilsner"}},
UserPersona{.name = "Sour Curious",
.description = "Seeks out wild ales, sours, and "
"barrel-aged experiments.",
.style_affinities = {"Sour", "Wild Ale"}},
},
forenames_by_country_{
{"US",
ForenameList{
ForenameEntry{.name = "James", .gender = "M"},
ForenameEntry{.name = "Mary", .gender = "F"},
}},
{"DE",
ForenameList{
ForenameEntry{.name = "Lukas", .gender = "M"},
ForenameEntry{.name = "Anna", .gender = "F"},
}},
{"FR",
ForenameList{
ForenameEntry{.name = "Lucas", .gender = "M"},
ForenameEntry{.name = "Camille", .gender = "F"},
}},
{"BE",
ForenameList{
ForenameEntry{.name = "Noah", .gender = "M"},
ForenameEntry{.name = "Emma", .gender = "F"},
}},
},
surnames_by_country_{
{"US", SurnameList{"Smith", "Johnson"}},
{"DE", SurnameList{"Muller", "Schmidt"}},
{"FR", SurnameList{"Martin", "Bernard"}},
{"BE", SurnameList{"Peeters", "Janssens"}},
} {}
const LocationsList& MockCuratedDataService::LoadLocations() {
return locations_;
}
const PersonasList& MockCuratedDataService::LoadPersonas() { return personas_; }
const ForenamesByCountryMap& MockCuratedDataService::LoadForenamesByCountry() {
return forenames_by_country_;
}
const SurnamesByCountryMap& MockCuratedDataService::LoadSurnamesByCountry() {
return surnames_by_country_;
}

View File

@@ -33,7 +33,7 @@ void ResetStatement(SqliteStatementHandle& statement) {
} }
void Bind(const SqliteStatementHandle& statement, void Bind(const SqliteStatementHandle& statement,
const BoundParam<std::string_view>& param) { const BindParam<std::string_view>& param) {
const auto byte_count = param.value.size(); const auto byte_count = param.value.size();
if (byte_count > static_cast<std::size_t>(std::numeric_limits<int>::max())) { if (byte_count > static_cast<std::size_t>(std::numeric_limits<int>::max())) {
ThrowSqliteError(sqlite3_db_handle(statement.get()), param.action); ThrowSqliteError(sqlite3_db_handle(statement.get()), param.action);
@@ -60,7 +60,7 @@ void Bind(const SqliteStatementHandle& statement,
} }
void Bind(const SqliteStatementHandle& statement, void Bind(const SqliteStatementHandle& statement,
const BoundParam<double>& param) { const BindParam<double>& param) {
if (sqlite3_bind_double(statement.get(), param.index, param.value) != if (sqlite3_bind_double(statement.get(), param.index, param.value) !=
SQLITE_OK) { SQLITE_OK) {
ThrowSqliteError(sqlite3_db_handle(statement.get()), param.action); ThrowSqliteError(sqlite3_db_handle(statement.get()), param.action);
@@ -68,7 +68,7 @@ void Bind(const SqliteStatementHandle& statement,
} }
void Bind(const SqliteStatementHandle& statement, void Bind(const SqliteStatementHandle& statement,
const BoundParam<sqlite3_int64>& param) { const BindParam<sqlite3_int64>& param) {
if (sqlite3_bind_int64(statement.get(), param.index, param.value) != if (sqlite3_bind_int64(statement.get(), param.index, param.value) !=
SQLITE_OK) { SQLITE_OK) {
ThrowSqliteError(sqlite3_db_handle(statement.get()), param.action); ThrowSqliteError(sqlite3_db_handle(statement.get()), param.action);

View File

@@ -42,50 +42,50 @@ sqlite3_int64 SqliteExportService::ResolveLocationId(const Location& location) {
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_location_stmt_, insert_location_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationCityBindIndex, .index = sqlite_export_service_internal::kLocationCityBindIndex,
.value = location.city, .value = location.city,
.action = "Failed to bind SQLite location city"}); .action = "Failed to bind SQLite location city"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_location_stmt_, insert_location_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BindParam<std::string_view>{
.index = .index =
sqlite_export_service_internal::kLocationStateProvinceBindIndex, sqlite_export_service_internal::kLocationStateProvinceBindIndex,
.value = location.state_province, .value = location.state_province,
.action = "Failed to bind SQLite location state/province"}); .action = "Failed to bind SQLite location state/province"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_location_stmt_, insert_location_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationIso31662BindIndex, .index = sqlite_export_service_internal::kLocationIso31662BindIndex,
.value = location.iso3166_2, .value = location.iso3166_2,
.action = "Failed to bind SQLite location ISO 3166-2 code"}); .action = "Failed to bind SQLite location ISO 3166-2 code"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_location_stmt_, insert_location_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationCountryBindIndex, .index = sqlite_export_service_internal::kLocationCountryBindIndex,
.value = location.country, .value = location.country,
.action = "Failed to bind SQLite location country"}); .action = "Failed to bind SQLite location country"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_location_stmt_, insert_location_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationIso31661BindIndex, .index = sqlite_export_service_internal::kLocationIso31661BindIndex,
.value = location.iso3166_1, .value = location.iso3166_1,
.action = "Failed to bind SQLite location ISO 3166-1 code"}); .action = "Failed to bind SQLite location ISO 3166-1 code"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_location_stmt_, insert_location_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationLanguagesBindIndex, .index = sqlite_export_service_internal::kLocationLanguagesBindIndex,
.value = local_languages_json, .value = local_languages_json,
.action = "Failed to bind SQLite location languages"}); .action = "Failed to bind SQLite location languages"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_location_stmt_, insert_location_stmt_,
sqlite_export_service_internal::BoundParam{ sqlite_export_service_internal::BindParam{
.index = sqlite_export_service_internal::kLocationLatitudeBindIndex, .index = sqlite_export_service_internal::kLocationLatitudeBindIndex,
.value = location.latitude, .value = location.latitude,
.action = "Failed to bind SQLite location latitude"}); .action = "Failed to bind SQLite location latitude"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_location_stmt_, insert_location_stmt_,
sqlite_export_service_internal::BoundParam{ sqlite_export_service_internal::BindParam{
.index = sqlite_export_service_internal::kLocationLongitudeBindIndex, .index = sqlite_export_service_internal::kLocationLongitudeBindIndex,
.value = location.longitude, .value = location.longitude,
.action = "Failed to bind SQLite location longitude"}); .action = "Failed to bind SQLite location longitude"});
@@ -111,21 +111,21 @@ uint64_t SqliteExportService::ProcessRecord(const BreweryRecord& brewery) {
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_stmt_, insert_brewery_stmt_,
sqlite_export_service_internal::BoundParam<sqlite3_int64>{ sqlite_export_service_internal::BindParam<sqlite3_int64>{
.index = sqlite_export_service_internal::kBreweryLocationIdBindIndex, .index = sqlite_export_service_internal::kBreweryLocationIdBindIndex,
.value = location_id, .value = location_id,
.action = "Failed to bind SQLite brewery location id"}); .action = "Failed to bind SQLite brewery location id"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_stmt_, insert_brewery_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kBreweryEnglishNameBindIndex, .index = sqlite_export_service_internal::kBreweryEnglishNameBindIndex,
.value = brewery.brewery.name_en, .value = brewery.brewery.name_en,
.action = "Failed to bind SQLite brewery English name"}); .action = "Failed to bind SQLite brewery English name"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_stmt_, insert_brewery_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal:: .index = sqlite_export_service_internal::
kBreweryEnglishDescriptionBindIndex, kBreweryEnglishDescriptionBindIndex,
.value = brewery.brewery.description_en, .value = brewery.brewery.description_en,
@@ -133,14 +133,14 @@ uint64_t SqliteExportService::ProcessRecord(const BreweryRecord& brewery) {
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_stmt_, insert_brewery_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kBreweryLocalNameBindIndex, .index = sqlite_export_service_internal::kBreweryLocalNameBindIndex,
.value = brewery.brewery.name_local, .value = brewery.brewery.name_local,
.action = "Failed to bind SQLite brewery local name"}); .action = "Failed to bind SQLite brewery local name"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_stmt_, insert_brewery_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BindParam<std::string_view>{
.index = .index =
sqlite_export_service_internal::kBreweryLocalDescriptionBindIndex, sqlite_export_service_internal::kBreweryLocalDescriptionBindIndex,
.value = brewery.brewery.description_local, .value = brewery.brewery.description_local,

View File

@@ -17,55 +17,55 @@ uint64_t SqliteExportService::ProcessRecord(const UserRecord& user) {
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_stmt_, insert_user_stmt_,
sqlite_export_service_internal::BoundParam<sqlite3_int64>{ sqlite_export_service_internal::BindParam<sqlite3_int64>{
.index = sqlite_export_service_internal::kUserLocationIdBindIndex, .index = sqlite_export_service_internal::kUserLocationIdBindIndex,
.value = location_id, .value = location_id,
.action = "Failed to bind SQLite user location id"}); .action = "Failed to bind SQLite user location id"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_stmt_, insert_user_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserFirstNameBindIndex, .index = sqlite_export_service_internal::kUserFirstNameBindIndex,
.value = user.user.first_name, .value = user.user.first_name,
.action = "Failed to bind SQLite user first name"}); .action = "Failed to bind SQLite user first name"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_stmt_, insert_user_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserLastNameBindIndex, .index = sqlite_export_service_internal::kUserLastNameBindIndex,
.value = user.user.last_name, .value = user.user.last_name,
.action = "Failed to bind SQLite user last name"}); .action = "Failed to bind SQLite user last name"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_stmt_, insert_user_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserGenderBindIndex, .index = sqlite_export_service_internal::kUserGenderBindIndex,
.value = user.user.gender, .value = user.user.gender,
.action = "Failed to bind SQLite user gender"}); .action = "Failed to bind SQLite user gender"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_stmt_, insert_user_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserUsernameBindIndex, .index = sqlite_export_service_internal::kUserUsernameBindIndex,
.value = user.user.username, .value = user.user.username,
.action = "Failed to bind SQLite user username"}); .action = "Failed to bind SQLite user username"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_stmt_, insert_user_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserBioBindIndex, .index = sqlite_export_service_internal::kUserBioBindIndex,
.value = user.user.bio, .value = user.user.bio,
.action = "Failed to bind SQLite user bio"}); .action = "Failed to bind SQLite user bio"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_stmt_, insert_user_stmt_,
sqlite_export_service_internal::BoundParam<double>{ sqlite_export_service_internal::BindParam<double>{
.index = sqlite_export_service_internal::kUserActivityWeightBindIndex, .index = sqlite_export_service_internal::kUserActivityWeightBindIndex,
.value = static_cast<double>(user.user.activity_weight), .value = static_cast<double>(user.user.activity_weight),
.action = "Failed to bind SQLite user activity weight"}); .action = "Failed to bind SQLite user activity weight"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_stmt_, insert_user_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserEmailBindIndex, .index = sqlite_export_service_internal::kUserEmailBindIndex,
.value = user.email, .value = user.email,
.action = "Failed to bind SQLite user email"}); .action = "Failed to bind SQLite user email"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_stmt_, insert_user_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserDateOfBirthBindIndex, .index = sqlite_export_service_internal::kUserDateOfBirthBindIndex,
.value = user.date_of_birth, .value = user.date_of_birth,
.action = "Failed to bind SQLite user date of birth"}); .action = "Failed to bind SQLite user date of birth"});