14 Commits

Author SHA1 Message Date
Aaron Po
3da0da6217 Do full Gemma-4 run of pipeline, transfer sqllite file to public dir of preview 2026-06-23 03:16:34 -04:00
Aaron Po
6883b05824 remove password field 2026-06-23 01:39:57 -04:00
Aaron Po
b631c0f5db add results viewer 2026-06-23 01:27:37 -04:00
Aaron Po
9a9ffd59e6 Formatted src/biergarten_pipeline_orchestrator 2026-06-23 01:04:52 -04:00
Aaron Po
35a5c0785c documentation updates, remove superfluous comments 2026-06-23 01:04:46 -04:00
Aaron Po
e590cf2543 edit .clang-format 2026-06-23 00:50:27 -04:00
Aaron Po
3b32766a83 edits 2026-06-23 00:48:20 -04:00
Aaron Po
8d6b83673b Update doxygen comments 2026-06-23 00:45:32 -04:00
Aaron Po
22b3f2b3f9 cleanup gbnf grammars 2026-06-23 00:38:53 -04:00
Aaron Po
ac701d661a Remove logger from JSON export 2026-06-23 00:19:28 -04:00
Aaron Po
3f978f4de4 Cleanup user and brewery generation exception logic 2026-06-21 23:09:29 -04:00
Aaron Po
ad97b0ea6c format 2026-06-21 22:14:42 -04:00
Aaron Po
4de0ea6638 Persist generated users to SQLite and code cleanup 2026-06-21 22:14:42 -04:00
Aaron Po
51b40a39c9 Add user generation feature 2026-06-21 22:14:42 -04:00
74 changed files with 1729 additions and 4144 deletions

View File

@@ -109,13 +109,13 @@ gender from the source data is preserved through to the sampled `Name`
persona/bio generation later.
The full multinational dataset is kept as-is (106 countries for forenames,
75 for surnames) rather than trimmed to `cities.json`'s current country
75 for surnames) rather than trimmed to `locations.json`'s current country
list, so it doesn't need re-sourcing if more countries are added later.
`SampleName()` (a free helper in `generate_users.cc`) returns no result for
a country present in neither file; of the countries in `cities.json`,
that's currently `KE`, `SE`, `SG`, `TH`, `VN`, and `ZA``GenerateUsers`
skips cities in those countries the same way brewery generation skips
cities whose enrichment lookup fails.
`NamesByCountry::SampleName()` returns no result for a country present in
neither file; of the countries in `locations.json`, that's currently `KE`,
`SE`, `SG`, `TH`, `VN`, and `ZA``GenerateUsers` skips cities in those
countries the same way brewery generation skips cities whose enrichment
lookup fails.
---
@@ -281,7 +281,7 @@ Output sample:
a location's code is unlisted, skip `description_local` generation and fall
back to English.
- **Upstream sanitization:** strip known low-resource language codes from the
`cities.json` payload before generation.
`locations.json` payload before generation.
- **Downstream flagging:** add a `description_local_confidence` column to the
SQLite schema so downstream applications can filter or flag potentially
hallucinated text by language tier.

View File

@@ -1,9 +1,8 @@
# Biergarten Pipeline
A C++20 command-line pipeline that samples city records from local JSON,
enriches each with Wikipedia context, and generates bilingual brewery names
and descriptions plus locale-grounded user profiles via a local GGUF model or
a deterministic mock.
enriches each with Wikipedia context, and generates bilingual brewery names and
descriptions via a local GGUF model or a deterministic mock.
> **This pipeline produces AI-generated data.** It is not a source of truth for
> brewing techniques, cultural representation, or local-language accuracy. See
@@ -46,7 +45,6 @@ step.
| Beer reviews and ratings | Stable brewery fixtures with enough context to anchor review pages |
| Social follow relationships | Repeatable brewery entities for feeds, follows, and saved lists |
| Geospatial brewery experiences | Latitude, longitude, and country-level metadata |
| User accounts and profiles | Locale-grounded names, bios, and an auth-ready email/date-of-birth pair for seeding real accounts |
---
@@ -86,7 +84,7 @@ curl -L \
### Run
Run from `build/` so the copied `cities.json` and `prompts/` are available.
Run from `build/` so the copied `locations.json` and `prompts/` are available.
Each run writes a fresh dated SQLite file such as
`biergarten_seed_2026-04-19T15-30-45.123456Z.sqlite` into the working directory.
@@ -109,7 +107,7 @@ Each run writes a fresh dated SQLite file such as
| `--prompt-dir` | Directory containing prompt files (e.g. `BREWERY_GENERATION.md`). Required unless `--mocked` is set. |
| `--output, -o` | Directory for generated SQLite artifacts. Default: `output`. |
| `--log-path` | Path for application logs. Default: `pipeline.log`. |
| `--location-count` | Number of cities to sample from `cities.json` per run. Default: `10`. |
| `--location-count` | Number of cities to sample from `locations.json` per run. Default: `10`. |
| `--temperature` | Sampling temperature. Default: `1.0`. |
| `--top-p` | Nucleus sampling. Default: `0.95`. |
| `--top-k` | Top-k sampling. Default: `64`. |
@@ -221,60 +219,34 @@ environment variables listed above to match your run.
### Pipeline Stages
| Stage | Implementation |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| Load | `ICuratedDataService` (`CuratedJsonDataService`) reads `cities.json`, `personas.json`, `forenames-by-country.json`, and `surnames-by-country.json` (paths supplied via a `CuratedDataFilePaths` DTO at construction) into typed records, caching each after its first load. `--mocked` runs use `MockCuratedDataService`'s fixed in-memory dataset instead. |
| -------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| Load | `JsonLoader::LoadLocations()` reads `locations.json` into typed `Location` records. |
| Sample | `BiergartenPipelineOrchestrator::QueryCitiesWithCountries()` samples `--location-count` locations per run (default `10`). |
| Enrich | `WikipediaEnrichmentService` fetches brewing and beer-related context. Keeps going when a lookup fails. `--mocked` runs use `MockEnrichmentService` instead and skip Wikipedia entirely. |
| Generate Users | `GenerateUsers()` samples a persona and a forename/surname pair per enriched city (skipping countries with no name data), then `MockGenerator` or `LlamaGenerator` produces a username, bio, and activity weight around the sampled name. |
| Generate Breweries | `MockGenerator` or `LlamaGenerator` produces brewery names and descriptions in English and the local language. |
| Store | `SqliteExportService` writes each successful user and brewery into a fresh dated `.sqlite` database with normalized `cities`, `users`, and `breweries` tables. |
| Generate | `MockGenerator` or `LlamaGenerator` produces brewery names and descriptions in English and the local language. |
| Store | `SqliteExportService` writes each successful brewery into a fresh dated `.sqlite` database with normalized location and brewery tables. |
| Log | `spdlog` writes results and warnings to the console. |
If name sampling, enrichment, or generation fails for a city, that city is
skipped and the pipeline continues. `GenerateUsers()` runs before
`GenerateBreweries()` in `BiergartenPipelineOrchestrator::Run()`.
If enrichment or generation fails for a city, that city is skipped and the
pipeline continues.
### Key Components
- `src/main.cc` — argument parsing and Boost.DI composition root.
- `CuratedJsonDataService` — implements `ICuratedDataService`; takes a
`CuratedDataFilePaths` DTO (locations/personas/forenames/surnames paths) in
its constructor, then parses and validates curated location, persona, and
forename/surname JSON, memoizing each result after its first load on a
given instance. Each `cities.json` entry's `postal_code.city_regex` and
`postal_code.examples` are parsed into `City::postal_regex` and
`City::postal_code_examples`. `MockCuratedDataService` is the in-memory
substitute (4 fixed locations, 3 personas, and name data for
`US`/`DE`/`FR`/`BE`) used in `--mocked` runs, and carries matching
`postal_regex`/`postal_code_examples` values for its 4 locations.
- `JsonLoader` — validates curated location input.
- `WikipediaEnrichmentService` — queries Wikipedia extracts, caches results,
returns empty context on failure. `MockEnrichmentService` is the no-op
substitute used in `--mocked` runs.
- `IPostalCodeService` — generates a postal code for a `City`, consumed by
`GenerateBreweries()` and stored on `BreweryRecord::address` (an `Address`
struct, currently just `postal_code`, mirroring the web backend's
`BreweryPostLocation`). Only `MockPostalCodeService` exists today, which
ignores `postal_regex` and returns `postal_code_examples.front()` — it's
wired into the Boost.DI graph unconditionally (no `--mocked` branch yet,
since there's no real implementation to switch to). A real implementation
still needs a **xeger**-style generator — turning a `postal_regex` pattern
into a random matching string — instead of always replaying a fixed
example; see [ROADMAP.md §9](./ROADMAP.md#9-postal-code-generation).
Street-address generation (`Address::address_line1`) has no fixture data or
service yet and remains future work.
- `LlamaGenerator` — formats prompts for Gemma 4, validates JSON output for
both `GenerateBrewery` and `GenerateUser`, retries malformed responses up
to three times with corrective feedback in the retry prompt. The token
budget is fixed across attempts; it is not raised automatically on
truncation.
- `MockGenerator` — stable hash-based output so the same city/persona/name
input always produces the same brewery or user.
- `SqliteExportService` — creates a dated SQLite file per run and persists
each successful user and brewery into normalized tables.
- `LlamaGenerator` — formats prompts for Gemma 4, validates JSON output, retries
malformed responses up to three times with corrective feedback in the
retry prompt. The token budget is fixed across attempts; it is not raised
automatically on truncation.
- `MockGenerator` — stable hash-based output so the same city input always
produces the same brewery.
- `SqliteExportService` — creates a dated SQLite file per run and persists each
successful brewery into normalized tables.
- Brewery payloads include English and local-language name and description
fields. User payloads carry a sampled first/last name and gender, an
LLM-generated username/bio/activity weight, and a programmatically
generated (not LLM-authored) unique email and date of birth.
fields.
### Runtime Behaviour
@@ -294,18 +266,6 @@ valid output on the first pass; the retry path is kept for resilience.
`MockGenerator` uses stable hashes for repeatable output in demos and Storybook
runs.
`CuratedJsonDataService` memoizes each of `LoadCities()`, `LoadPersonas()`,
`LoadForenamesByCountry()`, and `LoadSurnamesByCountry()` independently the
first time each is called, since `BiergartenPipelineOrchestrator` owns a
single `ICuratedDataService` instance for the whole run — later calls return
the cached result instead of re-parsing.
`GenerateUsers()` samples a forename/surname pair per city via `SampleName()`,
keyed by the city's ISO 3166-1 code. Countries present in `cities.json`
but absent from either name fixture (currently `KE`, `SE`, `SG`, `TH`, `VN`,
`ZA`) are skipped, the same way a failed enrichment or generation call skips
a city — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section.
### Process Flow - Activity Diagram
![An activity diagram](./diagrams/current/output/activity.svg)
@@ -318,10 +278,9 @@ a city — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section.
## Generated Output
Each successful run stores a `BreweryRecord` (source `City`, an `Address`,
and a `BreweryResult` payload), and a `UserRecord` pair with the source
`City` and a `UserResult` payload. The same generated records are also
written to a fresh SQLite export file named with the current UTC timestamp.
Each successful run stores a `GeneratedBrewery` pair with the source location
and a `BreweryResult` payload. The same generated records are also written to a
fresh SQLite export file named with the current UTC timestamp.
| Field | Meaning |
| ------------------- | ------------------------------------------ |
@@ -329,18 +288,6 @@ written to a fresh SQLite export file named with the current UTC timestamp.
| `description_en` | Brewery description in English. |
| `name_local` | Brewery name in the local language. |
| `description_local` | Brewery description in the local language. |
| `postal_code` | Postal code generated for the brewery's city (see `IPostalCodeService`, above). |
| Field | Meaning |
| ----------------- | ----------------------------------------------------------------- |
| `first_name` | Sampled forename, copied from the curated name data (not LLM-invented). |
| `last_name` | Sampled surname, copied from the curated name data (not LLM-invented). |
| `gender` | Gender associated with the sampled forename in the source dataset. |
| `username` | LLM-generated handle. |
| `bio` | LLM-generated short biography. |
| `activity_weight` | Relative check-in/activity weight, reserved for a future J-curve activity profile. |
| `email` | Unique `@thebiergarten.app` address, generated programmatically from the sampled name. |
| `date_of_birth` | Randomized date of birth (age 19-48), generated programmatically. |
The log dump also includes city, country, state or province, ISO subdivision
code, latitude, and longitude for each entry.
@@ -355,7 +302,6 @@ code, latitude, and longitude for each entry.
| `local_languages` | Locale-aware copy selection |
| `name_en`, `description_en` | Default English display content |
| `name_local`, `description_local` | Local-language display content |
| `postal_code` | Brewery address matching, mirrors the web backend's `BreweryPostLocation.PostalCode` |
---
@@ -421,17 +367,11 @@ Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present.
## Fixture Strategy
- `--mocked` for stable fixtures, repeatable screenshots, and Storybook runs.
`MockCuratedDataService` swaps in for `CuratedJsonDataService`, so no
fixture files need to be present on disk.
- `--model` when geographically grounded content matters for demos.
- Keep `cities.json` structured enough to support discovery and future
- Keep `locations.json` structured enough to support discovery and future
filtering.
- `personas.json`, `forenames-by-country.json`, and
`surnames-by-country.json` are curated/vendored fixture data, not
LLM-generated — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset
section for provenance.
- Treat SQLite output as seed material for the app's brewery and user
domains, not production data.
- Treat SQLite output as seed material for the app's brewery domain, not
production data.
---
@@ -441,10 +381,7 @@ Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present.
| -------------------------------------- | -------------------------------------------------- |
| `tooling/pipeline/includes/` | Public headers and shared models. |
| `tooling/pipeline/src/` | Implementation files. |
| `tooling/pipeline/cities.json` | Curated city input copied into the build tree. |
| `tooling/pipeline/personas.json` | Curated user persona archetypes copied into the build tree. |
| `tooling/pipeline/forenames-by-country.json` | Vendored (CC0) forename data by ISO 3166-1 country code. |
| `tooling/pipeline/surnames-by-country.json` | Vendored (CC0) surname data by ISO 3166-1 country code. |
| `tooling/pipeline/locations.json` | Curated city input copied into the build tree. |
| `tooling/pipeline/prompts/` | System prompts used by the model-backed path. |
| `tooling/pipeline/runpod/` | Dockerfile, launcher, and RunPod pod template. |
| `docs/pipeline/diagrams/` | Architecture and pipeline diagrams. |
@@ -459,14 +396,7 @@ Paths below are relative to `tooling/pipeline/`.
- `src/main.cc` — argument parsing and DI composition root.
- `src/biergarten_pipeline_orchestrator/` — orchestration, sampling, logging,
and export.
- `src/services/curated_data/``CuratedJsonDataService`, the file-backed
`ICuratedDataService`, and `MockCuratedDataService`, the in-memory
`ICuratedDataService` used in `--mocked` runs.
- `src/services/enrichment/wikipedia/` — enrichment service and cache.
- `includes/services/postal_code/``IPostalCodeService` and
`MockPostalCodeService` (header-only), consumed by `GenerateBreweries()`.
The real xeger-based implementation and its `--mocked`-aware DI wiring are
still to come.
- `src/services/sqlite/` — SQLite export implementation.
- `src/data_generation/llama/` — local inference, prompt loading, output
validation.
@@ -477,12 +407,11 @@ Paths below are relative to `tooling/pipeline/`.
## Next Steps
The pipeline currently produces city-aware brewery and user records and
dated SQLite exports. The next passes add additional fixture types so the
app can exercise the full brewery and social domains without live data. For
the detailed engineering breakdown of what's needed to reach the
architecture in [`diagrams/planned/`](./diagrams/planned/), see
[ROADMAP.md](./ROADMAP.md).
The pipeline currently produces city-aware brewery records and dated SQLite
exports. The next passes add additional fixture types so the app can exercise
the full brewery domain without live data. For the detailed engineering
breakdown of what's needed to reach the architecture in
[`diagrams/planned/`](./diagrams/planned/), see [ROADMAP.md](./ROADMAP.md).
### Testing — Very High Priority
@@ -504,6 +433,12 @@ Generate catalog entries with style, ABV, IBU, color, aroma notes, and food
pairing hints. Link beers back to breweries and cities. Keep style coverage wide
enough to exercise search, sort, and category filters.
### User Generation
Generate user profiles with stable names, bios, locale hints, and preference
signals. Include stable IDs for downstream fixture joins. Keep output
deterministic for screenshots while allowing larger randomized batches.
### Check-In System
Produce timestamped check-in events between users and breweries. Use a J-curve

View File

@@ -39,119 +39,54 @@ architecture needs.
not LLM-invented.
- [x] Add `Name` (`first_name`, `last_name`, `gender`) — the sampled result
handed to `DataGenerator::GenerateUser`. Add `ForenameEntry` (`name`,
`gender`). Landed as a flatter shape than originally planned here: no
`NamesByCountry` wrapper class. `curated_data_service.h` instead
declares `ForenameList = std::vector<ForenameEntry>` and
`SurnameList = std::vector<std::string>`, and
`ICuratedDataService` exposes two maps keyed by ISO 3166-1 code —
`ForenamesByCountryMap = unordered_map<string, ForenameList>` and
`SurnamesByCountryMap = unordered_map<string, SurnameList>`
directly (see §2). Sampling is
a free function, `SampleName(forenames_by_country,
surnames_by_country, iso3166_1, rng)`, in
`generate_users.cc`'s anonymous namespace, not a method on a class.
`gender`) and a `NamesByCountry` class (not a bare typedef) that owns
two maps — `std::unordered_map<std::string, std::vector<ForenameEntry>>`
and `std::unordered_map<std::string, std::vector<std::string>>`, both
keyed by ISO 3166-1 code — and exposes `SampleName(iso3166_1, rng)`.
Pairing a forename with a surname happens at sample time so gender
(present per-forename in the source data) is never discarded the way a
pre-flattened `{first_name, last_name}` list would lose it; see §2.
pre-flattened `{first_name, last_name}` list would lose it; see
`LoadNamesByCountry()` below.
- [ ] Extend `GeneratedBrewery` with `brewery_id`, `context_completeness`,
`metadata` (currently `{location, address, brewery}` — see the
`Location``City` rename and `Address` struct added in the
`postal_regex`/§9 entries below).
`metadata` (currently just `{location, brewery}`).
- [ ] Add `GeneratedBeer`, `GeneratedCheckin`, `GeneratedRating`,
`GeneratedFollow` aggregate structs.
- [x] Add `UserRecord` (`location`, `user : UserResult`, `email`,
`date_of_birth`) as the current stand-in for the planned
`GeneratedUser``email` and `date_of_birth` are programmatically
generated by the orchestrator, never LLM-authored, so a downstream
auth-account seeding consumer can register real accounts from the
pipeline's SQLite export. No `password`, `user_id`, or `metadata`
field yet, matching `BreweryRecord`'s equally pre-`metadata` shape.
Already wired into `IExportService`/`SqliteExportService`: a `users`
table exists and `GenerateUsers()` exports each successful record via
`ProcessRecord(const UserRecord&)` (see §5) in addition to logging and
holding `generated_users_` in memory. Still missing vs. the planned
`GeneratedUser`: `user_id`, `password`, and `metadata`.
- [x] Add `GeneratedUser` aggregate struct. Carries `email`, `date_of_birth`,
and `password` — programmatically generated by the orchestrator, never
LLM-authored — so a downstream auth-account seeding consumer (outside
this repo tree's docs) can register real accounts from the pipeline's
SQLite export. Skips `user_id`/`metadata` for now, matching
`GeneratedBrewery`'s current (pre-`brewery_id`/`metadata`) shape. Not
yet wired into `IExportService`/`SqliteExportService``GenerateUsers`
currently only logs and holds `generated_users_` in memory; no `users`
table exists in the SQLite export yet (see §5).
- [x] Add `UserPersona` (`name`, `description`, `style_affinities`).
- [x] Add `postal_regex` (`std::vector<std::string>`) and
`postal_code_examples` (`std::vector<std::string>`) to `Location`
(now `City` — see below), sourced from each `cities.json` entry's
`postal_code.city_regex` / `postal_code.examples`. Not part of the
originally planned `Location` shape in `diagrams/planned/class.puml`
added ahead of postal code generation work; see §9.
- [x] Renamed `Location``City` throughout the pipeline (struct, all
parameter/return types, `LocationsList``CityList`,
`LoadLocations()``LoadCities()`) so the domain vocabulary matches the
web backend's `City`/`CityId`. Field/variable names that merely hold a
`City` instance (e.g. `EnrichedCity::location`, `BreweryRecord::location`)
were left as-is — `City` itself has a `city` field (the city name
string), so renaming the container field too would read as
`enriched_city.city.city`. The SQLite export layer's own vocabulary
(`locations` table, `location_id` column, `location_cache_`, etc.) was
renamed to `cities`/`city_id` to match, since it's this pipeline's own
throwaway export, not the backend's SQL Server schema (see §5).
- [x] Added a new `Address` struct (`postal_code` only, for now) and a
`BreweryRecord::address` member, mirroring the web backend's
`BreweryPostLocation` (`AddressLine1`, `AddressLine2`, `PostalCode`,
`CityId`). Only `postal_code` is populated — `address_line1`/
`address_line2` generation has no fixture data or service yet. Users
do not get an `Address`: the backend's `UserAccount` has no address
fields, only `BreweryPostLocation` does.
## 2. Data Preloading
`tooling/pipeline/includes/services/curated_data/`, fixture files.
`tooling/pipeline/includes/json_handling/`, fixture files.
- [x] Extract an interface; have `CuratedJsonDataService` implement it.
Landed as `ICuratedDataService`
(`services/curated_data/curated_data_service.h`), not `DataPreloader`
`LoadCities()`, `LoadPersonas()`, `LoadForenamesByCountry()`, and
`LoadSurnamesByCountry()` are all virtual methods returning `const&`
and take no arguments. `CuratedJsonDataService`
(`services/curated_data/curated_json_data_service.h`, originally
landed as `JsonLoader` under `json_handling/` before being renamed and
moved) takes a `CuratedDataFilePaths` DTO (`locations_path`,
`personas_path`, `forenames_path`, `surnames_path`) in its
constructor rather than a path per `Load*()` call, and memoizes each
result in a private `cache` struct (`locations`, `personas`,
`forenames_by_country`, `surnames_by_country`), so a second call on
the same instance returns the cached result instead of re-parsing —
safe because `CuratedJsonDataService` outlives every call site (owned
by `BiergartenPipelineOrchestrator` via
`unique_ptr<ICuratedDataService>` for the whole run). `main.cc`'s DI
injector also gained a `MockCuratedDataService` (fixed in-memory
data: 4 locations across `US`/`DE`/`FR`/`BE`, 3 personas, and
matching forename/surname sets for those four countries), bound in
place of `CuratedJsonDataService` under `--mocked`, mirroring the
existing `MockEnrichmentService`/`MockGenerator` pattern.
- [ ] Extract a `DataPreloader` interface; have `JsonLoader` implement it.
`JsonLoader` now also exposes static `LoadPersonas()` and
`LoadNamesByCountry()` (see below), but as additional static methods,
not yet behind a formal interface.
- [ ] Add `LoadBeerStyles()`. `beer-styles.json` already exists in the repo
and is already copied into the Docker image
(`runpod/Dockerfile`), but no loader reads it yet, and the native
CMake build doesn't copy it into `build/` at all — `CMakeLists.txt`'s
"Runtime Assets" step only copies `cities.json` and `prompts/`.
"Runtime Assets" step only copies `locations.json` and `prompts/`.
- [x] Add `LoadPersonas()` and author `personas.json` (doesn't exist yet).
- [x] Add name-by-country loading, parsing
`tooling/pipeline/forenames-by-country.json` and
`surnames-by-country.json`. Landed as two separate methods —
`LoadForenamesByCountry()` and `LoadSurnamesByCountry()` — each
returning a flat `ForenamesByCountryMap` / `SurnamesByCountryMap`
(aliases for `unordered_map<string, ForenameList>` /
`unordered_map<string, SurnameList>`) keyed by ISO 3166-1 code, rather
than one combined `LoadNamesByCountry() : NamesByCountry` call. Both
files are vendored verbatim (unmodified,
full multinational coverage) from
- [x] Add `LoadNamesByCountry(forenames_filepath, surnames_filepath)`,
parsing `tooling/pipeline/forenames-by-country.json` and
`surnames-by-country.json` into a `NamesByCountry`. Both files are
vendored verbatim (unmodified, full multinational coverage) from
`sigpwned/popular-names-by-country-dataset` (CC0) — see
ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for
provenance. Deliberately not pre-paired or filtered down to just the
countries in `cities.json`: keeping the source shape (including
countries in `locations.json`: keeping the source shape (including
per-forename gender) intact means the loader can support more
countries later, or gender-aware persona/bio generation, without
re-sourcing data.
- [x] Parse `postal_code.city_regex` and `postal_code.examples` out of each
`cities.json` entry's nested `postal_code` object in
`CuratedJsonDataService::LoadCities()`, into `City::postal_regex`
/ `City::postal_code_examples` (see §1). `MockCuratedDataService`'s
4 fixed locations carry matching values pulled from the corresponding
`cities.json` entries.
## 3. Policy / Strategy Layer
@@ -201,25 +136,15 @@ Entirely new — no `includes/policy/` (or equivalent) directory exists today.
`tooling/pipeline/includes/services/database/`, `src/services/sqlite/`.
- [x] `IExportService::ProcessRecord(const UserRecord&)` and
`SqliteExportService`'s `users` table (see
`kCreateUsersTableSql` / `kInsertUserSql` in
`includes/services/database/sqlite_statement_helpers.h`) are
implemented — `GenerateUsers()` exports every successful user
alongside its resolved `city_id`.
- [x] `ProcessRecord(const BreweryRecord&)` now also binds
`BreweryRecord::address.postal_code` into a `postal_code` column on
`breweries` (see §9).
- [ ] Extend `IExportService` with `ProcessBeer`, `ProcessCheckin`,
`ProcessRating`, `ProcessFollow` (today `ProcessRecord(const
BreweryRecord&)` and `ProcessRecord(const UserRecord&)` exist).
- [ ] Add `beers`, `checkins`, `ratings`, `follows` tables to
`SqliteExportService::InitializeSchema()` (`cities`, `breweries`,
and `users` already exist) — see `kCreateCitiesTableSql` /
`kCreateBreweriesTableSql` / `kCreateUsersTableSql` in
- [ ] Extend `IExportService` with `ProcessBeer`, `ProcessUser`,
`ProcessCheckin`, `ProcessRating`, `ProcessFollow` (today only
`ProcessRecord(const GeneratedBrewery&)` exists).
- [ ] Add `beers`, `users`, `checkins`, `ratings`, `follows` tables to
`SqliteExportService::InitializeSchema()` — see
`kCreateLocationsTableSql` / `kCreateBreweriesTableSql` in
`includes/services/database/sqlite_statement_helpers.h` for the
existing pattern to follow.
- [ ] Add a `brewery_cache_` alongside the existing `city_cache_`, and
- [ ] Add a `brewery_cache_` alongside the existing `location_cache_`, and
move from one open transaction for the whole run to per-phase batched
commits (`BEGIN` / `COMMIT & BEGIN` on a batch-size threshold), as
shown in the planned activity diagram.
@@ -262,42 +187,11 @@ Entirely new — no `includes/policy/` (or equivalent) directory exists today.
## 8. Fixtures / Build
- [x] Author `personas.json`, `forenames-by-country.json`, and
`surnames-by-country.json` (see §2).
- [ ] Author `personas.json` and `names-by-country.json` (see §2).
- [ ] Fix the `beer-styles.json` build-tree gap: add it to the "Runtime
Assets" `configure_file` step in `CMakeLists.txt` so native builds and
the Docker image agree on what's available at runtime.
## 9. Postal Code Generation
`tooling/pipeline/includes/services/postal_code/`. Not part of the original
`diagrams/planned/class.puml` sketch — added once `Location` (now `City`)
gained `postal_regex`/`postal_code_examples` (§1, §2).
- [x] Add the `IPostalCodeService` interface —
`GeneratePostalCode(location : const City&) : std::string`.
- [x] Add `MockPostalCodeService`: ignores `location.postal_regex` entirely
and returns `location.postal_code_examples.front()`.
- [ ] **Implement a real, xeger-based `IPostalCodeService`.** A xeger
generator turns a regex pattern (here, one of
`City::postal_regex`) into a random string that matches it — the
inverse of matching a string against a regex. Without it, every
generated postal code is either a hardcoded mock value or absent
entirely; a real implementation is required before postal codes can
vary per generated brewery instead of always reusing the same
curated example.
- [x] Wire `IPostalCodeService` into `main.cc`'s Boost.DI injector — bound
unconditionally to `MockPostalCodeService` (no `--mocked` branch yet,
since a real implementation doesn't exist to switch to; add one once
the xeger-based generator above lands, mirroring `IEnrichmentService`)
— and into `BiergartenPipelineOrchestrator::GenerateBreweries()`,
which calls `GeneratePostalCode(location)` per brewery and stores the
result on `BreweryRecord::address` (a new `Address` struct — see §1),
exported to the `breweries.postal_code` SQLite column (§5).
Postal-code generation is per-brewery, not per-city or per-user: users
don't have an address concept in the web backend, only breweries do
(`BreweryPostLocation`), so `UserRecord` was deliberately not extended.
---
## Suggested build order
@@ -315,7 +209,3 @@ populated. Phase 1b may begin.", etc.) imply a dependency order. Roughly:
planned diagrams.
5. Enrichment cache pre-warming and the city/region query decision (§7) —
useful at any point, but most valuable once phases run concurrently.
6. Postal code generation (§9) is independent of the phases above — the
xeger-based generator just needs `City::postal_regex`, already in
place. Wiring the mock into brewery output has already landed (§9); the
real generator can be swapped in once built.

View File

@@ -69,14 +69,14 @@ end note
:QueryCitiesWithCountries();
|#E2EBDC|JsonLoader|
:JsonLoader::LoadCities("cities.json");
:JsonLoader::LoadLocations("locations.json");
:std::ranges::sample(all_locations, --location-count);
note right
--location-count defaults to 10.
end note
|#EAF0E8|BiergartenPipelineOrchestrator|
while (For each sampled City?) is (Remaining cities)
while (For each sampled Location?) is (Remaining cities)
|#DCE8D8|IEnrichmentService|
:GetLocationContext(loc);
note right
@@ -89,78 +89,7 @@ while (For each sampled City?) is (Remaining cities)
if (Lookup failed?) then (yes)
:Log warning, skip city;
else (no)
:Store EnrichedCity{City, region_context};
endif
endwhile (Done)
|#EAF0E8|BiergartenPipelineOrchestrator|
:GenerateUsers(enriched_cities);
|#E2EBDC|JsonLoader|
:LoadPersonas("personas.json");
:LoadForenamesByCountry("forenames-by-country.json");
:LoadSurnamesByCountry("surnames-by-country.json");
note right
Each call is cached after its first parse.
MockCuratedDataService (--mocked) returns a
fixed in-memory dataset instead and skips
these three JSON files entirely.
end note
|#EAF0E8|BiergartenPipelineOrchestrator|
while (For each EnrichedCity?) is (Remaining cities)
:SampleName(forenames_by_country,\n surnames_by_country, location.iso3166_1);
if (Names available for country?) then (no)
:Log warning, skip city;
else (yes)
:Sample UserPersona (uniform random);
|#E5EDE1|DataGenerator|
if (Generator Mode) then (MockGenerator)
:DeterministicHash & Format;
else (LlamaGenerator)
:prompt_directory_->Load("USER_GENERATION");
note right
Resolves to USER_GENERATION.md inside --prompt-dir.
end note
repeat
:Infer(system_prompt, user_prompt, max_tokens, kUserJsonGrammar);
:ValidateUserJson(raw, user);
if (Is JSON Valid?) then (yes)
break
else (no)
:Attempt++;
:Append validation error to retry prompt;
endif
repeat while (Attempt < 3?) is (yes)
endif
|#EAF0E8|BiergartenPipelineOrchestrator|
if (Generation successful?) then (yes)
:BuildEmail(name, used_email_local_parts);
:GenerateDateOfBirth(rng);
note right
email and date_of_birth are generated
programmatically, never LLM-authored.
end note
|#E0EAE0|SqliteExportService|
:ProcessRecord(UserRecord);
if (City in cache?) then (yes)
:Reuse city_id;
else (no)
:Insert City & Cache ID;
endif
:Insert User (FK: city_id);
if (Exception caught during insert?) then (yes)
|#EAF0E8|BiergartenPipelineOrchestrator|
:Log warning "SQLite export failed";
else (no)
endif
else (no)
:Log warning "Generation failed, skipping...";
endif
:Store EnrichedCity{Location, region_context};
endif
endwhile (Done)
@@ -197,12 +126,12 @@ while (For each EnrichedCity?) is (Remaining cities)
if (Generation successful?) then (yes)
|#E0EAE0|SqliteExportService|
:ProcessRecord(GeneratedBrewery);
if (City in cache?) then (yes)
:Reuse city_id;
if (Location in cache?) then (yes)
:Reuse location_id;
else (no)
:Insert City & Cache ID;
:Insert Location & Cache ID;
endif
:Insert Brewery (FK: city_id);
:Insert Brewery (FK: location_id);
if (Exception caught during insert?) then (yes)
|#EAF0E8|BiergartenPipelineOrchestrator|

View File

@@ -30,15 +30,11 @@ class BiergartenPipelineOrchestrator {
- context_service_ : std::unique_ptr<IEnrichmentService>
- generator_ : std::unique_ptr<DataGenerator>
- exporter_ : std::unique_ptr<IExportService>
- curated_data_service_ : std::unique_ptr<ICuratedDataService>
- postal_code_service_ : std::unique_ptr<IPostalCodeService>
- application_options_ : ApplicationOptions
- generated_breweries_ : std::vector<BreweryRecord>
- generated_users_ : std::vector<UserRecord>
- generated_breweries_ : std::vector<GeneratedBrewery>
+ Run() : bool
- QueryCitiesWithCountries() : std::vector<City>
- QueryCitiesWithCountries() : std::vector<Location>
- GenerateBreweries(cities : std::span<const EnrichedCity>) : void
- GenerateUsers(cities : std::span<const EnrichedCity>) : void
- LogResults() : void
}
@@ -91,17 +87,17 @@ class LogDispatcher {
}
interface IEnrichmentService <<interface>> {
+ GetLocationContext(loc : const City&) : std::string
+ GetLocationContext(loc : const Location&) : std::string
}
class MockEnrichmentService {
+ GetLocationContext(loc : const City&) : std::string
+ GetLocationContext(loc : const Location&) : std::string
}
class WikipediaEnrichmentService {
- client_ : std::unique_ptr<WebClient>
- extract_cache_ : std::unordered_map<std::string, std::string>
+ GetLocationContext(loc : const City&) : std::string
+ GetLocationContext(loc : const Location&) : std::string
- FetchExtract(query : std::string_view) : std::string
}
@@ -116,15 +112,14 @@ class HttpWebClient {
}
interface DataGenerator <<interface>> {
+ GenerateBrewery(location : const City&, region_context : const std::string&) : BreweryResult
+ GenerateUser(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult
+ GenerateBrewery(location : const Location&, region_context : const std::string&) : BreweryResult
+ GenerateUser(locale : const std::string&) : UserResult
}
class MockGenerator {
+ GenerateBrewery(location : const City&, region_context : const std::string&) : BreweryResult
+ GenerateUser(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult
- DeterministicHash(location : const City&) : size_t
- DeterministicHash(location : const City&, persona : const UserPersona&, name : const Name&) : size_t
+ GenerateBrewery(...) : BreweryResult
+ GenerateUser(...) : UserResult
- DeterministicHash(location : const Location&) : size_t
}
class LlamaGenerator {
@@ -158,70 +153,13 @@ class PromptDirectory {
+ Load(key : std::string_view) : std::string
}
interface ICuratedDataService <<interface>> {
+ LoadCities(filepath : const std::filesystem::path&) : const std::vector<City>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
}
class JsonLoader {
- cache_ : cache
+ LoadCities(filepath : const std::filesystem::path&) : const std::vector<City>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
+ {static} LoadLocations(filepath : const std::filesystem::path&) : std::vector<Location>
}
note right of JsonLoader
Each Load* method memoizes its result in
cache_ on first call; later calls on the
same instance return the cached value
without re-parsing.
end note
class MockCuratedDataService {
- locations_ : std::vector<City>
- personas_ : std::vector<UserPersona>
- forenames_by_country_ : std::unordered_map<std::string, forename_list>
- surnames_by_country_ : std::unordered_map<std::string, surname_list>
+ LoadCities(filepath : const std::filesystem::path&) : const std::vector<City>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
}
note right of MockCuratedDataService
Fixed 4-location / 3-persona dataset
(US, DE, FR, BE) used in --mocked runs;
filepath arguments are ignored.
end note
interface IPostalCodeService <<interface>> {
+ GeneratePostalCode(location : const City&) : std::string
}
class MockPostalCodeService {
+ GeneratePostalCode(location : const City&) : std::string
}
note right of MockPostalCodeService
Ignores location.postal_regex and returns
location.postal_code_examples.front(). Bound
unconditionally in main.cc's DI graph (no
--mocked branch yet -- no real implementation
exists to switch to) and called from
GenerateBreweries() per brewery, stored on
BreweryRecord::address. A real implementation
needs a xeger-style generator to turn
postal_regex into a synthesized matching
string. See ROADMAP.md §9.
end note
interface IExportService <<interface>> {
+ Initialize() : void
+ ProcessRecord(brewery : const BreweryRecord&) : uint64_t
+ ProcessRecord(user : const UserRecord&) : uint64_t
+ ProcessRecord(brewery : const GeneratedBrewery&) : void
+ Finalize() : void
}
@@ -229,18 +167,15 @@ class SqliteExportService {
- date_time_provider_ : std::unique_ptr<IDateTimeProvider>
- run_timestamp_utc_ : std::string
- database_path_ : std::filesystem::path
- db_handle_ : SqliteDatabaseHandle
- insert_city_stmt_ : SqliteStatementHandle
- insert_brewery_stmt_ : SqliteStatementHandle
- insert_user_stmt_ : SqliteStatementHandle
- db_handle_ : sqlite3*
- insert_location_stmt_ : sqlite3_stmt*
- insert_brewery_stmt_ : sqlite3_stmt*
- transaction_open_ : bool
- city_cache_ : std::unordered_map<std::string, sqlite3_int64>
- location_cache_ : std::unordered_map<std::string, sqlite3_int64>
+ Initialize() : void
+ ProcessRecord(brewery : const BreweryRecord&) : uint64_t
+ ProcessRecord(user : const UserRecord&) : uint64_t
+ ProcessRecord(brewery : const GeneratedBrewery&) : void
+ Finalize() : void
- InitializeSchema() : void
- ResolveCityId(location : const City&) : sqlite3_int64
}
interface IDateTimeProvider <<interface>> {
@@ -256,8 +191,6 @@ BiergartenPipelineOrchestrator *-- ILogger : owns
BiergartenPipelineOrchestrator *-- IEnrichmentService : owns
BiergartenPipelineOrchestrator *-- DataGenerator : owns
BiergartenPipelineOrchestrator *-- IExportService : owns
BiergartenPipelineOrchestrator *-- ICuratedDataService : owns
BiergartenPipelineOrchestrator *-- IPostalCodeService : owns
LogEntry *-- LogLevel
LogEntry *-- PipelinePhase
@@ -282,10 +215,7 @@ LlamaGenerator *-- IPromptDirectory : uses
IPromptFormatter <|.. Gemma4JinjaPromptFormatter : implements
IPromptDirectory <|.. PromptDirectory : implements
ICuratedDataService <|.. JsonLoader : implements
ICuratedDataService <|.. MockCuratedDataService : implements
IPostalCodeService <|.. MockPostalCodeService : implements
BiergartenPipelineOrchestrator ..> JsonLoader : uses
IExportService <|.. SqliteExportService : implements
SqliteExportService *-- IDateTimeProvider : owns

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 86 KiB

After

Width:  |  Height:  |  Size: 55 KiB

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 94 KiB

After

Width:  |  Height:  |  Size: 71 KiB

View File

@@ -39,22 +39,14 @@ fork
:JsonLoader::LoadBeerStyles("beer-styles.json");
:EnrichmentService::PreWarmBeerStyleCache(beer_styles);
fork again
:JsonLoader::LoadCities("cities.json");
:JsonLoader::LoadLocations("locations.json");
:EnrichmentService::PreWarmLocationCache(sampled_locations);
end fork
fork
:JsonLoader::LoadForenamesByCountry(\n "forenames-by-country.json");
fork again
:JsonLoader::LoadSurnamesByCountry(\n "surnames-by-country.json");
:JsonLoader::LoadNamesByCountry(\n "forenames-by-country.json",\n "surnames-by-country.json");
fork again
:JsonLoader::LoadPersonas("personas.json");
end fork
note right
Each call is memoized after its first parse
(implemented today). MockCuratedDataService
returns a fixed in-memory dataset instead
under --mocked, skipping these JSON files.
end note
' ═══════════════════════════════════════════
' PHASE 0 — USER GENERATION
@@ -75,7 +67,7 @@ fork
fork again
|LLM Worker|
while (loc_ch has items?) is (yes)
:Receive City;
:Receive Location;
:GetLocationContextFromCache(location);
note right
@@ -90,15 +82,11 @@ fork again
ibu_preference, checkin_weight.
end note
:SampleName(forenames_by_country,\n surnames_by_country, location.iso3166_1, rng);
:NamesByCountry::SampleName(\n location.iso3166_1);
note right
Deterministic lookup -- no LLM involved.
Free function (implemented today in
generate_users.cc), not a class method.
Name selected from pre-keyed maps
Name selected from pre-keyed table
and passed into the generation prompt.
Skips the city if either map has no
entry for iso3166_1.
end note
:GenerateUser(enriched_city, persona, sampled_name)\nvia DataGenerator;
@@ -146,13 +134,13 @@ end fork
fork
|Orchestrator|
:Loop: Sample User from user_pool_
and pair with City;
:Send BreweryTask(City, User) -> loc_ch;
and pair with Location;
:Send BreweryTask(Location, User) -> loc_ch;
:Close loc_ch;
fork again
|LLM Worker|
while (loc_ch has items?) is (yes)
:Receive BreweryTask(City, User);
:Receive BreweryTask(Location, User);
:GetLocationContextFromCache(task.location);
note right

View File

@@ -12,40 +12,17 @@ title Biergarten Data Pipeline — Class Diagram
package "Domain: Models" {
class City {
class Location {
+ city : std::string
+ state_province : std::string
+ iso3166_2 : std::string
+ country : std::string
+ iso3166_1 : std::string
+ postal_regex : std::vector<std::string>
+ postal_code_examples : std::vector<std::string>
+ local_languages : std::vector<std::string>
+ latitude : double
+ longitude : double
}
note right of City
postal_regex / postal_code_examples are
sourced from each cities.json entry's
postal_code.city_regex / postal_code.examples
(implemented today, ahead of the rest of this
diagram -- see ROADMAP.md §1/§2). Renamed from
Location to City to match the web backend's
City/CityId vocabulary -- see ROADMAP.md §1.
end note
class Address {
+ postal_code : std::string
}
note right of Address
Mirrors the web backend's BreweryPostLocation.
Only postal_code is populated today --
address_line1/address_line2 have no fixture
data or generator yet. See ROADMAP.md §9.
end note
class LocationContext {
+ text : std::string
+ completeness : Completeness
@@ -59,7 +36,7 @@ package "Domain: Models" {
}
class EnrichedCity {
+ location : City
+ location : Location
+ context : LocationContext
}
@@ -109,30 +86,27 @@ package "Domain: Models" {
+ gender : std::string
}
note right of ForenameEntry
forename_list = std::unordered_set<ForenameEntry>
surname_list = std::unordered_set<std::string>
(aliases declared in curated_data_service.h).
ICuratedDataService::LoadForenamesByCountry() /
LoadSurnamesByCountry() each return a flat
std::unordered_map<std::string, forename_list>
or <std::string, surname_list> keyed by ISO
3166-1 country code -- no NamesByCountry
wrapper class. SampleName(forenames_by_country,
surnames_by_country, iso3166_1, rng) is a free
function, not a method, so it can be called by
any per-city worker without owning the maps.
Sourced from popular-names-by-country-dataset
(CC0) -- see ETHICS-AND-KNOWN-ISSUES.md. Gender
is preserved per forename so it can be threaded
through to persona/bio generation later; surnames
carry no gender in the source data. Pairing
happens at sample time, not at fixture-authoring
time, so no information from the source dataset
is discarded up front.
class NamesByCountry {
- forenames_by_country_ : std::unordered_map<std::string, std::vector<ForenameEntry>>
- surnames_by_country_ : std::unordered_map<std::string, std::vector<std::string>>
+ SampleName(iso3166_1 : const std::string&,\n rng : std::mt19937&) : std::optional<Name>
}
note right of NamesByCountry
Backed by two source-shaped maps,
keyed by ISO 3166-1 country code,
sourced from popular-names-by-country-dataset
(CC0) -- see ETHICS-AND-KNOWN-ISSUES.md.
Gender is preserved per forename so it can
be threaded through to persona/bio generation
later; surnames carry no gender in the source
data. Pairing happens at sample time, not at
fixture-authoring time, so no information from
the source dataset is discarded up front.
end note
ForenameEntry ..> Name : SampleName() produces
NamesByCountry *-- "0..*" ForenameEntry
NamesByCountry ..> Name : produces
class CheckinResult {
+ checked_in_at : std::string
@@ -153,8 +127,7 @@ package "Domain: Models" {
class GeneratedBrewery {
+ brewery_id : uint64_t
+ location : City
+ address : Address
+ location : Location
+ brewery : BreweryResult
+ context_completeness : LocationContext::Completeness
+ metadata : GenerationMetadata
@@ -163,7 +136,7 @@ package "Domain: Models" {
class GeneratedBeer {
+ beer_id : uint64_t
+ brewery_id : uint64_t
+ location : City
+ location : Location
+ style : BeerStyle
+ beer : BeerResult
+ metadata : GenerationMetadata
@@ -171,7 +144,7 @@ package "Domain: Models" {
class GeneratedUser {
+ user_id : uint64_t
+ location : City
+ location : Location
+ user : UserResult
+ email : std::string
+ date_of_birth : std::string
@@ -251,27 +224,27 @@ package "Domain: Application Configuration" {
package "Domain: Policy" {
interface ContextStrategy <<interface>> {
+ QueriesFor(loc : const City&) : std::vector<std::string>
+ QueriesFor(loc : const Location&) : std::vector<std::string>
+ MaxContextChars() : size_t
}
class BreweryContextStrategy {
+ QueriesFor(loc : const City&) : std::vector<std::string>
+ QueriesFor(loc : const Location&) : std::vector<std::string>
+ MaxContextChars() : size_t
}
class BeerContextStrategy {
+ QueriesFor(loc : const City&) : std::vector<std::string>
+ QueriesFor(loc : const Location&) : std::vector<std::string>
+ MaxContextChars() : size_t
}
interface SamplingStrategy <<interface>> {
+ Sample(locations : const std::vector<City>&) : std::vector<City>
+ Sample(locations : const std::vector<Location>&) : std::vector<Location>
}
class UniformSamplingStrategy {
- sample_size_ : size_t
+ Sample(locations : const std::vector<City>&) : std::vector<City>
+ Sample(locations : const std::vector<Location>&) : std::vector<Location>
}
interface BeerSelectionStrategy <<interface>> {
@@ -392,55 +365,32 @@ package "Infrastructure: Pipeline Channel" {
package "Infrastructure: Data Preloading" {
interface ICuratedDataService <<interface>> {
+ LoadCities(filepath : const std::filesystem::path&) : const std::vector<City>&
+ LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
interface DataPreloader <<interface>> {
+ LoadLocations(filepath : const std::filesystem::path&) : std::vector<Location>
+ LoadBeerStyles(filepath : const std::filesystem::path&) : std::vector<BeerStyle>
+ LoadPersonas(filepath : const std::filesystem::path&) : std::vector<UserPersona>
+ LoadNamesByCountry(forenames_filepath : const std::filesystem::path&,\n surnames_filepath : const std::filesystem::path&) : NamesByCountry
}
class JsonLoader {
+ LoadCities(filepath : const std::filesystem::path&) : const std::vector<City>&
+ LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
+ LoadLocations(filepath : const std::filesystem::path&) : std::vector<Location>
+ LoadBeerStyles(filepath : const std::filesystem::path&) : std::vector<BeerStyle>
+ LoadPersonas(filepath : const std::filesystem::path&) : std::vector<UserPersona>
+ LoadNamesByCountry(forenames_filepath : const std::filesystem::path&,\n surnames_filepath : const std::filesystem::path&) : NamesByCountry
}
note right of JsonLoader
Each Load* call is memoized after its first
parse -- implemented today, ahead of the rest
of this package (LoadBeerStyles is still planned).
end note
class MockCuratedDataService {
+ LoadCities(filepath : const std::filesystem::path&) : const std::vector<City>&
+ LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
}
note right of MockCuratedDataService
Fixed in-memory dataset used in --mocked mode,
implemented today for Locations/Personas/
Forenames/Surnames; would need a LoadBeerStyles
fixture too once that method is implemented.
end note
}
package "Infrastructure: Enrichment" {
interface EnrichmentService <<interface>> {
+ GetLocationContext(loc : const City&,\n strategy : const ContextStrategy&) : LocationContext
+ GetLocationContext(loc : const Location&,\n strategy : const ContextStrategy&) : LocationContext
}
class WikipediaService {
- client_ : std::unique_ptr<WebClient>
- extract_cache_ : std::unordered_map<std::string, std::string>
+ GetLocationContext(loc : const City&,\n strategy : const ContextStrategy&) : LocationContext
+ GetLocationContext(loc : const Location&,\n strategy : const ContextStrategy&) : LocationContext
- FetchExtract(query : std::string_view) : std::string
}
@@ -456,39 +406,6 @@ package "Infrastructure: Enrichment" {
}
package "Infrastructure: Postal Code Generation" {
interface IPostalCodeService <<interface>> {
+ GeneratePostalCode(location : const City&) : std::string
}
class MockPostalCodeService {
+ GeneratePostalCode(location : const City&) : std::string
}
class XegerPostalCodeService {
+ GeneratePostalCode(location : const City&) : std::string
}
note right of MockPostalCodeService
Implemented today: ignores postal_regex and
returns postal_code_examples.front(). Not yet
bound in main.cc's DI graph or used by
BiergartenPipelineOrchestrator.
end note
note right of XegerPostalCodeService
Not implemented yet. Picks one of
location.postal_regex and generates a random
string matching it (a "xeger" -- the inverse of
regex matching) instead of always replaying a
fixed example. See ROADMAP.md §9.
end note
IPostalCodeService <|.. MockPostalCodeService
IPostalCodeService <|.. XegerPostalCodeService
}
package "Infrastructure: Prompting" {
interface IPromptDirectory <<interface>> {
@@ -508,8 +425,8 @@ package "Infrastructure: Prompting" {
package "Infrastructure: Data Generation" {
interface DataGenerator <<interface>> {
+ GenerateBrewery(location : const City&,\n context : const LocationContext&) : BreweryResult
+ GenerateBeer(brewery_id : uint64_t,\n location : const City&,\n context : const LocationContext&,\n style : const BeerStyle&) : BeerResult
+ GenerateBrewery(location : const Location&,\n context : const LocationContext&) : BreweryResult
+ GenerateBeer(brewery_id : uint64_t,\n location : const Location&,\n context : const LocationContext&,\n style : const BeerStyle&) : BeerResult
+ GenerateUser(city : const EnrichedCity&,\n persona : const UserPersona&,\n name : const Name&) : UserResult
+ GenerateCheckin(user : const GeneratedUser&,\n brewery : const GeneratedBrewery&,\n timestamp : const std::string&) : CheckinResult
+ GenerateRating(user : const GeneratedUser&,\n beer : const GeneratedBeer&,\n checkin_id : uint64_t) : RatingResult
@@ -521,7 +438,7 @@ package "Infrastructure: Data Generation" {
+ GenerateUser(...) : UserResult
+ GenerateCheckin(...) : CheckinResult
+ GenerateRating(...) : RatingResult
- DeterministicHash(location : const City&) : size_t
- DeterministicHash(location : const Location&) : size_t
}
class LlamaGenerator {
@@ -603,7 +520,7 @@ package "Infrastructure: Data Export" {
}
class BiergartenPipelineOrchestrator {
- curated_data_service_ : std::unique_ptr<ICuratedDataService>
- preloader_ : std::unique_ptr<DataPreloader>
- enrichment_service_ : std::unique_ptr<EnrichmentService>
- generator_ : std::unique_ptr<DataGenerator>
- logger_ : std::unique_ptr<Logger>
@@ -623,15 +540,15 @@ class BiergartenPipelineOrchestrator {
- follow_pool_ : std::vector<GeneratedFollow>
--
+ Run() : bool
- RunUserPhase(locations : const std::vector<City>&) : void
- RunBreweryAndBeerPhase(locations : const std::vector<City>&) : void
- RunUserPhase(locations : const std::vector<Location>&) : void
- RunBreweryAndBeerPhase(locations : const std::vector<Location>&) : void
- RunCheckinPhase() : void
- RunRatingPhase() : void
- RunFollowPhase() : void
}
' --- Orchestration Aggregations (Services & Strategies) ---
BiergartenPipelineOrchestrator *-- ICuratedDataService
BiergartenPipelineOrchestrator *-- DataPreloader
BiergartenPipelineOrchestrator *-- EnrichmentService
BiergartenPipelineOrchestrator *-- DataGenerator
BiergartenPipelineOrchestrator *-- ExportService
@@ -650,8 +567,7 @@ BiergartenPipelineOrchestrator *-- "0..*" GeneratedCheckin : checkin_pool_
BiergartenPipelineOrchestrator *-- "0..*" GeneratedFollow : follow_pool_
' --- Interfaces & Implementations ---
ICuratedDataService <|.. JsonLoader
ICuratedDataService <|.. MockCuratedDataService
DataPreloader <|.. JsonLoader
Logger <|.. PipelineLogger
ContextStrategy <|.. BreweryContextStrategy
ContextStrategy <|.. BeerContextStrategy
@@ -682,17 +598,16 @@ PipelineLogger o-- BoundedChannel : logs to
LogWorker o-- BoundedChannel : drains from
' --- Domain Containment ---
EnrichedCity *-- City
EnrichedCity *-- Location
EnrichedCity *-- LocationContext
GeneratedBrewery *-- City
GeneratedBrewery *-- Address
GeneratedBrewery *-- Location
GeneratedBrewery *-- BreweryResult
GeneratedBrewery *-- GenerationMetadata
GeneratedBeer *-- City
GeneratedBeer *-- Location
GeneratedBeer *-- BeerStyle
GeneratedBeer *-- BeerResult
GeneratedBeer *-- GenerationMetadata
GeneratedUser *-- City
GeneratedUser *-- Location
GeneratedUser *-- UserResult
GeneratedUser *-- GenerationMetadata
GeneratedCheckin *-- CheckinResult

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 142 KiB

After

Width:  |  Height:  |  Size: 139 KiB

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 257 KiB

After

Width:  |  Height:  |  Size: 234 KiB

View File

@@ -11,35 +11,7 @@
<div class="dialog-header">
<p>Pipeline Results Viewer</p>
</div>
<div class="tab-strip" role="tablist">
<button
type="button"
class="tab is-active"
id="tab-tables"
role="tab"
aria-selected="true"
aria-controls="panel-tables"
>
Tables
</button>
<button
type="button"
class="tab"
id="tab-query"
role="tab"
aria-selected="false"
aria-controls="panel-query"
>
Query
</button>
</div>
<div class="dialog-body">
<div
class="tab-panel"
id="panel-tables"
role="tabpanel"
aria-labelledby="tab-tables"
>
<div class="field-group">
<label for="table-select">Table</label>
<select id="table-select"></select>
@@ -47,27 +19,6 @@
<p id="status"></p>
<div id="table-container"></div>
</div>
<div
class="tab-panel"
id="panel-query"
role="tabpanel"
aria-labelledby="tab-query"
hidden
>
<div class="field-group field-group--stacked">
<label for="query-input">SQL Query</label>
<textarea
id="query-input"
rows="4"
spellcheck="false"
placeholder="SELECT * FROM ..."
></textarea>
</div>
<button type="button" id="query-run" class="btn">Run Query</button>
<p id="query-status"></p>
<div id="query-container"></div>
</div>
</div>
</div>
<script type="module" src="/src/main.ts"></script>
</body>

View File

@@ -8,11 +8,11 @@
"name": "pipeline-results-viewer",
"version": "0.0.0",
"dependencies": {
"95css": "^0.4.2",
"sql.js": "^1.14.1"
},
"devDependencies": {
"@types/sql.js": "^1.4.11",
"sass": "^1.101.0",
"typescript": "^6.0.3",
"vite": "^8.0.12"
}
@@ -80,316 +80,6 @@
"url": "https://github.com/sponsors/Boshen"
}
},
"node_modules/@parcel/watcher": {
"version": "2.5.6",
"resolved": "https://registry.npmjs.org/@parcel/watcher/-/watcher-2.5.6.tgz",
"integrity": "sha512-tmmZ3lQxAe/k/+rNnXQRawJ4NjxO2hqiOLTHvWchtGZULp4RyFeh6aU4XdOYBFe2KE1oShQTv4AblOs2iOrNnQ==",
"dev": true,
"hasInstallScript": true,
"license": "MIT",
"optional": true,
"dependencies": {
"detect-libc": "^2.0.3",
"is-glob": "^4.0.3",
"node-addon-api": "^7.0.0",
"picomatch": "^4.0.3"
},
"engines": {
"node": ">= 10.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
},
"optionalDependencies": {
"@parcel/watcher-android-arm64": "2.5.6",
"@parcel/watcher-darwin-arm64": "2.5.6",
"@parcel/watcher-darwin-x64": "2.5.6",
"@parcel/watcher-freebsd-x64": "2.5.6",
"@parcel/watcher-linux-arm-glibc": "2.5.6",
"@parcel/watcher-linux-arm-musl": "2.5.6",
"@parcel/watcher-linux-arm64-glibc": "2.5.6",
"@parcel/watcher-linux-arm64-musl": "2.5.6",
"@parcel/watcher-linux-x64-glibc": "2.5.6",
"@parcel/watcher-linux-x64-musl": "2.5.6",
"@parcel/watcher-win32-arm64": "2.5.6",
"@parcel/watcher-win32-ia32": "2.5.6",
"@parcel/watcher-win32-x64": "2.5.6"
}
},
"node_modules/@parcel/watcher-android-arm64": {
"version": "2.5.6",
"resolved": "https://registry.npmjs.org/@parcel/watcher-android-arm64/-/watcher-android-arm64-2.5.6.tgz",
"integrity": "sha512-YQxSS34tPF/6ZG7r/Ih9xy+kP/WwediEUsqmtf0cuCV5TPPKw/PQHRhueUo6JdeFJaqV3pyjm0GdYjZotbRt/A==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"android"
],
"engines": {
"node": ">= 10.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
}
},
"node_modules/@parcel/watcher-darwin-arm64": {
"version": "2.5.6",
"resolved": "https://registry.npmjs.org/@parcel/watcher-darwin-arm64/-/watcher-darwin-arm64-2.5.6.tgz",
"integrity": "sha512-Z2ZdrnwyXvvvdtRHLmM4knydIdU9adO3D4n/0cVipF3rRiwP+3/sfzpAwA/qKFL6i1ModaabkU7IbpeMBgiVEA==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"darwin"
],
"engines": {
"node": ">= 10.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
}
},
"node_modules/@parcel/watcher-darwin-x64": {
"version": "2.5.6",
"resolved": "https://registry.npmjs.org/@parcel/watcher-darwin-x64/-/watcher-darwin-x64-2.5.6.tgz",
"integrity": "sha512-HgvOf3W9dhithcwOWX9uDZyn1lW9R+7tPZ4sug+NGrGIo4Rk1hAXLEbcH1TQSqxts0NYXXlOWqVpvS1SFS4fRg==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"darwin"
],
"engines": {
"node": ">= 10.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
}
},
"node_modules/@parcel/watcher-freebsd-x64": {
"version": "2.5.6",
"resolved": "https://registry.npmjs.org/@parcel/watcher-freebsd-x64/-/watcher-freebsd-x64-2.5.6.tgz",
"integrity": "sha512-vJVi8yd/qzJxEKHkeemh7w3YAn6RJCtYlE4HPMoVnCpIXEzSrxErBW5SJBgKLbXU3WdIpkjBTeUNtyBVn8TRng==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"freebsd"
],
"engines": {
"node": ">= 10.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
}
},
"node_modules/@parcel/watcher-linux-arm-glibc": {
"version": "2.5.6",
"resolved": "https://registry.npmjs.org/@parcel/watcher-linux-arm-glibc/-/watcher-linux-arm-glibc-2.5.6.tgz",
"integrity": "sha512-9JiYfB6h6BgV50CCfasfLf/uvOcJskMSwcdH1PHH9rvS1IrNy8zad6IUVPVUfmXr+u+Km9IxcfMLzgdOudz9EQ==",
"cpu": [
"arm"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">= 10.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
}
},
"node_modules/@parcel/watcher-linux-arm-musl": {
"version": "2.5.6",
"resolved": "https://registry.npmjs.org/@parcel/watcher-linux-arm-musl/-/watcher-linux-arm-musl-2.5.6.tgz",
"integrity": "sha512-Ve3gUCG57nuUUSyjBq/MAM0CzArtuIOxsBdQ+ftz6ho8n7s1i9E1Nmk/xmP323r2YL0SONs1EuwqBp2u1k5fxg==",
"cpu": [
"arm"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">= 10.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
}
},
"node_modules/@parcel/watcher-linux-arm64-glibc": {
"version": "2.5.6",
"resolved": "https://registry.npmjs.org/@parcel/watcher-linux-arm64-glibc/-/watcher-linux-arm64-glibc-2.5.6.tgz",
"integrity": "sha512-f2g/DT3NhGPdBmMWYoxixqYr3v/UXcmLOYy16Bx0TM20Tchduwr4EaCbmxh1321TABqPGDpS8D/ggOTaljijOA==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">= 10.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
}
},
"node_modules/@parcel/watcher-linux-arm64-musl": {
"version": "2.5.6",
"resolved": "https://registry.npmjs.org/@parcel/watcher-linux-arm64-musl/-/watcher-linux-arm64-musl-2.5.6.tgz",
"integrity": "sha512-qb6naMDGlbCwdhLj6hgoVKJl2odL34z2sqkC7Z6kzir8b5W65WYDpLB6R06KabvZdgoHI/zxke4b3zR0wAbDTA==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">= 10.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
}
},
"node_modules/@parcel/watcher-linux-x64-glibc": {
"version": "2.5.6",
"resolved": "https://registry.npmjs.org/@parcel/watcher-linux-x64-glibc/-/watcher-linux-x64-glibc-2.5.6.tgz",
"integrity": "sha512-kbT5wvNQlx7NaGjzPFu8nVIW1rWqV780O7ZtkjuWaPUgpv2NMFpjYERVi0UYj1msZNyCzGlaCWEtzc+exjMGbQ==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">= 10.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
}
},
"node_modules/@parcel/watcher-linux-x64-musl": {
"version": "2.5.6",
"resolved": "https://registry.npmjs.org/@parcel/watcher-linux-x64-musl/-/watcher-linux-x64-musl-2.5.6.tgz",
"integrity": "sha512-1JRFeC+h7RdXwldHzTsmdtYR/Ku8SylLgTU/reMuqdVD7CtLwf0VR1FqeprZ0eHQkO0vqsbvFLXUmYm/uNKJBg==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">= 10.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
}
},
"node_modules/@parcel/watcher-win32-arm64": {
"version": "2.5.6",
"resolved": "https://registry.npmjs.org/@parcel/watcher-win32-arm64/-/watcher-win32-arm64-2.5.6.tgz",
"integrity": "sha512-3ukyebjc6eGlw9yRt678DxVF7rjXatWiHvTXqphZLvo7aC5NdEgFufVwjFfY51ijYEWpXbqF5jtrK275z52D4Q==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"win32"
],
"engines": {
"node": ">= 10.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
}
},
"node_modules/@parcel/watcher-win32-ia32": {
"version": "2.5.6",
"resolved": "https://registry.npmjs.org/@parcel/watcher-win32-ia32/-/watcher-win32-ia32-2.5.6.tgz",
"integrity": "sha512-k35yLp1ZMwwee3Ez/pxBi5cf4AoBKYXj00CZ80jUz5h8prpiaQsiRPKQMxoLstNuqe2vR4RNPEAEcjEFzhEz/g==",
"cpu": [
"ia32"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"win32"
],
"engines": {
"node": ">= 10.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
}
},
"node_modules/@parcel/watcher-win32-x64": {
"version": "2.5.6",
"resolved": "https://registry.npmjs.org/@parcel/watcher-win32-x64/-/watcher-win32-x64-2.5.6.tgz",
"integrity": "sha512-hbQlYcCq5dlAX9Qx+kFb0FHue6vbjlf0FrNzSKdYK2APUf7tGfGxQCk2ihEREmbR6ZMc0MVAD5RIX/41gpUzTw==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"win32"
],
"engines": {
"node": ">= 10.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
}
},
"node_modules/@rolldown/binding-android-arm64": {
"version": "1.0.3",
"resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.0.3.tgz",
@@ -693,21 +383,11 @@
"@types/node": "*"
}
},
"node_modules/chokidar": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/chokidar/-/chokidar-5.0.0.tgz",
"integrity": "sha512-TQMmc3w+5AxjpL8iIiwebF73dRDF4fBIieAqGn9RGCWaEVwQ6Fb2cGe31Yns0RRIzii5goJ1Y7xbMwo1TxMplw==",
"dev": true,
"license": "MIT",
"dependencies": {
"readdirp": "^5.0.0"
},
"engines": {
"node": ">= 20.19.0"
},
"funding": {
"url": "https://paulmillr.com/funding/"
}
"node_modules/95css": {
"version": "0.4.2",
"resolved": "https://registry.npmjs.org/95css/-/95css-0.4.2.tgz",
"integrity": "sha512-lzlM1rTgaGCxQL66qZTTzH7H+fHOMwwOZQsry3FcgoeUTrPp3mFReY4LUFPiTywC+Qeve0OH2VxrjDhHgRyu5A==",
"license": "MIT"
},
"node_modules/detect-libc": {
"version": "2.1.2",
@@ -752,38 +432,6 @@
"node": "^8.16.0 || ^10.6.0 || >=11.0.0"
}
},
"node_modules/immutable": {
"version": "5.1.8",
"resolved": "https://registry.npmjs.org/immutable/-/immutable-5.1.8.tgz",
"integrity": "sha512-TM5YqrGeTsVIPPpILzeqZ8D2Zc2TvNgSDi88zPF2a4cyqQdWV/wVWBDRDbNzzrLeRWScrFcOX9lW2iX6GOtUDw==",
"dev": true,
"license": "MIT"
},
"node_modules/is-extglob": {
"version": "2.1.1",
"resolved": "https://registry.npmjs.org/is-extglob/-/is-extglob-2.1.1.tgz",
"integrity": "sha512-SbKbANkN603Vi4jEZv49LeVJMn4yGwsbzZworEoyEiutsN3nJYdbO36zfhGJ6QEDpOZIFkDtnq5JRxmvl3jsoQ==",
"dev": true,
"license": "MIT",
"optional": true,
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/is-glob": {
"version": "4.0.3",
"resolved": "https://registry.npmjs.org/is-glob/-/is-glob-4.0.3.tgz",
"integrity": "sha512-xelSayHH36ZgE7ZWhli7pW34hNbNl8Ojv5KVmkJD4hBdD3th8Tfk9vYasLM+mXWOZhFkgZfxhLSnrwRr4elSSg==",
"dev": true,
"license": "MIT",
"optional": true,
"dependencies": {
"is-extglob": "^2.1.1"
},
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/lightningcss": {
"version": "1.32.0",
"resolved": "https://registry.npmjs.org/lightningcss/-/lightningcss-1.32.0.tgz",
@@ -1064,14 +712,6 @@
"node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1"
}
},
"node_modules/node-addon-api": {
"version": "7.1.1",
"resolved": "https://registry.npmjs.org/node-addon-api/-/node-addon-api-7.1.1.tgz",
"integrity": "sha512-5m3bsyrjFWE1xf7nz7YXdN4udnVtXK6/Yfgn5qnahL6bCkf2yKt4k3nuTKAtT4r3IG8JNR2ncsIMdZuAzJjHQQ==",
"dev": true,
"license": "MIT",
"optional": true
},
"node_modules/picocolors": {
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz",
@@ -1121,20 +761,6 @@
"node": "^10 || ^12 || >=14"
}
},
"node_modules/readdirp": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/readdirp/-/readdirp-5.0.0.tgz",
"integrity": "sha512-9u/XQ1pvrQtYyMpZe7DXKv2p5CNvyVwzUB6uhLAnQwHMSgKMBR62lc7AHljaeteeHXn11XTAaLLUVZYVZyuRBQ==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">= 20.19.0"
},
"funding": {
"type": "individual",
"url": "https://paulmillr.com/funding/"
}
},
"node_modules/rolldown": {
"version": "1.0.3",
"resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.0.3.tgz",
@@ -1169,27 +795,6 @@
"@rolldown/binding-win32-x64-msvc": "1.0.3"
}
},
"node_modules/sass": {
"version": "1.101.0",
"resolved": "https://registry.npmjs.org/sass/-/sass-1.101.0.tgz",
"integrity": "sha512-OL3GoQyoUdDt843DpVmDO6y2k1sc5IhUDSpu8XucEI+35neq5QivZ1iuegnpraEVTJXlQGK1gl27zKcTLEPbQw==",
"dev": true,
"license": "MIT",
"dependencies": {
"chokidar": "^5.0.0",
"immutable": "^5.1.5",
"source-map-js": ">=0.6.2 <2.0.0"
},
"bin": {
"sass": "sass.js"
},
"engines": {
"node": ">=20.19.0"
},
"optionalDependencies": {
"@parcel/watcher": "^2.4.1"
}
},
"node_modules/source-map-js": {
"version": "1.2.1",
"resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz",

View File

@@ -10,11 +10,11 @@
},
"devDependencies": {
"@types/sql.js": "^1.4.11",
"sass": "^1.101.0",
"typescript": "^6.0.3",
"vite": "^8.0.12"
},
"dependencies": {
"95css": "^0.4.2",
"sql.js": "^1.14.1"
}
}

View File

@@ -1,102 +1,41 @@
import initSqlJs, { type Database, type SqlValue } from "sql.js";
import "95css/css/95css.css";
import "./style.css";
import initSqlJs, { type Database } from "sql.js";
import sqlWasmUrl from "sql.js/dist/sql-wasm.wasm?url";
import "./style.scss";
const DB_URL = "/biergarten.sqlite";
const tabs = document.querySelectorAll<HTMLButtonElement>("[role='tab']");
const panels = document.querySelectorAll<HTMLDivElement>("[role='tabpanel']");
const tableSelect = document.querySelector<HTMLSelectElement>("#table-select")!;
const tableContainer =
document.querySelector<HTMLDivElement>("#table-container")!;
const tableContainer = document.querySelector<HTMLDivElement>("#table-container")!;
const status = document.querySelector<HTMLParagraphElement>("#status")!;
const queryInput = document.querySelector<HTMLTextAreaElement>("#query-input")!;
const queryRun = document.querySelector<HTMLButtonElement>("#query-run")!;
const queryStatus = document.querySelector<HTMLParagraphElement>("#query-status")!;
const queryContainer =
document.querySelector<HTMLDivElement>("#query-container")!;
function renderTable(db: Database, name: string): void {
const result = db.exec(`SELECT * FROM "${name}"`);
tableContainer.replaceChildren();
for (const tab of tabs) {
tab.addEventListener("click", () => {
for (const otherTab of tabs) {
const isActive = otherTab === tab;
otherTab.classList.toggle("is-active", isActive);
otherTab.setAttribute("aria-selected", String(isActive));
if (result.length === 0) {
tableContainer.append(Object.assign(document.createElement("p"), { textContent: "No rows." }));
return;
}
for (const panel of panels) {
panel.hidden = panel.id !== tab.getAttribute("aria-controls");
}
});
}
const renderResultTable = (
container: HTMLDivElement,
columns: string[],
values: SqlValue[][],
) => {
const { columns, values } = result[0];
const table = document.createElement("table");
const head = table.createTHead().insertRow();
for (const column of columns) {
const th = document.createElement("th");
th.textContent = column;
head.append(th);
head.append(Object.assign(document.createElement("th"), { textContent: column }));
}
const body = table.createTBody();
for (const row of values) {
const tr = body.insertRow();
for (const cell of row) {
const td = document.createElement("td");
td.textContent = String(cell ?? "");
tr.append(td);
tr.append(Object.assign(document.createElement("td"), { textContent: String(cell ?? "") }));
}
}
container.append(table);
};
const renderTable = (db: Database, name: string) => {
const result = db.exec(`SELECT * FROM "${name}"`);
tableContainer.replaceChildren();
if (result.length === 0) {
tableContainer.append(
Object.assign(document.createElement("p"), { textContent: "No rows." }),
);
return;
}
const { columns, values } = result[0];
renderResultTable(tableContainer, columns, values);
};
const runQuery = (db: Database) => {
const sql = queryInput.value.trim();
queryContainer.replaceChildren();
if (!sql) {
queryStatus.textContent = "Enter a query to run.";
return;
}
try {
const results = db.exec(sql);
if (results.length === 0) {
queryStatus.textContent = "Query executed. No rows returned.";
return;
}
const { columns, values } = results[results.length - 1];
renderResultTable(queryContainer, columns, values);
queryStatus.textContent = `${values.length} row${values.length === 1 ? "" : "s"} returned.`;
} catch (error) {
queryStatus.textContent = `Error: ${error instanceof Error ? error.message : String(error)}`;
}
};
tableContainer.append(table);
}
async function main(): Promise<void> {
status.textContent = "Loading database...";
@@ -108,38 +47,20 @@ async function main(): Promise<void> {
const tables =
db
.exec(
`SELECT name
FROM sqlite_master
WHERE
type = 'table'
AND
name NOT LIKE 'sqlite_%'
ORDER BY name`,
)[0]
.exec("SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%' ORDER BY name")[0]
?.values.map((row) => String(row[0])) ?? [];
for (const name of tables) {
tableSelect.append(new Option(name, name));
}
tableSelect.addEventListener("change", () =>
renderTable(db, tableSelect.value),
);
tableSelect.addEventListener("change", () => renderTable(db, tableSelect.value));
if (tables.length > 0) {
renderTable(db, tables[0]);
}
status.textContent = "";
queryRun.addEventListener("click", () => runQuery(db));
queryInput.addEventListener("keydown", (event) => {
if ((event.metaKey || event.ctrlKey) && event.key === "Enter") {
event.preventDefault();
runQuery(db);
}
});
}
main().catch((error: unknown) => {

View File

@@ -0,0 +1,28 @@
body {
margin: 0;
padding: 24px;
}
#window {
width: 100%;
max-width: 1600px;
margin: 0 auto;
}
#table-container {
overflow-x: auto;
margin-top: 8px;
}
table {
border-collapse: collapse;
}
th,
td {
border: 1px solid #808080;
padding: 4px 8px;
text-align: left;
font-size: 14px;
white-space: nowrap;
}

View File

@@ -1,267 +0,0 @@
$gray-100: #dedede;
$gray-300: #c3c3c3;
$gray-500: #808080;
$white: #ffffff;
$black: #000000;
$navy: #000080;
$blue: #1084d0;
$teal: #008080;
$color-surface: $gray-300;
$color-surface-alt: $gray-100;
$color-surface-sunken: $white;
$color-border-highlight: $gray-100;
$color-border-lowlight: $gray-500;
$color-border-light: $white;
$color-border-dark: $black;
$color-text: $black;
$color-text-on-accent: $white;
$color-text-status: $navy;
$color-accent-start: $navy;
$color-accent-end: $blue;
$color-desktop-bg: $teal;
$color-row-stripe: $gray-100;
$color-shadow: rgba($black, 0.4);
$color-focus-ring: $black;
$bevel-size: 2px;
$font-stack: Tahoma, "MS Sans Serif", Verdana, sans-serif;
@mixin bevel-raised($size: $bevel-size) {
border-style: solid;
border-width: $size;
border-top-color: $color-border-highlight;
border-left-color: $color-border-highlight;
border-right-color: $color-border-dark;
border-bottom-color: $color-border-dark;
box-shadow:
inset $size $size 0 0 $color-border-light,
inset (-$size) (-$size) 0 0 $color-border-lowlight;
}
@mixin bevel-sunken($size: $bevel-size) {
border-style: solid;
border-width: $size;
border-top-color: $color-border-lowlight;
border-left-color: $color-border-lowlight;
border-right-color: $color-border-light;
border-bottom-color: $color-border-light;
box-shadow:
inset $size $size 0 0 $color-border-dark,
inset (-$size) (-$size) 0 0 $color-surface-alt;
}
* {
box-sizing: border-box;
}
body {
margin: 0;
padding: 24px;
background: $color-desktop-bg;
font-family: $font-stack;
color: $color-text;
min-height: 100vh;
}
#window {
width: 100%;
max-width: 1600px;
margin: 0 auto;
background: $color-surface;
padding: 6px;
display: flex;
flex-direction: column;
gap: 6px;
@include bevel-raised(3px);
box-shadow: 8px 8px 0 0 $color-shadow;
}
.dialog-header {
background: linear-gradient(90deg, $color-accent-start, $color-accent-end);
color: $color-text-on-accent;
padding: 6px 8px;
display: flex;
align-items: center;
p {
margin: 0;
font-size: 14px;
font-weight: bold;
letter-spacing: 0.5px;
text-transform: uppercase;
}
}
.tab-strip {
display: flex;
gap: 4px;
padding: 8px 8px 0;
}
.tab {
position: relative;
top: 3px;
margin-bottom: -2px;
font-family: $font-stack;
font-size: 12px;
font-weight: bold;
text-transform: uppercase;
padding: 6px 14px;
background: $color-surface;
color: $color-text;
border-style: solid;
border-width: 2px;
border-top-color: $color-border-light;
border-left-color: $color-border-light;
border-right-color: $color-border-dark;
border-bottom-color: $color-border-dark;
border-radius: 0;
cursor: pointer;
&.is-active {
top: 0;
z-index: 1;
border-bottom-color: $color-surface;
}
&:focus {
outline: 1px dotted $color-focus-ring;
outline-offset: -4px;
}
}
.dialog-body {
border-top: 2px solid $color-border-dark;
padding: 12px;
}
.tab-panel {
display: flex;
flex-direction: column;
gap: 12px;
&[hidden] {
display: none;
}
}
.field-group {
display: flex;
align-items: center;
gap: 8px;
label {
font-size: 12px;
font-weight: bold;
text-transform: uppercase;
}
&.field-group--stacked {
flex-direction: column;
align-items: stretch;
}
}
select#table-select {
font-family: $font-stack;
font-size: 13px;
padding: 3px 6px;
background: $color-surface-sunken;
color: $color-text;
border-radius: 0;
@include bevel-sunken();
&:focus {
outline: 1px dotted $color-focus-ring;
outline-offset: 2px;
}
}
textarea#query-input {
font-family: $font-stack;
font-size: 13px;
padding: 6px;
background: $color-surface-sunken;
color: $color-text;
border-radius: 0;
resize: vertical;
@include bevel-sunken();
&:focus {
outline: 1px dotted $color-focus-ring;
outline-offset: 2px;
}
}
.btn {
align-self: flex-start;
font-family: $font-stack;
font-size: 12px;
font-weight: bold;
text-transform: uppercase;
padding: 6px 16px;
background: $color-surface;
color: $color-text;
cursor: pointer;
@include bevel-raised();
&:active {
@include bevel-sunken();
}
&:focus {
outline: 1px dotted $color-focus-ring;
outline-offset: -4px;
}
}
#status,
#query-status {
margin: 0;
font-size: 12px;
font-style: italic;
color: $color-text-status;
}
#table-container,
#query-container {
background: $color-surface-sunken;
padding: 4px;
overflow-x: auto;
@include bevel-sunken();
}
table {
border-collapse: collapse;
margin: 0 auto;
font-family: $font-stack;
}
th,
td {
border: 1px solid $color-border-lowlight;
padding: 4px 8px;
text-align: left;
font-size: 13px;
}
th {
background: $color-surface;
white-space: nowrap;
font-weight: bold;
text-transform: uppercase;
@include bevel-raised(1px);
}
td {
min-width: 120px;
max-width: 800px;
white-space: normal;
word-break: break-word;
background: $color-surface-sunken;
}
tbody tr:nth-child(even) td {
background: $color-row-stripe;
}

View File

@@ -139,9 +139,6 @@ FetchContent_MakeAvailable(cpp-httplib)
# 5. Executable & Sources
add_executable(${PROJECT_NAME}
includes/services/enrichment/mock_enrichment.h
includes/services/curated_data/mock_curated_data_service.h
includes/services/postal_code/postal_code_service.h
includes/services/postal_code/mock_postal_code_service.h
includes/json_handling/pretty_print.h)
# --- Entry point ---
@@ -151,12 +148,12 @@ target_sources(${PROJECT_NAME} PRIVATE
# --- json_handling ---
target_sources(${PROJECT_NAME} PRIVATE
src/services/curated_data/curated_json_data_service.cc
src/json_handling/json_loader.cc
)
# --- services: curated_data ---
# --- data_model ---
target_sources(${PROJECT_NAME} PRIVATE
src/services/curated_data/mock_curated_data_service.cc
src/data_model/names_by_country.cc
)
# --- application_options ---
@@ -266,8 +263,8 @@ target_compile_options(biergarten-pipeline PRIVATE
# 7. Runtime Assets
configure_file(
${CMAKE_SOURCE_DIR}/cities.json
${CMAKE_BINARY_DIR}/cities.json
${CMAKE_SOURCE_DIR}/locations.json
${CMAKE_BINARY_DIR}/locations.json
COPYONLY
)
configure_file(
@@ -290,3 +287,4 @@ add_custom_command(TARGET ${PROJECT_NAME} POST_BUILD
${CMAKE_SOURCE_DIR}/prompts
${CMAKE_BINARY_DIR}/prompts
)

File diff suppressed because it is too large Load Diff

View File

@@ -1,78 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_H_
#define BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_H_
/**
* @file biergarten_pipeline.h
* @brief Master umbrella header — includes every public header in the
* Biergarten pipeline.
*/
// --- orchestrator ---
#include "biergarten_pipeline_orchestrator.h"
// --- concurrency ---
#include "concurrency/bounded_channel.h"
// --- data_generation ---
#include "data_generation/data_generator.h"
#include "data_generation/json_grammars.h"
#ifndef BIERGARTEN_MOCK_ONLY
#include "data_generation/llama_generator.h"
#include "data_generation/llama_generator_helpers.h"
#endif
#include "data_generation/mock_generator.h"
#include "data_generation/prompt_formatting/gemma4_jinja_prompt_formatter.h"
#include "data_generation/prompt_formatting/prompt_formatter.h"
// --- data_model ---
#include "data_model/generated_models.h"
#include "data_model/models.h"
// --- json_handling ---
#include "json_handling/pretty_print.h"
#include "services/curated_data/curated_json_data_service.h"
// --- llama backend ---
#ifndef BIERGARTEN_MOCK_ONLY
#include "llama_backend_state.h"
#endif
// --- services: curated_data ---
#include "services/curated_data/curated_data_service.h"
#include "services/curated_data/mock_curated_data_service.h"
// --- services: database ---
#include "services/database/export_service.h"
#include "services/database/sqlite_connection_helpers.h"
#include "services/database/sqlite_export_service.h"
#include "services/database/sqlite_export_service_helpers.h"
#include "services/database/sqlite_handle_types.h"
#include "services/database/sqlite_statement_helpers.h"
// --- services: datetime ---
#include "services/datetime/date_time_provider.h"
#include "services/datetime/timer.h"
// --- services: enrichment ---
#include "services/enrichment/enrichment_service.h"
#include "services/enrichment/mock_enrichment.h"
#include "services/enrichment/wikipedia_service.h"
// --- services: logging ---
#include "services/logging/log_dispatcher.h"
#include "services/logging/log_entry.h"
#include "services/logging/log_producer.h"
#include "services/logging/logger.h"
// --- services: postal_code ---
#include "services/postal_code/mock_postal_code_service.h"
#include "services/postal_code/postal_code_service.h"
// --- services: prompting ---
#include "services/prompting/prompt_directory.h"
// --- web_client ---
#include "web_client/http_web_client.h"
#include "web_client/web_client.h"
#endif // BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_H_

View File

@@ -1,9 +1,12 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_ORCHESTRATOR_H_
#define BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_ORCHESTRATOR_H_
#ifndef BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_DATA_GENERATOR_H_
#define BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_DATA_GENERATOR_H_
/**
* @file biergarten_pipeline_orchestrator.h
* @brief Orchestration for end-to-end brewery and user data generation.
*
* Coordinates location loading, enrichment, and generation phases to produce
* a complete dataset, wiring dependencies selected at the composition root.
*/
#include <memory>
@@ -12,11 +15,10 @@
#include "data_generation/data_generator.h"
#include "data_model/generated_models.h"
#include "services/curated_data/curated_data_service.h"
#include "services/database/export_service.h"
#include "services/enrichment/enrichment_service.h"
#include "services/logging/logger.h"
#include "services/postal_code/postal_code_service.h"
/**
* @brief Main orchestrator for the Biergarten data generation pipeline.
*
@@ -32,9 +34,6 @@ class BiergartenPipelineOrchestrator {
* @param generator Implementation (Llama or Mock) for brewery/user
* generation.
* @param exporter Database backend for persisting generated records.
* @param curated_data_service Loads curated location, persona, and name
* data used to seed generation.
* @param postal_code_service
* @param application_options CLI configuration and paths.
*/
BiergartenPipelineOrchestrator(
@@ -42,8 +41,6 @@ class BiergartenPipelineOrchestrator {
std::unique_ptr<IEnrichmentService> context_service,
std::unique_ptr<DataGenerator> generator,
std::unique_ptr<IExportService> exporter,
std::unique_ptr<ICuratedDataService> curated_data_service,
std::unique_ptr<IPostalCodeService> postal_code_service,
const ApplicationOptions& application_options);
/**
@@ -79,8 +76,6 @@ class BiergartenPipelineOrchestrator {
* @brief Storage backend for generated brewery and user records.
*/
std::unique_ptr<IExportService> exporter_;
std::unique_ptr<ICuratedDataService> curated_data_service_;
std::unique_ptr<IPostalCodeService> postal_code_service_;
ApplicationOptions application_options_;
@@ -90,7 +85,7 @@ class BiergartenPipelineOrchestrator {
* @return Vector of locations randomly sampled per
* PipelineOptions::location_count.
*/
std::vector<City> QueryCitiesWithCountries();
std::vector<Location> QueryCitiesWithCountries();
/**
* @brief Generate breweries for enriched cities.
@@ -103,6 +98,10 @@ class BiergartenPipelineOrchestrator {
* @brief Generate users grounded in sampled names and personas for
* enriched cities.
*
* Loads personas.json / forenames-by-country.json /
* surnames-by-country.json itself, mirroring how QueryCitiesWithCountries()
* owns its own locations.json load.
*
* @param cities Span of enriched city data.
*/
void GenerateUsers(std::span<const EnrichedCity> cities);
@@ -115,5 +114,4 @@ class BiergartenPipelineOrchestrator {
std::vector<BreweryRecord> generated_breweries_;
std::vector<UserRecord> generated_users_;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_ORCHESTRATOR_H_
#endif // BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_DATA_GENERATOR_H_

View File

@@ -1,4 +1,4 @@
#include "concurrency/bounded_channel.h"
#include "bounded_channel.h"
template <typename T>
void BoundedChannel<T>::Send(T item) {

View File

@@ -24,7 +24,7 @@ class DataGenerator {
* @param region_context Additional regional context text.
* @return Brewery generation result.
*/
virtual BreweryResult GenerateBrewery(const City& location,
virtual BreweryResult GenerateBrewery(const Location& location,
const std::string& region_context) = 0;
/**
@@ -32,7 +32,7 @@ class DataGenerator {
*
* @param city Enriched city the user is associated with.
* @param persona Persona archetype grounding the generated bio.
* @param name Sampled first/last name (and gender)
* @param name Sampled first/last name (and gender) -- not LLM-invented.
* @return User generation result.
*/
virtual UserResult GenerateUser(const EnrichedCity& city,

View File

@@ -1,23 +1,24 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_LLAMA_GENERATOR_H_
#define BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_LLAMA_GENERATOR_H_
#include <filesystem>
/**
* @file data_generation/llama_generator.h
* @brief llama.cpp-backed implementation of DataGenerator.
*/
#include <cstdint>
#include <filesystem>
#include <memory>
#include <random>
#include <string>
#include <string_view>
#include "../services/prompting/prompt_directory.h"
#include "data_generation/data_generator.h"
#include "data_generation/prompt_formatting/prompt_formatter.h"
#include "data_model/models.h"
#include "services/logging/logger.h"
#include "services/prompting/prompt_directory.h"
struct llama_model;
struct llama_context;
@@ -55,7 +56,7 @@ class LlamaGenerator final : public DataGenerator {
* @param region_context Additional regional context.
* @return Generated brewery result.
*/
BreweryResult GenerateBrewery(const City& location,
BreweryResult GenerateBrewery(const Location& location,
const std::string& region_context) override;
/**

View File

@@ -24,7 +24,7 @@ class MockGenerator final : public DataGenerator {
* @param region_context Unused for mock generation.
* @return Generated brewery result.
*/
BreweryResult GenerateBrewery(const City& location,
BreweryResult GenerateBrewery(const Location& location,
const std::string& region_context) override;
/**
@@ -46,7 +46,7 @@ class MockGenerator final : public DataGenerator {
* @param location City and country names.
* @return Deterministic hash value.
*/
static size_t DeterministicHash(const City& location);
static size_t DeterministicHash(const Location& location);
/**
* @brief Combines city, persona, and name into a stable hash value.
@@ -56,7 +56,7 @@ class MockGenerator final : public DataGenerator {
* @param name Sampled first/last name.
* @return Deterministic hash value.
*/
static size_t DeterministicHash(const City& location,
static size_t DeterministicHash(const Location& location,
const UserPersona& persona, const Name& name);
// Hash stride constants for deterministic distribution across fixed-size

View File

@@ -7,7 +7,6 @@
* enriched data, and complete generation results.
*/
#include <optional>
#include <string>
#include "data_model/models.h"
@@ -88,45 +87,15 @@ struct UserResult {
* @brief Enriched city data with Wikipedia context.
*/
struct EnrichedCity {
City location;
Location location;
std::string region_context{};
};
/**
* @brief A brewery's address. Owns the `City` it belongs to alongside its
* street-level fields. Mirrors the `brewery_addresses` table and the
* backend's `BreweryPostLocation` shape.
*
* Only `city` and `postal_code` are populated today -- street-address
* generation has no fixture data or service yet, so the address lines stay
* empty.
*/
struct BreweryAddress {
City city{};
std::optional<std::string> address_line_1{};
std::optional<std::string> address_line_2{};
std::string postal_code{};
};
/**
* @brief A user's address. Owns the `City` it belongs to alongside its
* street-level fields. Mirrors the `user_addresses` table.
*
* Only `city` is populated today; the street-level columns exist for future
* enrichment.
*/
struct UserAddress {
City city{};
std::optional<std::string> address_line_1{};
std::optional<std::string> address_line_2{};
std::optional<std::string> postal_code{};
};
/**
* @brief Helper struct to store generated brewery data.
*/
struct BreweryRecord {
BreweryAddress address;
Location location;
BreweryResult brewery;
};
@@ -138,7 +107,7 @@ struct BreweryRecord {
* consumer can register real accounts from the pipeline's SQLite export.
*/
struct UserRecord {
UserAddress address{};
Location location;
UserResult user;
std::string email{};
std::string date_of_birth{};

View File

@@ -10,7 +10,6 @@
#include <boost/program_options.hpp>
#include <cstdint>
#include <filesystem>
#include <functional>
#include <memory>
#include <optional>
#include <string>
@@ -22,13 +21,13 @@ class ILogger;
namespace prog_opts = boost::program_options;
// ============================================================================
// City Models
// Location Models
// ============================================================================
/**
* @brief Canonical location record for city-level generation.
*/
struct City {
struct Location {
std::string city{};
std::string state_province{};
@@ -44,22 +43,20 @@ struct City {
*/
std::string iso3166_1{};
/**
* @brief A list of regular expressions used for valid postal codes in the
* region.
*
*/
std::vector<std::string> postal_regex{};
/**
* @brief Example postal codes for the region.
*/
std::vector<std::string> postal_code_examples{};
/**
* @brief Local language codes in priority order.
*/
std::vector<std::string> local_languages{};
/**
* @brief Latitude in decimal degrees.
*/
double latitude{};
/**
* @brief Longitude in decimal degrees.
*/
double longitude{};
};
// ============================================================================
@@ -69,7 +66,7 @@ struct City {
/**
* @brief A sampled first/last name pair, with the source forename's gender.
*
* Produced by the SampleName() helper in generate_users.cc.
* Produced by NamesByCountry::SampleName();
*/
struct Name {
std::string first_name{};
@@ -96,23 +93,8 @@ struct ForenameEntry {
* dataset (e.g. "M", "F").
*/
std::string gender{};
bool operator==(const ForenameEntry& other) const {
return name == other.name && gender == other.gender;
}
};
namespace std {
template <>
struct hash<ForenameEntry> {
size_t operator()(const ForenameEntry& entry) const noexcept {
const size_t name_hash = std::hash<std::string>{}(entry.name);
const size_t gender_hash = std::hash<std::string>{}(entry.gender);
return name_hash ^ (gender_hash << 1);
}
};
} // namespace std
/**
* @brief A persona archetype used to ground LLM-generated user bios.
*/

View File

@@ -0,0 +1,51 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_NAMES_BY_COUNTRY_H_
#define BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_NAMES_BY_COUNTRY_H_
/**
* @file data_model/names_by_country.h
* @brief Sampling over the names-by-country fixture data.
*/
#include <optional>
#include <random>
#include <string>
#include <unordered_map>
#include <vector>
#include "data_model/models.h"
/**
* @brief Samples locale-appropriate names from curated forename/surname
* fixture data, keyed by ISO 3166-1 country code.
*
* Forenames and surnames are kept in separate maps (mirroring the source
* dataset's own shape) so per-forename gender is preserved; pairing happens
* at sample time rather than being precomputed, so no information from the
* source dataset is discarded up front.
*/
class NamesByCountry {
public:
NamesByCountry(std::unordered_map<std::string, std::vector<ForenameEntry>>
forenames_by_country,
std::unordered_map<std::string, std::vector<std::string>>
surnames_by_country);
/**
* @brief Samples a first/last name pair for the given country.
*
* @param iso3166_1 ISO 3166-1 country code to sample for.
* @param rng Random source used for sampling.
* @return A sampled Name, or std::nullopt if the country has no forename
* or no surname entries.
*/
[[nodiscard]] std::optional<Name> SampleName(const std::string& iso3166_1,
std::mt19937& rng) const;
private:
std::unordered_map<std::string, std::vector<ForenameEntry>>
forenames_by_country_;
std::unordered_map<std::string, std::vector<std::string>>
surnames_by_country_;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_NAMES_BY_COUNTRY_H_

View File

@@ -0,0 +1,46 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_
#define BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_
/**
* @file json_handling/json_loader.h
* @brief Loader API for curated location data.
*/
#include <filesystem>
#include <memory>
#include <vector>
#include "data_model/models.h"
#include "data_model/names_by_country.h"
#include "services/logging/logger.h"
/**
* @brief Loads curated world locations from a JSON file into memory.
*/
class JsonLoader {
public:
/**
* @brief Parses a JSON array file and returns all location records.
*/
static std::vector<Location> LoadLocations(
const std::filesystem::path& filepath);
/**
* @brief Parses a JSON array file and returns all persona records.
*/
static std::vector<UserPersona> LoadPersonas(
const std::filesystem::path& filepath);
/**
* @brief Parses the names-by-country fixture pair into a sampling-capable
* NamesByCountry.
*
* @param forenames_filepath Path to forenames-by-country.json.
* @param surnames_filepath Path to surnames-by-country.json.
*/
static NamesByCountry LoadNamesByCountry(
const std::filesystem::path& forenames_filepath,
const std::filesystem::path& surnames_filepath);
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_

View File

@@ -106,4 +106,4 @@ inline void PrettyPrint(std::ostream& outstream,
}
}
#endif // BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_PRETTY_PRINT_H_
#endif

View File

@@ -1,50 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_CURATED_DATA_SERVICE_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_CURATED_DATA_SERVICE_H_
/**
* @file services/curated_data/curated_data_service.h
* @brief Abstraction for loading curated location, persona, and name data.
*/
#include <filesystem>
#include <unordered_map>
#include <unordered_set>
#include <vector>
#include "data_model/models.h"
using ForenameList = std::vector<ForenameEntry>;
using SurnameList = std::vector<std::string>;
using CityList = std::vector<City>;
using PersonasList = std::vector<UserPersona>;
using ForenamesByCountryMap = std::unordered_map<std::string, ForenameList>;
using SurnamesByCountryMap = std::unordered_map<std::string, SurnameList>;
/**
* @brief Interface for services that load curated data used to seed
* brewery/user generation.
*/
class ICuratedDataService {
public:
ICuratedDataService() = default;
virtual ~ICuratedDataService() = default;
ICuratedDataService(const ICuratedDataService&) = delete;
ICuratedDataService& operator=(const ICuratedDataService&) = delete;
ICuratedDataService(ICuratedDataService&&) = delete;
ICuratedDataService& operator=(ICuratedDataService&&) = delete;
/**
* @brief Loads all curated location records.
*/
virtual const CityList& LoadCities() = 0;
/**
* @brief Loads all curated persona records.
*/
virtual const PersonasList& LoadPersonas() = 0;
virtual const ForenamesByCountryMap& LoadForenamesByCountry() = 0;
virtual const SurnamesByCountryMap& LoadSurnamesByCountry() = 0;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_CURATED_DATA_SERVICE_H_

View File

@@ -1,63 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_
#define BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_
/**
* @file json_handling/json_loader.h
* @brief JSON-backed implementation of ICuratedDataService.
*/
#include <filesystem>
#include "data_model/models.h"
#include "services/curated_data/curated_data_service.h"
/**
* @brief File locations for the curated JSON fixtures consumed by
* CuratedJsonDataService.
*/
struct CuratedDataFilePaths {
std::filesystem::path cities_path;
std::filesystem::path personas_path;
std::filesystem::path forenames_path;
std::filesystem::path surnames_path;
};
/**
* @brief Loads curated location, persona, and name data from JSON files.
*/
class CuratedJsonDataService final : public ICuratedDataService {
struct cache {
CityList locations;
PersonasList personas;
ForenamesByCountryMap forenames_by_country;
SurnamesByCountryMap surnames_by_country;
cache() = default;
~cache() = default;
};
CuratedDataFilePaths filepaths_;
cache cache_;
public:
explicit CuratedJsonDataService(CuratedDataFilePaths filepaths);
/**
* @brief Parses a JSON array file and returns all city records.
*/
const CityList& LoadCities() override;
/**
* @brief Parses a JSON array file and returns all persona records.
*/
const PersonasList& LoadPersonas() override;
const ForenamesByCountryMap& LoadForenamesByCountry() override;
/**
* @brief Parses a JSON file and returns all the forenames per country.
*/
const SurnamesByCountryMap& LoadSurnamesByCountry() override;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_

View File

@@ -1,38 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_MOCK_CURATED_DATA_SERVICE_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_MOCK_CURATED_DATA_SERVICE_H_
/**
* @file services/curated_data/mock_curated_data_service.h
* @brief In-memory ICuratedDataService backed by a small fixed dataset, used
* when file-backed curated data is disabled (mock mode).
*/
#include <filesystem>
#include "data_model/models.h"
#include "services/curated_data/curated_data_service.h"
/**
* @brief Curated data service returning a small fixed in-memory dataset in
* place of the JSON fixture files used by JsonLoader.
*/
class MockCuratedDataService final : public ICuratedDataService {
public:
MockCuratedDataService();
const CityList& LoadCities() override;
const PersonasList& LoadPersonas() override;
const ForenamesByCountryMap& LoadForenamesByCountry() override;
const SurnamesByCountryMap& LoadSurnamesByCountry() override;
private:
CityList locations_;
PersonasList personas_;
ForenamesByCountryMap forenames_by_country_;
SurnamesByCountryMap surnames_by_country_;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_MOCK_CURATED_DATA_SERVICE_H_

View File

@@ -12,7 +12,7 @@
#include <string>
#include <string_view>
#include "services/database/sqlite_handle_types.h"
#include "sqlite_handle_types.h"
namespace sqlite_export_service_internal {

View File

@@ -11,10 +11,10 @@
#include <string>
#include <unordered_map>
#include "../datetime/date_time_provider.h"
#include "data_model/models.h"
#include "services/database/export_service.h"
#include "services/database/sqlite_export_service_helpers.h"
#include "services/datetime/date_time_provider.h"
#include "export_service.h"
#include "sqlite_export_service_helpers.h"
/**
* @brief Persists generated brewery records into a fresh SQLite database.
@@ -45,7 +45,7 @@ class SqliteExportService final : public IExportService {
void RollbackAndCloseNoThrow() noexcept;
[[nodiscard]] std::filesystem::path BuildDatabasePath() const;
[[nodiscard]] static std::string BuildCityKey(const City& location);
[[nodiscard]] static std::string BuildLocationKey(const Location& location);
/**
* @brief Returns the row id for @p location, inserting it first if it has
@@ -54,20 +54,18 @@ class SqliteExportService final : public IExportService {
* Shared by both ProcessRecord() overloads so breweries and users
* referencing the same location resolve to the same row.
*/
[[nodiscard]] sqlite3_int64 ResolveCityId(const City& location);
[[nodiscard]] sqlite3_int64 ResolveLocationId(const Location& location);
std::unique_ptr<IDateTimeProvider> date_time_provider_;
std::filesystem::path output_path_;
std::string run_timestamp_utc_;
std::filesystem::path database_path_;
SqliteDatabaseHandle db_handle_;
SqliteStatementHandle insert_city_stmt_;
SqliteStatementHandle insert_location_stmt_;
SqliteStatementHandle insert_brewery_stmt_;
SqliteStatementHandle insert_brewery_address_stmt_;
SqliteStatementHandle insert_user_stmt_;
SqliteStatementHandle insert_user_address_stmt_;
bool transaction_open_ = false;
std::unordered_map<std::string, sqlite3_int64> city_cache_;
std::unordered_map<std::string, sqlite3_int64> location_cache_;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_SQLITE_EXPORT_SERVICE_H_

View File

@@ -3,8 +3,8 @@
/* Umbrella header for backward compatibility. */
#include "services/database/sqlite_connection_helpers.h"
#include "services/database/sqlite_handle_types.h"
#include "services/database/sqlite_statement_helpers.h"
#include "sqlite_connection_helpers.h"
#include "sqlite_handle_types.h"
#include "sqlite_statement_helpers.h"
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_SQLITE_EXPORT_SERVICE_HELPERS_H_

View File

@@ -24,10 +24,8 @@ using SqliteDatabaseHandle = std::unique_ptr<sqlite3, SqliteDatabaseDeleter>;
using SqliteStatementHandle =
std::unique_ptr<sqlite3_stmt, SqliteStatementDeleter>;
// Represents a parameter that is bound to a prepared SQLite statement.
// N.B. indices are 1 based.
template <typename T>
struct BoundParam {
struct BindParam {
int index;
T value;
std::string_view action;

View File

@@ -13,13 +13,13 @@
#include <string_view>
#include <vector>
#include "services/database/sqlite_handle_types.h"
#include "sqlite_handle_types.h"
namespace sqlite_export_service_internal {
inline constexpr std::string_view kCreateCitiesTableSql = R"sql(
inline constexpr std::string_view kCreateLocationsTableSql = R"sql(
CREATE TABLE IF NOT EXISTS cities (
CREATE TABLE IF NOT EXISTS locations (
id INTEGER PRIMARY KEY AUTOINCREMENT,
city TEXT NOT NULL,
state_province TEXT NOT NULL,
@@ -27,7 +27,9 @@ CREATE TABLE IF NOT EXISTS cities (
country TEXT NOT NULL,
iso3166_1 TEXT NOT NULL,
local_languages_json TEXT NOT NULL,
UNIQUE(city, state_province, iso3166_2, country)
latitude REAL NOT NULL,
longitude REAL NOT NULL,
UNIQUE(city, state_province, iso3166_2, country, latitude, longitude)
);
)sql";
@@ -36,21 +38,23 @@ inline constexpr std::string_view kCreateBreweriesTableSql = R"sql(
CREATE TABLE IF NOT EXISTS breweries (
id INTEGER PRIMARY KEY AUTOINCREMENT,
city_id INTEGER NOT NULL,
location_id INTEGER NOT NULL,
name_en TEXT NOT NULL,
description_en TEXT NOT NULL,
name_local TEXT NOT NULL,
description_local TEXT NOT NULL,
FOREIGN KEY(city_id) REFERENCES cities(id) ON DELETE CASCADE
FOREIGN KEY(location_id) REFERENCES locations(id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_breweries_city_id ON breweries(city_id);
CREATE INDEX IF NOT EXISTS idx_breweries_location_id ON breweries(location_id);
)sql";
inline constexpr std::string_view kCreateUsersTableSql = R"sql(
CREATE TABLE IF NOT EXISTS users (
id INTEGER PRIMARY KEY AUTOINCREMENT,
location_id INTEGER NOT NULL,
first_name TEXT NOT NULL,
last_name TEXT NOT NULL,
gender TEXT NOT NULL,
@@ -58,61 +62,30 @@ CREATE TABLE IF NOT EXISTS users (
bio TEXT NOT NULL,
activity_weight REAL NOT NULL,
email TEXT NOT NULL UNIQUE,
date_of_birth TEXT NOT NULL
date_of_birth TEXT NOT NULL,
FOREIGN KEY(location_id) REFERENCES locations(id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_users_location_id ON users(location_id);
)sql";
inline constexpr std::string_view kCreateBreweryAddressTableSql = R"sql(
CREATE TABLE IF NOT EXISTS brewery_addresses (
id INTEGER PRIMARY KEY AUTOINCREMENT,
brewery_id INTEGER NOT NULL,
address_line_1 TEXT,
address_line_2 TEXT,
postal_code TEXT NOT NULL,
city_id INTEGER NOT NULL,
FOREIGN KEY(brewery_id) REFERENCES breweries(id) ON DELETE CASCADE,
FOREIGN KEY(city_id) REFERENCES cities(id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_brewery_addresses_brewery_id
ON brewery_addresses(brewery_id);
CREATE INDEX IF NOT EXISTS idx_brewery_addresses_city_id
ON brewery_addresses(city_id);
)sql";
inline constexpr std::string_view kCreateUserAddressTableSql = R"sql(
CREATE TABLE IF NOT EXISTS user_addresses (
id INTEGER PRIMARY KEY AUTOINCREMENT,
user_id INTEGER NOT NULL,
address_line_1 TEXT,
address_line_2 TEXT,
postal_code TEXT,
city_id INTEGER NOT NULL,
FOREIGN KEY(user_id) REFERENCES users(id) ON DELETE CASCADE,
FOREIGN KEY(city_id) REFERENCES cities(id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_user_addresses_user_id
ON user_addresses(user_id);
CREATE INDEX IF NOT EXISTS idx_user_addresses_city_id
ON user_addresses(city_id);
)sql";
inline constexpr std::string_view kInsertCitySql = R"sql(
INSERT INTO cities (
inline constexpr std::string_view kInsertLocationSql = R"sql(
INSERT INTO locations (
city,
state_province,
iso3166_2,
country,
iso3166_1,
local_languages_json
) VALUES (?, ?, ?, ?, ?, ?);
local_languages_json,
latitude,
longitude
) VALUES (?, ?, ?, ?, ?, ?, ?, ?);
)sql";
inline constexpr std::string_view kInsertBrewerySql = R"sql(
INSERT INTO breweries (
city_id,
location_id,
name_en,
description_en,
name_local,
@@ -122,6 +95,7 @@ INSERT INTO breweries (
inline constexpr std::string_view kInsertUserSql = R"sql(
INSERT INTO users (
location_id,
first_name,
last_name,
gender,
@@ -130,64 +104,35 @@ INSERT INTO users (
activity_weight,
email,
date_of_birth
) VALUES (?, ?, ?, ?, ?, ?, ?, ?);
)sql";
inline constexpr std::string_view kInsertBreweryAddressSql = R"sql(
INSERT INTO brewery_addresses (
brewery_id,
postal_code,
city_id
) VALUES (?, ?, ?);
)sql";
inline constexpr std::string_view kInsertUserAddressSql = R"sql(
INSERT INTO user_addresses (
user_id,
city_id
) VALUES (?, ?);
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?);
)sql";
// sqlite3_bind_*() parameter indices are 1-based, matching the "?"
// placeholder order in the SQL above.
enum CityBindIndex {
kCityNameBindIndex = 1,
kCityStateProvinceBindIndex,
kCityIso31662BindIndex,
kCityCountryBindIndex,
kCityIso31661BindIndex,
kCityLanguagesBindIndex,
};
inline constexpr int kLocationCityBindIndex = 1;
inline constexpr int kLocationStateProvinceBindIndex = 2;
inline constexpr int kLocationIso31662BindIndex = 3;
inline constexpr int kLocationCountryBindIndex = 4;
inline constexpr int kLocationIso31661BindIndex = 5;
inline constexpr int kLocationLanguagesBindIndex = 6;
inline constexpr int kLocationLatitudeBindIndex = 7;
inline constexpr int kLocationLongitudeBindIndex = 8;
enum BreweryBindIndex {
kBreweryCityIdBindIndex = 1,
kBreweryEnglishNameBindIndex,
kBreweryEnglishDescriptionBindIndex,
kBreweryLocalNameBindIndex,
kBreweryLocalDescriptionBindIndex,
};
inline constexpr int kBreweryLocationIdBindIndex = 1;
inline constexpr int kBreweryEnglishNameBindIndex = 2;
inline constexpr int kBreweryEnglishDescriptionBindIndex = 3;
inline constexpr int kBreweryLocalNameBindIndex = 4;
inline constexpr int kBreweryLocalDescriptionBindIndex = 5;
enum UserBindIndex {
kUserFirstNameBindIndex = 1,
kUserLastNameBindIndex,
kUserGenderBindIndex,
kUserUsernameBindIndex,
kUserBioBindIndex,
kUserActivityWeightBindIndex,
kUserEmailBindIndex,
kUserDateOfBirthBindIndex,
};
enum BreweryAddressBindIndex {
kBreweryAddressBreweryIdBindIndex = 1,
kBreweryAddressPostalCodeBindIndex,
kBreweryAddressCityIdBindIndex,
};
enum UserAddressBindIndex {
kUserAddressUserIdBindIndex = 1,
kUserAddressCityIdBindIndex,
};
inline constexpr int kUserLocationIdBindIndex = 1;
inline constexpr int kUserFirstNameBindIndex = 2;
inline constexpr int kUserLastNameBindIndex = 3;
inline constexpr int kUserGenderBindIndex = 4;
inline constexpr int kUserUsernameBindIndex = 5;
inline constexpr int kUserBioBindIndex = 6;
inline constexpr int kUserActivityWeightBindIndex = 7;
inline constexpr int kUserEmailBindIndex = 8;
inline constexpr int kUserDateOfBirthBindIndex = 9;
SqliteStatementHandle PrepareStatement(const SqliteDatabaseHandle& db_handle,
std::string_view sql,
@@ -196,13 +141,13 @@ SqliteStatementHandle PrepareStatement(const SqliteDatabaseHandle& db_handle,
void ResetStatement(SqliteStatementHandle& statement);
void Bind(const SqliteStatementHandle& statement,
const BoundParam<std::string_view>& param);
const BindParam<std::string_view>& param);
void Bind(const SqliteStatementHandle& statement,
const BoundParam<double>& param);
const BindParam<double>& param);
void Bind(const SqliteStatementHandle& statement,
const BoundParam<sqlite3_int64>& param);
const BindParam<sqlite3_int64>& param);
void StepStatement(const SqliteDatabaseHandle& db_handle,
const SqliteStatementHandle& statement,

View File

@@ -23,7 +23,7 @@ class IEnrichmentService {
* @param loc Location to enrich.
* @return Context text, or an empty string if unavailable.
*/
virtual std::string GetLocationContext(const City& loc) = 0;
virtual std::string GetLocationContext(const Location& loc) = 0;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_ENRICHMENT_SERVICE_H_

View File

@@ -8,14 +8,14 @@
#include <string>
#include "services/enrichment/enrichment_service.h"
#include "enrichment_service.h"
/**
* @brief Enrichment service that returns no context for any location.
*/
class MockEnrichmentService final : public IEnrichmentService {
public:
std::string GetLocationContext(const City& /*loc*/) override {
std::string GetLocationContext(const Location& /*loc*/) override {
return {};
}
};

View File

@@ -11,7 +11,7 @@
#include <string_view>
#include <unordered_map>
#include "services/enrichment/enrichment_service.h"
#include "enrichment_service.h"
#include "services/logging/logger.h"
#include "web_client/web_client.h"
@@ -29,7 +29,7 @@ class WikipediaEnrichmentService final : public IEnrichmentService {
/**
* @brief Returns the Wikipedia-derived context for a location.
*/
[[nodiscard]] std::string GetLocationContext(const City& loc) override;
[[nodiscard]] std::string GetLocationContext(const Location& loc) override;
private:
std::string FetchExtract(std::string_view query);

View File

@@ -6,8 +6,8 @@
* a bounded channel for later processing by the dispatcher.
*/
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_PRODUCER_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_PRODUCER_H_
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_CHANNEL_LOGGER_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_CHANNEL_LOGGER_H_
#include <string_view>
@@ -50,4 +50,4 @@ class LogProducer final : public ILogger {
BoundedChannel<LogEntry>& channel_;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_PRODUCER_H_
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_CHANNEL_LOGGER_H_

View File

@@ -1,32 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_POSTAL_CODE_MOCK_POSTAL_CODE_SERVICE_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_POSTAL_CODE_MOCK_POSTAL_CODE_SERVICE_H_
/**
* @file services/postal_code/mock_postal_code_service.h
* @brief Placeholder IPostalCodeService used until xeger-based generation is
* implemented.
*/
#include <stdexcept>
#include <string>
#include "data_model/models.h"
#include "services/postal_code/postal_code_service.h"
/**
* @brief Postal code service that returns the location's first known
* example postal code rather than generating one from its regex.
*/
class MockPostalCodeService final : public IPostalCodeService {
public:
std::string GeneratePostalCode(const City& location) override {
if (location.postal_code_examples.empty()) {
throw std::runtime_error(
"MockPostalCodeService: location has no postal_code_examples: " +
location.city);
}
return location.postal_code_examples.front();
}
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_POSTAL_CODE_MOCK_POSTAL_CODE_SERVICE_H_

View File

@@ -1,30 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_POSTAL_CODE_POSTAL_CODE_SERVICE_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_POSTAL_CODE_POSTAL_CODE_SERVICE_H_
/**
* @file services/postal_code/postal_code_service.h
* @brief Abstraction for generating a postal code for a location.
*/
#include <string>
#include "data_model/models.h"
/**
* @brief Interface for services that generate a postal code string matching
* a location's postal code format (`City::postal_regex`).
*/
class IPostalCodeService {
public:
virtual ~IPostalCodeService() = default;
/**
* @brief Generates a postal code for the given location.
*
* @param location Location whose postal code format to generate for.
* @return A postal code string matching `location.postal_regex`.
*/
virtual std::string GeneratePostalCode(const City& location) = 0;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_POSTAL_CODE_POSTAL_CODE_SERVICE_H_

View File

@@ -52,4 +52,4 @@ class HttpWebClient final : public WebClient {
std::shared_ptr<ILogger> logger_;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_WEB_CLIENT_HTTP_WEB_CLIENT_H_
#endif

File diff suppressed because it is too large Load Diff

View File

@@ -1,8 +0,0 @@
./biergarten-pipeline \
--model ../models/google_gemma-4-E4B-it-Q6_K.gguf \
--temperature 1.0 \
--top-p 0.95 \
--top-k 64 \
--n-ctx 8192 \
--location-count 15 \
--prompt-dir ../prompts

View File

@@ -52,7 +52,7 @@ WORKDIR /app/build
COPY --from=builder /app/build/biergarten-pipeline ./
# Copy required config files
COPY cities.json /app/build/
COPY locations.json /app/build/
COPY beer-styles.json /app/build/
# Copy prompt templates

View File

@@ -7,20 +7,14 @@
#include <utility>
#include "services/postal_code/postal_code_service.h"
BiergartenPipelineOrchestrator::BiergartenPipelineOrchestrator(
std::shared_ptr<ILogger> logger,
std::unique_ptr<IEnrichmentService> context_service,
std::unique_ptr<DataGenerator> generator,
std::unique_ptr<IExportService> exporter,
std::unique_ptr<ICuratedDataService> curated_data_service,
std::unique_ptr<IPostalCodeService> postal_code_service,
const ApplicationOptions& application_options)
const ApplicationOptions& app_options)
: logger_(std::move(logger)),
context_service_(std::move(context_service)),
generator_(std::move(generator)),
exporter_(std::move(exporter)),
curated_data_service_(std::move(curated_data_service)),
postal_code_service_(std::move(postal_code_service)),
application_options_(application_options) {}
application_options_(app_options) {}

View File

@@ -22,17 +22,12 @@ void BiergartenPipelineOrchestrator::GenerateBreweries(
const auto generate_record =
[this, &skipped_count](
const City& location,
const Location& location,
const std::string& region_context) -> std::optional<BreweryRecord> {
try {
const BreweryResult brewery =
generator_->GenerateBrewery(location, region_context);
const std::string postal_code =
postal_code_service_->GeneratePostalCode(location);
return BreweryRecord{
.address =
BreweryAddress{.city = location, .postal_code = postal_code},
.brewery = brewery};
return BreweryRecord{.location = location, .brewery = brewery};
} catch (const std::exception& e) {
++skipped_count;
@@ -52,12 +47,12 @@ void BiergartenPipelineOrchestrator::GenerateBreweries(
exporter_->ProcessRecord(record);
} catch (const std::exception& export_exception) {
++export_failed_count;
logger_->Log({.level = LogLevel::Warn,
logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::BreweryAndBeerGeneration,
.message = std::format(
"[Pipeline] Generated brewery for '{}' ({}) "
.message = std::format("[Pipeline] Generated brewery for '{}' ({}) "
"but SQLite export failed: {}",
record.address.city.city, record.address.city.country,
record.location.city, record.location.country,
export_exception.what())});
}
};

View File

@@ -6,7 +6,6 @@
#include <cctype>
#include <chrono>
#include <format>
#include <iterator>
#include <optional>
#include <random>
#include <stdexcept>
@@ -15,10 +14,11 @@
#include <unordered_set>
#include "biergarten_pipeline_orchestrator.h"
#include "services/curated_data/curated_json_data_service.h"
#include "json_handling/json_loader.h"
#include "services/logging/logger.h"
namespace {
std::string Sanitize(std::string_view value) {
std::string out;
out.reserve(value.size());
@@ -71,35 +71,6 @@ std::string GenerateDateOfBirth(std::mt19937& rng) {
static_cast<unsigned>(birth_ymd.day()));
}
std::optional<Name> SampleName(
const ForenamesByCountryMap& forenames_by_country,
const SurnamesByCountryMap& surnames_by_country,
const std::string& iso3166_1, std::mt19937& rng) {
const auto forenames_it = forenames_by_country.find(iso3166_1);
const auto surnames_it = surnames_by_country.find(iso3166_1);
if (forenames_it == forenames_by_country.end() ||
surnames_it == surnames_by_country.end() ||
forenames_it->second.empty() || surnames_it->second.empty()) {
return std::nullopt;
}
const ForenameList& forenames = forenames_it->second;
const SurnameList& surnames = surnames_it->second;
std::uniform_int_distribution<size_t> forename_dist(0, forenames.size() - 1);
std::uniform_int_distribution<size_t> surname_dist(0, surnames.size() - 1);
auto forename_it = forenames.begin();
std::advance(forename_it, forename_dist(rng));
auto surname_it = surnames.begin();
std::advance(surname_it, surname_dist(rng));
return Name{.first_name = forename_it->name,
.last_name = *surname_it,
.gender = forename_it->gender};
}
} // namespace
void BiergartenPipelineOrchestrator::GenerateUsers(
@@ -108,17 +79,16 @@ void BiergartenPipelineOrchestrator::GenerateUsers(
.phase = PipelinePhase::UserGeneration,
.message = "=== SAMPLE USER GENERATION ==="});
const PersonasList& personas = curated_data_service_->LoadPersonas();
const std::vector<UserPersona> personas =
JsonLoader::LoadPersonas("personas.json");
if (personas.empty()) {
throw std::runtime_error(
"No personas available in personas.json for user generation");
}
const ForenamesByCountryMap& forenames_by_country =
curated_data_service_->LoadForenamesByCountry();
const SurnamesByCountryMap& surnames_by_country =
curated_data_service_->LoadSurnamesByCountry();
const NamesByCountry names_by_country = JsonLoader::LoadNamesByCountry(
"forenames-by-country.json", "surnames-by-country.json");
std::mt19937 rng(std::random_device{}());
std::uniform_int_distribution<size_t> persona_dist(0, personas.size() - 1);
@@ -126,18 +96,19 @@ void BiergartenPipelineOrchestrator::GenerateUsers(
generated_users_.clear();
size_t skipped_count = 0;
size_t export_failed_count = 0;
std::unordered_set<std::string> used_email_local_parts;
const auto generate_record =
[this, &rng, &skipped_count, &used_email_local_parts](
[this, &rng, &skipped_count](
const EnrichedCity& city, const UserPersona& persona,
const Name& sampled_name) -> std::optional<UserRecord> {
try {
std::unordered_set<std::string> used_email_local_parts;
const UserResult user =
generator_->GenerateUser(city, persona, sampled_name);
return UserRecord{
.address = UserAddress{.city = city.location},
.location = city.location,
.user = user,
.email = BuildEmail(sampled_name, used_email_local_parts),
.date_of_birth = GenerateDateOfBirth(rng),
@@ -161,20 +132,19 @@ void BiergartenPipelineOrchestrator::GenerateUsers(
} catch (const std::exception& export_exception) {
++export_failed_count;
logger_->Log({.level = LogLevel::Warn,
logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration,
.message = std::format(
"[Pipeline] Generated user for '{}' ({}) but "
.message = std::format("[Pipeline] Generated user for '{}' ({}) but "
"SQLite export failed: {}",
record.address.city.city, record.address.city.country,
record.location.city, record.location.country,
export_exception.what())});
}
};
for (const auto& city : cities) {
const std::optional<Name> sampled_name =
SampleName(forenames_by_country, surnames_by_country,
city.location.iso3166_1, rng);
names_by_country.SampleName(city.location.iso3166_1, rng);
if (!sampled_name.has_value()) {
++skipped_count;

View File

@@ -7,26 +7,26 @@
#include <chrono>
#include <format>
#include "../../includes/json_handling/pretty_print.h"
#include "biergarten_pipeline_orchestrator.h"
#include "json_handling/pretty_print.h"
#include "services/logging/logger.h"
void BiergartenPipelineOrchestrator::LogResults() const {
boost::json::array brewery_output;
for (const auto& [address, brewery] : generated_breweries_) {
const City& location = address.city;
for (const auto& [location, brewery] : generated_breweries_) {
brewery_output.push_back(boost::json::object{
{"name_en", brewery.name_en},
{"description_en", brewery.description_en},
{"name_local", brewery.name_local},
{"description_local", brewery.description_local},
{"postal_code", address.postal_code},
{"location", boost::json::object{
{"city", location.city},
{"country", location.country},
{"state_province", location.state_province},
{"iso3166_2", location.iso3166_2},
{"latitude", location.latitude},
{"longitude", location.longitude},
}}});
}

View File

@@ -6,22 +6,24 @@
#include <algorithm>
#include <chrono>
#include <filesystem>
#include <format>
#include <iterator>
#include <random>
#include "biergarten_pipeline_orchestrator.h"
#include "services/curated_data/curated_json_data_service.h"
#include "json_handling/json_loader.h"
#include "services/logging/logger.h"
std::vector<City>
std::vector<Location>
BiergartenPipelineOrchestrator::QueryCitiesWithCountries() {
logger_->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "=== GEOGRAPHIC DATA OVERVIEW ==="});
const std::vector<City>& all_locations =
curated_data_service_->LoadCities();
const std::filesystem::path locations_path = "locations.json";
auto all_locations = JsonLoader::LoadLocations(locations_path);
const size_t sample_count = std::min(
static_cast<size_t>(application_options_.pipeline.location_count),
@@ -31,7 +33,7 @@ BiergartenPipelineOrchestrator::QueryCitiesWithCountries() {
static_cast<std::iter_difference_t<decltype(all_locations.cbegin())>>(
sample_count);
std::vector<City> sampled_locations;
std::vector<Location> sampled_locations;
sampled_locations.reserve(sample_count);
std::random_device random_generator;

View File

@@ -14,7 +14,7 @@ bool BiergartenPipelineOrchestrator::Run() {
try {
exporter_->Initialize();
std::vector<City> cities = QueryCitiesWithCountries();
std::vector<Location> cities = QueryCitiesWithCountries();
std::vector<EnrichedCity> enriched;
enriched.reserve(cities.size());

View File

@@ -36,7 +36,7 @@ static std::string FormatLocalLanguageCodes(
static constexpr int kBreweryInitialMaxTokens = 2800;
BreweryResult LlamaGenerator::GenerateBrewery(
const City& location, const std::string& region_context) {
const Location& location, const std::string& region_context) {
/**
* Preprocess and truncate region context to manageable size
*/

View File

@@ -30,13 +30,10 @@ UserResult LlamaGenerator::GenerateUser(const EnrichedCity& city,
const std::string system_prompt = prompt_directory_->Load("USER_GENERATION");
std::string user_prompt = std::format(
"## NAME:\n\n{} {}\n\n"
"## GENDER:\n\n{}\n\n"
"## CITY:\n\n{}\n\n"
"## COUNTRY:\n\n{}\n\n"
"## PERSONA:\n\n{}\n\n"
"## PERSONA DESCRIPTION:\n\n{}\n\n"
"## STYLE AFFINITIES:\n\n{}",
"## NAME:\n{} {}\n\n## GENDER:\n{}\n\n## CITY:\n{}\n\n## "
"COUNTRY:\n{}\n\n"
"## PERSONA:\n{}\n\n## PERSONA DESCRIPTION:\n{}\n\n## STYLE "
"AFFINITIES:\n{}",
name.first_name, name.last_name, name.gender, city.location.city,
city.location.country, persona.name, persona.description,
style_affinities);

View File

@@ -15,10 +15,8 @@
#include <string_view>
#include <vector>
#include <llama.h>
#include "data_generation/llama_generator_helpers.h"
#include "llama.h"
namespace {
/**
* String trimming: removes leading and trailing whitespace

View File

@@ -14,10 +14,9 @@
#include <string_view>
#include <vector>
#include <llama.h>
#include "data_generation/llama_generator.h"
#include "data_generation/llama_generator_helpers.h"
#include "llama.h"
static constexpr size_t kPromptTokenSlack = 8;
// Minimum tokens to keep when using top-p sampling. Ensures at least one

View File

@@ -11,9 +11,8 @@
#include <stdexcept>
#include <string>
#include <llama.h>
#include "data_model/models.h"
#include "llama.h"
static constexpr uint32_t kMaxContextSize = 32768U;

View File

@@ -10,10 +10,9 @@
#include <string>
#include <utility>
#include <ggml-backend.h>
#include <llama.h>
#include "data_generation/llama_generator.h"
#include "ggml-backend.h"
#include "llama.h"
// Maximum batch size for decode operations. Capping the batch prevents
// excessive memory allocation while maintaining inference performance.

View File

@@ -8,14 +8,14 @@
#include "data_generation/mock_generator.h"
size_t MockGenerator::DeterministicHash(const City& location) {
size_t MockGenerator::DeterministicHash(const Location& location) {
size_t seed = 0;
boost::hash_combine(seed, location.city);
boost::hash_combine(seed, location.country);
return seed;
}
size_t MockGenerator::DeterministicHash(const City& location,
size_t MockGenerator::DeterministicHash(const Location& location,
const UserPersona& persona,
const Name& name) {
size_t seed = DeterministicHash(location);

View File

@@ -11,7 +11,7 @@
#include "data_generation/mock_generator.h"
BreweryResult MockGenerator::GenerateBrewery(
const City& location, const std::string& /*region_context*/) {
const Location& location, const std::string& /*region_context*/) {
const size_t hash = DeterministicHash(location);
const std::string_view adjective =

View File

@@ -0,0 +1,40 @@
/**
* @file data_model/names_by_country.cc
* @brief NamesByCountry::SampleName() implementation.
*/
#include "data_model/names_by_country.h"
#include <utility>
NamesByCountry::NamesByCountry(
std::unordered_map<std::string, std::vector<ForenameEntry>>
forenames_by_country,
std::unordered_map<std::string, std::vector<std::string>>
surnames_by_country)
: forenames_by_country_(std::move(forenames_by_country)),
surnames_by_country_(std::move(surnames_by_country)) {}
std::optional<Name> NamesByCountry::SampleName(const std::string& iso3166_1,
std::mt19937& rng) const {
const auto forenames_it = forenames_by_country_.find(iso3166_1);
const auto surnames_it = surnames_by_country_.find(iso3166_1);
if (forenames_it == forenames_by_country_.end() ||
surnames_it == surnames_by_country_.end() ||
forenames_it->second.empty() || surnames_it->second.empty()) {
return std::nullopt;
}
const std::vector<ForenameEntry>& forenames = forenames_it->second;
const std::vector<std::string>& surnames = surnames_it->second;
std::uniform_int_distribution<size_t> forename_dist(0, forenames.size() - 1);
std::uniform_int_distribution<size_t> surname_dist(0, surnames.size() - 1);
const ForenameEntry& forename = forenames[forename_dist(rng)];
return Name{.first_name = forename.name,
.last_name = surnames[surname_dist(rng)],
.gender = forename.gender};
}

View File

@@ -1,10 +1,10 @@
/**
* @file json_handling/json_loader.cc
* @brief Parses curated location JSON input into strongly typed City
* @brief Parses curated location JSON input into strongly typed Location
* records with strict field validation and descriptive error reporting.
*/
#include "services/curated_data/curated_json_data_service.h"
#include "json_handling/json_loader.h"
#include <boost/json.hpp>
#include <format>
@@ -14,8 +14,8 @@
#include <stdexcept>
#include <string>
#include <string_view>
#include <unordered_map>
#include <utility>
#include <vector>
#include "services/logging/logger.h"
@@ -106,89 +106,51 @@ std::string ReadFirstOfStringArray(const boost::json::object& object,
} // namespace
CuratedJsonDataService::CuratedJsonDataService(CuratedDataFilePaths filepaths)
: filepaths_(std::move(filepaths)) {}
const CityList& CuratedJsonDataService::LoadCities() {
if (!cache_.locations.empty()) {
return cache_.locations;
}
const boost::json::value root =
ParseJsonFile(filepaths_.cities_path, "cities");
std::vector<Location> JsonLoader::LoadLocations(
const std::filesystem::path& filepath) {
const boost::json::value root = ParseJsonFile(filepath, "locations");
if (!root.is_array()) {
throw std::runtime_error(
"Invalid cities JSON: root element must be an array");
"Invalid locations JSON: root element must be an array");
}
CityList locations;
std::vector<Location> locations;
const auto& items = root.as_array();
locations.reserve(items.size());
for (const auto& item : items) {
if (!item.is_object()) {
throw std::runtime_error(
"Invalid cities JSON: each entry must be an object");
"Invalid locations JSON: each entry must be an object");
}
const auto& object = item.as_object();
const boost::json::value* postal_code = object.if_contains("postal_code");
if (postal_code == nullptr || !postal_code->is_object()) {
throw std::runtime_error("Missing or invalid object field: postal_code");
}
locations.push_back(City{
locations.push_back(Location{
.city = ReadRequiredString(object, "city"),
.state_province = ReadRequiredString(object, "state_province"),
.iso3166_2 = ReadRequiredString(object, "iso3166_2"),
.country = ReadRequiredString(object, "country"),
.iso3166_1 = ReadRequiredString(object, "iso3166_1"),
.local_languages = ReadRequiredStringArray(object, "local_languages"),
.postal_regex =
ReadRequiredStringArray(postal_code->as_object(), "city_regex"),
.postal_code_examples =
ReadRequiredStringArray(postal_code->as_object(), "examples"),
.latitude = ReadRequiredNumber(object, "latitude"),
.longitude = ReadRequiredNumber(object, "longitude"),
});
}
for (auto location : locations) {
std::cout << "Location: " << location.city << ", "
<< location.state_province << ", " << location.iso3166_2 << ", "
<< location.country << ", " << location.iso3166_1 << "\n";
for (const auto& regex : location.postal_regex) {
std::cout << " Postal regex: " << regex << "\n";
}
for (const auto& example : location.postal_code_examples) {
std::cout << " Postal example: " << example << "\n";
}
for (const auto& lang : location.local_languages) {
std::cout << " Local language: " << lang << "\n";
}
}
cache_.locations = std::move(locations);
return cache_.locations;
return locations;
}
const PersonasList& CuratedJsonDataService::LoadPersonas() {
if (!cache_.personas.empty()) {
return cache_.personas;
}
const boost::json::value root =
ParseJsonFile(filepaths_.personas_path, "personas");
std::vector<UserPersona> JsonLoader::LoadPersonas(
const std::filesystem::path& filepath) {
const boost::json::value root = ParseJsonFile(filepath, "personas");
if (!root.is_array()) {
throw std::runtime_error(
"Invalid personas JSON: root element must be an array");
}
PersonasList personas;
std::vector<UserPersona> personas;
const auto& items = root.as_array();
personas.reserve(items.size());
@@ -206,30 +168,30 @@ const PersonasList& CuratedJsonDataService::LoadPersonas() {
});
}
cache_.personas = std::move(personas);
return cache_.personas;
return personas;
}
const ForenamesByCountryMap& CuratedJsonDataService::LoadForenamesByCountry() {
if (!cache_.forenames_by_country.empty()) {
return cache_.forenames_by_country;
}
NamesByCountry JsonLoader::LoadNamesByCountry(
const std::filesystem::path& forenames_filepath,
const std::filesystem::path& surnames_filepath) {
const boost::json::value forenames_root =
ParseJsonFile(forenames_filepath, "forenames-by-country");
const boost::json::value surnames_root =
ParseJsonFile(surnames_filepath, "surnames-by-country");
const boost::json::value root =
ParseJsonFile(filepaths_.forenames_path, "forenames-by-country");
if (!root.is_object()) {
if (!forenames_root.is_object() || !surnames_root.is_object()) {
throw std::runtime_error(
"Invalid forenames-by-country JSON: root element must be an object "
"Invalid names-by-country JSON: root element must be an object "
"keyed by ISO 3166-1 country code");
}
ForenamesByCountryMap forenames_by_country;
for (const auto& [country_code, regions] : root.as_object()) {
std::unordered_map<std::string, std::vector<ForenameEntry>>
forenames_by_country;
for (const auto& [country_code, regions] : forenames_root.as_object()) {
if (!regions.is_array()) {
continue;
}
ForenameList entries;
std::vector<ForenameEntry> entries;
for (const auto& region : regions.as_array()) {
if (!region.is_object()) {
continue;
@@ -243,7 +205,7 @@ const ForenamesByCountryMap& CuratedJsonDataService::LoadForenamesByCountry() {
continue;
}
const auto& name_object = name_value.as_object();
entries.emplace_back(ForenameEntry{
entries.push_back(ForenameEntry{
.name =
ReadFirstOfStringArray(name_object, "romanized", "localized"),
.gender = ReadRequiredString(name_object, "gender"),
@@ -253,39 +215,22 @@ const ForenamesByCountryMap& CuratedJsonDataService::LoadForenamesByCountry() {
forenames_by_country.emplace(country_code, std::move(entries));
}
cache_.forenames_by_country = std::move(forenames_by_country);
return cache_.forenames_by_country;
}
const SurnamesByCountryMap& CuratedJsonDataService::LoadSurnamesByCountry() {
if (!cache_.surnames_by_country.empty()) {
return cache_.surnames_by_country;
std::unordered_map<std::string, std::vector<std::string>> surnames_by_country;
for (const auto& [country_code, name_entries] : surnames_root.as_object()) {
if (!name_entries.is_array()) {
continue;
}
const boost::json::value root =
ParseJsonFile(filepaths_.surnames_path, "surnames-by-country");
if (!root.is_object()) {
throw std::runtime_error(
"Invalid surnames-by-country JSON: root element must be an object "
"keyed by ISO 3166-1 country code");
}
SurnamesByCountryMap surnames_by_country;
for (const auto& [country_code, name_entries] : root.as_object()) {
if (!name_entries.is_array()) continue;
SurnameList surnames;
std::vector<std::string> surnames;
for (const auto& name_value : name_entries.as_array()) {
if (!name_value.is_object()) {
continue;
}
surnames.emplace_back(ReadFirstOfStringArray(name_value.as_object(),
surnames.push_back(ReadFirstOfStringArray(name_value.as_object(),
"romanized", "localized"));
}
surnames_by_country.emplace(country_code, std::move(surnames));
}
cache_.surnames_by_country = std::move(surnames_by_country);
return cache_.surnames_by_country;
return NamesByCountry(std::move(forenames_by_country),
std::move(surnames_by_country));
}

View File

@@ -18,7 +18,25 @@
#include <string>
#include <thread>
#include "biergarten_pipeline.h"
#include "biergarten_pipeline_orchestrator.h"
#include "concurrency/bounded_channel.h"
#include "data_generation/llama_generator.h"
#include "data_generation/mock_generator.h"
#include "data_generation/prompt_formatting/gemma4_jinja_prompt_formatter.h"
#include "data_model/models.h"
#include "llama_backend_state.h"
#include "services/database/export_service.h"
#include "services/database/sqlite_export_service.h"
#include "services/datetime/timer.h"
#include "services/enrichment/enrichment_service.h"
#include "services/enrichment/mock_enrichment.h"
#include "services/enrichment/wikipedia_service.h"
#include "services/logging/log_dispatcher.h"
#include "services/logging/log_entry.h"
#include "services/logging/log_producer.h"
#include "services/logging/logger.h"
#include "services/prompting/prompt_directory.h"
#include "web_client/http_web_client.h"
namespace di = boost::di;
@@ -86,30 +104,6 @@ int main(const int argc, char** argv) {
di::bind<ApplicationOptions>().to(options),
di::bind<std::string>().to(model_path),
di::bind<IExportService>().to<SqliteExportService>(),
di::bind<ICuratedDataService>().to(
[options, &log_producer]() -> std::unique_ptr<ICuratedDataService> {
if (options.generator.use_mocked) {
log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "Curated data: mock"});
return std::make_unique<MockCuratedDataService>();
}
log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "Curated data: JsonLoader"});
return std::make_unique<CuratedJsonDataService>(
CuratedDataFilePaths{
.cities_path = "cities.json",
.personas_path = "personas.json",
.forenames_path = "forenames-by-country.json",
.surnames_path = "surnames-by-country.json",
});
}),
di::bind<IPostalCodeService>().to<MockPostalCodeService>(),
di::bind<IPromptFormatter>().to([options, log_producer] {
if (options.generator.use_mocked) {
{
@@ -138,11 +132,11 @@ int main(const int argc, char** argv) {
}
return std::unique_ptr<WebClient>(nullptr);
}
{
log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "Web client: HttpWebClient"});
}
return std::unique_ptr<WebClient>(
std::make_unique<HttpWebClient>(log_producer));
}),
@@ -150,17 +144,18 @@ int main(const int argc, char** argv) {
[options, &log_producer](
const auto& inj) -> std::unique_ptr<IEnrichmentService> {
if (options.generator.use_mocked) {
{
log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "Enrichment: mock"});
}
return std::make_unique<MockEnrichmentService>();
}
{
log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "Enrichment: Wikipedia"});
}
return std::make_unique<WikipediaEnrichmentService>(
inj.template create<std::unique_ptr<WebClient>>(),
log_producer);
@@ -169,35 +164,28 @@ int main(const int argc, char** argv) {
[&options, &model_path, &sampling, &prompt_directory,
&log_producer](const auto& inj) -> std::unique_ptr<DataGenerator> {
if (options.generator.use_mocked) {
{
log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "Generator: mock"});
}
return std::make_unique<MockGenerator>();
}
#ifdef BIERGARTEN_MOCK_ONLY
// Unreachable: ParseArguments requires --mocked when
// BIERGARTEN_MOCK_ONLY is compiled in, so LlamaGenerator is
// never constructed and is not linked into this binary.
throw std::runtime_error(
"LlamaGenerator is unavailable in a BIERGARTEN_MOCK_ONLY "
"build");
#else
{
log_producer->Log(
{.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = std::format(
"Generator: LlamaGenerator | model={} | "
"temp={:.2f} top_p={:.2f} top_k={} n_ctx={} seed={}",
"temp={:.2f} "
"top_p={:.2f} top_k={} n_ctx={} seed={}",
model_path, sampling.temperature, sampling.top_p,
sampling.top_k, sampling.n_ctx, sampling.seed)});
}
return std::make_unique<LlamaGenerator>(
options, model_path, log_producer,
inj.template create<std::unique_ptr<IPromptFormatter>>(),
std::move(prompt_directory));
#endif
}));
const auto orchestrator =
@@ -216,6 +204,7 @@ int main(const int argc, char** argv) {
timer.Elapsed())});
return shutdown(EXIT_SUCCESS);
} catch (const std::exception& exception) {
const LogDTO log_entry{.level = LogLevel::Error,
.phase = PipelinePhase::Teardown,

View File

@@ -1,96 +0,0 @@
/**
* @file services/curated_data/mock_curated_data_service.cc
* @brief Fixed in-memory location, persona, and name dataset for mock mode.
*/
#include "services/curated_data/mock_curated_data_service.h"
MockCuratedDataService::MockCuratedDataService()
: locations_{
{.city = "Portland",
.state_province = "Oregon",
.iso3166_2 = "US-OR",
.country = "United States",
.iso3166_1 = "US",
.local_languages = {"en"},
.postal_regex = {"^972[0-9]{2}(?:-[0-9]{4})?$"},
.postal_code_examples = {"97201", "97294"}},
{.city = "Munich",
.state_province = "Bavaria",
.iso3166_2 = "DE-BY",
.country = "Germany",
.iso3166_1 = "DE",
.local_languages = {"de"},
.postal_regex = {"^8[01][0-9]{3}$"},
.postal_code_examples = {"80331", "81929"}},
{.city = "Lyon",
.state_province = "Auvergne-Rhone-Alpes",
.iso3166_2 = "FR-ARA",
.country = "France",
.iso3166_1 = "FR",
.local_languages = {"fr"},
.postal_regex = {"^6900[1-9]$"},
.postal_code_examples = {"69001", "69009"}},
{.city = "Brussels",
.state_province = "Brussels-Capital",
.iso3166_2 = "BE-BRU",
.country = "Belgium",
.iso3166_1 = "BE",
.local_languages = {"nl", "fr"},
.postal_regex = {"^1(?:0[0-9]{2}|1[0-9]{2}|20[0-9]|210)$"},
.postal_code_examples = {"1000", "1210"}},
},
personas_{
UserPersona{.name = "Hophead Explorer",
.description = "Chases hop-forward IPAs and seeks out "
"taprooms with rotating drafts.",
.style_affinities = {"IPA", "Pale Ale"}},
UserPersona{.name = "Lager Traditionalist",
.description = "Prefers clean, balanced lagers and "
"classic pub styles.",
.style_affinities = {"Lager", "Pilsner"}},
UserPersona{.name = "Sour Curious",
.description = "Seeks out wild ales, sours, and "
"barrel-aged experiments.",
.style_affinities = {"Sour", "Wild Ale"}},
},
forenames_by_country_{
{"US",
ForenameList{
ForenameEntry{.name = "James", .gender = "M"},
ForenameEntry{.name = "Mary", .gender = "F"},
}},
{"DE",
ForenameList{
ForenameEntry{.name = "Lukas", .gender = "M"},
ForenameEntry{.name = "Anna", .gender = "F"},
}},
{"FR",
ForenameList{
ForenameEntry{.name = "Lucas", .gender = "M"},
ForenameEntry{.name = "Camille", .gender = "F"},
}},
{"BE",
ForenameList{
ForenameEntry{.name = "Noah", .gender = "M"},
ForenameEntry{.name = "Emma", .gender = "F"},
}},
},
surnames_by_country_{
{"US", SurnameList{"Smith", "Johnson"}},
{"DE", SurnameList{"Muller", "Schmidt"}},
{"FR", SurnameList{"Martin", "Bernard"}},
{"BE", SurnameList{"Peeters", "Janssens"}},
} {}
const CityList& MockCuratedDataService::LoadCities() { return locations_; }
const PersonasList& MockCuratedDataService::LoadPersonas() { return personas_; }
const ForenamesByCountryMap& MockCuratedDataService::LoadForenamesByCountry() {
return forenames_by_country_;
}
const SurnamesByCountryMap& MockCuratedDataService::LoadSurnamesByCountry() {
return surnames_by_country_;
}

View File

@@ -11,7 +11,7 @@
#include "services/enrichment/wikipedia_service.h"
std::string WikipediaEnrichmentService::GetLocationContext(
const City& loc) {
const Location& loc) {
using namespace std::literals::chrono_literals;
if (!this->client_) {
if (logger_) {

View File

@@ -14,11 +14,9 @@ void SqliteExportService::Finalize() {
}
try {
insert_user_address_stmt_.reset();
insert_user_stmt_.reset();
insert_brewery_address_stmt_.reset();
insert_brewery_stmt_.reset();
insert_city_stmt_.reset();
insert_location_stmt_.reset();
if (transaction_open_) {
sqlite_export_service_internal::ExecSql(
db_handle_, "COMMIT;", "Failed to commit SQLite transaction");
@@ -26,7 +24,7 @@ void SqliteExportService::Finalize() {
}
db_handle_.reset();
city_cache_.clear();
location_cache_.clear();
} catch (...) {
RollbackAndCloseNoThrow();
throw;

View File

@@ -33,7 +33,7 @@ void ResetStatement(SqliteStatementHandle& statement) {
}
void Bind(const SqliteStatementHandle& statement,
const BoundParam<std::string_view>& param) {
const BindParam<std::string_view>& param) {
const auto byte_count = param.value.size();
if (byte_count > static_cast<std::size_t>(std::numeric_limits<int>::max())) {
ThrowSqliteError(sqlite3_db_handle(statement.get()), param.action);
@@ -60,7 +60,7 @@ void Bind(const SqliteStatementHandle& statement,
}
void Bind(const SqliteStatementHandle& statement,
const BoundParam<double>& param) {
const BindParam<double>& param) {
if (sqlite3_bind_double(statement.get(), param.index, param.value) !=
SQLITE_OK) {
ThrowSqliteError(sqlite3_db_handle(statement.get()), param.action);
@@ -68,7 +68,7 @@ void Bind(const SqliteStatementHandle& statement,
}
void Bind(const SqliteStatementHandle& statement,
const BoundParam<sqlite3_int64>& param) {
const BindParam<sqlite3_int64>& param) {
if (sqlite3_bind_int64(statement.get(), param.index, param.value) !=
SQLITE_OK) {
ThrowSqliteError(sqlite3_db_handle(statement.get()), param.action);

View File

@@ -28,39 +28,26 @@ std::filesystem::path SqliteExportService::BuildDatabasePath() const {
void SqliteExportService::InitializeSchema() const {
sqlite_export_service_internal::ExecSql(
db_handle_, sqlite_export_service_internal::kCreateCitiesTableSql,
"Failed to create SQLite cities table");
db_handle_, sqlite_export_service_internal::kCreateLocationsTableSql,
"Failed to create SQLite locations table");
sqlite_export_service_internal::ExecSql(
db_handle_, sqlite_export_service_internal::kCreateBreweriesTableSql,
"Failed to create SQLite breweries table");
sqlite_export_service_internal::ExecSql(
db_handle_, sqlite_export_service_internal::kCreateUsersTableSql,
"Failed to create SQLite users table");
sqlite_export_service_internal::ExecSql(
db_handle_, sqlite_export_service_internal::kCreateBreweryAddressTableSql,
"Failed to create SQLite brewery addresses table");
sqlite_export_service_internal::ExecSql(
db_handle_, sqlite_export_service_internal::kCreateUserAddressTableSql,
"Failed to create SQLite user addresses table");
}
void SqliteExportService::PrepareStatements() {
insert_city_stmt_ = sqlite_export_service_internal::PrepareStatement(
db_handle_, sqlite_export_service_internal::kInsertCitySql,
"Failed to prepare SQLite city insert statement");
insert_location_stmt_ = sqlite_export_service_internal::PrepareStatement(
db_handle_, sqlite_export_service_internal::kInsertLocationSql,
"Failed to prepare SQLite location insert statement");
insert_brewery_stmt_ = sqlite_export_service_internal::PrepareStatement(
db_handle_, sqlite_export_service_internal::kInsertBrewerySql,
"Failed to prepare SQLite brewery insert statement");
insert_brewery_address_stmt_ =
sqlite_export_service_internal::PrepareStatement(
db_handle_, sqlite_export_service_internal::kInsertBreweryAddressSql,
"Failed to prepare SQLite brewery address insert statement");
insert_user_stmt_ = sqlite_export_service_internal::PrepareStatement(
db_handle_, sqlite_export_service_internal::kInsertUserSql,
"Failed to prepare SQLite user insert statement");
insert_user_address_stmt_ = sqlite_export_service_internal::PrepareStatement(
db_handle_, sqlite_export_service_internal::kInsertUserAddressSql,
"Failed to prepare SQLite user address insert statement");
}
void SqliteExportService::RollbackAndCloseNoThrow() noexcept {
@@ -73,13 +60,11 @@ void SqliteExportService::RollbackAndCloseNoThrow() noexcept {
transaction_open_ = false;
}
insert_user_address_stmt_.reset();
insert_user_stmt_.reset();
insert_brewery_address_stmt_.reset();
insert_brewery_stmt_.reset();
insert_city_stmt_.reset();
insert_location_stmt_.reset();
db_handle_.reset();
city_cache_.clear();
location_cache_.clear();
}
void SqliteExportService::Initialize() {

View File

@@ -1,9 +1,10 @@
/**
* @file services/sqlite/process_record.cc
* @brief SqliteExportService::ProcessRecord() implementation
* and the shared city-resolution helper.
* and the shared location-resolution helper.
*/
#include <iomanip>
#include <sstream>
#include <stdexcept>
#include <string>
@@ -11,74 +12,94 @@
#include "services/database/sqlite_export_service.h"
#include "services/database/sqlite_export_service_helpers.h"
std::string SqliteExportService::BuildCityKey(const City& location) {
constexpr int kLocationPrecision = 17;
std::string SqliteExportService::BuildLocationKey(const Location& location) {
std::ostringstream key_stream;
key_stream << location.city << '\n'
<< location.state_province << '\n'
<< location.iso3166_2 << '\n'
<< location.country << '\n'
<< location.iso3166_1 << '\n'
<< std::setprecision(kLocationPrecision) << location.latitude
<< '\n'
<< std::setprecision(kLocationPrecision) << location.longitude
<< '\n'
<< sqlite_export_service_internal::SerializeVector(
location.local_languages);
return key_stream.str();
}
sqlite3_int64 SqliteExportService::ResolveCityId(const City& location) {
const std::string city_key = BuildCityKey(location);
const auto cached_city = city_cache_.find(city_key);
if (cached_city != city_cache_.end()) {
return cached_city->second;
sqlite3_int64 SqliteExportService::ResolveLocationId(const Location& location) {
const std::string location_key = BuildLocationKey(location);
const auto cached_location = location_cache_.find(location_key);
if (cached_location != location_cache_.end()) {
return cached_location->second;
}
const std::string local_languages_json =
sqlite_export_service_internal::SerializeVector(location.local_languages);
sqlite_export_service_internal::Bind(
insert_city_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kCityNameBindIndex,
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationCityBindIndex,
.value = location.city,
.action = "Failed to bind SQLite city name"});
.action = "Failed to bind SQLite location city"});
sqlite_export_service_internal::Bind(
insert_city_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kCityStateProvinceBindIndex,
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index =
sqlite_export_service_internal::kLocationStateProvinceBindIndex,
.value = location.state_province,
.action = "Failed to bind SQLite city state/province"});
.action = "Failed to bind SQLite location state/province"});
sqlite_export_service_internal::Bind(
insert_city_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kCityIso31662BindIndex,
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationIso31662BindIndex,
.value = location.iso3166_2,
.action = "Failed to bind SQLite city ISO 3166-2 code"});
.action = "Failed to bind SQLite location ISO 3166-2 code"});
sqlite_export_service_internal::Bind(
insert_city_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kCityCountryBindIndex,
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationCountryBindIndex,
.value = location.country,
.action = "Failed to bind SQLite city country"});
.action = "Failed to bind SQLite location country"});
sqlite_export_service_internal::Bind(
insert_city_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kCityIso31661BindIndex,
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationIso31661BindIndex,
.value = location.iso3166_1,
.action = "Failed to bind SQLite city ISO 3166-1 code"});
.action = "Failed to bind SQLite location ISO 3166-1 code"});
sqlite_export_service_internal::Bind(
insert_city_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kCityLanguagesBindIndex,
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationLanguagesBindIndex,
.value = local_languages_json,
.action = "Failed to bind SQLite city languages"});
.action = "Failed to bind SQLite location languages"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam{
.index = sqlite_export_service_internal::kLocationLatitudeBindIndex,
.value = location.latitude,
.action = "Failed to bind SQLite location latitude"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam{
.index = sqlite_export_service_internal::kLocationLongitudeBindIndex,
.value = location.longitude,
.action = "Failed to bind SQLite location longitude"});
sqlite_export_service_internal::StepStatement(
db_handle_, insert_city_stmt_, "Failed to insert SQLite city row");
db_handle_, insert_location_stmt_,
"Failed to insert SQLite location row");
const sqlite3_int64 city_id =
const sqlite3_int64 location_id =
sqlite_export_service_internal::LastInsertRowId(db_handle_);
city_cache_.emplace(city_key, city_id);
sqlite_export_service_internal::ResetStatement(insert_city_stmt_);
location_cache_.emplace(location_key, location_id);
sqlite_export_service_internal::ResetStatement(insert_location_stmt_);
return city_id;
return location_id;
}
uint64_t SqliteExportService::ProcessRecord(const BreweryRecord& brewery) {
@@ -86,25 +107,25 @@ uint64_t SqliteExportService::ProcessRecord(const BreweryRecord& brewery) {
throw std::runtime_error("SQLite export service is not initialized");
}
const sqlite3_int64 city_id = ResolveCityId(brewery.address.city);
const sqlite3_int64 location_id = ResolveLocationId(brewery.location);
sqlite_export_service_internal::Bind(
insert_brewery_stmt_,
sqlite_export_service_internal::BoundParam<sqlite3_int64>{
.index = sqlite_export_service_internal::kBreweryCityIdBindIndex,
.value = city_id,
.action = "Failed to bind SQLite brewery city id"});
sqlite_export_service_internal::BindParam<sqlite3_int64>{
.index = sqlite_export_service_internal::kBreweryLocationIdBindIndex,
.value = location_id,
.action = "Failed to bind SQLite brewery location id"});
sqlite_export_service_internal::Bind(
insert_brewery_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kBreweryEnglishNameBindIndex,
.value = brewery.brewery.name_en,
.action = "Failed to bind SQLite brewery English name"});
sqlite_export_service_internal::Bind(
insert_brewery_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::
kBreweryEnglishDescriptionBindIndex,
.value = brewery.brewery.description_en,
@@ -112,14 +133,14 @@ uint64_t SqliteExportService::ProcessRecord(const BreweryRecord& brewery) {
sqlite_export_service_internal::Bind(
insert_brewery_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kBreweryLocalNameBindIndex,
.value = brewery.brewery.name_local,
.action = "Failed to bind SQLite brewery local name"});
sqlite_export_service_internal::Bind(
insert_brewery_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{
sqlite_export_service_internal::BindParam<std::string_view>{
.index =
sqlite_export_service_internal::kBreweryLocalDescriptionBindIndex,
.value = brewery.brewery.description_local,
@@ -128,37 +149,7 @@ uint64_t SqliteExportService::ProcessRecord(const BreweryRecord& brewery) {
sqlite_export_service_internal::StepStatement(
db_handle_, insert_brewery_stmt_, "Failed to insert SQLite brewery row");
const sqlite3_int64 brewery_id =
sqlite_export_service_internal::LastInsertRowId(db_handle_);
sqlite_export_service_internal::ResetStatement(insert_brewery_stmt_);
sqlite_export_service_internal::Bind(
insert_brewery_address_stmt_,
sqlite_export_service_internal::BoundParam<sqlite3_int64>{
.index =
sqlite_export_service_internal::kBreweryAddressBreweryIdBindIndex,
.value = brewery_id,
.action = "Failed to bind SQLite brewery address brewery id"});
sqlite_export_service_internal::Bind(
insert_brewery_address_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::
kBreweryAddressPostalCodeBindIndex,
.value = brewery.address.postal_code,
.action = "Failed to bind SQLite brewery address postal code"});
sqlite_export_service_internal::Bind(
insert_brewery_address_stmt_,
sqlite_export_service_internal::BoundParam<sqlite3_int64>{
.index =
sqlite_export_service_internal::kBreweryAddressCityIdBindIndex,
.value = city_id,
.action = "Failed to bind SQLite brewery address city id"});
sqlite_export_service_internal::StepStatement(
db_handle_, insert_brewery_address_stmt_,
"Failed to insert SQLite brewery address row");
sqlite_export_service_internal::ResetStatement(insert_brewery_address_stmt_);
return brewery_id;
return sqlite_export_service_internal::LastInsertRowId(db_handle_);
}

View File

@@ -13,53 +13,59 @@ uint64_t SqliteExportService::ProcessRecord(const UserRecord& user) {
throw std::runtime_error("SQLite export service is not initialized");
}
const sqlite3_int64 city_id = ResolveCityId(user.address.city);
const sqlite3_int64 location_id = ResolveLocationId(user.location);
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{
sqlite_export_service_internal::BindParam<sqlite3_int64>{
.index = sqlite_export_service_internal::kUserLocationIdBindIndex,
.value = location_id,
.action = "Failed to bind SQLite user location id"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserFirstNameBindIndex,
.value = user.user.first_name,
.action = "Failed to bind SQLite user first name"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserLastNameBindIndex,
.value = user.user.last_name,
.action = "Failed to bind SQLite user last name"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserGenderBindIndex,
.value = user.user.gender,
.action = "Failed to bind SQLite user gender"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserUsernameBindIndex,
.value = user.user.username,
.action = "Failed to bind SQLite user username"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserBioBindIndex,
.value = user.user.bio,
.action = "Failed to bind SQLite user bio"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BoundParam<double>{
sqlite_export_service_internal::BindParam<double>{
.index = sqlite_export_service_internal::kUserActivityWeightBindIndex,
.value = static_cast<double>(user.user.activity_weight),
.action = "Failed to bind SQLite user activity weight"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserEmailBindIndex,
.value = user.email,
.action = "Failed to bind SQLite user email"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserDateOfBirthBindIndex,
.value = user.date_of_birth,
.action = "Failed to bind SQLite user date of birth"});
@@ -67,28 +73,7 @@ uint64_t SqliteExportService::ProcessRecord(const UserRecord& user) {
sqlite_export_service_internal::StepStatement(
db_handle_, insert_user_stmt_, "Failed to insert SQLite user row");
const sqlite3_int64 user_id =
sqlite_export_service_internal::LastInsertRowId(db_handle_);
sqlite_export_service_internal::ResetStatement(insert_user_stmt_);
sqlite_export_service_internal::Bind(
insert_user_address_stmt_,
sqlite_export_service_internal::BoundParam<sqlite3_int64>{
.index = sqlite_export_service_internal::kUserAddressUserIdBindIndex,
.value = user_id,
.action = "Failed to bind SQLite user address user id"});
sqlite_export_service_internal::Bind(
insert_user_address_stmt_,
sqlite_export_service_internal::BoundParam<sqlite3_int64>{
.index = sqlite_export_service_internal::kUserAddressCityIdBindIndex,
.value = city_id,
.action = "Failed to bind SQLite user address city id"});
sqlite_export_service_internal::StepStatement(
db_handle_, insert_user_address_stmt_,
"Failed to insert SQLite user address row");
sqlite_export_service_internal::ResetStatement(insert_user_address_stmt_);
return user_id;
return sqlite_export_service_internal::LastInsertRowId(db_handle_);
}

View File

@@ -1 +0,0 @@
/Users/aaronpo/projects/the-biergarten-app/docs/website