AaronPo/the-biergarten-app
1
0
Fork 0
mirror of https://github.com/aaronpo97/the-biergarten-app.git synced 2026-10-14 17:54:42 +00:00
Code Issues Packages Projects Releases Wiki Activity

Formatting

Browse Source
This commit is contained in:
Aaron Po
2026-07-19 04:05:30 -04:00
parent 8e7ecf2fc3
commit ec770aa724
78 changed files with 13388 additions and 32143 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
docs/pipeline/ETHICS-AND-KNOWN-ISSUES.md
View File

@@ -102,21 +102,19 @@ vendored verbatim, unmodified, from
(the `common-forenames-by-country.json` / `common-surnames-by-country.json` (the `common-forenames-by-country.json` / `common-surnames-by-country.json`
release assets), released under **CC0** (public domain). That dataset's own release assets), released under **CC0** (public domain). That dataset's own
forename/surname lists are pulled from Wikipedia's "Lists of most common forename/surname lists are pulled from Wikipedia's "Lists of most common
surnames" and "List of most popular given names" as of the week of surnames" and "List of most popular given names" as of the week of 2023-07-08 —
2023-07-08 — see that project's README for full provenance. Names are not see that project's README for full provenance. Names are not LLM-generated; this
LLM-generated; this is curated fixture data per ROADMAP.md §2. Per-forename is curated fixture data per ROADMAP.md §2. Per-forename gender from the source
gender from the source data is preserved through to the sampled `Name` data is preserved through to the sampled `Name` (rather than discarded during
(rather than discarded during loading) so it's available for gender-aware loading) so it's available for gender-aware persona/bio generation later.
persona/bio generation later.
The full multinational dataset is kept as-is (106 countries for forenames, The full multinational dataset is kept as-is (106 countries for forenames, 75
75 for surnames) rather than trimmed to `locations.json`'s current country for surnames) rather than trimmed to `locations.json`'s current country list, so
list, so it doesn't need re-sourcing if more countries are added later. it doesn't need re-sourcing if more countries are added later. `SampleName()` (a
`SampleName()` (a free helper in `generate_users.cc`) returns no result for free helper in `generate_users.cc`) returns no result for a country present in
a country present in neither file; of the countries in `locations.json`, neither file; of the countries in `locations.json`, that's currently `KE`, `SE`,
that's currently `KE`, `SE`, `SG`, `TH`, `VN`, and `ZA` — `GenerateUsers` `SG`, `TH`, `VN`, and `ZA` — `GenerateUsers` skips cities in those countries the
skips cities in those countries the same way brewery generation skips same way brewery generation skips cities whose enrichment lookup fails.
cities whose enrichment lookup fails.
--- ---
@@ -273,8 +271,7 @@ For languages such as Welsh (Wales), Māori (Aotearoa/New Zealand), or Sicilian
but is semantically incoherent. This comes from limited training-data coverage but is semantically incoherent. This comes from limited training-data coverage
rather than prompt engineering. rather than prompt engineering.
Output sample: Output sample: [./french-cities.example](french-cities.example)
[./french-cities.example](french-cities.example)
#### Proposed Mitigations #### Proposed Mitigations
@@ -298,9 +295,9 @@ curated real-world examples.
The generated code is **format-conformant by construction** — it always The generated code is **format-conformant by construction** — it always
satisfies the regex it was generated from — but it is not looked up from any satisfies the regex it was generated from — but it is not looked up from any
real postal registry. It does not correspond to a real address, business, or real postal registry. It does not correspond to a real address, business, or
resident, and should not be treated as accurate geocoding or delivery data. resident, and should not be treated as accurate geocoding or delivery data. This
This is consistent with the rest of the dataset: fixture data for a is consistent with the rest of the dataset: fixture data for a proof-of-concept,
proof-of-concept, not a source of truth. not a source of truth.
--- ---

219
docs/pipeline/README.md
View File

@@ -1,9 +1,9 @@
# Biergarten Pipeline # Biergarten Pipeline
A C++20 command-line pipeline that samples city records from local JSON, A C++20 command-line pipeline that samples city records from local JSON,
enriches each with Wikipedia context, and generates bilingual brewery names enriches each with Wikipedia context, and generates bilingual brewery names and
and descriptions plus locale-grounded user profiles via a local GGUF model or descriptions plus locale-grounded user profiles via a local GGUF model or a
a deterministic mock. deterministic mock.
> **This pipeline produces AI-generated data.** It is not a source of truth for > **This pipeline produces AI-generated data.** It is not a source of truth for
> brewing techniques, cultural representation, or local-language accuracy. See > brewing techniques, cultural representation, or local-language accuracy. See
@@ -40,12 +40,12 @@ The pipeline is a data ingestion layer. It sits outside the web app runtime and
produces seed records the app imports at startup or during a dedicated seed produces seed records the app imports at startup or during a dedicated seed
step. step.
| Planned app area | Pipeline contribution | | Planned app area | Pipeline contribution |
| -------------------------------- | ------------------------------------------------------------------ | | -------------------------------- | ------------------------------------------------------------------------------------------------- |
| Brewery discovery and management | Sampled city records, localized names, long-form descriptions | | Brewery discovery and management | Sampled city records, localized names, long-form descriptions |
| Beer reviews and ratings | Stable brewery fixtures with enough context to anchor review pages | | Beer reviews and ratings | Stable brewery fixtures with enough context to anchor review pages |
| Social follow relationships | Repeatable brewery entities for feeds, follows, and saved lists | | Social follow relationships | Repeatable brewery entities for feeds, follows, and saved lists |
| Geospatial brewery experiences | Latitude, longitude, and country-level metadata | | Geospatial brewery experiences | Latitude, longitude, and country-level metadata |
| User accounts and profiles | Locale-grounded names, bios, and an auth-ready email/date-of-birth pair for seeding real accounts | | User accounts and profiles | Locale-grounded names, bios, and an auth-ready email/date-of-birth pair for seeding real accounts |
--- ---
@@ -65,10 +65,9 @@ cmake --build build
CMake automatically detects whether a compatible llama.cpp installation is CMake automatically detects whether a compatible llama.cpp installation is
present on the system (`libllama`, `libggml`, `libggml-base`, and `llama.h` present on the system (`libllama`, `libggml`, `libggml-base`, and `llama.h`
visible on the default search paths). If found, it links against those visible on the default search paths). If found, it links against those libraries
libraries and skips the FetchContent build. If not found, it fetches and builds and skips the FetchContent build. If not found, it fetches and builds llama.cpp
llama.cpp from source at tag `b9012`. No additional flags are required in from source at tag `b9012`. No additional flags are required in either case.
either case.
Metal is enabled automatically on Apple Silicon. CUDA or HIP/ROCm is detected Metal is enabled automatically on Apple Silicon. CUDA or HIP/ROCm is detected
automatically on Linux when the relevant toolkit is present. automatically on Linux when the relevant toolkit is present.
@@ -138,17 +137,18 @@ NVIDIA GPU.
The container uses a two-stage build. The builder stage installs CMake/Ninja, The container uses a two-stage build. The builder stage installs CMake/Ninja,
clones the matching llama.cpp release tag for its headers only (installed into clones the matching llama.cpp release tag for its headers only (installed into
`/usr/local/include`), and copies prebuilt shared libraries (`libllama`, `/usr/local/include`), and copies prebuilt shared libraries (`libllama`,
`libggml`, and CUDA/CPU backend plugins) from `ghcr.io/ggml-org/llama.cpp:full-cuda` `libggml`, and CUDA/CPU backend plugins) from
into `/usr/local/lib`. With both headers and libraries present, CMake's `ghcr.io/ggml-org/llama.cpp:full-cuda` into `/usr/local/lib`. With both headers
system-library detection (see [Build](#build) above) finds them and skips the and libraries present, CMake's system-library detection (see [Build](#build)
FetchContent source build, keeping image build times short. above) finds them and skips the FetchContent source build, keeping image build
times short.
The runtime stage copies the compiled binary, the same prebuilt shared The runtime stage copies the compiled binary, the same prebuilt shared
libraries, and config/prompt assets into a slim CUDA runtime image. It sets libraries, and config/prompt assets into a slim CUDA runtime image. It sets
`LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH` so the dynamic linker `LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH` so the dynamic linker resolves
resolves `libllama`/`libggml` at startup, and also co-locates `libllama`/`libggml` at startup, and also co-locates `libggml-cuda.so` and the
`libggml-cuda.so` and the CPU backend plugins next to the binary for CPU backend plugins next to the binary for `ggml_backend_load_all()`'s `dlopen`
`ggml_backend_load_all()`'s `dlopen` scan. scan.
### Build the image ### Build the image
@@ -177,8 +177,8 @@ output to confirm the fast path was taken.
The container always runs the model-backed path; there is no `--mocked` The container always runs the model-backed path; there is no `--mocked`
container mode (use a native build for that — see [Quick Start](#quick-start)). container mode (use a native build for that — see [Quick Start](#quick-start)).
The entrypoint, `runpod/start.sh`, downloads the GGUF model automatically if The entrypoint, `runpod/start.sh`, downloads the GGUF model automatically if it
it is not already present at the configured path. is not already present at the configured path.
```bash ```bash
docker run --rm \ docker run --rm \
@@ -189,20 +189,20 @@ docker run --rm \
biergarten-pipeline:latest biergarten-pipeline:latest
``` ```
By default this downloads `google_gemma-4-E4B-it-Q6_K.gguf` to By default this downloads `google_gemma-4-E4B-it-Q6_K.gguf` to `./models/` on
`./models/` on first run if it isn't already there. To use a pre-downloaded first run if it isn't already there. To use a pre-downloaded model, place it at
model, place it at that path first — see [Model](#model) above. that path first — see [Model](#model) above.
#### Environment variables #### Environment variables
| Variable | Purpose | | Variable | Purpose |
| ------------------------- | ----------------------------------------------------------------- | | ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| `BIERGARTEN_MODEL_PATH` | GGUF model path. Default: `/workspace/models/google_gemma-4-E4B-it-Q6_K.gguf`. | | `BIERGARTEN_MODEL_PATH` | GGUF model path. Default: `/workspace/models/google_gemma-4-E4B-it-Q6_K.gguf`. |
| `BIERGARTEN_OUTPUT_DIR` | SQLite output directory. Default: `/workspace/output`. | | `BIERGARTEN_OUTPUT_DIR` | SQLite output directory. Default: `/workspace/output`. |
| `BIERGARTEN_LOG_PATH` | Log file path. Default: `/workspace/logs/pipeline.log`. | | `BIERGARTEN_LOG_PATH` | Log file path. Default: `/workspace/logs/pipeline.log`. |
| `BIERGARTEN_GL_LAYERS` | GPU layers to offload (`--n-gpu-layers`). Default: `40`. | | `BIERGARTEN_GL_LAYERS` | GPU layers to offload (`--n-gpu-layers`). Default: `40`. |
| `BIERGARTEN_TEMPERATURE`, `BIERGARTEN_TOP_P`, `BIERGARTEN_TOP_K`, `BIERGARTEN_N_CTX`, `BIERGARTEN_SEED` | Optional sampling overrides, unset by default (binary defaults apply). | | `BIERGARTEN_TEMPERATURE`, `BIERGARTEN_TOP_P`, `BIERGARTEN_TOP_K`, `BIERGARTEN_N_CTX`, `BIERGARTEN_SEED` | Optional sampling overrides, unset by default (binary defaults apply). |
| `BIERGARTEN_EXTRA_ARGS` | Additional raw CLI args appended verbatim. | | `BIERGARTEN_EXTRA_ARGS` | Additional raw CLI args appended verbatim. |
`--prompt-dir` is hardcoded to `/app/prompts` inside the container and is not `--prompt-dir` is hardcoded to `/app/prompts` inside the container and is not
configurable via environment variable. configurable via environment variable.
@@ -220,15 +220,15 @@ environment variables listed above to match your run.
### Pipeline Stages ### Pipeline Stages
| Stage | Implementation | | Stage | Implementation |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Load | `ICuratedDataService` (`CuratedJsonDataService`) reads `locations.json`, `personas.json`, `forenames-by-country.json`, and `surnames-by-country.json` (paths supplied via a `CuratedDataFilePaths` DTO at construction) into typed records, caching each after its first load. `--mocked` runs use `MockCuratedDataService`'s fixed in-memory dataset instead. | | Load | `ICuratedDataService` (`CuratedJsonDataService`) reads `locations.json`, `personas.json`, `forenames-by-country.json`, and `surnames-by-country.json` (paths supplied via a `CuratedDataFilePaths` DTO at construction) into typed records, caching each after its first load. `--mocked` runs use `MockCuratedDataService`'s fixed in-memory dataset instead. |
| Sample | `BiergartenPipelineOrchestrator::QueryCitiesWithCountries()` samples `--location-count` locations per run (default `10`). | | Sample | `BiergartenPipelineOrchestrator::QueryCitiesWithCountries()` samples `--location-count` locations per run (default `10`). |
| Enrich | `WikipediaEnrichmentService` fetches brewing and beer-related context. Keeps going when a lookup fails. `--mocked` runs use `MockEnrichmentService` instead and skip Wikipedia entirely. | | Enrich | `WikipediaEnrichmentService` fetches brewing and beer-related context. Keeps going when a lookup fails. `--mocked` runs use `MockEnrichmentService` instead and skip Wikipedia entirely. |
| Generate Users | `GenerateUsers()` samples a persona and a forename/surname pair per enriched city (skipping countries with no name data), then `MockGenerator` or `LlamaGenerator` produces a username, bio, and activity weight around the sampled name. | | Generate Users | `GenerateUsers()` samples a persona and a forename/surname pair per enriched city (skipping countries with no name data), then `MockGenerator` or `LlamaGenerator` produces a username, bio, and activity weight around the sampled name. |
| Generate Breweries | `MockGenerator` or `LlamaGenerator` produces brewery names and descriptions in English and the local language. | | Generate Breweries | `MockGenerator` or `LlamaGenerator` produces brewery names and descriptions in English and the local language. |
| Store | `SqliteExportService` writes each successful user and brewery into a fresh dated `.sqlite` database with normalized `locations`, `users`, and `breweries` tables. | | Store | `SqliteExportService` writes each successful user and brewery into a fresh dated `.sqlite` database with normalized `locations`, `users`, and `breweries` tables. |
| Log | `spdlog` writes results and warnings to the console. | | Log | `spdlog` writes results and warnings to the console. |
If name sampling, enrichment, or generation fails for a city, that city is If name sampling, enrichment, or generation fails for a city, that city is
skipped and the pipeline continues. `GenerateUsers()` runs before skipped and the pipeline continues. `GenerateUsers()` runs before
@@ -240,33 +240,32 @@ skipped and the pipeline continues. `GenerateUsers()` runs before
- `CuratedJsonDataService` — implements `ICuratedDataService`; takes a - `CuratedJsonDataService` — implements `ICuratedDataService`; takes a
`CuratedDataFilePaths` DTO (locations/personas/forenames/surnames paths) in `CuratedDataFilePaths` DTO (locations/personas/forenames/surnames paths) in
its constructor, then parses and validates curated location, persona, and its constructor, then parses and validates curated location, persona, and
forename/surname JSON, memoizing each result after its first load on a forename/surname JSON, memoizing each result after its first load on a given
given instance. `MockCuratedDataService` is the in-memory substitute (4 instance. `MockCuratedDataService` is the in-memory substitute (4 fixed
fixed locations, 3 personas, and name data for `US`/`DE`/`FR`/`BE`) used in locations, 3 personas, and name data for `US`/`DE`/`FR`/`BE`) used in
`--mocked` runs. `--mocked` runs.
- `WikipediaEnrichmentService` — queries Wikipedia extracts, caches results, - `WikipediaEnrichmentService` — queries Wikipedia extracts, caches results,
returns empty context on failure. `MockEnrichmentService` is the no-op returns empty context on failure. `MockEnrichmentService` is the no-op
substitute used in `--mocked` runs. substitute used in `--mocked` runs.
- `LlamaGenerator` — formats prompts for Gemma 4, validates JSON output for - `LlamaGenerator` — formats prompts for Gemma 4, validates JSON output for both
both `GenerateBrewery` and `GenerateUser`, retries malformed responses up `GenerateBrewery` and `GenerateUser`, retries malformed responses up to three
to three times with corrective feedback in the retry prompt. The token times with corrective feedback in the retry prompt. The token budget is fixed
budget is fixed across attempts; it is not raised automatically on across attempts; it is not raised automatically on truncation.
truncation. - `MockGenerator` — stable hash-based output so the same city/persona/name input
- `MockGenerator` — stable hash-based output so the same city/persona/name always produces the same brewery or user.
input always produces the same brewery or user. - `SqliteExportService` — creates a dated SQLite file per run and persists each
- `SqliteExportService` — creates a dated SQLite file per run and persists successful user and brewery into normalized tables.
each successful user and brewery into normalized tables.
- Brewery payloads include English and local-language name and description - Brewery payloads include English and local-language name and description
fields. User payloads carry a sampled first/last name and gender, an fields. User payloads carry a sampled first/last name and gender, an
LLM-generated username/bio/activity weight, and a programmatically LLM-generated username/bio/activity weight, and a programmatically generated
generated (not LLM-authored) unique email and date of birth. (not LLM-authored) unique email and date of birth.
### Runtime Behaviour ### Runtime Behaviour
`WikipediaEnrichmentService` fetches two Wikipedia extracts per city: a `WikipediaEnrichmentService` fetches two Wikipedia extracts per city: a generic
generic "brewing" extract and a "beer in `{country}`" extract. It does not "brewing" extract and a "beer in `{country}`" extract. It does not currently
currently query a city- or region-specific page. Each query string is cached query a city- or region-specific page. Each query string is cached after its
after its first successful (or empty) lookup. first successful (or empty) lookup.
`GetLocationContext()` returns an empty string when the web client is `GetLocationContext()` returns an empty string when the web client is
unavailable or when lookup/parsing fails. unavailable or when lookup/parsing fails.
@@ -281,15 +280,15 @@ runs.
`CuratedJsonDataService` memoizes each of `LoadLocations()`, `LoadPersonas()`, `CuratedJsonDataService` memoizes each of `LoadLocations()`, `LoadPersonas()`,
`LoadForenamesByCountry()`, and `LoadSurnamesByCountry()` independently the `LoadForenamesByCountry()`, and `LoadSurnamesByCountry()` independently the
first time each is called, since `BiergartenPipelineOrchestrator` owns a first time each is called, since `BiergartenPipelineOrchestrator` owns a single
single `ICuratedDataService` instance for the whole run — later calls return `ICuratedDataService` instance for the whole run — later calls return the cached
the cached result instead of re-parsing. result instead of re-parsing.
`GenerateUsers()` samples a forename/surname pair per city via `SampleName()`, `GenerateUsers()` samples a forename/surname pair per city via `SampleName()`,
keyed by the city's ISO 3166-1 code. Countries present in `locations.json` keyed by the city's ISO 3166-1 code. Countries present in `locations.json` but
but absent from either name fixture (currently `KE`, `SE`, `SG`, `TH`, `VN`, absent from either name fixture (currently `KE`, `SE`, `SG`, `TH`, `VN`, `ZA`)
`ZA`) are skipped, the same way a failed enrichment or generation call skips are skipped, the same way a failed enrichment or generation call skips a city —
a city — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section. see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section.
### Process Flow - Activity Diagram ### Process Flow - Activity Diagram
@@ -303,10 +302,10 @@ a city — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section.
## Generated Output ## Generated Output
Each successful run stores a `BreweryRecord` pair with the source location Each successful run stores a `BreweryRecord` pair with the source location and a
and a `BreweryResult` payload, and a `UserRecord` pair with the source `BreweryResult` payload, and a `UserRecord` pair with the source location and a
location and a `UserResult` payload. The same generated records are also `UserResult` payload. The same generated records are also written to a fresh
written to a fresh SQLite export file named with the current UTC timestamp. SQLite export file named with the current UTC timestamp.
| Field | Meaning | | Field | Meaning |
| ------------------- | ------------------------------------------ | | ------------------- | ------------------------------------------ |
@@ -315,16 +314,16 @@ written to a fresh SQLite export file named with the current UTC timestamp.
| `name_local` | Brewery name in the local language. | | `name_local` | Brewery name in the local language. |
| `description_local` | Brewery description in the local language. | | `description_local` | Brewery description in the local language. |
| Field | Meaning | | Field | Meaning |
| ----------------- | ----------------------------------------------------------------- | | ----------------- | -------------------------------------------------------------------------------------- |
| `first_name` | Sampled forename, copied from the curated name data (not LLM-invented). | | `first_name` | Sampled forename, copied from the curated name data (not LLM-invented). |
| `last_name` | Sampled surname, copied from the curated name data (not LLM-invented). | | `last_name` | Sampled surname, copied from the curated name data (not LLM-invented). |
| `gender` | Gender associated with the sampled forename in the source dataset. | | `gender` | Gender associated with the sampled forename in the source dataset. |
| `username` | LLM-generated handle. | | `username` | LLM-generated handle. |
| `bio` | LLM-generated short biography. | | `bio` | LLM-generated short biography. |
| `activity_weight` | Relative check-in/activity weight, reserved for a future J-curve activity profile. | | `activity_weight` | Relative check-in/activity weight, reserved for a future J-curve activity profile. |
| `email` | Unique `@thebiergarten.app` address, generated programmatically from the sampled name. | | `email` | Unique `@thebiergarten.app` address, generated programmatically from the sampled name. |
| `date_of_birth` | Randomized date of birth (age 19-48), generated programmatically. | | `date_of_birth` | Randomized date of birth (age 19-48), generated programmatically. |
The log dump also includes city, country, state or province, ISO subdivision The log dump also includes city, country, state or province, ISO subdivision
code, latitude, and longitude for each entry. code, latitude, and longitude for each entry.
@@ -358,11 +357,9 @@ The build fetches Boost.DI, spdlog, and SQLite via CMake. llama.cpp is fetched
only when a system installation is not detected. Metal is enabled on Apple only when a system installation is not detected. Metal is enabled on Apple
Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present. Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present.
> **Code Style:** Modern C++20 throughout — RAII for ownership, > **Code Style:** Code style generally follows the Google Style Guide and
> `std::unique_ptr` for injected dependencies, `std::optional` for parse > features modern C++20 throughout. Formatting follows the Google C++ Style
> outcomes, `std::span` for read-only views over generated city data, structured > Guide, with the exception of indentation width (3 spaces vs. Google's 2).
> bindings in pipeline loops. Formatting follows the Google C++ Style Guide via
> `.clang-format` with a narrow column limit and two-space indentation.
--- ---
@@ -404,34 +401,33 @@ Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present.
## Fixture Strategy ## Fixture Strategy
- `--mocked` for stable fixtures, repeatable screenshots, and Storybook runs. - `--mocked` for stable fixtures, repeatable screenshots, and Storybook runs.
`MockCuratedDataService` swaps in for `CuratedJsonDataService`, so no `MockCuratedDataService` swaps in for `CuratedJsonDataService`, so no fixture
fixture files need to be present on disk. files need to be present on disk.
- `--model` when geographically grounded content matters for demos. - `--model` when geographically grounded content matters for demos.
- Keep `locations.json` structured enough to support discovery and future - Keep `locations.json` structured enough to support discovery and future
filtering. filtering.
- `personas.json`, `forenames-by-country.json`, and - `personas.json`, `forenames-by-country.json`, and `surnames-by-country.json`
`surnames-by-country.json` are curated/vendored fixture data, not are curated/vendored fixture data, not LLM-generated — see
LLM-generated — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for provenance.
section for provenance. - Treat SQLite output as seed material for the app's brewery and user domains,
- Treat SQLite output as seed material for the app's brewery and user not production data.
domains, not production data.
--- ---
## Repo Layout ## Repo Layout
| Path | Purpose | | Path | Purpose |
| -------------------------------------- | -------------------------------------------------- | | -------------------------------------------- | ----------------------------------------------------------- |
| `tooling/pipeline/includes/` | Public headers and shared models. | | `tooling/pipeline/includes/` | Public headers and shared models. |
| `tooling/pipeline/src/` | Implementation files. | | `tooling/pipeline/src/` | Implementation files. |
| `tooling/pipeline/locations.json` | Curated city input copied into the build tree. | | `tooling/pipeline/locations.json` | Curated city input copied into the build tree. |
| `tooling/pipeline/personas.json` | Curated user persona archetypes copied into the build tree. | | `tooling/pipeline/personas.json` | Curated user persona archetypes copied into the build tree. |
| `tooling/pipeline/forenames-by-country.json` | Vendored (CC0) forename data by ISO 3166-1 country code. | | `tooling/pipeline/forenames-by-country.json` | Vendored (CC0) forename data by ISO 3166-1 country code. |
| `tooling/pipeline/surnames-by-country.json` | Vendored (CC0) surname data by ISO 3166-1 country code. | | `tooling/pipeline/surnames-by-country.json` | Vendored (CC0) surname data by ISO 3166-1 country code. |
| `tooling/pipeline/prompts/` | System prompts used by the model-backed path. | | `tooling/pipeline/prompts/` | System prompts used by the model-backed path. |
| `tooling/pipeline/runpod/` | Dockerfile, launcher, and RunPod pod template. | | `tooling/pipeline/runpod/` | Dockerfile, launcher, and RunPod pod template. |
| `docs/pipeline/diagrams/` | Architecture and pipeline diagrams. | | `docs/pipeline/diagrams/` | Architecture and pipeline diagrams. |
| `docs/pipeline/ETHICS-AND-KNOWN-ISSUES.md` | Ethics, bias, hallucination analysis, mitigations. | | `docs/pipeline/ETHICS-AND-KNOWN-ISSUES.md` | Ethics, bias, hallucination analysis, mitigations. |
--- ---
@@ -456,12 +452,11 @@ Paths below are relative to `tooling/pipeline/`.
## Next Steps ## Next Steps
The pipeline currently produces city-aware brewery and user records and The pipeline currently produces city-aware brewery and user records and dated
dated SQLite exports. The next passes add additional fixture types so the SQLite exports. The next passes add additional fixture types so the app can
app can exercise the full brewery and social domains without live data. For exercise the full brewery and social domains without live data. For the detailed
the detailed engineering breakdown of what's needed to reach the engineering breakdown of what's needed to reach the architecture in
architecture in [`diagrams/planned/`](./diagrams/planned/), see [`diagrams/planned/`](./diagrams/planned/), see [ROADMAP.md](./ROADMAP.md).
[ROADMAP.md](./ROADMAP.md).
### Testing — Very High Priority ### Testing — Very High Priority

256
docs/pipeline/ROADMAP.md
View File

@@ -3,13 +3,12 @@
This is the engineering breakdown for closing the gap between the **current** This is the engineering breakdown for closing the gap between the **current**
pipeline and the **planned** architecture documented in pipeline and the **planned** architecture documented in
[`diagrams/planned/class.puml`](./diagrams/planned/class.puml) and [`diagrams/planned/class.puml`](./diagrams/planned/class.puml) and
[`diagrams/planned/activity.puml`](./diagrams/planned/activity.puml). Nothing [`diagrams/planned/activity.puml`](./diagrams/planned/activity.puml). Nothing in
in `diagrams/planned/` is implemented yet — this file tracks what it would `diagrams/planned/` is implemented yet — this file tracks what it would take to
take to get there. For the current implementation, see get there. For the current implementation, see [README.md](./README.md).
[README.md](./README.md).
Items are grouped by layer and roughly ordered by dependency: later groups Items are grouped by layer and roughly ordered by dependency: later groups build
build on types and services introduced earlier. on types and services introduced earlier.
The only concurrency in the current pipeline is the log dispatcher thread The only concurrency in the current pipeline is the log dispatcher thread
(`main.cc` spawns it to drain a `BoundedChannel<LogEntry>` into spdlog for the (`main.cc` spawns it to drain a `BoundedChannel<LogEntry>` into spdlog for the
@@ -25,31 +24,30 @@ architecture needs.
`tooling/pipeline/includes/data_model/generated_models.h` and `models.h`. `tooling/pipeline/includes/data_model/generated_models.h` and `models.h`.
- [ ] Add `Completeness` enum (`Full`, `Partial`, `Absent`) and a - [ ] Add `Completeness` enum (`Full`, `Partial`, `Absent`) and a
`LocationContext` struct (`text`, `completeness`, `char_count`). `LocationContext` struct (`text`, `completeness`, `char_count`). Replace
Replace `EnrichedCity::region_context` (currently a plain `EnrichedCity::region_context` (currently a plain `std::string`) with
`std::string`) with `context : LocationContext`. `context : LocationContext`.
- [ ] Add `BeerStyle` (`name`, `description`, `min_abv`, `max_abv`, - [ ] Add `BeerStyle` (`name`, `description`, `min_abv`, `max_abv`, `min_ibu`,
`min_ibu`, `max_ibu`). `max_ibu`).
- [ ] Add `BeerResult`, `CheckinResult`, `RatingResult` result payloads. - [ ] Add `BeerResult`, `CheckinResult`, `RatingResult` result payloads.
- [ ] Add `GenerationMetadata` (`generation_id`, `generated_time`, - [ ] Add `GenerationMetadata` (`generation_id`, `generated_time`,
`context_provided`, `generated_with`). `context_provided`, `generated_with`).
- [x] Add `first_name`, `last_name`, `gender`, and `activity_weight` to - [x] Add `first_name`, `last_name`, `gender`, and `activity_weight` to
`UserResult` (currently just `username`, `bio`). `first_name`/ `UserResult` (currently just `username`, `bio`). `first_name`/
`last_name`/`gender` are copied from the sampled `Name` (see below), `last_name`/`gender` are copied from the sampled `Name` (see below), not
not LLM-invented. LLM-invented.
- [x] Add `Name` (`first_name`, `last_name`, `gender`) — the sampled result - [x] Add `Name` (`first_name`, `last_name`, `gender`) — the sampled result
handed to `DataGenerator::GenerateUser`. Add `ForenameEntry` (`name`, handed to `DataGenerator::GenerateUser`. Add `ForenameEntry` (`name`,
`gender`). Landed as a flatter shape than originally planned here: no `gender`). Landed as a flatter shape than originally planned here: no
`NamesByCountry` wrapper class. `curated_data_service.h` instead `NamesByCountry` wrapper class. `curated_data_service.h` instead declares
declares `ForenameList = std::vector<ForenameEntry>` and `ForenameList = std::vector<ForenameEntry>` and
`SurnameList = std::vector<std::string>`, and `SurnameList = std::vector<std::string>`, and `ICuratedDataService`
`ICuratedDataService` exposes two maps keyed by ISO 3166-1 code — exposes two maps keyed by ISO 3166-1 code —
`ForenamesByCountryMap = unordered_map<string, ForenameList>` and `ForenamesByCountryMap = unordered_map<string, ForenameList>` and
`SurnamesByCountryMap = unordered_map<string, SurnameList>` — `SurnamesByCountryMap = unordered_map<string, SurnameList>` — directly
directly (see §2). Sampling is (see §2). Sampling is a free function,
a free function, `SampleName(forenames_by_country, `SampleName(forenames_by_country, surnames_by_country, iso3166_1, rng)`,
surnames_by_country, iso3166_1, rng)`, in in `generate_users.cc`'s anonymous namespace, not a method on a class.
`generate_users.cc`'s anonymous namespace, not a method on a class.
Pairing a forename with a surname happens at sample time so gender Pairing a forename with a surname happens at sample time so gender
(present per-forename in the source data) is never discarded the way a (present per-forename in the source data) is never discarded the way a
pre-flattened `{first_name, last_name}` list would lose it; see §2. pre-flattened `{first_name, last_name}` list would lose it; see §2.
@@ -58,14 +56,14 @@ architecture needs.
- [ ] Add `GeneratedBeer`, `GeneratedCheckin`, `GeneratedRating`, - [ ] Add `GeneratedBeer`, `GeneratedCheckin`, `GeneratedRating`,
`GeneratedFollow` aggregate structs. `GeneratedFollow` aggregate structs.
- [x] Add `UserRecord` (`location`, `user : UserResult`, `email`, - [x] Add `UserRecord` (`location`, `user : UserResult`, `email`,
`date_of_birth`) as the current stand-in for the planned `date_of_birth`) as the current stand-in for the planned `GeneratedUser` —
`GeneratedUser` — `email` and `date_of_birth` are programmatically `email` and `date_of_birth` are programmatically generated by the
generated by the orchestrator, never LLM-authored, so a downstream orchestrator, never LLM-authored, so a downstream auth-account seeding
auth-account seeding consumer can register real accounts from the consumer can register real accounts from the pipeline's SQLite export. No
pipeline's SQLite export. No `password`, `user_id`, or `metadata` `password`, `user_id`, or `metadata` field yet, matching `BreweryRecord`'s
field yet, matching `BreweryRecord`'s equally pre-`metadata` shape. equally pre-`metadata` shape. Already wired into
Already wired into `IExportService`/`SqliteExportService`: a `users` `IExportService`/`SqliteExportService`: a `users` table exists and
table exists and `GenerateUsers()` exports each successful record via `GenerateUsers()` exports each successful record via
`ProcessRecord(const UserRecord&)` (see §5) in addition to logging and `ProcessRecord(const UserRecord&)` (see §5) in addition to logging and
holding `generated_users_` in memory. Still missing vs. the planned holding `generated_users_` in memory. Still missing vs. the planned
`GeneratedUser`: `user_id`, `password`, and `metadata`. `GeneratedUser`: `user_id`, `password`, and `metadata`.
@@ -75,66 +73,62 @@ architecture needs.
`tooling/pipeline/includes/services/curated_data/`, fixture files. `tooling/pipeline/includes/services/curated_data/`, fixture files.
- [x] Extract an interface; have `CuratedJsonDataService` implement it. - [x] Extract an interface; have `CuratedJsonDataService` implement it. Landed
Landed as `ICuratedDataService` as `ICuratedDataService` (`services/curated_data/curated_data_service.h`),
(`services/curated_data/curated_data_service.h`), not `DataPreloader` not `DataPreloader` — `LoadLocations()`, `LoadPersonas()`,
— `LoadLocations()`, `LoadPersonas()`, `LoadForenamesByCountry()`, and `LoadForenamesByCountry()`, and `LoadSurnamesByCountry()` are all virtual
`LoadSurnamesByCountry()` are all virtual methods returning `const&` methods returning `const&` and take no arguments. `CuratedJsonDataService`
and take no arguments. `CuratedJsonDataService` (`services/curated_data/curated_json_data_service.h`, originally landed as
(`services/curated_data/curated_json_data_service.h`, originally `JsonLoader` under `json_handling/` before being renamed and moved) takes
landed as `JsonLoader` under `json_handling/` before being renamed and a `CuratedDataFilePaths` DTO (`locations_path`, `personas_path`,
moved) takes a `CuratedDataFilePaths` DTO (`locations_path`, `forenames_path`, `surnames_path`) in its constructor rather than a path
`personas_path`, `forenames_path`, `surnames_path`) in its per `Load*()` call, and memoizes each result in a private `cache` struct
constructor rather than a path per `Load*()` call, and memoizes each (`locations`, `personas`, `forenames_by_country`, `surnames_by_country`),
result in a private `cache` struct (`locations`, `personas`, so a second call on the same instance returns the cached result instead of
`forenames_by_country`, `surnames_by_country`), so a second call on re-parsing — safe because `CuratedJsonDataService` outlives every call
the same instance returns the cached result instead of re-parsing — site (owned by `BiergartenPipelineOrchestrator` via
safe because `CuratedJsonDataService` outlives every call site (owned
by `BiergartenPipelineOrchestrator` via
`unique_ptr<ICuratedDataService>` for the whole run). `main.cc`'s DI `unique_ptr<ICuratedDataService>` for the whole run). `main.cc`'s DI
injector also gained a `MockCuratedDataService` (fixed in-memory injector also gained a `MockCuratedDataService` (fixed in-memory data: 4
data: 4 locations across `US`/`DE`/`FR`/`BE`, 3 personas, and locations across `US`/`DE`/`FR`/`BE`, 3 personas, and matching
matching forename/surname sets for those four countries), bound in forename/surname sets for those four countries), bound in place of
place of `CuratedJsonDataService` under `--mocked`, mirroring the `CuratedJsonDataService` under `--mocked`, mirroring the existing
existing `MockEnrichmentService`/`MockGenerator` pattern. `MockEnrichmentService`/`MockGenerator` pattern.
- [ ] Add `LoadBeerStyles()`. `beer-styles.json` already exists in the repo - [ ] Add `LoadBeerStyles()`. `beer-styles.json` already exists in the repo and
and is already copied into the Docker image is already copied into the Docker image (`runpod/Dockerfile`), but no
(`runpod/Dockerfile`), but no loader reads it yet, and the native loader reads it yet, and the native CMake build doesn't copy it into
CMake build doesn't copy it into `build/` at all — `CMakeLists.txt`'s `build/` at all — `CMakeLists.txt`'s "Runtime Assets" step only copies
"Runtime Assets" step only copies `locations.json` and `prompts/`. `locations.json` and `prompts/`.
- [x] Add `LoadPersonas()` and author `personas.json` (doesn't exist yet). - [x] Add `LoadPersonas()` and author `personas.json` (doesn't exist yet).
- [x] Add name-by-country loading, parsing - [x] Add name-by-country loading, parsing
`tooling/pipeline/forenames-by-country.json` and `tooling/pipeline/forenames-by-country.json` and
`surnames-by-country.json`. Landed as two separate methods — `surnames-by-country.json`. Landed as two separate methods —
`LoadForenamesByCountry()` and `LoadSurnamesByCountry()` — each `LoadForenamesByCountry()` and `LoadSurnamesByCountry()` — each returning
returning a flat `ForenamesByCountryMap` / `SurnamesByCountryMap` a flat `ForenamesByCountryMap` / `SurnamesByCountryMap` (aliases for
(aliases for `unordered_map<string, ForenameList>` / `unordered_map<string, ForenameList>` /
`unordered_map<string, SurnameList>`) keyed by ISO 3166-1 code, rather `unordered_map<string, SurnameList>`) keyed by ISO 3166-1 code, rather
than one combined `LoadNamesByCountry() : NamesByCountry` call. Both than one combined `LoadNamesByCountry() : NamesByCountry` call. Both files
files are vendored verbatim (unmodified, are vendored verbatim (unmodified, full multinational coverage) from
full multinational coverage) from
`sigpwned/popular-names-by-country-dataset` (CC0) — see `sigpwned/popular-names-by-country-dataset` (CC0) — see
ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for
provenance. Deliberately not pre-paired or filtered down to just the provenance. Deliberately not pre-paired or filtered down to just the
countries in `locations.json`: keeping the source shape (including countries in `locations.json`: keeping the source shape (including
per-forename gender) intact means the loader can support more per-forename gender) intact means the loader can support more countries
countries later, or gender-aware persona/bio generation, without later, or gender-aware persona/bio generation, without re-sourcing data.
re-sourcing data.
## 3. Policy / Strategy Layer ## 3. Policy / Strategy Layer
Entirely new — no `includes/policy/` (or equivalent) directory exists today. Entirely new — no `includes/policy/` (or equivalent) directory exists today.
- [ ] `ContextStrategy` interface + `BreweryContextStrategy` / - [ ] `ContextStrategy` interface + `BreweryContextStrategy` /
`BeerContextStrategy`. Today `WikipediaEnrichmentService::GetLocationContext` `BeerContextStrategy`. Today
hardcodes a generic `"brewing"` query and a `"beer in {country}"` `WikipediaEnrichmentService::GetLocationContext` hardcodes a generic
query directly — no per-phase strategy selection. `"brewing"` query and a `"beer in {country}"` query directly — no
- [ ] `SamplingStrategy` interface + `UniformSamplingStrategy`, replacing per-phase strategy selection.
the inline `std::ranges::sample(...)` call in - [ ] `SamplingStrategy` interface + `UniformSamplingStrategy`, replacing the
inline `std::ranges::sample(...)` call in
`BiergartenPipelineOrchestrator::QueryCitiesWithCountries()`. `BiergartenPipelineOrchestrator::QueryCitiesWithCountries()`.
- [ ] `BeerSelectionStrategy` interface + `RandomBeerSelectionStrategy`, to - [ ] `BeerSelectionStrategy` interface + `RandomBeerSelectionStrategy`, to pick
pick styles per brewery from the `BeerStyle` palette (depends on §1 styles per brewery from the `BeerStyle` palette (depends on §1 and §2).
and §2).
- [ ] `CheckinDistributionStrategy` interface + `JCurveCheckinStrategy` / - [ ] `CheckinDistributionStrategy` interface + `JCurveCheckinStrategy` /
`RandomCheckinStrategy` — this is the "Check-In System" item already `RandomCheckinStrategy` — this is the "Check-In System" item already
called out in README.md's Next Steps, made concrete. called out in README.md's Next Steps, made concrete.
@@ -150,88 +144,87 @@ Entirely new — no `includes/policy/` (or equivalent) directory exists today.
`GenerateUser`). `GenerateUser`).
- [x] Implement `LlamaGenerator::GenerateUser` for real, with a retry loop - [x] Implement `LlamaGenerator::GenerateUser` for real, with a retry loop
mirroring `GenerateBrewery` (GBNF grammar, `ValidateUserJson`, up to 3 mirroring `GenerateBrewery` (GBNF grammar, `ValidateUserJson`, up to 3
attempts with corrective feedback) and a new attempts with corrective feedback) and a new `prompts/USER_GENERATION.md`.
`prompts/USER_GENERATION.md`. Changed the `DataGenerator::GenerateUser` Changed the `DataGenerator::GenerateUser` signature to
signature to
`(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult`, `(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult`,
matching what the activity diagram actually passes. `MockGenerator::GenerateUser` matching what the activity diagram actually passes.
updated to match the new signature too. `MockGenerator::GenerateUser` updated to match the new signature too.
- [ ] Implement `MockGenerator::GenerateBeer` / `GenerateCheckin` / - [ ] Implement `MockGenerator::GenerateBeer` / `GenerateCheckin` /
`GenerateRating`. `GenerateRating`.
- [ ] Add `IPromptFormatter::ExpectedArchitecture()` and - [ ] Add `IPromptFormatter::ExpectedArchitecture()` and
`LlamaGenerator::ValidateModelArchitecture()` so loading a GGUF that `LlamaGenerator::ValidateModelArchitecture()` so loading a GGUF that
doesn't match the configured chat template fails fast instead of doesn't match the configured chat template fails fast instead of silently
silently producing degraded output. producing degraded output.
- [ ] Add prompt template files for beer/checkin/rating generation next to - [ ] Add prompt template files for beer/checkin/rating generation next to the
the existing `prompts/BREWERY_GENERATION.md`. existing `prompts/BREWERY_GENERATION.md`.
## 5. Export Service ## 5. Export Service
`tooling/pipeline/includes/services/database/`, `src/services/sqlite/`. `tooling/pipeline/includes/services/database/`, `src/services/sqlite/`.
- [x] `IExportService::ProcessRecord(const UserRecord&)` and - [x] `IExportService::ProcessRecord(const UserRecord&)` and
`SqliteExportService`'s `users` table (see `SqliteExportService`'s `users` table (see `kCreateUsersTableSql` /
`kCreateUsersTableSql` / `kInsertUserSql` in `kInsertUserSql` in
`includes/services/database/sqlite_statement_helpers.h`) are `includes/services/database/sqlite_statement_helpers.h`) are implemented —
implemented — `GenerateUsers()` exports every successful user `GenerateUsers()` exports every successful user alongside its resolved
alongside its resolved `location_id`. `location_id`.
- [ ] Extend `IExportService` with `ProcessBeer`, `ProcessCheckin`, - [ ] Extend `IExportService` with `ProcessBeer`, `ProcessCheckin`,
`ProcessRating`, `ProcessFollow` (today `ProcessRecord(const `ProcessRating`, `ProcessFollow` (today
BreweryRecord&)` and `ProcessRecord(const UserRecord&)` exist). `ProcessRecord(const BreweryRecord&)` and
`ProcessRecord(const UserRecord&)` exist).
- [ ] Add `beers`, `checkins`, `ratings`, `follows` tables to - [ ] Add `beers`, `checkins`, `ratings`, `follows` tables to
`SqliteExportService::InitializeSchema()` (`locations`, `breweries`, `SqliteExportService::InitializeSchema()` (`locations`, `breweries`, and
and `users` already exist) — see `kCreateLocationsTableSql` / `users` already exist) — see `kCreateLocationsTableSql` /
`kCreateBreweriesTableSql` / `kCreateUsersTableSql` in `kCreateBreweriesTableSql` / `kCreateUsersTableSql` in
`includes/services/database/sqlite_statement_helpers.h` for the `includes/services/database/sqlite_statement_helpers.h` for the existing
existing pattern to follow. pattern to follow.
- [ ] Add a `brewery_cache_` alongside the existing `location_cache_`, and - [ ] Add a `brewery_cache_` alongside the existing `location_cache_`, and move
move from one open transaction for the whole run to per-phase batched from one open transaction for the whole run to per-phase batched commits
commits (`BEGIN` / `COMMIT & BEGIN` on a batch-size threshold), as (`BEGIN` / `COMMIT & BEGIN` on a batch-size threshold), as shown in the
shown in the planned activity diagram. planned activity diagram.
## 6. Concurrency & Orchestration ## 6. Concurrency & Orchestration
`tooling/pipeline/includes/concurrency/`, `tooling/pipeline/includes/concurrency/`,
`src/biergarten_pipeline_orchestrator/`. `src/biergarten_pipeline_orchestrator/`.
- [ ] `BoundedChannel<T>` already exists and is production-tested — but - [ ] `BoundedChannel<T>` already exists and is production-tested — but it's
it's only wired to the single log channel today. Stand up the only wired to the single log channel today. Stand up the per-phase
per-phase producer/consumer channels (`loc_ch`, `exp_ch`, etc.) the producer/consumer channels (`loc_ch`, `exp_ch`, etc.) the planned activity
planned activity diagram describes, each with a dedicated LLM worker diagram describes, each with a dedicated LLM worker thread and SQLite
thread and SQLite worker thread. worker thread.
- [ ] Rewrite `BiergartenPipelineOrchestrator::Run()` from one synchronous - [ ] Rewrite `BiergartenPipelineOrchestrator::Run()` from one synchronous pass
pass over `generated_breweries_` into phased methods — over `generated_breweries_` into phased methods — `RunUserPhase` →
`RunUserPhase` → `RunBreweryAndBeerPhase` (with a `RunBeerPhase` `RunBreweryAndBeerPhase` (with a `RunBeerPhase` sub-step once
sub-step once `brewery_pool_` is populated) → `RunCheckinPhase` / `brewery_pool_` is populated) → `RunCheckinPhase` / `RunFollowPhase`
`RunFollowPhase` (these two can run in parallel) → `RunRatingPhase` — (these two can run in parallel) → `RunRatingPhase` — backed by
backed by `user_pool_`, `brewery_pool_`, `beer_pool_`, `user_pool_`, `brewery_pool_`, `beer_pool_`, `checkin_pool_`,
`checkin_pool_`, `follow_pool_`. `follow_pool_`.
- [ ] Honor the structural-concurrency requirement already called out in a - [ ] Honor the structural-concurrency requirement already called out in a
comment on `BiergartenPipelineOrchestrator::Run()` in comment on `BiergartenPipelineOrchestrator::Run()` in
`biergarten_pipeline_orchestrator.h`: once real worker threads exist, `biergarten_pipeline_orchestrator.h`: once real worker threads exist, they
they must be structurally joined (e.g. via `std::jthread`) before must be structurally joined (e.g. via `std::jthread`) before `Run()`
`Run()` returns, so no worker logs to a closed channel during returns, so no worker logs to a closed channel during teardown.
teardown.
## 7. Enrichment ## 7. Enrichment
- [ ] Decide whether to restore the city/region-specific Wikipedia query - [ ] Decide whether to restore the city/region-specific Wikipedia query that's
that's currently commented out in currently commented out in
`src/services/enrichment/wikipedia/get_summary.cc` `src/services/enrichment/wikipedia/get_summary.cc` (`GetLocationContext`).
(`GetLocationContext`). The `ContextStrategy` work in §3 is a natural The `ContextStrategy` work in §3 is a natural place to reintroduce it via
place to reintroduce it via `BreweryContextStrategy::QueriesFor()`. `BreweryContextStrategy::QueriesFor()`.
- [ ] Pre-warm caches at startup (`PreWarmBeerStyleCache`, - [ ] Pre-warm caches at startup (`PreWarmBeerStyleCache`,
`PreWarmLocationCache` in the planned activity diagram) instead of `PreWarmLocationCache` in the planned activity diagram) instead of
fetching lazily per record, so the streaming phases never block on a fetching lazily per record, so the streaming phases never block on a cold
cold cache mid-run. cache mid-run.
## 8. Fixtures / Build ## 8. Fixtures / Build
- [x] Author `personas.json`, `forenames-by-country.json`, and - [x] Author `personas.json`, `forenames-by-country.json`, and
`surnames-by-country.json` (see §2). `surnames-by-country.json` (see §2).
- [ ] Fix the `beer-styles.json` build-tree gap: add it to the "Runtime - [ ] Fix the `beer-styles.json` build-tree gap: add it to the "Runtime Assets"
Assets" `configure_file` step in `CMakeLists.txt` so native builds and `configure_file` step in `CMakeLists.txt` so native builds and the Docker
the Docker image agree on what's available at runtime. image agree on what's available at runtime.
--- ---
@@ -240,13 +233,12 @@ Entirely new — no `includes/policy/` (or equivalent) directory exists today.
The planned activity diagram's own phase notes ("brewery*pool* is now fully The planned activity diagram's own phase notes ("brewery*pool* is now fully
populated. Phase 1b may begin.", etc.) imply a dependency order. Roughly: populated. Phase 1b may begin.", etc.) imply a dependency order. Roughly:
1. Domain models (§1) and data preloading (§2) — nothing else compiles 1. Domain models (§1) and data preloading (§2) — nothing else compiles without
without these. these.
2. Export schema (§5) — so every later generation phase has somewhere to 2. Export schema (§5) — so every later generation phase has somewhere to land
land its output. its output.
3. Policy/strategy layer (§3) and generator interface (§4). 3. Policy/strategy layer (§3) and generator interface (§4).
4. Concurrency/orchestration rewrite (§6), which is the only piece that 4. Concurrency/orchestration rewrite (§6), which is the only piece that actually
actually wires §1–§5 into the phased, parallel pipeline shown in the wires §1–§5 into the phased, parallel pipeline shown in the planned diagrams.
planned diagrams. 5. Enrichment cache pre-warming and the city/region query decision (§7) — useful
5. Enrichment cache pre-warming and the city/region query decision (§7) — at any point, but most valuable once phases run concurrently.
useful at any point, but most valuable once phases run concurrently.

2
tooling/pipeline/.clang-format
View File

@@ -1,5 +1,5 @@
--- ---
BasedOnStyle: Google BasedOnStyle: Google
ColumnLimit: 80 ColumnLimit: 80
IndentWidth: 2 IndentWidth: 3
... ...

10
tooling/pipeline/.clang-tidy
View File

@@ -1,12 +1,6 @@
Checks: > Checks: >
-*, -*, bugprone-*, google-*, modernize-*, readability-*, cppcoreguidelines-*,
bugprone-*, -modernize-use-trailing-return-type, -google-runtime-references
google-*,
modernize-*,
readability-*,
cppcoreguidelines-*,
-modernize-use-trailing-return-type,
-google-runtime-references
CheckOptions: CheckOptions:
# Enforce Google Naming Conventions with valid clang-tidy strings # Enforce Google Naming Conventions with valid clang-tidy strings

18976
tooling/pipeline/forenames-by-country.json
View File

File diff suppressed because it is too large Load Diff

164
tooling/pipeline/includes/biergarten_pipeline_orchestrator.h
View File

@@ -24,98 +24,98 @@
* Handles location loading, city enrichment, and brewery/user generation. * Handles location loading, city enrichment, and brewery/user generation.
*/ */
class BiergartenPipelineOrchestrator { class BiergartenPipelineOrchestrator {
public: public:
/** /**
* @brief Constructs the orchestrator with injected pipeline dependencies. * @brief Constructs the orchestrator with injected pipeline dependencies.
* *
* @param logger Sink for pipeline diagnostics. * @param logger Sink for pipeline diagnostics.
* @param context_service Provides regional context for locations. * @param context_service Provides regional context for locations.
* @param generator Implementation (Llama or Mock) for brewery/user * @param generator Implementation (Llama or Mock) for brewery/user
* generation. * generation.
* @param exporter Database backend for persisting generated records. * @param exporter Database backend for persisting generated records.
* @param curated_data_service Loads curated location, persona, and name * @param curated_data_service Loads curated location, persona, and name
* data used to seed generation. * data used to seed generation.
* @param postal_code_service Resolves a postal code for a city from its * @param postal_code_service Resolves a postal code for a city from its
* curated postal-code data. * curated postal-code data.
* @param application_options CLI configuration and paths. * @param application_options CLI configuration and paths.
*/ */
BiergartenPipelineOrchestrator( BiergartenPipelineOrchestrator(
std::shared_ptr<ILogger> logger, std::shared_ptr<ILogger> logger,
std::unique_ptr<IEnrichmentService> context_service, std::unique_ptr<IEnrichmentService> context_service,
std::unique_ptr<DataGenerator> generator, std::unique_ptr<DataGenerator> generator,
std::unique_ptr<IExportService> exporter, std::unique_ptr<IExportService> exporter,
std::unique_ptr<ICuratedDataService> curated_data_service, std::unique_ptr<ICuratedDataService> curated_data_service,
std::unique_ptr<IPostalCodeService> postal_code_service, std::unique_ptr<IPostalCodeService> postal_code_service,
const ApplicationOptions& application_options); const ApplicationOptions& application_options);
/** /**
* @brief Run the data generation pipeline. * @brief Run the data generation pipeline.
* *
* Performs the following steps: * Performs the following steps:
* 1. Load curated locations from JSON * 1. Load curated locations from JSON
* 2. Resolve context for each city using the injected context service * 2. Resolve context for each city using the injected context service
* 3. Generate brewery and user data for sampled cities * 3. Generate brewery and user data for sampled cities
* *
* @note STRUCTURAL CONCURRENCY REQUIREMENT: * @note STRUCTURAL CONCURRENCY REQUIREMENT:
* When transitioned to a multithreaded design, this method MUST * When transitioned to a multithreaded design, this method MUST
* structurally enforce that all deployed worker threads are joined before * structurally enforce that all deployed worker threads are joined before
* returning (e.g. by using std::jthread or a structured concurrency * returning (e.g. by using std::jthread or a structured concurrency
* primitive). This ensures workers do not attempt to log to a closed * primitive). This ensures workers do not attempt to log to a closed
* channel during application teardown. * channel during application teardown.
* *
* @return true if successful, false if not * @return true if successful, false if not
*/ */
bool Run(); bool Run();
private: private:
std::shared_ptr<ILogger> logger_; std::shared_ptr<ILogger> logger_;
std::unique_ptr<IEnrichmentService> context_service_; std::unique_ptr<IEnrichmentService> context_service_;
/** /**
* @brief Generator implementation selected at the composition root (Llama * @brief Generator implementation selected at the composition root (Llama
* or Mock). * or Mock).
*/ */
std::unique_ptr<DataGenerator> generator_; std::unique_ptr<DataGenerator> generator_;
/** /**
* @brief Storage backend for generated brewery and user records. * @brief Storage backend for generated brewery and user records.
*/ */
std::unique_ptr<IExportService> exporter_; std::unique_ptr<IExportService> exporter_;
std::unique_ptr<ICuratedDataService> curated_data_service_; std::unique_ptr<ICuratedDataService> curated_data_service_;
std::unique_ptr<IPostalCodeService> postal_code_service_; std::unique_ptr<IPostalCodeService> postal_code_service_;
ApplicationOptions application_options_; ApplicationOptions application_options_;
/** /**
* @brief Load locations from JSON and sample cities. * @brief Load locations from JSON and sample cities.
* *
* @return Vector of locations randomly sampled per * @return Vector of locations randomly sampled per
* PipelineOptions::location_count. * PipelineOptions::location_count.
*/ */
std::vector<City> QueryLocations(); std::vector<City> QueryLocations();
/** /**
* @brief Generate breweries for enriched cities. * @brief Generate breweries for enriched cities.
* *
* @param cities Span of enriched city data. * @param cities Span of enriched city data.
*/ */
void GenerateBreweries(std::span<const EnrichedCity> cities); void GenerateBreweries(std::span<const EnrichedCity> cities);
/** /**
* @brief Generate users grounded in sampled names and personas for * @brief Generate users grounded in sampled names and personas for
* enriched cities. * enriched cities.
* *
* @param cities Span of enriched city data. * @param cities Span of enriched city data.
*/ */
void GenerateUsers(std::span<const EnrichedCity> cities); void GenerateUsers(std::span<const EnrichedCity> cities);
/** /**
* @brief Log the generated brewery and user results. * @brief Log the generated brewery and user results.
*/ */
void LogResults() const; void LogResults() const;
std::vector<BreweryRecord> generated_breweries_; std::vector<BreweryRecord> generated_breweries_;
std::vector<UserRecord> generated_users_; std::vector<UserRecord> generated_users_;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_ORCHESTRATOR_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_ORCHESTRATOR_H_

58
tooling/pipeline/includes/concurrency/bounded_channel.h
View File

@@ -26,45 +26,45 @@
*/ */
template <typename T> template <typename T>
class BoundedChannel { class BoundedChannel {
// Internal state — all access must be guarded by mutex_. // Internal state — all access must be guarded by mutex_.
std::queue<T> queue_; std::queue<T> queue_;
std::mutex mutex_; std::mutex mutex_;
std::condition_variable not_full_; std::condition_variable not_full_;
std::condition_variable not_empty_; std::condition_variable not_empty_;
std::size_t capacity_; std::size_t capacity_;
bool closed_ = false; bool closed_ = false;
public: public:
/** /**
* @brief Construct a bounded channel with the given capacity. * @brief Construct a bounded channel with the given capacity.
* @param capacity Maximum number of items the channel may hold. * @param capacity Maximum number of items the channel may hold.
*/ */
explicit BoundedChannel(std::size_t capacity) : capacity_(capacity) {} explicit BoundedChannel(std::size_t capacity) : capacity_(capacity) {}
/** /**
* @brief Send an item into the channel. Blocks when the channel is full. * @brief Send an item into the channel. Blocks when the channel is full.
* @param item Move-only item to enqueue. * @param item Move-only item to enqueue.
*/ */
void Send(T item); void Send(T item);
/** /**
* @brief Receive an item from the channel. Blocks when the channel is * @brief Receive an item from the channel. Blocks when the channel is
* empty. * empty.
* @return std::optional<T> containing the item, or std::nullopt when the * @return std::optional<T> containing the item, or std::nullopt when the
* channel is closed and drained. * channel is closed and drained.
*/ */
std::optional<T> Receive(); std::optional<T> Receive();
/** /**
* @brief Close the channel and unblock all waiting threads. Idempotent. * @brief Close the channel and unblock all waiting threads. Idempotent.
*/ */
void Close(); void Close();
}; };
// Include the template implementation // Include the template implementation

40
tooling/pipeline/includes/data_generation/data_generator.h
View File

@@ -14,28 +14,28 @@
* @brief Interface for data generator implementations. * @brief Interface for data generator implementations.
*/ */
class DataGenerator { class DataGenerator {
public: public:
virtual ~DataGenerator() = default; virtual ~DataGenerator() = default;
/** /**
* @brief Generates brewery data for a location. * @brief Generates brewery data for a location.
* *
* @param city Enriched city the brewery is associated with. * @param city Enriched city the brewery is associated with.
* @return Brewery generation result. * @return Brewery generation result.
*/ */
virtual BreweryResult GenerateBrewery(const EnrichedCity& city) = 0; virtual BreweryResult GenerateBrewery(const EnrichedCity& city) = 0;
/** /**
* @brief Generates a user profile grounded in a sampled name and persona. * @brief Generates a user profile grounded in a sampled name and persona.
* *
* @param city Enriched city the user is associated with. * @param city Enriched city the user is associated with.
* @param persona Persona archetype grounding the generated bio. * @param persona Persona archetype grounding the generated bio.
* @param name Sampled first/last name (and gender) * @param name Sampled first/last name (and gender)
* @return User generation result. * @return User generation result.
*/ */
virtual UserResult GenerateUser(const EnrichedCity& city, virtual UserResult GenerateUser(const EnrichedCity& city,
const UserPersona& persona, const UserPersona& persona,
const Name& name) = 0; const Name& name) = 0;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_DATA_GENERATOR_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_DATA_GENERATOR_H_

188
tooling/pipeline/includes/data_generation/llama_generator.h
View File

@@ -26,110 +26,112 @@ struct llama_context;
* @brief Data generator implementation backed by llama.cpp. * @brief Data generator implementation backed by llama.cpp.
*/ */
class LlamaGenerator final : public DataGenerator { class LlamaGenerator final : public DataGenerator {
public: public:
/** /**
* @brief Constructs a generator using parsed application options and loads * @brief Constructs a generator using parsed application options and loads
* the configured model immediately. * the configured model immediately.
* *
* @param options Parsed application options. * @param options Parsed application options.
* @param model_path Filesystem path to GGUF model assets. * @param model_path Filesystem path to GGUF model assets.
* @param prompt_formatter Formatter that produces model-specific prompts. * @param prompt_formatter Formatter that produces model-specific prompts.
* @param prompt_directory Directory service for loading named prompt files. * @param prompt_directory Directory service for loading named prompt files.
*/ */
LlamaGenerator(const ApplicationOptions& options, LlamaGenerator(const ApplicationOptions& options,
const std::string& model_path, std::shared_ptr<ILogger> logger, const std::string& model_path,
std::unique_ptr<IPromptFormatter> prompt_formatter, std::shared_ptr<ILogger> logger,
std::unique_ptr<IPromptDirectory> prompt_directory); std::unique_ptr<IPromptFormatter> prompt_formatter,
std::unique_ptr<IPromptDirectory> prompt_directory);
~LlamaGenerator() override; ~LlamaGenerator() override;
LlamaGenerator(const LlamaGenerator&) = delete; LlamaGenerator(const LlamaGenerator&) = delete;
LlamaGenerator& operator=(const LlamaGenerator&) = delete; LlamaGenerator& operator=(const LlamaGenerator&) = delete;
LlamaGenerator(LlamaGenerator&&) = delete; LlamaGenerator(LlamaGenerator&&) = delete;
LlamaGenerator& operator=(LlamaGenerator&&) = delete; LlamaGenerator& operator=(LlamaGenerator&&) = delete;
/** /**
* @brief Generates brewery data for a specific location. * @brief Generates brewery data for a specific location.
* *
* @param location City object. * @param location City object.
* @param region_context Additional regional context. * @param region_context Additional regional context.
* @return Generated brewery result. * @return Generated brewery result.
*/ */
BreweryResult GenerateBrewery(const EnrichedCity&) override; BreweryResult GenerateBrewery(const EnrichedCity&) override;
/** /**
* @brief Generates a user profile grounded in a sampled name and persona. * @brief Generates a user profile grounded in a sampled name and persona.
* *
* @param city Enriched city the user is associated with. * @param city Enriched city the user is associated with.
* @param persona Persona archetype grounding the generated bio. * @param persona Persona archetype grounding the generated bio.
* @param name Sampled first/last name -- not LLM-invented. * @param name Sampled first/last name -- not LLM-invented.
* @return Generated user profile. * @return Generated user profile.
*/ */
UserResult GenerateUser(const EnrichedCity& city, const UserPersona& persona, UserResult GenerateUser(const EnrichedCity& city, const UserPersona& persona,
const Name& name) override; const Name& name) override;
private: private:
static constexpr int32_t kDefaultMaxTokens = 10000; static constexpr int32_t kDefaultMaxTokens = 10000;
static constexpr float kDefaultSamplingTopP = 0.95F; static constexpr float kDefaultSamplingTopP = 0.95F;
static constexpr uint32_t kDefaultSamplingTopK = 64; static constexpr uint32_t kDefaultSamplingTopK = 64;
static constexpr uint32_t kDefaultContextSize = 8192; static constexpr uint32_t kDefaultContextSize = 8192;
struct ModelDeleter { struct ModelDeleter {
void operator()(llama_model* model) const noexcept; void operator()(llama_model* model) const noexcept;
}; };
struct ContextDeleter { struct ContextDeleter {
void operator()(llama_context* context) const noexcept; void operator()(llama_context* context) const noexcept;
}; };
using ModelHandle = std::unique_ptr<llama_model, ModelDeleter>; using ModelHandle = std::unique_ptr<llama_model, ModelDeleter>;
using ContextHandle = std::unique_ptr<llama_context, ContextDeleter>; using ContextHandle = std::unique_ptr<llama_context, ContextDeleter>;
/** /**
* @brief Loads model and prepares inference context. * @brief Loads model and prepares inference context.
* *
* @param model_path Filesystem path to GGUF model. * @param model_path Filesystem path to GGUF model.
*/ */
void Load(const std::string& model_path); void Load(const std::string& model_path);
/** /**
* @brief Infers text from separate system and user prompts. * @brief Infers text from separate system and user prompts.
* *
* This helps chat-capable models preserve system-role behavior instead of * This helps chat-capable models preserve system-role behavior instead of
* concatenating system text into user input. * concatenating system text into user input.
* *
* @param system_prompt System role prompt. * @param system_prompt System role prompt.
* @param prompt User prompt. * @param prompt User prompt.
* @param max_tokens Maximum tokens to generate. * @param max_tokens Maximum tokens to generate.
* @param grammar Optional GBNF grammar constraining generated output. * @param grammar Optional GBNF grammar constraining generated output.
* @return Generated text. * @return Generated text.
*/ */
std::string Infer(const std::string& system_prompt, const std::string& prompt, std::string Infer(const std::string& system_prompt,
int max_tokens = kDefaultMaxTokens, const std::string& prompt,
std::string_view grammar = {}); int max_tokens = kDefaultMaxTokens,
std::string_view grammar = {});
/** /**
* @brief Runs inference on an already-formatted prompt. * @brief Runs inference on an already-formatted prompt.
* *
* @param formatted_prompt Prompt preformatted for model chat template. * @param formatted_prompt Prompt preformatted for model chat template.
* @param max_tokens Maximum tokens to generate. * @param max_tokens Maximum tokens to generate.
* @param grammar Optional GBNF grammar constraining generated output. * @param grammar Optional GBNF grammar constraining generated output.
* @return Generated text. * @return Generated text.
*/ */
std::string InferFormatted(const std::string& formatted_prompt, std::string InferFormatted(const std::string& formatted_prompt,
int max_tokens = kDefaultMaxTokens, int max_tokens = kDefaultMaxTokens,
std::string_view grammar = {}); std::string_view grammar = {});
ModelHandle model_; ModelHandle model_;
ContextHandle context_; ContextHandle context_;
float sampling_temperature_ = 1.0F; float sampling_temperature_ = 1.0F;
float sampling_top_p_ = kDefaultSamplingTopP; float sampling_top_p_ = kDefaultSamplingTopP;
uint32_t sampling_top_k_ = kDefaultSamplingTopK; uint32_t sampling_top_k_ = kDefaultSamplingTopK;
std::mt19937 rng_; std::mt19937 rng_;
uint32_t n_ctx_ = kDefaultContextSize; uint32_t n_ctx_ = kDefaultContextSize;
int n_gpu_layers_ = 0; int n_gpu_layers_ = 0;
std::shared_ptr<ILogger> logger_; std::shared_ptr<ILogger> logger_;
std::unique_ptr<IPromptFormatter> prompt_formatter_; std::unique_ptr<IPromptFormatter> prompt_formatter_;
std::unique_ptr<IPromptDirectory> prompt_directory_; std::unique_ptr<IPromptDirectory> prompt_directory_;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_LLAMA_GENERATOR_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_LLAMA_GENERATOR_H_

225
tooling/pipeline/includes/data_generation/mock_generator.h
View File

@@ -16,128 +16,129 @@
* @brief Mock generator used for deterministic, model-free outputs. * @brief Mock generator used for deterministic, model-free outputs.
*/ */
class MockGenerator final : public DataGenerator { class MockGenerator final : public DataGenerator {
public: public:
/** /**
* @brief Generates deterministic brewery data for a location. * @brief Generates deterministic brewery data for a location.
* *
* @param enriched_city * @param enriched_city
* @return Generated brewery result. * @return Generated brewery result.
*/ */
BreweryResult GenerateBrewery(const EnrichedCity& enriched_city) override; BreweryResult GenerateBrewery(const EnrichedCity& enriched_city) override;
/** /**
* @brief Generates deterministic user data grounded in a sampled name and * @brief Generates deterministic user data grounded in a sampled name and
* persona. * persona.
* @return Generated user result. * @return Generated user result.
*/ */
UserResult GenerateUser(const EnrichedCity& city, const UserPersona& persona, UserResult GenerateUser(const EnrichedCity& city, const UserPersona& persona,
const Name& name) override; const Name& name) override;
private: private:
/** /**
* @brief Combines two strings into a stable hash value. * @brief Combines two strings into a stable hash value.
* *
* @param location City and country names. * @param location City and country names.
* @return Deterministic hash value. * @return Deterministic hash value.
*/ */
static size_t DeterministicHash(const City& location); static size_t DeterministicHash(const City& location);
/** /**
* @brief Combines city, persona, and name into a stable hash value. * @brief Combines city, persona, and name into a stable hash value.
* *
* @param location City and country names. * @param location City and country names.
* @param persona Persona archetype. * @param persona Persona archetype.
* @param name Sampled first/last name. * @param name Sampled first/last name.
* @return Deterministic hash value. * @return Deterministic hash value.
*/ */
static size_t DeterministicHash(const City& location, static size_t DeterministicHash(const City& location,
const UserPersona& persona, const Name& name); const UserPersona& persona,
const Name& name);
// Hash stride constants for deterministic distribution across fixed-size // Hash stride constants for deterministic distribution across fixed-size
// arrays. These coprime strides spread hash values uniformly without // arrays. These coprime strides spread hash values uniformly without
// clustering, ensuring diverse output across different hash inputs. // clustering, ensuring diverse output across different hash inputs.
static constexpr size_t kNounHashStride = 7; static constexpr size_t kNounHashStride = 7;
static constexpr size_t kDescriptionHashStride = 13; static constexpr size_t kDescriptionHashStride = 13;
static constexpr size_t kBioHashStride = 11; static constexpr size_t kBioHashStride = 11;
static constexpr size_t kActivityWeightHashStride = 17; static constexpr size_t kActivityWeightHashStride = 17;
static constexpr size_t kActivityWeightHashRange = 1000; static constexpr size_t kActivityWeightHashRange = 1000;
static constexpr std::array<std::string_view, 18> kBreweryAdjectives = { static constexpr std::array<std::string_view, 18> kBreweryAdjectives = {
"Craft", "Heritage", "Local", "Artisan", "Pioneer", "Golden", "Craft", "Heritage", "Local", "Artisan", "Pioneer", "Golden",
"Modern", "Classic", "Summit", "Northern", "Riverstone", "Barrel", "Modern", "Classic", "Summit", "Northern", "Riverstone", "Barrel",
"Hinterland", "Harbor", "Wild", "Granite", "Copper", "Maple"}; "Hinterland", "Harbor", "Wild", "Granite", "Copper", "Maple"};
static constexpr std::array<std::string_view, 18> kBreweryNouns = { static constexpr std::array<std::string_view, 18> kBreweryNouns = {
"Brewing Co.", "Brewery", "Bier Haus", "Taproom", "Works", "Brewing Co.", "Brewery", "Bier Haus", "Taproom", "Works",
"House", "Fermentery", "Ale Co.", "Cellars", "Collective", "House", "Fermentery", "Ale Co.", "Cellars", "Collective",
"Project", "Foundry", "Malthouse", "Public House", "Co-op", "Project", "Foundry", "Malthouse", "Public House", "Co-op",
"Lab", "Beer Hall", "Guild"}; "Lab", "Beer Hall", "Guild"};
static constexpr std::array<std::string_view, 18> kBreweryDescriptions = { static constexpr std::array<std::string_view, 18> kBreweryDescriptions = {
"Handcrafted pale ales and seasonal IPAs with local ingredients.", "Handcrafted pale ales and seasonal IPAs with local ingredients.",
"Traditional lagers and experimental sours in small batches.", "Traditional lagers and experimental sours in small batches.",
"Award-winning stouts and wildly hoppy blonde ales.", "Award-winning stouts and wildly hoppy blonde ales.",
"Craft brewery specializing in Belgian-style triples and dark " "Craft brewery specializing in Belgian-style triples and dark "
"porters.", "porters.",
"Modern brewery blending tradition with bold experimental flavors.", "Modern brewery blending tradition with bold experimental flavors.",
"Neighborhood-focused taproom pouring crisp pilsners and citrusy " "Neighborhood-focused taproom pouring crisp pilsners and citrusy "
"pale " "pale "
"ales.", "ales.",
"Small-batch brewery known for barrel-aged releases and smoky " "Small-batch brewery known for barrel-aged releases and smoky "
"lagers.", "lagers.",
"Independent brewhouse pairing farmhouse ales with rotating food " "Independent brewhouse pairing farmhouse ales with rotating food "
"pop-ups.", "pop-ups.",
"Community brewpub making balanced bitters, saisons, and hazy IPAs.", "Community brewpub making balanced bitters, saisons, and hazy IPAs.",
"Experimental nanobrewery exploring local yeast and regional " "Experimental nanobrewery exploring local yeast and regional "
"grains.", "grains.",
"Family-run brewery producing smooth amber ales and robust porters.", "Family-run brewery producing smooth amber ales and robust porters.",
"Urban brewery crafting clean lagers and bright, fruit-forward " "Urban brewery crafting clean lagers and bright, fruit-forward "
"sours.", "sours.",
"Riverfront brewhouse featuring oak-matured ales and seasonal " "Riverfront brewhouse featuring oak-matured ales and seasonal "
"blends.", "blends.",
"Modern taproom focused on sessionable lagers and classic pub " "Modern taproom focused on sessionable lagers and classic pub "
"styles.", "styles.",
"Brewery rooted in tradition with a lineup of malty reds and crisp " "Brewery rooted in tradition with a lineup of malty reds and crisp "
"lagers.", "lagers.",
"Creative brewery offering rotating collaborations and limited " "Creative brewery offering rotating collaborations and limited "
"draft-only " "draft-only "
"pours.", "pours.",
"Locally inspired brewery serving approachable ales with bold hop " "Locally inspired brewery serving approachable ales with bold hop "
"character.", "character.",
"Destination taproom known for balanced IPAs and cocoa-rich " "Destination taproom known for balanced IPAs and cocoa-rich "
"stouts."}; "stouts."};
static constexpr std::array<std::string_view, 18> kUsernames = { static constexpr std::array<std::string_view, 18> kUsernames = {
"hopseeker", "malttrail", "yeastwhisper", "lagerlane", "hopseeker", "malttrail", "yeastwhisper", "lagerlane",
"barrelbound", "foamfinder", "taphunter", "graingeist", "barrelbound", "foamfinder", "taphunter", "graingeist",
"brewscout", "aleatlas", "caskcompass", "hopsandmaps", "brewscout", "aleatlas", "caskcompass", "hopsandmaps",
"mashpilot", "pintnomad", "fermentfriend", "stoutsignal", "mashpilot", "pintnomad", "fermentfriend", "stoutsignal",
"sessionwander", "kettlekeeper"}; "sessionwander", "kettlekeeper"};
static constexpr std::array<std::string_view, 18> kBios = { static constexpr std::array<std::string_view, 18> kBios = {
"Always chasing balanced IPAs and crisp lagers across local taprooms.", "Always chasing balanced IPAs and crisp lagers across local taprooms.",
"Weekend brewery explorer with a soft spot for dark, roasty stouts.", "Weekend brewery explorer with a soft spot for dark, roasty stouts.",
"Documenting tiny brewpubs, fresh pours, and unforgettable beer " "Documenting tiny brewpubs, fresh pours, and unforgettable beer "
"gardens.", "gardens.",
"Fan of farmhouse ales, food pairings, and long tasting flights.", "Fan of farmhouse ales, food pairings, and long tasting flights.",
"Collecting favorite pilsners one city at a time.", "Collecting favorite pilsners one city at a time.",
"Hops-first drinker who still saves room for classic malt-forward " "Hops-first drinker who still saves room for classic malt-forward "
"styles.", "styles.",
"Finding hidden tap lists and sharing the best seasonal releases.", "Finding hidden tap lists and sharing the best seasonal releases.",
"Brewery road-tripper focused on local ingredients and clean " "Brewery road-tripper focused on local ingredients and clean "
"fermentation.", "fermentation.",
"Always comparing house lagers and ranking patio pint vibes.", "Always comparing house lagers and ranking patio pint vibes.",
"Curious about yeast strains, barrel programs, and cellar experiments.", "Curious about yeast strains, barrel programs, and cellar experiments.",
"Believes every neighborhood deserves a great community taproom.", "Believes every neighborhood deserves a great community taproom.",
"Looking for session beers that taste great from first sip to last.", "Looking for session beers that taste great from first sip to last.",
"Belgian ale enthusiast who never skips a new saison.", "Belgian ale enthusiast who never skips a new saison.",
"Hazy IPA critic with deep respect for a perfectly clear pilsner.", "Hazy IPA critic with deep respect for a perfectly clear pilsner.",
"Visits breweries for the stories, stays for the flagship pours.", "Visits breweries for the stories, stays for the flagship pours.",
"Craft beer fan mapping tasting notes and favorite brew routes.", "Craft beer fan mapping tasting notes and favorite brew routes.",
"Always ready to trade recommendations for underrated local breweries.", "Always ready to trade recommendations for underrated local breweries.",
"Keeping a running list of must-try collab releases and tap " "Keeping a running list of must-try collab releases and tap "
"takeovers."}; "takeovers."};
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_MOCK_GENERATOR_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_MOCK_GENERATOR_H_

11
tooling/pipeline/includes/data_generation/prompt_formatting/gemma4_jinja_prompt_formatter.h
View File

@@ -7,12 +7,13 @@
#include "data_generation/prompt_formatting/prompt_formatter.h" #include "data_generation/prompt_formatting/prompt_formatter.h"
class Gemma4JinjaPromptFormatter final : public IPromptFormatter { class Gemma4JinjaPromptFormatter final : public IPromptFormatter {
public: public:
Gemma4JinjaPromptFormatter() = default; Gemma4JinjaPromptFormatter() = default;
~Gemma4JinjaPromptFormatter() override = default; ~Gemma4JinjaPromptFormatter() override = default;
[[nodiscard]] std::string Format(std::string_view system_prompt, [[nodiscard]] std::string Format(
std::string_view user_prompt) const override; std::string_view system_prompt,
std::string_view user_prompt) const override;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_PROMPT_FORMATTING_GEMMA4_JINJA_PROMPT_FORMATTER_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_PROMPT_FORMATTING_GEMMA4_JINJA_PROMPT_FORMATTER_H_

18
tooling/pipeline/includes/data_generation/prompt_formatting/prompt_formatter.h
View File

@@ -5,16 +5,16 @@
#include <string_view> #include <string_view>
class IPromptFormatter { class IPromptFormatter {
public: public:
IPromptFormatter() = default; IPromptFormatter() = default;
IPromptFormatter(const IPromptFormatter&) = delete; IPromptFormatter(const IPromptFormatter&) = delete;
IPromptFormatter& operator=(const IPromptFormatter&) = delete; IPromptFormatter& operator=(const IPromptFormatter&) = delete;
IPromptFormatter(IPromptFormatter&&) = delete; IPromptFormatter(IPromptFormatter&&) = delete;
IPromptFormatter& operator=(IPromptFormatter&&) = delete; IPromptFormatter& operator=(IPromptFormatter&&) = delete;
virtual ~IPromptFormatter() = default; virtual ~IPromptFormatter() = default;
[[nodiscard]] virtual std::string Format( [[nodiscard]] virtual std::string Format(
std::string_view system_prompt, std::string_view user_prompt) const = 0; std::string_view system_prompt, std::string_view user_prompt) const = 0;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_PROMPT_FORMATTING_PROMPT_FORMATTER_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_PROMPT_FORMATTING_PROMPT_FORMATTER_H_

112
tooling/pipeline/includes/data_model/generated_models.h
View File

@@ -17,64 +17,64 @@
* @brief Generated brewery payload. * @brief Generated brewery payload.
*/ */
struct BreweryResult { struct BreweryResult {
/** /**
* @brief Brewery display name in English. * @brief Brewery display name in English.
*/ */
std::string name_en; std::string name_en;
/** /**
* @brief Brewery description text in English. * @brief Brewery description text in English.
*/ */
std::string description_en; std::string description_en;
/** /**
* @brief Brewery display name in the local language. * @brief Brewery display name in the local language.
*/ */
std::string name_local; std::string name_local;
/** /**
* @brief Brewery description text in the local language. * @brief Brewery description text in the local language.
*/ */
std::string description_local; std::string description_local;
}; };
/** /**
* @brief Generated user profile payload. * @brief Generated user profile payload.
*/ */
struct UserResult { struct UserResult {
/** /**
* @brief First (given) name, copied from the sampled Name -- not * @brief First (given) name, copied from the sampled Name -- not
* LLM-invented. * LLM-invented.
*/ */
std::string first_name{}; std::string first_name{};
/** /**
* @brief Last (family) name, copied from the sampled Name -- not * @brief Last (family) name, copied from the sampled Name -- not
* LLM-invented. * LLM-invented.
*/ */
std::string last_name{}; std::string last_name{};
/** /**
* @brief Gender associated with the sampled first name, copied from the * @brief Gender associated with the sampled first name, copied from the
* sampled Name -- not LLM-invented. * sampled Name -- not LLM-invented.
*/ */
std::string gender{}; std::string gender{};
/** /**
* @brief Username handle. * @brief Username handle.
*/ */
std::string username{}; std::string username{};
/** /**
* @brief Short user biography. * @brief Short user biography.
*/ */
std::string bio{}; std::string bio{};
/** /**
* @brief Relative check-in/activity weight for this user, used to drive * @brief Relative check-in/activity weight for this user, used to drive
* a J-curve activity profile in later pipeline phases. * a J-curve activity profile in later pipeline phases.
*/ */
float activity_weight{}; float activity_weight{};
}; };
// Pipeline Data Models // Pipeline Data Models
@@ -83,8 +83,8 @@ struct UserResult {
* @brief Enriched city data with Wikipedia context. * @brief Enriched city data with Wikipedia context.
*/ */
struct EnrichedCity { struct EnrichedCity {
City location; City location;
std::string region_context{}; std::string region_context{};
}; };
/** /**
@@ -92,8 +92,8 @@ struct EnrichedCity {
* from that city's curated postal-code examples. * from that city's curated postal-code examples.
*/ */
struct BreweryAddress { struct BreweryAddress {
City city; City city;
std::string postal_code{}; std::string postal_code{};
}; };
/** /**
@@ -101,16 +101,16 @@ struct BreweryAddress {
* postal code selected from that city's curated postal-code examples. * postal code selected from that city's curated postal-code examples.
*/ */
struct UserAddress { struct UserAddress {
City city; City city;
std::string postal_code{}; std::string postal_code{};
}; };
/** /**
* @brief Helper struct to store generated brewery data. * @brief Helper struct to store generated brewery data.
*/ */
struct BreweryRecord { struct BreweryRecord {
BreweryAddress address; BreweryAddress address;
BreweryResult brewery; BreweryResult brewery;
}; };
/** /**
@@ -121,10 +121,10 @@ struct BreweryRecord {
* consumer can register real accounts from the pipeline's SQLite export. * consumer can register real accounts from the pipeline's SQLite export.
*/ */
struct UserRecord { struct UserRecord {
UserAddress address; UserAddress address;
UserResult user; UserResult user;
std::string email{}; std::string email{};
std::string date_of_birth{}; std::string date_of_birth{};
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_GENERATED_MODELS_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_GENERATED_MODELS_H_

250
tooling/pipeline/includes/data_model/models.h
View File

@@ -28,54 +28,54 @@ namespace prog_opts = boost::program_options;
* `postal_code` object in the locations dataset. * `postal_code` object in the locations dataset.
*/ */
struct PostalCodeSpec { struct PostalCodeSpec {
/** /**
* @brief Permissive country-wide postal-code format regex, used as the * @brief Permissive country-wide postal-code format regex, used as the
* validation fallback for a selected example. * validation fallback for a selected example.
*/ */
std::string country_format_regex{}; std::string country_format_regex{};
/** /**
* @brief City-specific postal-code regexes (more precise than the country * @brief City-specific postal-code regexes (more precise than the country
* format). * format).
*/ */
std::vector<std::string> city_regexes{}; std::vector<std::string> city_regexes{};
/** /**
* @brief Concrete example postal codes for this city. The postal code * @brief Concrete example postal codes for this city. The postal code
* service selects from these rather than synthesizing codes. * service selects from these rather than synthesizing codes.
*/ */
std::vector<std::string> examples{}; std::vector<std::string> examples{};
}; };
/** /**
* @brief Canonical city record for city-level generation. * @brief Canonical city record for city-level generation.
*/ */
struct City { struct City {
std::string city{}; std::string city{};
std::string state_province{}; std::string state_province{};
/** /**
* @brief ISO 3166-2 subdivision code. * @brief ISO 3166-2 subdivision code.
*/ */
std::string iso3166_2{}; std::string iso3166_2{};
std::string country{}; std::string country{};
/** /**
* @brief ISO 3166-1 country code. * @brief ISO 3166-1 country code.
*/ */
std::string iso3166_1{}; std::string iso3166_1{};
/** /**
* @brief Local language codes in priority order. * @brief Local language codes in priority order.
*/ */
std::vector<std::string> local_languages{}; std::vector<std::string> local_languages{};
/** /**
* @brief Postal-code metadata (format regex, city regexes, and concrete * @brief Postal-code metadata (format regex, city regexes, and concrete
* examples) sourced directly from the curated locations dataset. * examples) sourced directly from the curated locations dataset.
*/ */
PostalCodeSpec postal_code{}; PostalCodeSpec postal_code{};
}; };
// Name / Persona Models // Name / Persona Models
@@ -86,44 +86,44 @@ struct City {
* Produced by the SampleName() helper in generate_users.cc. * Produced by the SampleName() helper in generate_users.cc.
*/ */
struct Name { struct Name {
std::string first_name{}; std::string first_name{};
std::string last_name{}; std::string last_name{};
/** /**
* @brief Gender associated with the sampled forename (e.g. "M", "F"), as * @brief Gender associated with the sampled forename (e.g. "M", "F"), as
* reported by the source dataset. * reported by the source dataset.
*/ */
std::string gender{}; std::string gender{};
}; };
/** /**
* @brief A single forename entry from the names-by-country fixture data. * @brief A single forename entry from the names-by-country fixture data.
*/ */
struct ForenameEntry { struct ForenameEntry {
/** /**
* @brief Romanized forename. * @brief Romanized forename.
*/ */
std::string name{}; std::string name{};
/** /**
* @brief Gender associated with this forename, as reported by the source * @brief Gender associated with this forename, as reported by the source
* dataset (e.g. "M", "F"). * dataset (e.g. "M", "F").
*/ */
std::string gender{}; std::string gender{};
bool operator==(const ForenameEntry& other) const { bool operator==(const ForenameEntry& other) const {
return name == other.name && gender == other.gender; return name == other.name && gender == other.gender;
} }
}; };
namespace std { namespace std {
template <> template <>
struct hash<ForenameEntry> { struct hash<ForenameEntry> {
size_t operator()(const ForenameEntry& entry) const noexcept { size_t operator()(const ForenameEntry& entry) const noexcept {
const size_t name_hash = std::hash<std::string>{}(entry.name); const size_t name_hash = std::hash<std::string>{}(entry.name);
const size_t gender_hash = std::hash<std::string>{}(entry.gender); const size_t gender_hash = std::hash<std::string>{}(entry.gender);
return name_hash ^ (gender_hash << 1); return name_hash ^ (gender_hash << 1);
} }
}; };
} // namespace std } // namespace std
@@ -131,20 +131,20 @@ struct hash<ForenameEntry> {
* @brief A persona archetype used to ground LLM-generated user bios. * @brief A persona archetype used to ground LLM-generated user bios.
*/ */
struct UserPersona { struct UserPersona {
/** /**
* @brief Persona display name (e.g. "Hophead Explorer"). * @brief Persona display name (e.g. "Hophead Explorer").
*/ */
std::string name{}; std::string name{};
/** /**
* @brief Short description of the persona's interests and voice. * @brief Short description of the persona's interests and voice.
*/ */
std::string description{}; std::string description{};
/** /**
* @brief Beer styles this persona gravitates toward. * @brief Beer styles this persona gravitates toward.
*/ */
std::vector<std::string> style_affinities{}; std::vector<std::string> style_affinities{};
}; };
// Configuration Models // Configuration Models
@@ -153,88 +153,88 @@ struct UserPersona {
* @brief LLM sampling parameters. * @brief LLM sampling parameters.
*/ */
struct SamplingOptions { struct SamplingOptions {
/** /**
* @brief LLM sampling temperature (higher = more random). * @brief LLM sampling temperature (higher = more random).
*/ */
float temperature = 1.0F; float temperature = 1.0F;
/** /**
* @brief LLM nucleus sampling top-p parameter. * @brief LLM nucleus sampling top-p parameter.
*/ */
float top_p = 0.95F; float top_p = 0.95F;
/** /**
* @brief LLM top-k sampling parameter. * @brief LLM top-k sampling parameter.
*/ */
uint32_t top_k = 64; uint32_t top_k = 64;
/** /**
* @brief Context window size (tokens). * @brief Context window size (tokens).
*/ */
uint32_t n_ctx = 8192; uint32_t n_ctx = 8192;
/** /**
* @brief Random seed (-1 for random, otherwise non-negative). * @brief Random seed (-1 for random, otherwise non-negative).
*/ */
int seed = -1; int seed = -1;
/** /**
* @brief Number of layers to offload to GPU. * @brief Number of layers to offload to GPU.
*/ */
int n_gpu_layers = 0; int n_gpu_layers = 0;
}; };
/** /**
* @brief Configuration for the LLM generator component. * @brief Configuration for the LLM generator component.
*/ */
struct GeneratorOptions { struct GeneratorOptions {
/** /**
* @brief Path to the LLM model file (gguf format). * @brief Path to the LLM model file (gguf format).
*/ */
std::filesystem::path model_path; std::filesystem::path model_path;
/** /**
* @brief Use mocked generator instead of actual LLM inference. * @brief Use mocked generator instead of actual LLM inference.
*/ */
bool use_mocked = false; bool use_mocked = false;
/** /**
* @brief Specific sampling parameters for this generator. * @brief Specific sampling parameters for this generator.
* If nullopt, the application should use global defaults. * If nullopt, the application should use global defaults.
*/ */
std::optional<SamplingOptions> sampling; std::optional<SamplingOptions> sampling;
}; };
/** /**
* @brief Configuration for the pipeline execution and output. * @brief Configuration for the pipeline execution and output.
*/ */
struct PipelineOptions { struct PipelineOptions {
/** /**
* @brief Directory for generated artifacts. * @brief Directory for generated artifacts.
*/ */
std::filesystem::path output_path; std::filesystem::path output_path;
/** /**
* @brief Directory that contains named prompt files (e.g. * @brief Directory that contains named prompt files (e.g.
* BREWERY_GENERATION.md). * BREWERY_GENERATION.md).
*/ */
std::filesystem::path prompt_dir; std::filesystem::path prompt_dir;
std::filesystem::path log_path; std::filesystem::path log_path;
/** /**
* @brief Number of locations to sample from the dataset * @brief Number of locations to sample from the dataset
* More locations -> more users/more breweries * More locations -> more users/more breweries
*/ */
uint32_t location_count; uint32_t location_count;
}; };
/** /**
* @brief Root configuration object for the Biergarten pipeline. * @brief Root configuration object for the Biergarten pipeline.
*/ */
struct ApplicationOptions { struct ApplicationOptions {
GeneratorOptions generator; GeneratorOptions generator;
PipelineOptions pipeline; PipelineOptions pipeline;
}; };
// Function Declarations // Function Declarations

149
tooling/pipeline/includes/json_handling/pretty_print.h
View File

@@ -27,83 +27,84 @@
inline void PrettyPrint(std::ostream& outstream, inline void PrettyPrint(std::ostream& outstream,
boost::json::value const& json_val, boost::json::value const& json_val,
std::string* indent = nullptr) { std::string* indent = nullptr) {
std::string str; std::string str;
if (indent == nullptr) { if (indent == nullptr) {
indent = &str; indent = &str;
} }
switch (json_val.kind()) { switch (json_val.kind()) {
case boost::json::kind::object: { case boost::json::kind::object: {
outstream << "{\n"; outstream << "{\n";
indent->append(4, ' '); indent->append(4, ' ');
auto const& obj = json_val.get_object(); auto const& obj = json_val.get_object();
if (!obj.empty()) { if (!obj.empty()) {
const auto* iter = obj.begin(); const auto* iter = obj.begin();
for (;;) { for (;;) {
outstream << *indent << boost::json::serialize(iter->key()) << " : "; outstream << *indent << boost::json::serialize(iter->key())
PrettyPrint(outstream, iter->value(), indent); << " : ";
iter = std::next(iter); PrettyPrint(outstream, iter->value(), indent);
if (iter == obj.end()) { iter = std::next(iter);
break; if (iter == obj.end()) {
} break;
}
outstream << ",\n"; outstream << ",\n";
} }
}
outstream << "\n";
indent->resize(indent->size() - 4);
outstream << *indent << "}";
break;
} }
case boost::json::kind::array: {
outstream << "[\n";
indent->append(4, ' ');
auto const& arr = json_val.get_array();
if (!arr.empty()) {
const auto* iter = arr.begin();
for (;;) {
outstream << *indent;
PrettyPrint(outstream, *iter, indent);
iter = std::next(iter);
if (iter == arr.end()) {
break;
}
outstream << ",\n";
}
}
outstream << "\n";
indent->resize(indent->size() - 4);
outstream << *indent << "]";
break;
}
case boost::json::kind::string: {
outstream << serialize(json_val.get_string());
break;
}
case boost::json::kind::uint64:
case boost::json::kind::int64:
case boost::json::kind::double_:
outstream << json_val;
break;
case boost::json::kind::bool_:
if (json_val.get_bool()) {
outstream << "true";
} else {
outstream << "false";
}
break;
case boost::json::kind::null:
outstream << "null";
break;
}
if (indent->empty()) {
outstream << "\n"; outstream << "\n";
indent->resize(indent->size() - 4); }
outstream << *indent << "}";
break;
}
case boost::json::kind::array: {
outstream << "[\n";
indent->append(4, ' ');
auto const& arr = json_val.get_array();
if (!arr.empty()) {
const auto* iter = arr.begin();
for (;;) {
outstream << *indent;
PrettyPrint(outstream, *iter, indent);
iter = std::next(iter);
if (iter == arr.end()) {
break;
}
outstream << ",\n";
}
}
outstream << "\n";
indent->resize(indent->size() - 4);
outstream << *indent << "]";
break;
}
case boost::json::kind::string: {
outstream << serialize(json_val.get_string());
break;
}
case boost::json::kind::uint64:
case boost::json::kind::int64:
case boost::json::kind::double_:
outstream << json_val;
break;
case boost::json::kind::bool_:
if (json_val.get_bool()) {
outstream << "true";
} else {
outstream << "false";
}
break;
case boost::json::kind::null:
outstream << "null";
break;
}
if (indent->empty()) {
outstream << "\n";
}
} }
#endif // BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_PRETTY_PRINT_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_PRETTY_PRINT_H_

10
tooling/pipeline/includes/llama_backend_state.h
View File

@@ -15,12 +15,12 @@
* it alive for application lifetime. * it alive for application lifetime.
*/ */
class LlamaBackendState { class LlamaBackendState {
public: public:
LlamaBackendState() { llama_backend_init(); } LlamaBackendState() { llama_backend_init(); }
~LlamaBackendState() { llama_backend_free(); } ~LlamaBackendState() { llama_backend_free(); }
LlamaBackendState(const LlamaBackendState&) = delete; LlamaBackendState(const LlamaBackendState&) = delete;
LlamaBackendState& operator=(const LlamaBackendState&) = delete; LlamaBackendState& operator=(const LlamaBackendState&) = delete;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_LLAMA_BACKEND_STATE_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_LLAMA_BACKEND_STATE_H_

34
tooling/pipeline/includes/services/curated_data/curated_data_service.h
View File

@@ -25,26 +25,26 @@ using SurnamesByCountryMap = std::unordered_map<std::string, SurnameList>;
* brewery/user generation. * brewery/user generation.
*/ */
class ICuratedDataService { class ICuratedDataService {
public: public:
ICuratedDataService() = default; ICuratedDataService() = default;
virtual ~ICuratedDataService() = default; virtual ~ICuratedDataService() = default;
ICuratedDataService(const ICuratedDataService&) = delete; ICuratedDataService(const ICuratedDataService&) = delete;
ICuratedDataService& operator=(const ICuratedDataService&) = delete; ICuratedDataService& operator=(const ICuratedDataService&) = delete;
ICuratedDataService(ICuratedDataService&&) = delete; ICuratedDataService(ICuratedDataService&&) = delete;
ICuratedDataService& operator=(ICuratedDataService&&) = delete; ICuratedDataService& operator=(ICuratedDataService&&) = delete;
/** /**
* @brief Loads all curated location records. * @brief Loads all curated location records.
*/ */
virtual const LocationsList& LoadLocations() = 0; virtual const LocationsList& LoadLocations() = 0;
/** /**
* @brief Loads all curated persona records. * @brief Loads all curated persona records.
*/ */
virtual const PersonasList& LoadPersonas() = 0; virtual const PersonasList& LoadPersonas() = 0;
virtual const ForenamesByCountryMap& LoadForenamesByCountry() = 0; virtual const ForenamesByCountryMap& LoadForenamesByCountry() = 0;
virtual const SurnamesByCountryMap& LoadSurnamesByCountry() = 0; virtual const SurnamesByCountryMap& LoadSurnamesByCountry() = 0;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_CURATED_DATA_SERVICE_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_CURATED_DATA_SERVICE_H_

58
tooling/pipeline/includes/services/curated_data/curated_json_data_service.h
View File

@@ -16,48 +16,48 @@
* CuratedJsonDataService. * CuratedJsonDataService.
*/ */
struct CuratedDataFilePaths { struct CuratedDataFilePaths {
std::filesystem::path locations_path; std::filesystem::path locations_path;
std::filesystem::path personas_path; std::filesystem::path personas_path;
std::filesystem::path forenames_path; std::filesystem::path forenames_path;
std::filesystem::path surnames_path; std::filesystem::path surnames_path;
}; };
/** /**
* @brief Loads curated location, persona, and name data from JSON files. * @brief Loads curated location, persona, and name data from JSON files.
*/ */
class CuratedJsonDataService final : public ICuratedDataService { class CuratedJsonDataService final : public ICuratedDataService {
struct cache { struct cache {
LocationsList locations; LocationsList locations;
PersonasList personas; PersonasList personas;
ForenamesByCountryMap forenames_by_country; ForenamesByCountryMap forenames_by_country;
SurnamesByCountryMap surnames_by_country; SurnamesByCountryMap surnames_by_country;
cache() = default; cache() = default;
~cache() = default; ~cache() = default;
}; };
CuratedDataFilePaths filepaths_; CuratedDataFilePaths filepaths_;
cache cache_; cache cache_;
public: public:
explicit CuratedJsonDataService(CuratedDataFilePaths filepaths); explicit CuratedJsonDataService(CuratedDataFilePaths filepaths);
/** /**
* @brief Parses a JSON array file and returns all location records. * @brief Parses a JSON array file and returns all location records.
*/ */
const LocationsList& LoadLocations() override; const LocationsList& LoadLocations() override;
/** /**
* @brief Parses a JSON array file and returns all persona records. * @brief Parses a JSON array file and returns all persona records.
*/ */
const PersonasList& LoadPersonas() override; const PersonasList& LoadPersonas() override;
const ForenamesByCountryMap& LoadForenamesByCountry() override; const ForenamesByCountryMap& LoadForenamesByCountry() override;
/** /**
* @brief Parses a JSON file and returns all the forenames per country. * @brief Parses a JSON file and returns all the forenames per country.
*/ */
const SurnamesByCountryMap& LoadSurnamesByCountry() override; const SurnamesByCountryMap& LoadSurnamesByCountry() override;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_

22
tooling/pipeline/includes/services/curated_data/mock_curated_data_service.h
View File

@@ -17,22 +17,22 @@
* place of the JSON fixture files used by JsonLoader. * place of the JSON fixture files used by JsonLoader.
*/ */
class MockCuratedDataService final : public ICuratedDataService { class MockCuratedDataService final : public ICuratedDataService {
public: public:
MockCuratedDataService(); MockCuratedDataService();
const LocationsList& LoadLocations() override; const LocationsList& LoadLocations() override;
const PersonasList& LoadPersonas() override; const PersonasList& LoadPersonas() override;
const ForenamesByCountryMap& LoadForenamesByCountry() override; const ForenamesByCountryMap& LoadForenamesByCountry() override;
const SurnamesByCountryMap& LoadSurnamesByCountry() override; const SurnamesByCountryMap& LoadSurnamesByCountry() override;
private: private:
LocationsList locations_; LocationsList locations_;
PersonasList personas_; PersonasList personas_;
ForenamesByCountryMap forenames_by_country_; ForenamesByCountryMap forenames_by_country_;
SurnamesByCountryMap surnames_by_country_; SurnamesByCountryMap surnames_by_country_;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_MOCK_CURATED_DATA_SERVICE_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_MOCK_CURATED_DATA_SERVICE_H_

54
tooling/pipeline/includes/services/database/export_service.h
View File

@@ -14,38 +14,38 @@
* @brief Interface for services that persist generated brewery records. * @brief Interface for services that persist generated brewery records.
*/ */
class IExportService { class IExportService {
public: public:
IExportService() = default; IExportService() = default;
virtual ~IExportService() = default; virtual ~IExportService() = default;
IExportService(const IExportService&) = delete; IExportService(const IExportService&) = delete;
IExportService& operator=(const IExportService&) = delete; IExportService& operator=(const IExportService&) = delete;
IExportService(IExportService&&) = delete; IExportService(IExportService&&) = delete;
IExportService& operator=(IExportService&&) = delete; IExportService& operator=(IExportService&&) = delete;
/** /**
* @brief Prepares the export destination for a new run. * @brief Prepares the export destination for a new run.
*/ */
virtual void Initialize() = 0; virtual void Initialize() = 0;
/** /**
* @brief Persists one generated brewery record. * @brief Persists one generated brewery record.
* *
* @param brewery Generated brewery payload to store. * @param brewery Generated brewery payload to store.
*/ */
virtual uint64_t ProcessRecord(const BreweryRecord& brewery) = 0; virtual uint64_t ProcessRecord(const BreweryRecord& brewery) = 0;
/** /**
* @brief Persists one generated user record. * @brief Persists one generated user record.
* *
* @param user Generated user payload to store. * @param user Generated user payload to store.
*/ */
virtual uint64_t ProcessRecord(const UserRecord& user) = 0; virtual uint64_t ProcessRecord(const UserRecord& user) = 0;
/** /**
* @brief Finalizes the export destination. * @brief Finalizes the export destination.
*/ */
virtual void Finalize() = 0; virtual void Finalize() = 0;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_EXPORT_SERVICE_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_EXPORT_SERVICE_H_

82
tooling/pipeline/includes/services/database/sqlite_export_service.h
View File

@@ -20,54 +20,54 @@
* @brief Persists generated brewery records into a fresh SQLite database. * @brief Persists generated brewery records into a fresh SQLite database.
*/ */
class SqliteExportService final : public IExportService { class SqliteExportService final : public IExportService {
public: public:
explicit SqliteExportService(const ApplicationOptions& options); explicit SqliteExportService(const ApplicationOptions& options);
~SqliteExportService() override; ~SqliteExportService() override;
SqliteExportService(const SqliteExportService&) = delete; SqliteExportService(const SqliteExportService&) = delete;
SqliteExportService& operator=(const SqliteExportService&) = delete; SqliteExportService& operator=(const SqliteExportService&) = delete;
SqliteExportService(SqliteExportService&&) = delete; SqliteExportService(SqliteExportService&&) = delete;
SqliteExportService& operator=(SqliteExportService&&) = delete; SqliteExportService& operator=(SqliteExportService&&) = delete;
void Initialize() override; void Initialize() override;
uint64_t ProcessRecord(const BreweryRecord& brewery) override; uint64_t ProcessRecord(const BreweryRecord& brewery) override;
uint64_t ProcessRecord(const UserRecord& user) override; uint64_t ProcessRecord(const UserRecord& user) override;
void Finalize() override; void Finalize() override;
private: private:
using SqliteDatabaseHandle = using SqliteDatabaseHandle =
sqlite_export_service_internal::SqliteDatabaseHandle; sqlite_export_service_internal::SqliteDatabaseHandle;
using SqliteStatementHandle = using SqliteStatementHandle =
sqlite_export_service_internal::SqliteStatementHandle; sqlite_export_service_internal::SqliteStatementHandle;
void InitializeSchema() const; void InitializeSchema() const;
void PrepareStatements(); void PrepareStatements();
void RollbackAndCloseNoThrow() noexcept; void RollbackAndCloseNoThrow() noexcept;
[[nodiscard]] std::filesystem::path BuildDatabasePath() const; [[nodiscard]] std::filesystem::path BuildDatabasePath() const;
[[nodiscard]] static std::string BuildCityKey(const City& city); [[nodiscard]] static std::string BuildCityKey(const City& city);
/** /**
* @brief Returns the row id for @p city, inserting it first if it has * @brief Returns the row id for @p city, inserting it first if it has
* not already been seen during this run. * not already been seen during this run.
* *
* Shared by both ProcessRecord() overloads so brewery and user addresses * Shared by both ProcessRecord() overloads so brewery and user addresses
* referencing the same city resolve to the same row. * referencing the same city resolve to the same row.
*/ */
[[nodiscard]] sqlite3_int64 ResolveCityId(const City& city); [[nodiscard]] sqlite3_int64 ResolveCityId(const City& city);
std::unique_ptr<IDateTimeProvider> date_time_provider_; std::unique_ptr<IDateTimeProvider> date_time_provider_;
std::filesystem::path output_path_; std::filesystem::path output_path_;
std::string run_timestamp_utc_; std::string run_timestamp_utc_;
std::filesystem::path database_path_; std::filesystem::path database_path_;
SqliteDatabaseHandle db_handle_; SqliteDatabaseHandle db_handle_;
SqliteStatementHandle insert_city_stmt_; SqliteStatementHandle insert_city_stmt_;
SqliteStatementHandle insert_brewery_stmt_; SqliteStatementHandle insert_brewery_stmt_;
SqliteStatementHandle insert_brewery_address_stmt_; SqliteStatementHandle insert_brewery_address_stmt_;
SqliteStatementHandle insert_user_stmt_; SqliteStatementHandle insert_user_stmt_;
SqliteStatementHandle insert_user_address_stmt_; SqliteStatementHandle insert_user_address_stmt_;
bool transaction_open_ = false; bool transaction_open_ = false;
std::unordered_map<std::string, sqlite3_int64> city_cache_; std::unordered_map<std::string, sqlite3_int64> city_cache_;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_SQLITE_EXPORT_SERVICE_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_SQLITE_EXPORT_SERVICE_H_

10
tooling/pipeline/includes/services/database/sqlite_handle_types.h
View File

@@ -13,11 +13,11 @@
namespace sqlite_export_service_internal { namespace sqlite_export_service_internal {
struct SqliteDatabaseDeleter { struct SqliteDatabaseDeleter {
void operator()(sqlite3* handle) const noexcept; void operator()(sqlite3* handle) const noexcept;
}; };
struct SqliteStatementDeleter { struct SqliteStatementDeleter {
void operator()(sqlite3_stmt* statement) const noexcept; void operator()(sqlite3_stmt* statement) const noexcept;
}; };
using SqliteDatabaseHandle = std::unique_ptr<sqlite3, SqliteDatabaseDeleter>; using SqliteDatabaseHandle = std::unique_ptr<sqlite3, SqliteDatabaseDeleter>;
@@ -28,9 +28,9 @@ using SqliteStatementHandle =
// N.B. indices are 1 based. // N.B. indices are 1 based.
template <typename T> template <typename T>
struct BoundParam { struct BoundParam {
int index; int index;
T value; T value;
std::string_view action; std::string_view action;
}; };
} // namespace sqlite_export_service_internal } // namespace sqlite_export_service_internal

52
tooling/pipeline/includes/services/database/sqlite_statement_helpers.h
View File

@@ -152,44 +152,44 @@ INSERT INTO user_addresses (
// sqlite3_bind_*() parameter indices are 1-based, matching the "?" // sqlite3_bind_*() parameter indices are 1-based, matching the "?"
// placeholder order in the SQL above. // placeholder order in the SQL above.
enum CityBindIndex { enum CityBindIndex {
kCityCityBindIndex = 1, kCityCityBindIndex = 1,
kCityStateProvinceBindIndex, kCityStateProvinceBindIndex,
kCityIso31662BindIndex, kCityIso31662BindIndex,
kCityCountryBindIndex, kCityCountryBindIndex,
kCityIso31661BindIndex, kCityIso31661BindIndex,
kCityLanguagesBindIndex, kCityLanguagesBindIndex,
kCityPostalCodeCountryFormatRegexBindIndex, kCityPostalCodeCountryFormatRegexBindIndex,
kCityPostalCodeCityRegexJsonBindIndex, kCityPostalCodeCityRegexJsonBindIndex,
}; };
enum BreweryBindIndex { enum BreweryBindIndex {
kBreweryEnglishNameBindIndex = 1, kBreweryEnglishNameBindIndex = 1,
kBreweryEnglishDescriptionBindIndex, kBreweryEnglishDescriptionBindIndex,
kBreweryLocalNameBindIndex, kBreweryLocalNameBindIndex,
kBreweryLocalDescriptionBindIndex, kBreweryLocalDescriptionBindIndex,
}; };
enum BreweryAddressBindIndex { enum BreweryAddressBindIndex {
kBreweryAddressBreweryIdBindIndex = 1, kBreweryAddressBreweryIdBindIndex = 1,
kBreweryAddressCityIdBindIndex, kBreweryAddressCityIdBindIndex,
kBreweryAddressPostalCodeBindIndex, kBreweryAddressPostalCodeBindIndex,
}; };
enum UserBindIndex { enum UserBindIndex {
kUserFirstNameBindIndex = 1, kUserFirstNameBindIndex = 1,
kUserLastNameBindIndex, kUserLastNameBindIndex,
kUserGenderBindIndex, kUserGenderBindIndex,
kUserUsernameBindIndex, kUserUsernameBindIndex,
kUserBioBindIndex, kUserBioBindIndex,
kUserActivityWeightBindIndex, kUserActivityWeightBindIndex,
kUserEmailBindIndex, kUserEmailBindIndex,
kUserDateOfBirthBindIndex, kUserDateOfBirthBindIndex,
}; };
enum UserAddressBindIndex { enum UserAddressBindIndex {
kUserAddressUserIdBindIndex = 1, kUserAddressUserIdBindIndex = 1,
kUserAddressCityIdBindIndex, kUserAddressCityIdBindIndex,
kUserAddressPostalCodeBindIndex, kUserAddressPostalCodeBindIndex,
}; };
SqliteStatementHandle PrepareStatement(const SqliteDatabaseHandle& db_handle, SqliteStatementHandle PrepareStatement(const SqliteDatabaseHandle& db_handle,

64
tooling/pipeline/includes/services/datetime/date_time_provider.h
View File

@@ -20,49 +20,49 @@
* directly * directly
*/ */
class IDateTimeProvider { class IDateTimeProvider {
public: public:
virtual ~IDateTimeProvider() = default; virtual ~IDateTimeProvider() = default;
IDateTimeProvider() = default; IDateTimeProvider() = default;
IDateTimeProvider(const IDateTimeProvider&) = delete; IDateTimeProvider(const IDateTimeProvider&) = delete;
IDateTimeProvider& operator=(const IDateTimeProvider&) = delete; IDateTimeProvider& operator=(const IDateTimeProvider&) = delete;
IDateTimeProvider(IDateTimeProvider&&) = delete; IDateTimeProvider(IDateTimeProvider&&) = delete;
IDateTimeProvider& operator=(IDateTimeProvider&&) = delete; IDateTimeProvider& operator=(IDateTimeProvider&&) = delete;
/** /**
* @brief Returns the current UTC timestamp in a file-safe format. * @brief Returns the current UTC timestamp in a file-safe format.
* *
* @return UTC timestamp string. * @return UTC timestamp string.
*/ */
virtual std::string GetUtcTimestamp() = 0; virtual std::string GetUtcTimestamp() = 0;
}; };
/** /**
* @brief Timestamp provider backed by the system clock. * @brief Timestamp provider backed by the system clock.
*/ */
class SystemDateTimeProvider final : public IDateTimeProvider { class SystemDateTimeProvider final : public IDateTimeProvider {
public: public:
std::string GetUtcTimestamp() override { std::string GetUtcTimestamp() override {
constexpr int kFractionalSecondWidth = 6; constexpr int kFractionalSecondWidth = 6;
const auto now = std::chrono::system_clock::now(); const auto now = std::chrono::system_clock::now();
const auto now_time = std::chrono::system_clock::to_time_t(now); const auto now_time = std::chrono::system_clock::to_time_t(now);
const std::tm* utc_time_ptr = std::gmtime(&now_time); const std::tm* utc_time_ptr = std::gmtime(&now_time);
if (utc_time_ptr == nullptr) { if (utc_time_ptr == nullptr) {
throw std::runtime_error("Failed to format UTC timestamp"); throw std::runtime_error("Failed to format UTC timestamp");
} }
const auto fractional_seconds = const auto fractional_seconds =
std::chrono::duration_cast<std::chrono::microseconds>( std::chrono::duration_cast<std::chrono::microseconds>(
now.time_since_epoch()) % now.time_since_epoch()) %
std::chrono::seconds(1); std::chrono::seconds(1);
std::ostringstream output; std::ostringstream output;
output << std::put_time(utc_time_ptr, "%Y-%m-%dT%H-%M-%S"); output << std::put_time(utc_time_ptr, "%Y-%m-%dT%H-%M-%S");
output << '.' << std::setw(kFractionalSecondWidth) << std::setfill('0') output << '.' << std::setw(kFractionalSecondWidth) << std::setfill('0')
<< fractional_seconds.count() << 'Z'; << fractional_seconds.count() << 'Z';
return output.str(); return output.str();
} }
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATETIME_DATE_TIME_PROVIDER_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATETIME_DATE_TIME_PROVIDER_H_

38
tooling/pipeline/includes/services/datetime/timer.h
View File

@@ -8,28 +8,28 @@
* @brief Simple timer utility for measuring elapsed time. * @brief Simple timer utility for measuring elapsed time.
*/ */
class Timer { class Timer {
std::chrono::steady_clock::time_point start_time = std::chrono::steady_clock::time_point start_time =
std::chrono::steady_clock::now(); std::chrono::steady_clock::now();
public: public:
Timer(const Timer&) = delete; Timer(const Timer&) = delete;
Timer& operator=(const Timer&) = delete; Timer& operator=(const Timer&) = delete;
Timer(Timer&&) = delete; Timer(Timer&&) = delete;
Timer& operator=(Timer&&) = delete; Timer& operator=(Timer&&) = delete;
Timer() = default; Timer() = default;
~Timer() = default; ~Timer() = default;
[[nodiscard]] int64_t Elapsed() const { [[nodiscard]] int64_t Elapsed() const {
return std::chrono::duration_cast<std::chrono::milliseconds>( return std::chrono::duration_cast<std::chrono::milliseconds>(
std::chrono::steady_clock::now() - start_time) std::chrono::steady_clock::now() - start_time)
.count(); .count();
} }
[[nodiscard]] int64_t Reset() { [[nodiscard]] int64_t Reset() {
auto previous_elapsed = Elapsed(); auto previous_elapsed = Elapsed();
start_time = std::chrono::steady_clock::now(); start_time = std::chrono::steady_clock::now();
return previous_elapsed; return previous_elapsed;
} }
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATETIME_TIMER_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATETIME_TIMER_H_

18
tooling/pipeline/includes/services/enrichment/enrichment_service.h
View File

@@ -14,16 +14,16 @@
* @brief Interface for services that can enrich a location with context. * @brief Interface for services that can enrich a location with context.
*/ */
class IEnrichmentService { class IEnrichmentService {
public: public:
virtual ~IEnrichmentService() = default; virtual ~IEnrichmentService() = default;
/** /**
* @brief Resolves contextual enrichment for a location. * @brief Resolves contextual enrichment for a location.
* *
* @param loc City to enrich. * @param loc City to enrich.
* @return Context text, or an empty string if unavailable. * @return Context text, or an empty string if unavailable.
*/ */
virtual std::string GetLocationContext(const City& loc) = 0; virtual std::string GetLocationContext(const City& loc) = 0;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_ENRICHMENT_SERVICE_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_ENRICHMENT_SERVICE_H_

6
tooling/pipeline/includes/services/enrichment/mock_enrichment.h
View File

@@ -14,9 +14,7 @@
* @brief Enrichment service that returns no context for any location. * @brief Enrichment service that returns no context for any location.
*/ */
class MockEnrichmentService final : public IEnrichmentService { class MockEnrichmentService final : public IEnrichmentService {
public: public:
std::string GetLocationContext(const City& /*loc*/) override { std::string GetLocationContext(const City& /*loc*/) override { return {}; }
return {};
}
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_MOCK_ENRICHMENT_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_MOCK_ENRICHMENT_H_

44
tooling/pipeline/includes/services/enrichment/wikipedia_service.h
View File

@@ -19,30 +19,30 @@
* @brief Provides Wikipedia summary lookups backed by cached raw extracts. * @brief Provides Wikipedia summary lookups backed by cached raw extracts.
*/ */
class WikipediaEnrichmentService final : public IEnrichmentService { class WikipediaEnrichmentService final : public IEnrichmentService {
public: public:
/** /**
* @brief Creates a new Wikipedia service with the provided web client. * @brief Creates a new Wikipedia service with the provided web client.
*/ */
explicit WikipediaEnrichmentService(std::unique_ptr<WebClient> client, explicit WikipediaEnrichmentService(std::unique_ptr<WebClient> client,
std::shared_ptr<ILogger> logger); std::shared_ptr<ILogger> logger);
/** /**
* @brief Returns the Wikipedia-derived context for a location. * @brief Returns the Wikipedia-derived context for a location.
*/ */
[[nodiscard]] std::string GetLocationContext(const City& loc) override; [[nodiscard]] std::string GetLocationContext(const City& loc) override;
private: private:
std::string FetchExtract(std::string_view query); std::string FetchExtract(std::string_view query);
std::unique_ptr<WebClient> client_; std::unique_ptr<WebClient> client_;
std::shared_ptr<ILogger> logger_; std::shared_ptr<ILogger> logger_;
/** /**
* @brief Cache for raw Wikipedia query extracts, keyed by query string. * @brief Cache for raw Wikipedia query extracts, keyed by query string.
* *
* GetLocationContext() always queries "brewing" and reuses "beer in * GetLocationContext() always queries "brewing" and reuses "beer in
* {country}" for every city in that country, so caching avoids refetching * {country}" for every city in that country, so caching avoids refetching
* the same extract across locations in a run. * the same extract across locations in a run.
*/ */
std::unordered_map<std::string, std::string> extract_cache_; std::unordered_map<std::string, std::string> extract_cache_;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_WIKIPEDIA_SERVICE_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_WIKIPEDIA_SERVICE_H_

44
tooling/pipeline/includes/services/logging/log_dispatcher.h
View File

@@ -22,32 +22,32 @@
* and exit once the channel has been closed and drained. * and exit once the channel has been closed and drained.
*/ */
class LogDispatcher { class LogDispatcher {
public: public:
/** /**
* @brief Construct a log dispatcher. * @brief Construct a log dispatcher.
* *
* @param channel Reference to the bounded channel used for log retrieval. * @param channel Reference to the bounded channel used for log retrieval.
*/ */
explicit LogDispatcher(BoundedChannel<LogEntry>& channel); explicit LogDispatcher(BoundedChannel<LogEntry>& channel);
LogDispatcher(const LogDispatcher&) = delete; LogDispatcher(const LogDispatcher&) = delete;
LogDispatcher& operator=(const LogDispatcher&) = delete; LogDispatcher& operator=(const LogDispatcher&) = delete;
LogDispatcher(LogDispatcher&&) = delete; LogDispatcher(LogDispatcher&&) = delete;
LogDispatcher& operator=(LogDispatcher&&) = delete; LogDispatcher& operator=(LogDispatcher&&) = delete;
~LogDispatcher() = default; ~LogDispatcher() = default;
/** /**
* @brief Drain the channel and forward entries to spdlog. * @brief Drain the channel and forward entries to spdlog.
* *
* Intended to be called once on a dedicated thread. The loop returns after * Intended to be called once on a dedicated thread. The loop returns after
* the channel has been closed and all queued entries have been processed. * the channel has been closed and all queued entries have been processed.
*/ */
void Run(); void Run();
private: private:
BoundedChannel<LogEntry>& channel_; BoundedChannel<LogEntry>& channel_;
static spdlog::level::level_enum ToSpdlogLevel(LogLevel level); static spdlog::level::level_enum ToSpdlogLevel(LogLevel level);
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_DISPATCHER_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_DISPATCHER_H_

150
tooling/pipeline/includes/services/logging/log_entry.h
View File

@@ -20,22 +20,22 @@
* @brief Severity levels supported by the logging infra. * @brief Severity levels supported by the logging infra.
*/ */
enum class LogLevel { enum class LogLevel {
/** /**
* @brief Development/debugging information. * @brief Development/debugging information.
*/ */
Debug, Debug,
/** /**
* @brief General informational messages. * @brief General informational messages.
*/ */
Info, Info,
/** /**
* @brief Warning conditions. * @brief Warning conditions.
*/ */
Warn, Warn,
/** /**
* @brief Error conditions. * @brief Error conditions.
*/ */
Error, Error,
}; };
/** /**
@@ -46,38 +46,38 @@ enum class LogLevel {
* pipeline that emitted it. * pipeline that emitted it.
*/ */
enum class PipelinePhase { enum class PipelinePhase {
/** /**
* @brief Initialization and validation. * @brief Initialization and validation.
*/ */
Startup, Startup,
/** /**
* @brief City/context enrichment (e.g. Wikipedia lookups). * @brief City/context enrichment (e.g. Wikipedia lookups).
*/ */
Enrichment, Enrichment,
/** /**
* @brief User profile generation. * @brief User profile generation.
*/ */
UserGeneration, UserGeneration,
/** /**
* @brief Brewery and beer data generation. * @brief Brewery and beer data generation.
*/ */
BreweryAndBeerGeneration, BreweryAndBeerGeneration,
/** /**
* @brief Checkin (visit) record generation. * @brief Checkin (visit) record generation.
*/ */
CheckinGeneration, CheckinGeneration,
/** /**
* @brief Rating and review generation. * @brief Rating and review generation.
*/ */
RatingGeneration, RatingGeneration,
/** /**
* @brief Follow relationship generation. * @brief Follow relationship generation.
*/ */
FollowGeneration, FollowGeneration,
/** /**
* @brief Finalization and cleanup. * @brief Finalization and cleanup.
*/ */
Teardown, Teardown,
}; };
/** /**
@@ -86,9 +86,9 @@ enum class PipelinePhase {
* transparently. * transparently.
*/ */
struct LogDTO { struct LogDTO {
LogLevel level; LogLevel level;
PipelinePhase phase; PipelinePhase phase;
std::string message; std::string message;
}; };
/** /**
@@ -102,35 +102,35 @@ struct LogDTO {
* before the entry is dispatched. * before the entry is dispatched.
*/ */
struct LogEntry { struct LogEntry {
/** /**
* @brief Timestamp when the entry was created. * @brief Timestamp when the entry was created.
*/ */
std::chrono::system_clock::time_point timestamp{}; std::chrono::system_clock::time_point timestamp{};
/** /**
* @brief Source location where the log call was made. * @brief Source location where the log call was made.
*/ */
std::source_location origin{}; std::source_location origin{};
/** /**
* @brief Thread responsible for emitting the log. * @brief Thread responsible for emitting the log.
*/ */
std::thread::id thread_id{}; std::thread::id thread_id{};
/** /**
* @brief Severity level of this entry. * @brief Severity level of this entry.
*/ */
LogLevel level; LogLevel level;
/** /**
* @brief Pipeline phase associated with the entry. * @brief Pipeline phase associated with the entry.
*/ */
PipelinePhase phase; PipelinePhase phase;
/** /**
* @brief Log message text. * @brief Log message text.
*/ */
std::string message; std::string message;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_ENTRY_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_ENTRY_H_

42
tooling/pipeline/includes/services/logging/log_producer.h
View File

@@ -23,31 +23,31 @@
* remain valid for the lifetime of the producer. * remain valid for the lifetime of the producer.
*/ */
class LogProducer final : public ILogger { class LogProducer final : public ILogger {
public: public:
/** /**
* @brief Construct a channel-backed producer. * @brief Construct a channel-backed producer.
* *
* @param channel Reference to the bounded channel used for log transfer. * @param channel Reference to the bounded channel used for log transfer.
*/ */
explicit LogProducer(BoundedChannel<LogEntry>& channel); explicit LogProducer(BoundedChannel<LogEntry>& channel);
LogProducer(const LogProducer&) = delete; LogProducer(const LogProducer&) = delete;
LogProducer& operator=(const LogProducer&) = delete; LogProducer& operator=(const LogProducer&) = delete;
LogProducer(LogProducer&&) = delete; LogProducer(LogProducer&&) = delete;
LogProducer& operator=(LogProducer&&) = delete; LogProducer& operator=(LogProducer&&) = delete;
~LogProducer() override = default; ~LogProducer() override = default;
/** /**
* @brief Queue a log message for asynchronous processing. * @brief Queue a log message for asynchronous processing.
* *
* Blocks while the channel applies backpressure. This blocking behavior * Blocks while the channel applies backpressure. This blocking behavior
* under heavy load is an accepted trade-off for simplicity. * under heavy load is an accepted trade-off for simplicity.
*/ */
void DoLog(LogEntry log_entry) override; void DoLog(LogEntry log_entry) override;
private: private:
BoundedChannel<LogEntry>& channel_; BoundedChannel<LogEntry>& channel_;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_PRODUCER_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_PRODUCER_H_

70
tooling/pipeline/includes/services/logging/logger.h
View File

@@ -23,43 +23,43 @@
* by the composition root and injected into services that emit diagnostics. * by the composition root and injected into services that emit diagnostics.
*/ */
class ILogger { class ILogger {
public: public:
ILogger() = default; ILogger() = default;
ILogger(const ILogger&) = delete; ILogger(const ILogger&) = delete;
ILogger& operator=(const ILogger&) = delete; ILogger& operator=(const ILogger&) = delete;
ILogger(ILogger&&) = delete; ILogger(ILogger&&) = delete;
ILogger& operator=(ILogger&&) = delete; ILogger& operator=(ILogger&&) = delete;
virtual ~ILogger() = default; virtual ~ILogger() = default;
/** /**
* @brief Submit a log message to the logging subsystem. * @brief Submit a log message to the logging subsystem.
* *
* @param payload User-provided log data (level, phase, message). * @param payload User-provided log data (level, phase, message).
* @param origin Auto-captured source location of the call site. * @param origin Auto-captured source location of the call site.
*/ */
void Log(LogDTO payload, void Log(LogDTO payload,
std::source_location origin = std::source_location::current(), std::source_location origin = std::source_location::current(),
std::chrono::system_clock::time_point timestamp = std::chrono::system_clock::time_point timestamp =
std::chrono::system_clock::now(), std::chrono::system_clock::now(),
std::thread::id thread_id = std::this_thread::get_id()) { std::thread::id thread_id = std::this_thread::get_id()) {
LogEntry entry; LogEntry entry;
entry.timestamp = timestamp; entry.timestamp = timestamp;
entry.thread_id = thread_id; entry.thread_id = thread_id;
entry.level = payload.level; entry.level = payload.level;
entry.phase = payload.phase; entry.phase = payload.phase;
entry.message = std::move(payload.message); entry.message = std::move(payload.message);
entry.origin = origin; entry.origin = origin;
DoLog(std::move(entry)); DoLog(std::move(entry));
} }
protected: protected:
/** /**
* @brief Underlying implementation to transport the log entry. * @brief Underlying implementation to transport the log entry.
* *
* Implementations must be thread-safe as DoLog can be called concurrently * Implementations must be thread-safe as DoLog can be called concurrently
* from multiple worker threads. * from multiple worker threads.
*/ */
virtual void DoLog(LogEntry log_entry) = 0; virtual void DoLog(LogEntry log_entry) = 0;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOGGER_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOGGER_H_

93
tooling/pipeline/includes/services/postal_code/mock_postal_code_service.h
View File

@@ -24,56 +24,57 @@
* example that fails validation, throws so the caller can skip the city. * example that fails validation, throws so the caller can skip the city.
*/ */
class MockPostalCodeService final : public IPostalCodeService { class MockPostalCodeService final : public IPostalCodeService {
public: public:
std::string GeneratePostalCode(const City& city) override { std::string GeneratePostalCode(const City& city) override {
const PostalCodeSpec& spec = city.postal_code; const PostalCodeSpec& spec = city.postal_code;
if (spec.examples.empty()) { if (spec.examples.empty()) {
throw std::runtime_error(std::format( throw std::runtime_error(std::format(
"No postal-code examples available for city '{}'", city.city)); "No postal-code examples available for city '{}'", city.city));
}
const std::string& candidate = spec.examples.front();
if (!MatchesAnyPattern(candidate, spec)) {
throw std::runtime_error(
std::format("Postal code '{}' for city '{}' does not match the "
"curated postal-code regex",
candidate, city.city));
}
return candidate;
}
private:
/**
* @brief Returns true if @p candidate matches any city regex, or the
* country format regex when no city regex matches. When the spec carries no
* usable regex at all, validation is skipped (returns true).
*/
static bool MatchesAnyPattern(const std::string& candidate,
const PostalCodeSpec& spec) {
bool has_pattern = false;
for (const std::string& pattern : spec.city_regexes) {
if (pattern.empty()) {
continue;
} }
has_pattern = true;
if (std::regex_match(candidate, std::regex(pattern))) {
return true;
}
}
if (!spec.country_format_regex.empty()) { const std::string& candidate = spec.examples.front();
has_pattern = true;
if (std::regex_match(candidate, std::regex(spec.country_format_regex))) {
return true;
}
}
return !has_pattern; if (!MatchesAnyPattern(candidate, spec)) {
} throw std::runtime_error(
std::format("Postal code '{}' for city '{}' does not match the "
"curated postal-code regex",
candidate, city.city));
}
return candidate;
}
private:
/**
* @brief Returns true if @p candidate matches any city regex, or the
* country format regex when no city regex matches. When the spec carries no
* usable regex at all, validation is skipped (returns true).
*/
static bool MatchesAnyPattern(const std::string& candidate,
const PostalCodeSpec& spec) {
bool has_pattern = false;
for (const std::string& pattern : spec.city_regexes) {
if (pattern.empty()) {
continue;
}
has_pattern = true;
if (std::regex_match(candidate, std::regex(pattern))) {
return true;
}
}
if (!spec.country_format_regex.empty()) {
has_pattern = true;
if (std::regex_match(candidate,
std::regex(spec.country_format_regex))) {
return true;
}
}
return !has_pattern;
}
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_POSTAL_CODE_MOCK_POSTAL_CODE_SERVICE_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_POSTAL_CODE_MOCK_POSTAL_CODE_SERVICE_H_

28
tooling/pipeline/includes/services/postal_code/postal_code_service.h
View File

@@ -14,22 +14,22 @@
* @brief Interface for services that resolve a postal code for a city. * @brief Interface for services that resolve a postal code for a city.
*/ */
class IPostalCodeService { class IPostalCodeService {
public: public:
IPostalCodeService() = default; IPostalCodeService() = default;
virtual ~IPostalCodeService() = default; virtual ~IPostalCodeService() = default;
IPostalCodeService(const IPostalCodeService&) = delete; IPostalCodeService(const IPostalCodeService&) = delete;
IPostalCodeService& operator=(const IPostalCodeService&) = delete; IPostalCodeService& operator=(const IPostalCodeService&) = delete;
IPostalCodeService(IPostalCodeService&&) = delete; IPostalCodeService(IPostalCodeService&&) = delete;
IPostalCodeService& operator=(IPostalCodeService&&) = delete; IPostalCodeService& operator=(IPostalCodeService&&) = delete;
/** /**
* @brief Resolves a postal code for @p city. * @brief Resolves a postal code for @p city.
* *
* @param city City whose curated postal-code examples supply the result. * @param city City whose curated postal-code examples supply the result.
* @return A postal code string, or an empty string if none is available. * @return A postal code string, or an empty string if none is available.
*/ */
virtual std::string GeneratePostalCode(const City& city) = 0; virtual std::string GeneratePostalCode(const City& city) = 0;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_POSTAL_CODE_POSTAL_CODE_SERVICE_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_POSTAL_CODE_POSTAL_CODE_SERVICE_H_

10
tooling/pipeline/includes/services/postal_code/xeger_postal_code_service.h
View File

@@ -20,13 +20,13 @@
* via GenerateStringFromRegex(). * via GenerateStringFromRegex().
*/ */
class XegerPostalCodeService final : public IPostalCodeService { class XegerPostalCodeService final : public IPostalCodeService {
public: public:
XegerPostalCodeService(); XegerPostalCodeService();
std::string GeneratePostalCode(const City& city) override; std::string GeneratePostalCode(const City& city) override;
private: private:
std::mt19937 rng_; std::mt19937 rng_;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_POSTAL_CODE_XEGER_POSTAL_CODE_SERVICE_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_POSTAL_CODE_XEGER_POSTAL_CODE_SERVICE_H_

80
tooling/pipeline/includes/services/prompting/prompt_directory.h
View File

@@ -24,22 +24,22 @@
* @brief Interface for loading named prompt files. * @brief Interface for loading named prompt files.
*/ */
class IPromptDirectory { class IPromptDirectory {
public: public:
IPromptDirectory() = default; IPromptDirectory() = default;
IPromptDirectory(const IPromptDirectory&) = delete; IPromptDirectory(const IPromptDirectory&) = delete;
IPromptDirectory& operator=(const IPromptDirectory&) = delete; IPromptDirectory& operator=(const IPromptDirectory&) = delete;
IPromptDirectory(IPromptDirectory&&) = delete; IPromptDirectory(IPromptDirectory&&) = delete;
IPromptDirectory& operator=(IPromptDirectory&&) = delete; IPromptDirectory& operator=(IPromptDirectory&&) = delete;
virtual ~IPromptDirectory() = default; virtual ~IPromptDirectory() = default;
/** /**
* @brief Loads the prompt associated with @p key. * @brief Loads the prompt associated with @p key.
* *
* @param key Logical prompt key, e.g. "BREWERY_GENERATION". * @param key Logical prompt key, e.g. "BREWERY_GENERATION".
* @return Prompt text. * @return Prompt text.
* @throws std::runtime_error if the prompt file cannot be found or read. * @throws std::runtime_error if the prompt file cannot be found or read.
*/ */
[[nodiscard]] virtual std::string Load(std::string_view key) = 0; [[nodiscard]] virtual std::string Load(std::string_view key) = 0;
}; };
/** /**
@@ -50,33 +50,33 @@ class IPromptDirectory {
* at construction time; individual file absence is reported lazily at Load(). * at construction time; individual file absence is reported lazily at Load().
*/ */
class PromptDirectory final : public IPromptDirectory { class PromptDirectory final : public IPromptDirectory {
public: public:
/** /**
* @brief Constructs a PromptDirectory rooted at @p prompt_dir. * @brief Constructs a PromptDirectory rooted at @p prompt_dir.
* *
* @param prompt_dir Absolute or relative path to the prompt directory. * @param prompt_dir Absolute or relative path to the prompt directory.
* @throws std::runtime_error if @p prompt_dir does not exist or is not a * @throws std::runtime_error if @p prompt_dir does not exist or is not a
* directory. * directory.
*/ */
explicit PromptDirectory(const std::filesystem::path& prompt_dir); explicit PromptDirectory(const std::filesystem::path& prompt_dir);
PromptDirectory(const std::filesystem::path& prompt_dir, PromptDirectory(const std::filesystem::path& prompt_dir,
std::shared_ptr<ILogger> logger); std::shared_ptr<ILogger> logger);
/** /**
* @brief Loads the prompt for @p key, caching the result. * @brief Loads the prompt for @p key, caching the result.
* *
* Maps @p key → <prompt_dir>/<key>.md. * Maps @p key → <prompt_dir>/<key>.md.
* *
* @param key Logical prompt key. * @param key Logical prompt key.
* @return Prompt text. * @return Prompt text.
* @throws std::runtime_error if the file does not exist or is empty. * @throws std::runtime_error if the file does not exist or is empty.
*/ */
[[nodiscard]] std::string Load(std::string_view key) override; [[nodiscard]] std::string Load(std::string_view key) override;
private: private:
std::filesystem::path prompt_dir_; std::filesystem::path prompt_dir_;
std::shared_ptr<ILogger> logger_; std::shared_ptr<ILogger> logger_;
std::unordered_map<std::string, std::string> cache_; std::unordered_map<std::string, std::string> cache_;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_PROMPTING_PROMPT_DIRECTORY_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_PROMPTING_PROMPT_DIRECTORY_H_

134
tooling/pipeline/includes/services/xeger/xeger_internal.h
View File

@@ -55,33 +55,33 @@ struct Alternation;
struct Repeat; struct Repeat;
struct Visitor { struct Visitor {
virtual void Visit(const Literal&) = 0; virtual void Visit(const Literal&) = 0;
virtual void Visit(const AnyChar&) = 0; virtual void Visit(const AnyChar&) = 0;
virtual void Visit(const CharClass&) = 0; virtual void Visit(const CharClass&) = 0;
virtual void Visit(const Concat&) = 0; virtual void Visit(const Concat&) = 0;
virtual void Visit(const Alternation&) = 0; virtual void Visit(const Alternation&) = 0;
virtual void Visit(const Repeat&) = 0; virtual void Visit(const Repeat&) = 0;
virtual ~Visitor() = default; virtual ~Visitor() = default;
}; };
struct Node { struct Node {
virtual void Accept(Visitor&) const = 0; virtual void Accept(Visitor&) const = 0;
virtual ~Node() = default; virtual ~Node() = default;
}; };
using NodePtr = std::unique_ptr<Node>; using NodePtr = std::unique_ptr<Node>;
template <typename Derived> template <typename Derived>
struct Visitable : Node { struct Visitable : Node {
void Accept(Visitor& visitor) const override { void Accept(Visitor& visitor) const override {
visitor.Visit(static_cast<const Derived&>(*this)); visitor.Visit(static_cast<const Derived&>(*this));
} }
}; };
// One fixed character, e.g. 'a'. // One fixed character, e.g. 'a'.
struct Literal : Visitable<Literal> { struct Literal : Visitable<Literal> {
char ch; char ch;
explicit Literal(const char c) : ch(c) {} explicit Literal(const char c) : ch(c) {}
}; };
// The "." wildcard: any single printable character. // The "." wildcard: any single printable character.
@@ -91,18 +91,18 @@ struct AnyChar : Visitable<AnyChar> {};
// character 'x' is the range {'x', 'x'}. `negated` flips the meaning to // character 'x' is the range {'x', 'x'}. `negated` flips the meaning to
// "any character NOT listed" (i.e. `[^...]`). // "any character NOT listed" (i.e. `[^...]`).
struct CharClass : Visitable<CharClass> { struct CharClass : Visitable<CharClass> {
std::vector<std::pair<char, char>> ranges; std::vector<std::pair<char, char>> ranges;
bool negated = false; bool negated = false;
}; };
// "Do these children one after another." // "Do these children one after another."
struct Concat : Visitable<Concat> { struct Concat : Visitable<Concat> {
std::vector<NodePtr> parts; std::vector<NodePtr> parts;
}; };
// "Pick exactly one of these options at random" -- the `|` operator. // "Pick exactly one of these options at random" -- the `|` operator.
struct Alternation : Visitable<Alternation> { struct Alternation : Visitable<Alternation> {
std::vector<NodePtr> options; std::vector<NodePtr> options;
}; };
// "Repeat one child between min and max times." // "Repeat one child between min and max times."
@@ -111,9 +111,9 @@ struct Alternation : Visitable<Alternation> {
// a? -> min 0, max 1 // a? -> min 0, max 1
// a{2,5} -> min 2, max 5 // a{2,5} -> min 2, max 5
struct Repeat : Visitable<Repeat> { struct Repeat : Visitable<Repeat> {
NodePtr child; NodePtr child;
unsigned int min = 0; unsigned int min = 0;
std::optional<unsigned int> max; // std::nullopt means "no upper bound" std::optional<unsigned int> max; // std::nullopt means "no upper bound"
}; };
// Parser (recursive descent) // Parser (recursive descent)
@@ -125,75 +125,75 @@ struct Repeat : Visitable<Repeat> {
// ParseAtom ::= a . [..] (..) \d // ParseAtom ::= a . [..] (..) \d
class Parser { class Parser {
public: public:
explicit Parser(std::string_view pattern); explicit Parser(std::string_view pattern);
NodePtr Parse(); NodePtr Parse();
private: private:
std::string src_; std::string src_;
size_t pos_ = 0; size_t pos_ = 0;
// Strips anchors (leading '^' and trailing '$') as they are not needed for // Strips anchors (leading '^' and trailing '$') as they are not needed for
// generation. // generation.
static std::string StripAnchors(std::string_view pattern); static std::string StripAnchors(std::string_view pattern);
[[nodiscard]] bool AtEnd() const; [[nodiscard]] bool AtEnd() const;
[[nodiscard]] char Peek() const; [[nodiscard]] char Peek() const;
char Advance(); char Advance();
bool Match(char c); bool Match(char c);
void Expect(char c); void Expect(char c);
// Level 1: alternation -- ( concat | concat | concat ) // Level 1: alternation -- ( concat | concat | concat )
NodePtr ParseAlternation(); NodePtr ParseAlternation();
// Level 2: concatenation -- a run of items back to back // Level 2: concatenation -- a run of items back to back
NodePtr ParseConcat(); NodePtr ParseConcat();
// Level 3: an atom plus an optional quantifier // Level 3: an atom plus an optional quantifier
NodePtr ParseRepeat(); NodePtr ParseRepeat();
// Curly-brace quantifiers: {n}, {n,}, {n,m} // Curly-brace quantifiers: {n}, {n,}, {n,m}
NodePtr ParseBrace(NodePtr atom); NodePtr ParseBrace(NodePtr atom);
// Level 4: an atom -- the smallest standalone unit // Level 4: an atom -- the smallest standalone unit
NodePtr ParseAtom(); NodePtr ParseAtom();
static NodePtr MakeClass(std::vector<std::pair<char, char>> ranges, static NodePtr MakeClass(std::vector<std::pair<char, char>> ranges,
bool negated); bool negated);
NodePtr ParseEscape(); NodePtr ParseEscape();
NodePtr ParseCharClass(); NodePtr ParseCharClass();
// Reads one character inside [ ], turning \] \n etc. into the real // Reads one character inside [ ], turning \] \n etc. into the real
// character they represent. // character they represent.
char ClassChar(); char ClassChar();
}; };
// Generator -- a Visitor that emits a random matching string // Generator -- a Visitor that emits a random matching string
class XegerGenerator final : public Visitor { class XegerGenerator final : public Visitor {
public: public:
XegerGenerator(std::mt19937& rng, int unbounded_cap); XegerGenerator(std::mt19937& rng, int unbounded_cap);
std::string Run(const Node& root); std::string Run(const Node& root);
void Visit(const Literal& node) override; void Visit(const Literal& node) override;
void Visit(const AnyChar&) override; void Visit(const AnyChar&) override;
void Visit(const CharClass& node) override; void Visit(const CharClass& node) override;
void Visit(const Concat& node) override; void Visit(const Concat& node) override;
void Visit(const Alternation& node) override; void Visit(const Alternation& node) override;
void Visit(const Repeat& node) override; void Visit(const Repeat& node) override;
private: private:
std::mt19937& rng_; std::mt19937& rng_;
int unbounded_cap_; int unbounded_cap_;
std::string out_; std::string out_;
static bool InRanges(const CharClass& node, char c); static bool InRanges(const CharClass& node, char c);
char PickFromClass(const CharClass& node); char PickFromClass(const CharClass& node);
}; };
} // namespace xeger_internal } // namespace xeger_internal

44
tooling/pipeline/includes/web_client/http_web_client.h
View File

@@ -25,31 +25,31 @@
* bound to a single origin at construction time. * bound to a single origin at construction time.
*/ */
class HttpWebClient final : public WebClient { class HttpWebClient final : public WebClient {
public: public:
explicit HttpWebClient(std::shared_ptr<ILogger> logger) explicit HttpWebClient(std::shared_ptr<ILogger> logger)
: logger_(std::move(logger)) {} : logger_(std::move(logger)) {}
~HttpWebClient() override = default; ~HttpWebClient() override = default;
/** /**
* @brief Executes a blocking HTTP/HTTPS GET request against a full URL. * @brief Executes a blocking HTTP/HTTPS GET request against a full URL.
* *
* @param url Fully-qualified URL, e.g. * @param url Fully-qualified URL, e.g.
* "https://en.wikipedia.org/api/rest_v1/page/summary/Berlin" * "https://en.wikipedia.org/api/rest_v1/page/summary/Berlin"
* @return Response body on HTTP 2xx; throws std::runtime_error otherwise. * @return Response body on HTTP 2xx; throws std::runtime_error otherwise.
*/ */
std::string Get(const std::string& url) override; std::string Get(const std::string& url) override;
/** /**
* @brief Percent-encodes a single URI component (query parameter value or * @brief Percent-encodes a single URI component (query parameter value or
* path segment). Delegates to httplib::encode_uri_component(). * path segment). Delegates to httplib::encode_uri_component().
* *
* @param value Raw string to encode. * @param value Raw string to encode.
* @return Percent-encoded string safe for use in a URL. * @return Percent-encoded string safe for use in a URL.
*/ */
std::string EncodeURL(const std::string& value) override; std::string EncodeURL(const std::string& value) override;
private: private:
std::shared_ptr<ILogger> logger_; std::shared_ptr<ILogger> logger_;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_WEB_CLIENT_HTTP_WEB_CLIENT_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_WEB_CLIENT_HTTP_WEB_CLIENT_H_

32
tooling/pipeline/includes/web_client/web_client.h
View File

@@ -12,24 +12,24 @@
* @brief Abstract web client interface. * @brief Abstract web client interface.
*/ */
class WebClient { class WebClient {
public: public:
virtual ~WebClient() = default; virtual ~WebClient() = default;
/** /**
* @brief Executes an HTTP GET request. * @brief Executes an HTTP GET request.
* *
* @param url Request URL. * @param url Request URL.
* @return Response body. * @return Response body.
*/ */
virtual std::string Get(const std::string& url) = 0; virtual std::string Get(const std::string& url) = 0;
/** /**
* @brief URL-encodes a string value. * @brief URL-encodes a string value.
* *
* @param value Raw string value. * @param value Raw string value.
* @return Encoded value safe for URL usage. * @return Encoded value safe for URL usage.
*/ */
virtual std::string EncodeURL(const std::string& value) = 0; virtual std::string EncodeURL(const std::string& value) = 0;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_WEB_CLIENT_WEB_CLIENT_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_WEB_CLIENT_WEB_CLIENT_H_

42
tooling/pipeline/personas.json
View File

@@ -2,7 +2,12 @@
{ {
"name": "Hophead Explorer", "name": "Hophead Explorer",
"description": "Chases the newest hop varieties and double-digit IBUs, tracking hazy releases and limited tap takeovers across town.", "description": "Chases the newest hop varieties and double-digit IBUs, tracking hazy releases and limited tap takeovers across town.",
"style_affinities": ["American IPA", "Double IPA", "New England IPA", "Black IPA"] "style_affinities": [
"American IPA",
"Double IPA",
"New England IPA",
"Black IPA"
]
}, },
{ {
"name": "Malt & Tradition Loyalist", "name": "Malt & Tradition Loyalist",
@@ -12,7 +17,12 @@
{ {
"name": "Dark & Roasty Devotee", "name": "Dark & Roasty Devotee",
"description": "Orders the darkest beer on the board year-round, drawn to roasted malt, coffee, and chocolate notes over carbonated lightness.", "description": "Orders the darkest beer on the board year-round, drawn to roasted malt, coffee, and chocolate notes over carbonated lightness.",
"style_affinities": ["Imperial Stout", "Robust Porter", "Baltic Porter", "Oatmeal Stout"] "style_affinities": [
"Imperial Stout",
"Robust Porter",
"Baltic Porter",
"Oatmeal Stout"
]
}, },
{ {
"name": "Sour & Wild Forager", "name": "Sour & Wild Forager",
@@ -22,21 +32,41 @@
{ {
"name": "Session Sipper", "name": "Session Sipper",
"description": "Values an easy-drinking pint that holds up over a long afternoon at the taproom more than raw intensity or novelty.", "description": "Values an easy-drinking pint that holds up over a long afternoon at the taproom more than raw intensity or novelty.",
"style_affinities": ["Session IPA", "Ordinary Bitter", "Cream Ale", "Blonde Ale"] "style_affinities": [
"Session IPA",
"Ordinary Bitter",
"Cream Ale",
"Blonde Ale"
]
}, },
{ {
"name": "Belgian Abbey Pilgrim", "name": "Belgian Abbey Pilgrim",
"description": "Collects abbey and trappist releases, drawn to the dried-fruit esters and warming strength of classic Belgian ales.", "description": "Collects abbey and trappist releases, drawn to the dried-fruit esters and warming strength of classic Belgian ales.",
"style_affinities": ["Belgian Tripel", "Belgian Dubbel", "Belgian Quadrupel", "Trappist Single"] "style_affinities": [
"Belgian Tripel",
"Belgian Dubbel",
"Belgian Quadrupel",
"Trappist Single"
]
}, },
{ {
"name": "Wheat & Citrus Casual", "name": "Wheat & Citrus Casual",
"description": "Reaches for something light, cloudy, and refreshing, usually with a citrus or coriander note, especially on a patio in summer.", "description": "Reaches for something light, cloudy, and refreshing, usually with a citrus or coriander note, especially on a patio in summer.",
"style_affinities": ["Hefeweizen", "Witbier", "American Wheat Beer", "Berliner Weisse"] "style_affinities": [
"Hefeweizen",
"Witbier",
"American Wheat Beer",
"Berliner Weisse"
]
}, },
{ {
"name": "Big & Boozy Connoisseur", "name": "Big & Boozy Connoisseur",
"description": "Sips slowly from a snifter, seeking out high-ABV releases meant for aging and savoring rather than quick refreshment.", "description": "Sips slowly from a snifter, seeking out high-ABV releases meant for aging and savoring rather than quick refreshment.",
"style_affinities": ["English Barleywine", "Wee Heavy", "Doppelbock", "Old Ale"] "style_affinities": [
"English Barleywine",
"Wee Heavy",
"Doppelbock",
"Old Ale"
]
} }
] ]

50
tooling/pipeline/prompts/USER_GENERATION.md
View File

@@ -41,11 +41,11 @@ You will receive the inputs like this:
ABSOLUTELY NO MARKDOWN FORMATTING. Do NOT wrap your response in json or ``` ABSOLUTELY NO MARKDOWN FORMATTING. Do NOT wrap your response in json or ```
blocks. blocks.
Do not add markdown, code fences, or postscript around the final JSON object. Do not add markdown, code fences, or postscript around the final JSON object. Do
Do not say "Here is the JSON" or "Enjoy!". not say "Here is the JSON" or "Enjoy!".
The JSON must contain exactly three keys ("username", "bio", The JSON must contain exactly three keys ("username", "bio", "activity_weight")
"activity_weight") in that order. Do not rename or add any other keys. in that order. Do not rename or add any other keys.
ESCAPE ALL QUOTES inside the bio field using \", or use single quotes (' ') ESCAPE ALL QUOTES inside the bio field using \", or use single quotes (' ')
instead. instead.
@@ -53,15 +53,15 @@ instead.
DO NOT use actual line breaks (\n) inside any string. Keep the bio as one DO NOT use actual line breaks (\n) inside any string. Keep the bio as one
continuous string. continuous string.
The bio must be between 25 and 60 words, written in first person, and must The bio must be between 25 and 60 words, written in first person, and must read
read as something this specific person would write about themselves -- not a as something this specific person would write about themselves -- not a generic
generic profile blurb. profile blurb.
`activity_weight` is a number between 0 and 1 (inclusive) representing how `activity_weight` is a number between 0 and 1 (inclusive) representing how often
often this persona checks in at breweries relative to other users. Persona this persona checks in at breweries relative to other users. Persona archetypes
archetypes implying frequent, habitual brewery visits should skew higher; implying frequent, habitual brewery visits should skew higher; casual or
casual or occasional-visitor personas should skew lower. Do not default to occasional-visitor personas should skew lower. Do not default to 0.5 -- vary it
0.5 -- vary it meaningfully based on the persona description. meaningfully based on the persona description.
Expected JSON format: Expected JSON format:
@@ -78,8 +78,8 @@ Expected JSON format:
### THE USERNAME: ### THE USERNAME:
Derive the username from the given first and/or last name (e.g. lowercase, Derive the username from the given first and/or last name (e.g. lowercase,
abbreviated, or combined with a short word or number tied to the persona's abbreviated, or combined with a short word or number tied to the persona's style
style affinities). Do not invent an unrelated handle. Do not include spaces. affinities). Do not invent an unrelated handle. Do not include spaces.
### THE BIO: ### THE BIO:
@@ -90,14 +90,14 @@ city or country naturally if it fits, but do not force it.
### VOICE & PERSPECTIVE: ### VOICE & PERSPECTIVE:
Write in the first person ("I"/"my"). The tone should match the persona: Write in the first person ("I"/"my"). The tone should match the persona: an
an enthusiastic explorer sounds different from a relaxed, easy-drinking enthusiastic explorer sounds different from a relaxed, easy-drinking regular. Do
regular. Do not write in second or third person. not write in second or third person.
### THE BLOCKLIST (FORBIDDEN CONCEPTS): ### THE BLOCKLIST (FORBIDDEN CONCEPTS):
You absolutely cannot use the following words and phrases. Make sure your You absolutely cannot use the following words and phrases. Make sure your final
final output doesn't have any of these: output doesn't have any of these:
- "hidden gem" - "hidden gem"
- "passion" - "passion"
@@ -108,14 +108,14 @@ final output doesn't have any of these:
#### FORBIDDEN WRITING PATTERNS #### FORBIDDEN WRITING PATTERNS
The following patterns are common AI writing pitfalls and must not appear in The following patterns are common AI writing pitfalls and must not appear in the
the bio: bio:
- Negative parallelism constructions: "It's not X, it's Y" - Negative parallelism constructions: "It's not X, it's Y"
- Inflated significance phrases: "stands as a testament," "plays a vital - Inflated significance phrases: "stands as a testament," "plays a vital role,"
role," "deeply rooted" "deeply rooted"
- Superficial trailing analyses: sentences ending in -ing words that add - Superficial trailing analyses: sentences ending in -ing words that add opinion
opinion without content without content
- Promotional travel-copy tone: "breathtaking," "must-visit," "vibrant" - Promotional travel-copy tone: "breathtaking," "must-visit," "vibrant"
- Overused conjunctive transitions used as sentence openers: "Moreover," - Overused conjunctive transitions used as sentence openers: "Moreover,"
"Furthermore," "In addition" "Furthermore," "In addition"

373
tooling/pipeline/src/application_options/parse_arguments.cc
View File

@@ -10,207 +10,212 @@
std::optional<ApplicationOptions> ParseArguments( std::optional<ApplicationOptions> ParseArguments(
const int argc, char** argv, std::shared_ptr<ILogger> logger) { const int argc, char** argv, std::shared_ptr<ILogger> logger) {
prog_opts::options_description desc("Pipeline Options"); prog_opts::options_description desc("Pipeline Options");
auto opt = desc.add_options(); auto opt = desc.add_options();
opt("help,h", "Produce help message"); opt("help,h", "Produce help message");
auto add_sampling_options = [&]() -> void { auto add_sampling_options = [&]() -> void {
const SamplingOptions sampling_defaults{}; const SamplingOptions sampling_defaults{};
opt("temperature", opt("temperature",
prog_opts::value<float>()->default_value(sampling_defaults.temperature), prog_opts::value<float>()->default_value(
"Sampling temperature (higher = more random)"); sampling_defaults.temperature),
opt("top-p", "Sampling temperature (higher = more random)");
prog_opts::value<float>()->default_value(sampling_defaults.top_p), opt("top-p",
"Nucleus sampling top-p in (0,1] (higher = more random)"); prog_opts::value<float>()->default_value(sampling_defaults.top_p),
opt("top-k", "Nucleus sampling top-p in (0,1] (higher = more random)");
prog_opts::value<uint32_t>()->default_value(sampling_defaults.top_k), opt("top-k",
"Top-k sampling parameter (higher = more candidate tokens)"); prog_opts::value<uint32_t>()->default_value(sampling_defaults.top_k),
opt("n-ctx", "Top-k sampling parameter (higher = more candidate tokens)");
prog_opts::value<uint32_t>()->default_value(sampling_defaults.n_ctx), opt("n-ctx",
"Context window size in tokens"); prog_opts::value<uint32_t>()->default_value(sampling_defaults.n_ctx),
opt("seed", prog_opts::value<int>()->default_value(sampling_defaults.seed), "Context window size in tokens");
"Sampler seed: -1 for random, otherwise non-negative integer"); opt("seed",
opt("n-gpu-layers", prog_opts::value<int>()->default_value(0), prog_opts::value<int>()->default_value(sampling_defaults.seed),
"Number of layers to offload to GPU"); "Sampler seed: -1 for random, otherwise non-negative integer");
}; opt("n-gpu-layers", prog_opts::value<int>()->default_value(0),
"Number of layers to offload to GPU");
};
// --mocked and --model are mutually exclusive; validation is enforced below // --mocked and --model are mutually exclusive; validation is enforced below
// rather than at registration to produce a clear diagnostic message. // rather than at registration to produce a clear diagnostic message.
auto add_generator_options = [&]() -> void { auto add_generator_options = [&]() -> void {
opt("mocked", prog_opts::bool_switch(), opt("mocked", prog_opts::bool_switch(),
"Use mocked generator for brewery/user data"); "Use mocked generator for brewery/user data");
opt("model,m", prog_opts::value<std::string>()->default_value(""), opt("model,m", prog_opts::value<std::string>()->default_value(""),
"Path to LLM model (gguf)"); "Path to LLM model (gguf)");
}; };
auto add_pipeline_options = [&]() -> void { auto add_pipeline_options = [&]() -> void {
opt("output,o", prog_opts::value<std::string>()->default_value("output"), opt("output,o", prog_opts::value<std::string>()->default_value("output"),
"Directory for generated artifacts"); "Directory for generated artifacts");
opt("log-path", opt("log-path",
prog_opts::value<std::string>()->default_value("pipeline.log"), prog_opts::value<std::string>()->default_value("pipeline.log"),
"Path for application logs"); "Path for application logs");
opt("prompt-dir", prog_opts::value<std::string>()->default_value(""), opt("prompt-dir", prog_opts::value<std::string>()->default_value(""),
"Directory containing named prompt files (e.g. " "Directory containing named prompt files (e.g. "
"BREWERY_GENERATION.md)." "BREWERY_GENERATION.md)."
" Required when not using --mocked."); " Required when not using --mocked.");
opt("location-count", prog_opts::value<uint32_t>()->default_value(10)); opt("location-count", prog_opts::value<uint32_t>()->default_value(10));
}; };
add_sampling_options(); add_sampling_options();
add_generator_options(); add_generator_options();
add_pipeline_options(); add_pipeline_options();
// No flags provided — treat as a help request rather than an error. // No flags provided — treat as a help request rather than an error.
if (argc == 1) { if (argc == 1) {
const std::string title = "Biergarten Pipeline"; const std::string title = "Biergarten Pipeline";
const std::string usage = ([&] { const std::string usage = ([&] {
std::stringstream usage_stream; std::stringstream usage_stream;
usage_stream << "\nUsage: biergarten-pipeline [options]\n\n" << desc; usage_stream << "\nUsage: biergarten-pipeline [options]\n\n" << desc;
return usage_stream.str(); return usage_stream.str();
})(); })();
if (logger) {
logger->Log(LogDTO{.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = title});
logger->Log(LogDTO{.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = usage});
}
return std::nullopt;
}
try {
prog_opts::variables_map var_map;
prog_opts::store(prog_opts::parse_command_line(argc, argv, desc), var_map);
prog_opts::notify(var_map);
if (var_map.contains("help")) {
std::stringstream help_stream;
help_stream << "\n" << desc;
if (logger) { if (logger) {
logger->Log(LogDTO{.level = LogLevel::Info, logger->Log(LogDTO{.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = help_stream.str()}); .message = title});
logger->Log(LogDTO{.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = usage});
} }
return std::nullopt; return std::nullopt;
} }
ApplicationOptions options; try {
prog_opts::variables_map var_map;
prog_opts::store(prog_opts::parse_command_line(argc, argv, desc),
var_map);
prog_opts::notify(var_map);
options.pipeline.output_path = var_map["output"].as<std::string>(); if (var_map.contains("help")) {
options.pipeline.log_path = var_map["log-path"].as<std::string>(); std::stringstream help_stream;
options.pipeline.prompt_dir = var_map["prompt-dir"].as<std::string>(); help_stream << "\n" << desc;
options.pipeline.location_count = var_map["location-count"].as<uint32_t>(); if (logger) {
logger->Log(LogDTO{.level = LogLevel::Info,
const bool use_mocked = var_map["mocked"].as<bool>(); .phase = PipelinePhase::Startup,
const std::string model_path = var_map["model"].as<std::string>(); .message = help_stream.str()});
const int n_gpu_layers = var_map["n-gpu-layers"].as<int>(); }
return std::nullopt;
// Enforce mutual exclusivity before any further configuration is
// applied.
if (use_mocked && !model_path.empty()) {
const std::string msg =
"Invalid arguments: --mocked and --model are mutually "
"exclusive";
if (logger) {
logger->Log(LogDTO{.level = LogLevel::Error,
.phase = PipelinePhase::Startup,
.message = msg});
} else {
std::cerr << msg << std::endl;
} }
return std::nullopt;
}
if (!use_mocked && model_path.empty()) { ApplicationOptions options;
const std::string msg =
"Invalid arguments: either --mocked or --model must be " options.pipeline.output_path = var_map["output"].as<std::string>();
"specified"; options.pipeline.log_path = var_map["log-path"].as<std::string>();
if (logger) { options.pipeline.prompt_dir = var_map["prompt-dir"].as<std::string>();
logger->Log(LogDTO{.level = LogLevel::Error, options.pipeline.location_count =
.phase = PipelinePhase::Startup, var_map["location-count"].as<uint32_t>();
.message = msg});
} else { const bool use_mocked = var_map["mocked"].as<bool>();
std::cerr << msg << std::endl; const std::string model_path = var_map["model"].as<std::string>();
const int n_gpu_layers = var_map["n-gpu-layers"].as<int>();
// Enforce mutual exclusivity before any further configuration is
// applied.
if (use_mocked && !model_path.empty()) {
const std::string msg =
"Invalid arguments: --mocked and --model are mutually "
"exclusive";
if (logger) {
logger->Log(LogDTO{.level = LogLevel::Error,
.phase = PipelinePhase::Startup,
.message = msg});
} else {
std::cerr << msg << std::endl;
}
return std::nullopt;
} }
return std::nullopt;
}
// Prompt directory is only meaningful for live inference — the mock if (!use_mocked && model_path.empty()) {
// generator has no use for it and should not require it to be present. const std::string msg =
if (!use_mocked && options.pipeline.prompt_dir.empty()) { "Invalid arguments: either --mocked or --model must be "
const std::string msg = "specified";
"Invalid arguments: --prompt-dir is required when not using " if (logger) {
"--mocked"; logger->Log(LogDTO{.level = LogLevel::Error,
if (logger) { .phase = PipelinePhase::Startup,
logger->Log({.level = LogLevel::Error, .message = msg});
.phase = PipelinePhase::Startup, } else {
.message = msg}); std::cerr << msg << std::endl;
} else { }
std::cerr << msg << std::endl; return std::nullopt;
} }
return std::nullopt;
}
options.generator.use_mocked = use_mocked; // Prompt directory is only meaningful for live inference — the mock
options.generator.model_path = model_path; // generator has no use for it and should not require it to be present.
if (!use_mocked && options.pipeline.prompt_dir.empty()) {
// Only populate sampling config when the user explicitly overrides at const std::string msg =
// least one value. Leaving it as std::nullopt lets LlamaGenerator fall "Invalid arguments: --prompt-dir is required when not using "
// back to its own SamplingOptions{} defaults, keeping the two paths "--mocked";
// consistent without redundant copies. if (logger) {
const bool user_provided_sampling = logger->Log({.level = LogLevel::Error,
!var_map["temperature"].defaulted() || !var_map["top-p"].defaulted() ||
!var_map["top-k"].defaulted() || !var_map["n-ctx"].defaulted() ||
!var_map["seed"].defaulted() || !var_map["n_gpu_layers"].defaulted();
if (user_provided_sampling) {
// Warn but do not fail — the run is still valid, the flags are just
// silently irrelevant when no model is loaded.
if (use_mocked) {
const std::string msg =
"Sampling parameters are ignored when using --mocked";
if (logger) {
logger->Log(LogDTO{.level = LogLevel::Warn,
.phase = PipelinePhase::Startup,
.message = msg});
} else {
std::cerr << msg << std::endl;
}
} else {
SamplingOptions sampling;
sampling.temperature = var_map["temperature"].as<float>();
sampling.top_p = var_map["top-p"].as<float>();
sampling.top_k = var_map["top-k"].as<uint32_t>();
sampling.n_ctx = var_map["n-ctx"].as<uint32_t>();
sampling.seed = var_map["seed"].as<int>();
sampling.n_gpu_layers = var_map["n-gpu-layers"].as<int>();
options.generator.sampling = sampling;
}
}
return options;
} catch (const std::exception& exception) {
const std::string msg =
std::string("Failed to parse command-line arguments: ") +
exception.what();
if (logger) {
logger->Log(LogDTO{.level = LogLevel::Error,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = msg}); .message = msg});
} } else {
return std::nullopt; std::cerr << msg << std::endl;
} catch (...) { }
const std::string msg = return std::nullopt;
"Failed to parse command-line arguments: unknown error"; }
if (logger) {
logger->Log(LogDTO{.level = LogLevel::Error, options.generator.use_mocked = use_mocked;
.phase = PipelinePhase::Startup, options.generator.model_path = model_path;
.message = msg});
} // Only populate sampling config when the user explicitly overrides at
return std::nullopt; // least one value. Leaving it as std::nullopt lets LlamaGenerator fall
} // back to its own SamplingOptions{} defaults, keeping the two paths
// consistent without redundant copies.
const bool user_provided_sampling =
!var_map["temperature"].defaulted() ||
!var_map["top-p"].defaulted() || !var_map["top-k"].defaulted() ||
!var_map["n-ctx"].defaulted() || !var_map["seed"].defaulted() ||
!var_map["n_gpu_layers"].defaulted();
if (user_provided_sampling) {
// Warn but do not fail — the run is still valid, the flags are just
// silently irrelevant when no model is loaded.
if (use_mocked) {
const std::string msg =
"Sampling parameters are ignored when using --mocked";
if (logger) {
logger->Log(LogDTO{.level = LogLevel::Warn,
.phase = PipelinePhase::Startup,
.message = msg});
} else {
std::cerr << msg << std::endl;
}
} else {
SamplingOptions sampling;
sampling.temperature = var_map["temperature"].as<float>();
sampling.top_p = var_map["top-p"].as<float>();
sampling.top_k = var_map["top-k"].as<uint32_t>();
sampling.n_ctx = var_map["n-ctx"].as<uint32_t>();
sampling.seed = var_map["seed"].as<int>();
sampling.n_gpu_layers = var_map["n-gpu-layers"].as<int>();
options.generator.sampling = sampling;
}
}
return options;
} catch (const std::exception& exception) {
const std::string msg =
std::string("Failed to parse command-line arguments: ") +
exception.what();
if (logger) {
logger->Log(LogDTO{.level = LogLevel::Error,
.phase = PipelinePhase::Startup,
.message = msg});
}
return std::nullopt;
} catch (...) {
const std::string msg =
"Failed to parse command-line arguments: unknown error";
if (logger) {
logger->Log(LogDTO{.level = LogLevel::Error,
.phase = PipelinePhase::Startup,
.message = msg});
}
return std::nullopt;
}
} }

141
tooling/pipeline/src/biergarten_pipeline_orchestrator/generate_breweries.cc
View File

@@ -12,80 +12,83 @@
void BiergartenPipelineOrchestrator::GenerateBreweries( void BiergartenPipelineOrchestrator::GenerateBreweries(
std::span<const EnrichedCity> cities) { std::span<const EnrichedCity> cities) {
logger_->Log({.level = LogLevel::Info, logger_->Log({.level = LogLevel::Info,
.phase = PipelinePhase::BreweryAndBeerGeneration, .phase = PipelinePhase::BreweryAndBeerGeneration,
.message = "=== SAMPLE BREWERY GENERATION ==="}); .message = "=== SAMPLE BREWERY GENERATION ==="});
generated_breweries_.clear(); generated_breweries_.clear();
size_t skipped_count = 0; size_t skipped_count = 0;
size_t export_failed_count = 0; size_t export_failed_count = 0;
const auto generate_record = const auto generate_record =
[this, &skipped_count]( [this, &skipped_count](
const EnrichedCity& enriched_city) -> std::optional<BreweryRecord> { const EnrichedCity& enriched_city) -> std::optional<BreweryRecord> {
try { try {
const BreweryResult brewery = generator_->GenerateBrewery(enriched_city); const BreweryResult brewery =
const std::string postal_code = generator_->GenerateBrewery(enriched_city);
postal_code_service_->GeneratePostalCode(enriched_city.location); const std::string postal_code =
return BreweryRecord{ postal_code_service_->GeneratePostalCode(enriched_city.location);
.address = BreweryAddress{.city = enriched_city.location, return BreweryRecord{
.postal_code = postal_code}, .address = BreweryAddress{.city = enriched_city.location,
.brewery = brewery}; .postal_code = postal_code},
} catch (const std::exception& e) { .brewery = brewery};
++skipped_count; } catch (const std::exception& e) {
++skipped_count;
logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::BreweryAndBeerGeneration,
.message = std::format(
"[Pipeline] Skipping city '{}' ({}): brewery "
"generation failed: {}",
enriched_city.location.city,
enriched_city.location.country, e.what())});
return std::nullopt;
}
};
const auto export_record =
[this, &export_failed_count](const BreweryRecord& record) {
try {
exporter_->ProcessRecord(record);
} catch (const std::exception& export_exception) {
++export_failed_count;
logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::BreweryAndBeerGeneration,
.message = std::format(
"[Pipeline] Generated brewery for '{}' ({}) "
"but SQLite export failed: {}",
record.address.city.city, record.address.city.country,
export_exception.what())});
}
};
for (const EnrichedCity& enriched_city : cities) {
const std::optional<BreweryRecord> record =
generate_record(enriched_city);
if (!record.has_value()) {
continue;
}
generated_breweries_.push_back(*record);
export_record(*record);
}
if (skipped_count > 0) {
logger_->Log( logger_->Log(
{.level = LogLevel::Warn, {.level = LogLevel::Warn,
.phase = PipelinePhase::BreweryAndBeerGeneration, .phase = PipelinePhase::BreweryAndBeerGeneration,
.message = std::format("[Pipeline] Skipping city '{}' ({}): brewery " .message = std::format(
"generation failed: {}", "[Pipeline] Skipped {} city/cities due to generation errors",
enriched_city.location.city, skipped_count)});
enriched_city.location.country, e.what())}); }
return std::nullopt;
}
};
const auto export_record = [this, &export_failed_count]( if (export_failed_count > 0) {
const BreweryRecord& record) { logger_->Log(
try { {.level = LogLevel::Warn,
exporter_->ProcessRecord(record); .phase = PipelinePhase::Teardown,
} catch (const std::exception& export_exception) { .message = std::format("[Pipeline] Failed to export {} generated "
++export_failed_count; "brewery/breweries to SQLite",
logger_->Log({.level = LogLevel::Warn, export_failed_count)});
.phase = PipelinePhase::BreweryAndBeerGeneration, }
.message = std::format(
"[Pipeline] Generated brewery for '{}' ({}) "
"but SQLite export failed: {}",
record.address.city.city, record.address.city.country,
export_exception.what())});
}
};
for (const EnrichedCity& enriched_city : cities) {
const std::optional<BreweryRecord> record = generate_record(enriched_city);
if (!record.has_value()) {
continue;
}
generated_breweries_.push_back(*record);
export_record(*record);
}
if (skipped_count > 0) {
logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::BreweryAndBeerGeneration,
.message = std::format(
"[Pipeline] Skipped {} city/cities due to generation errors",
skipped_count)});
}
if (export_failed_count > 0) {
logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::Teardown,
.message = std::format("[Pipeline] Failed to export {} generated "
"brewery/breweries to SQLite",
export_failed_count)});
}
} }

320
tooling/pipeline/src/biergarten_pipeline_orchestrator/generate_users.cc
View File

@@ -20,203 +20,205 @@
namespace { namespace {
std::string Sanitize(std::string_view value) { std::string Sanitize(std::string_view value) {
std::string out; std::string out;
out.reserve(value.size()); out.reserve(value.size());
for (const char character : value) { for (const char character : value) {
if (std::isalnum(static_cast<unsigned char>(character)) != 0) { if (std::isalnum(static_cast<unsigned char>(character)) != 0) {
out.push_back(static_cast<char>( out.push_back(static_cast<char>(
std::tolower(static_cast<unsigned char>(character)))); std::tolower(static_cast<unsigned char>(character))));
} }
} }
return out; return out;
} }
std::string BuildEmail(const Name& name, std::string BuildEmail(const Name& name,
std::unordered_set<std::string>& used_local_parts) { std::unordered_set<std::string>& used_local_parts) {
const std::string base = const std::string base = std::format("{}.{}", Sanitize(name.first_name),
std::format("{}.{}", Sanitize(name.first_name), Sanitize(name.last_name)); Sanitize(name.last_name));
std::string local_part = base; std::string local_part = base;
uint32_t suffix = 1; uint32_t suffix = 1;
while (used_local_parts.contains(local_part)) { while (used_local_parts.contains(local_part)) {
local_part = std::format("{}{}", base, suffix); local_part = std::format("{}{}", base, suffix);
++suffix; ++suffix;
} }
used_local_parts.insert(local_part); used_local_parts.insert(local_part);
return std::format("{}@thebiergarten.app", local_part); return std::format("{}@thebiergarten.app", local_part);
} }
std::string GenerateDateOfBirth(std::mt19937& rng) { std::string GenerateDateOfBirth(std::mt19937& rng) {
using namespace std::chrono; using namespace std::chrono;
constexpr int kMinAge = 19; constexpr int kMinAge = 19;
constexpr int kMaxAge = 48; constexpr int kMaxAge = 48;
constexpr int kMaxDayOffset = 364; constexpr int kMaxDayOffset = 364;
std::uniform_int_distribution<int> age_dist(kMinAge, kMaxAge); std::uniform_int_distribution<int> age_dist(kMinAge, kMaxAge);
std::uniform_int_distribution<int> day_offset_dist(0, kMaxDayOffset); std::uniform_int_distribution<int> day_offset_dist(0, kMaxDayOffset);
const year_month_day today{floor<days>(system_clock::now())}; const year_month_day today{floor<days>(system_clock::now())};
const year_month_day birth_year_anchor{today.year() - years{age_dist(rng)}, const year_month_day birth_year_anchor{today.year() - years{age_dist(rng)},
today.month(), today.day()}; today.month(), today.day()};
const sys_days birth_date = const sys_days birth_date =
sys_days{birth_year_anchor} - days{day_offset_dist(rng)}; sys_days{birth_year_anchor} - days{day_offset_dist(rng)};
const year_month_day birth_ymd{birth_date}; const year_month_day birth_ymd{birth_date};
return std::format("{:04}-{:02}-{:02}", static_cast<int>(birth_ymd.year()), return std::format("{:04}-{:02}-{:02}", static_cast<int>(birth_ymd.year()),
static_cast<unsigned>(birth_ymd.month()), static_cast<unsigned>(birth_ymd.month()),
static_cast<unsigned>(birth_ymd.day())); static_cast<unsigned>(birth_ymd.day()));
} }
std::optional<Name> SampleName( std::optional<Name> SampleName(
const ForenamesByCountryMap& forenames_by_country, const ForenamesByCountryMap& forenames_by_country,
const SurnamesByCountryMap& surnames_by_country, const SurnamesByCountryMap& surnames_by_country,
const std::string& iso3166_1, std::mt19937& rng) { const std::string& iso3166_1, std::mt19937& rng) {
const auto forenames_it = forenames_by_country.find(iso3166_1); const auto forenames_it = forenames_by_country.find(iso3166_1);
const auto surnames_it = surnames_by_country.find(iso3166_1); const auto surnames_it = surnames_by_country.find(iso3166_1);
if (forenames_it == forenames_by_country.end() || if (forenames_it == forenames_by_country.end() ||
surnames_it == surnames_by_country.end() || surnames_it == surnames_by_country.end() ||
forenames_it->second.empty() || surnames_it->second.empty()) { forenames_it->second.empty() || surnames_it->second.empty()) {
return std::nullopt; return std::nullopt;
} }
const ForenameList& forenames = forenames_it->second; const ForenameList& forenames = forenames_it->second;
const SurnameList& surnames = surnames_it->second; const SurnameList& surnames = surnames_it->second;
std::uniform_int_distribution<size_t> forename_dist(0, forenames.size() - 1); std::uniform_int_distribution<size_t> forename_dist(0, forenames.size() - 1);
std::uniform_int_distribution<size_t> surname_dist(0, surnames.size() - 1); std::uniform_int_distribution<size_t> surname_dist(0, surnames.size() - 1);
auto forename_it = forenames.begin(); auto forename_it = forenames.begin();
std::advance(forename_it, forename_dist(rng)); std::advance(forename_it, forename_dist(rng));
auto surname_it = surnames.begin(); auto surname_it = surnames.begin();
std::advance(surname_it, surname_dist(rng)); std::advance(surname_it, surname_dist(rng));
return Name{.first_name = forename_it->name, return Name{.first_name = forename_it->name,
.last_name = *surname_it, .last_name = *surname_it,
.gender = forename_it->gender}; .gender = forename_it->gender};
} }
} // namespace } // namespace
void BiergartenPipelineOrchestrator::GenerateUsers( void BiergartenPipelineOrchestrator::GenerateUsers(
std::span<const EnrichedCity> cities) { std::span<const EnrichedCity> cities) {
logger_->Log({.level = LogLevel::Info, logger_->Log({.level = LogLevel::Info,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::UserGeneration,
.message = "=== SAMPLE USER GENERATION ==="}); .message = "=== SAMPLE USER GENERATION ==="});
const PersonasList& personas = curated_data_service_->LoadPersonas(); const PersonasList& personas = curated_data_service_->LoadPersonas();
if (personas.empty()) { if (personas.empty()) {
throw std::runtime_error( throw std::runtime_error(
"No personas available in personas.json for user generation"); "No personas available in personas.json for user generation");
} }
const ForenamesByCountryMap& forenames_by_country = const ForenamesByCountryMap& forenames_by_country =
curated_data_service_->LoadForenamesByCountry(); curated_data_service_->LoadForenamesByCountry();
const SurnamesByCountryMap& surnames_by_country = const SurnamesByCountryMap& surnames_by_country =
curated_data_service_->LoadSurnamesByCountry(); curated_data_service_->LoadSurnamesByCountry();
std::mt19937 rng(std::random_device{}()); std::mt19937 rng(std::random_device{}());
std::uniform_int_distribution<size_t> persona_dist(0, personas.size() - 1); std::uniform_int_distribution<size_t> persona_dist(0, personas.size() - 1);
generated_users_.clear(); generated_users_.clear();
size_t skipped_count = 0; size_t skipped_count = 0;
size_t export_failed_count = 0; size_t export_failed_count = 0;
std::unordered_set<std::string> used_email_local_parts; std::unordered_set<std::string> used_email_local_parts;
const auto generate_record = const auto generate_record =
[this, &rng, &skipped_count, &used_email_local_parts]( [this, &rng, &skipped_count, &used_email_local_parts](
const EnrichedCity& city, const UserPersona& persona, const EnrichedCity& city, const UserPersona& persona,
const Name& sampled_name) -> std::optional<UserRecord> { const Name& sampled_name) -> std::optional<UserRecord> {
try { try {
const UserResult user = const UserResult user =
generator_->GenerateUser(city, persona, sampled_name); generator_->GenerateUser(city, persona, sampled_name);
const std::string postal_code = const std::string postal_code =
postal_code_service_->GeneratePostalCode(city.location); postal_code_service_->GeneratePostalCode(city.location);
return UserRecord{ return UserRecord{
.address = .address =
UserAddress{.city = city.location, .postal_code = postal_code}, UserAddress{.city = city.location, .postal_code = postal_code},
.user = user, .user = user,
.email = BuildEmail(sampled_name, used_email_local_parts), .email = BuildEmail(sampled_name, used_email_local_parts),
.date_of_birth = GenerateDateOfBirth(rng), .date_of_birth = GenerateDateOfBirth(rng),
}; };
} catch (const std::exception& e) { } catch (const std::exception& e) {
++skipped_count; ++skipped_count;
logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration,
.message = std::format("[Pipeline] Skipping city '{}' ({}): "
"user generation failed: {}",
city.location.city, city.location.country,
e.what())});
return std::nullopt;
}
};
const auto export_record = [this,
&export_failed_count](const UserRecord& record) {
try {
exporter_->ProcessRecord(record);
} catch (const std::exception& export_exception) {
++export_failed_count;
logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration,
.message = std::format(
"[Pipeline] Generated user for '{}' ({}) but "
"SQLite export failed: {}",
record.address.city.city, record.address.city.country,
export_exception.what())});
}
};
for (const auto& city : cities) {
const std::optional<Name> sampled_name =
SampleName(forenames_by_country, surnames_by_country,
city.location.iso3166_1, rng);
if (!sampled_name.has_value()) {
++skipped_count;
logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration,
.message = std::format(
"[Pipeline] Skipping city '{}' ({}): no names "
"available for country '{}'",
city.location.city, city.location.country,
city.location.iso3166_1)});
continue;
}
const UserPersona& persona = personas[persona_dist(rng)];
const std::optional<UserRecord> record =
generate_record(city, persona, *sampled_name);
if (!record.has_value()) {
continue;
}
generated_users_.push_back(*record);
export_record(*record);
}
if (skipped_count > 0) {
logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration,
.message = std::format(
"[Pipeline] Skipped {} city/cities during user generation",
skipped_count)});
}
if (export_failed_count > 0) {
logger_->Log({.level = LogLevel::Warn, logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::Teardown,
.message = std::format( .message = std::format("[Pipeline] Failed to export {} "
"[Pipeline] Skipping city '{}' ({}): " "generated user/users to SQLite",
"user generation failed: {}", export_failed_count)});
city.location.city, city.location.country, e.what())}); }
return std::nullopt;
}
};
const auto export_record = [this,
&export_failed_count](const UserRecord& record) {
try {
exporter_->ProcessRecord(record);
} catch (const std::exception& export_exception) {
++export_failed_count;
logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration,
.message = std::format(
"[Pipeline] Generated user for '{}' ({}) but "
"SQLite export failed: {}",
record.address.city.city, record.address.city.country,
export_exception.what())});
}
};
for (const auto& city : cities) {
const std::optional<Name> sampled_name =
SampleName(forenames_by_country, surnames_by_country,
city.location.iso3166_1, rng);
if (!sampled_name.has_value()) {
++skipped_count;
logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration,
.message = std::format(
"[Pipeline] Skipping city '{}' ({}): no names "
"available for country '{}'",
city.location.city, city.location.country,
city.location.iso3166_1)});
continue;
}
const UserPersona& persona = personas[persona_dist(rng)];
const std::optional<UserRecord> record =
generate_record(city, persona, *sampled_name);
if (!record.has_value()) {
continue;
}
generated_users_.push_back(*record);
export_record(*record);
}
if (skipped_count > 0) {
logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration,
.message = std::format(
"[Pipeline] Skipped {} city/cities during user generation",
skipped_count)});
}
if (export_failed_count > 0) {
logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::Teardown,
.message = std::format("[Pipeline] Failed to export {} "
"generated user/users to SQLite",
export_failed_count)});
}
} }

78
tooling/pipeline/src/biergarten_pipeline_orchestrator/log_results.cc
View File

@@ -12,48 +12,48 @@
#include "services/logging/logger.h" #include "services/logging/logger.h"
void BiergartenPipelineOrchestrator::LogResults() const { void BiergartenPipelineOrchestrator::LogResults() const {
boost::json::array brewery_output; boost::json::array brewery_output;
for (const auto& [address, brewery] : generated_breweries_) { for (const auto& [address, brewery] : generated_breweries_) {
const City& city = address.city; const City& city = address.city;
brewery_output.push_back(boost::json::object{ brewery_output.push_back(boost::json::object{
{"name_en", brewery.name_en}, {"name_en", brewery.name_en},
{"description_en", brewery.description_en}, {"description_en", brewery.description_en},
{"name_local", brewery.name_local}, {"name_local", brewery.name_local},
{"description_local", brewery.description_local}, {"description_local", brewery.description_local},
{"address", boost::json::object{ {"address", boost::json::object{
{"city", city.city}, {"city", city.city},
{"country", city.country}, {"country", city.country},
{"state_province", city.state_province}, {"state_province", city.state_province},
{"iso3166_2", city.iso3166_2}, {"iso3166_2", city.iso3166_2},
{"postal_code", address.postal_code}, {"postal_code", address.postal_code},
}}}); }}});
} }
std::ostringstream brewery_oss; std::ostringstream brewery_oss;
PrettyPrint(brewery_oss, brewery_output); PrettyPrint(brewery_oss, brewery_output);
logger_->Log({.level = LogLevel::Info, logger_->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Teardown, .phase = PipelinePhase::Teardown,
.message = brewery_oss.str()}); .message = brewery_oss.str()});
boost::json::array user_output; boost::json::array user_output;
for (const auto& generated_user : generated_users_) { for (const auto& generated_user : generated_users_) {
user_output.push_back(boost::json::object{ user_output.push_back(boost::json::object{
{"first_name", generated_user.user.first_name}, {"first_name", generated_user.user.first_name},
{"last_name", generated_user.user.last_name}, {"last_name", generated_user.user.last_name},
{"gender", generated_user.user.gender}, {"gender", generated_user.user.gender},
{"username", generated_user.user.username}, {"username", generated_user.user.username},
{"bio", generated_user.user.bio}, {"bio", generated_user.user.bio},
{"activity_weight", generated_user.user.activity_weight}, {"activity_weight", generated_user.user.activity_weight},
{"email", generated_user.email}, {"email", generated_user.email},
{"date_of_birth", generated_user.date_of_birth}, {"date_of_birth", generated_user.date_of_birth},
}); });
} }
std::ostringstream user_oss; std::ostringstream user_oss;
PrettyPrint(user_oss, user_output); PrettyPrint(user_oss, user_output);
logger_->Log({.level = LogLevel::Info, logger_->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Teardown, .phase = PipelinePhase::Teardown,
.message = user_oss.str()}); .message = user_oss.str()});
} }

53
tooling/pipeline/src/biergarten_pipeline_orchestrator/query_cities_with_countries.cc
View File

@@ -14,37 +14,36 @@
#include "services/curated_data/curated_json_data_service.h" #include "services/curated_data/curated_json_data_service.h"
#include "services/logging/logger.h" #include "services/logging/logger.h"
std::vector<City> std::vector<City> BiergartenPipelineOrchestrator::QueryLocations() {
BiergartenPipelineOrchestrator::QueryLocations() { logger_->Log({.level = LogLevel::Info,
logger_->Log({.level = LogLevel::Info, .phase = PipelinePhase::Startup,
.phase = PipelinePhase::Startup, .message = "=== GEOGRAPHIC DATA OVERVIEW ==="});
.message = "=== GEOGRAPHIC DATA OVERVIEW ==="});
const std::vector<City>& all_locations = const std::vector<City>& all_locations =
curated_data_service_->LoadLocations(); curated_data_service_->LoadLocations();
const size_t sample_count = std::min( const size_t sample_count = std::min(
static_cast<size_t>(application_options_.pipeline.location_count), static_cast<size_t>(application_options_.pipeline.location_count),
all_locations.size()); all_locations.size());
const auto sample_count_signed = const auto sample_count_signed =
static_cast<std::iter_difference_t<decltype(all_locations.cbegin())>>( static_cast<std::iter_difference_t<decltype(all_locations.cbegin())>>(
sample_count); sample_count);
std::vector<City> sampled_locations; std::vector<City> sampled_locations;
sampled_locations.reserve(sample_count); sampled_locations.reserve(sample_count);
std::random_device random_generator; std::random_device random_generator;
std::ranges::sample(all_locations, std::back_inserter(sampled_locations), std::ranges::sample(all_locations, std::back_inserter(sampled_locations),
sample_count_signed, random_generator); sample_count_signed, random_generator);
logger_->Log({.level = LogLevel::Info, logger_->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = std::format(" Locations available: {}", .message = std::format(" Locations available: {}",
all_locations.size())}); all_locations.size())});
logger_->Log({.level = LogLevel::Info, logger_->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = std::format(" Sampled locations: {}", .message = std::format(" Sampled locations: {}",
sampled_locations.size())}); sampled_locations.size())});
return sampled_locations; return sampled_locations;
} }

83
tooling/pipeline/src/biergarten_pipeline_orchestrator/run.cc
View File

@@ -11,51 +11,52 @@
#include "services/logging/logger.h" #include "services/logging/logger.h"
bool BiergartenPipelineOrchestrator::Run() { bool BiergartenPipelineOrchestrator::Run() {
try { try {
exporter_->Initialize(); exporter_->Initialize();
std::vector<City> cities = QueryLocations(); std::vector<City> cities = QueryLocations();
std::vector<EnrichedCity> enriched; std::vector<EnrichedCity> enriched;
enriched.reserve(cities.size()); enriched.reserve(cities.size());
size_t skipped_count = 0; size_t skipped_count = 0;
for (auto& city : cities) { for (auto& city : cities) {
try { try {
std::string region_context = context_service_->GetLocationContext(city); std::string region_context =
context_service_->GetLocationContext(city);
enriched.push_back( enriched.push_back(
EnrichedCity{.location = std::move(city), EnrichedCity{.location = std::move(city),
.region_context = std::move(region_context)}); .region_context = std::move(region_context)});
} catch (const std::exception& exception) { } catch (const std::exception& exception) {
++skipped_count; ++skipped_count;
logger_->Log({.level = LogLevel::Warn, logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment, .phase = PipelinePhase::Enrichment,
.message = std::format( .message = std::format(
"[Pipeline] Skipping city '{}' ({}): context " "[Pipeline] Skipping city '{}' ({}): context "
"lookup failed: {}", "lookup failed: {}",
city.city, city.country, exception.what())}); city.city, city.country, exception.what())});
}
} }
}
if (skipped_count > 0) { if (skipped_count > 0) {
logger_->Log( logger_->Log(
{.level = LogLevel::Warn, {.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment, .phase = PipelinePhase::Enrichment,
.message = std::format("[Pipeline] Skipped {} city/cities due " .message = std::format("[Pipeline] Skipped {} city/cities due "
"to context lookup errors", "to context lookup errors",
skipped_count)}); skipped_count)});
} }
this->GenerateUsers(enriched); this->GenerateUsers(enriched);
this->GenerateBreweries(enriched); this->GenerateBreweries(enriched);
exporter_->Finalize(); exporter_->Finalize();
this->LogResults(); this->LogResults();
return true; return true;
} catch (const std::exception& e) { } catch (const std::exception& e) {
logger_->Log({.level = LogLevel::Error, logger_->Log({.level = LogLevel::Error,
.phase = PipelinePhase::Teardown, .phase = PipelinePhase::Teardown,
.message = std::format( .message = std::format(
"Pipeline execution failed with error: {}", e.what())}); "Pipeline execution failed with error: {}", e.what())});
return false; return false;
} }
} }

236
tooling/pipeline/src/data_generation/llama/generate_brewery.cc
View File

@@ -18,144 +18,144 @@
static std::string FormatLocalLanguageCodes( static std::string FormatLocalLanguageCodes(
const std::vector<std::string>& codes) { const std::vector<std::string>& codes) {
if (codes.empty()) { if (codes.empty()) {
return "Not provided"; return "Not provided";
} }
std::string formatted; std::string formatted;
for (const std::string& code : codes) { for (const std::string& code : codes) {
if (!formatted.empty()) { if (!formatted.empty()) {
formatted += ", "; formatted += ", ";
} }
formatted += code; formatted += code;
} }
return formatted; return formatted;
} }
static constexpr int kBreweryInitialMaxTokens = 2800; static constexpr int kBreweryInitialMaxTokens = 2800;
BreweryResult LlamaGenerator::GenerateBrewery( BreweryResult LlamaGenerator::GenerateBrewery(
const EnrichedCity& enriched_city) { const EnrichedCity& enriched_city) {
const City& location = enriched_city.location; const City& location = enriched_city.location;
const std::string& region_context = enriched_city.region_context; const std::string& region_context = enriched_city.region_context;
/** /**
* Preprocess and truncate region context to manageable size * Preprocess and truncate region context to manageable size
*/ */
const std::string safe_region_context = PrepareRegionContext(region_context); const std::string safe_region_context = PrepareRegionContext(region_context);
const std::string local_language_codes = const std::string local_language_codes =
FormatLocalLanguageCodes(location.local_languages); FormatLocalLanguageCodes(location.local_languages);
const std::string country_suffix = const std::string country_suffix =
location.country.empty() ? std::string{} location.country.empty() ? std::string{}
: std::format(", {}", location.country); : std::format(", {}", location.country);
/** /**
* Load brewery system prompt via the injected prompt directory. * Load brewery system prompt via the injected prompt directory.
* The key "BREWERY_GENERATION" resolves to BREWERY_GENERATION.md inside * The key "BREWERY_GENERATION" resolves to BREWERY_GENERATION.md inside
* the configured --prompt-dir. Throws on missing or empty file. * the configured --prompt-dir. Throws on missing or empty file.
*/ */
const std::string system_prompt = const std::string system_prompt =
prompt_directory_->Load("BREWERY_GENERATION"); prompt_directory_->Load("BREWERY_GENERATION");
std::string user_prompt = std::format( std::string user_prompt = std::format(
"## CITY:\n{}\n\n## COUNTRY:\n{}\n\n## LOCAL LANGUAGE CODES:\n{}\n\n## " "## CITY:\n{}\n\n## COUNTRY:\n{}\n\n## LOCAL LANGUAGE CODES:\n{}\n\n## "
"CONTEXT:\n{}", "CONTEXT:\n{}",
location.city, location.country, local_language_codes, location.city, location.country, local_language_codes,
safe_region_context); safe_region_context);
/** /**
* Store location context for retry prompts (without repeating full context) * Store location context for retry prompts (without repeating full context)
*/ */
const std::string retry_location = const std::string retry_location =
std::format("City: {}{}\nLocal language codes: {}", location.city, std::format("City: {}{}\nLocal language codes: {}", location.city,
country_suffix, local_language_codes); country_suffix, local_language_codes);
/** /**
* RETRY LOOP with validation and error correction * RETRY LOOP with validation and error correction
* Attempts to generate valid brewery data up to 3 times, with * Attempts to generate valid brewery data up to 3 times, with
* feedback-based refinement * feedback-based refinement
*/ */
constexpr int max_attempts = 3; constexpr int max_attempts = 3;
std::string raw; std::string raw;
std::string last_error; std::string last_error;
// Token budget: too small risks truncating valid JSON mid-string. // Token budget: too small risks truncating valid JSON mid-string.
int max_tokens = kBreweryInitialMaxTokens; int max_tokens = kBreweryInitialMaxTokens;
// Limit output length to keep it concise and focused
for (int attempt = 0; attempt < max_attempts; ++attempt) {
// Generate brewery data from LLM
raw = this->Infer(system_prompt, user_prompt, max_tokens,
kBreweryJsonGrammar);
if (logger_) {
logger_->Log(
{.level = LogLevel::Debug,
.phase = PipelinePhase::BreweryAndBeerGeneration,
.message = std::format("LlamaGenerator: raw output (attempt {}): {}",
attempt + 1, raw)});
}
// Validate output: parse JSON and check required fields
BreweryResult brewery;
const std::optional<std::string> validation_error =
ValidateBreweryJson(raw, brewery);
if (!validation_error.has_value()) {
// Success: return parsed brewery data
// Limit output length to keep it concise and focused
for (int attempt = 0; attempt < max_attempts; ++attempt) {
// Generate brewery data from LLM
raw = this->Infer(system_prompt, user_prompt, max_tokens,
kBreweryJsonGrammar);
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Debug,
{.level = LogLevel::Info, .phase = PipelinePhase::BreweryAndBeerGeneration,
.phase = PipelinePhase::BreweryAndBeerGeneration, .message = std::format(
.message = std::format("LlamaGenerator: successfully generated " "LlamaGenerator: raw output (attempt {}): {}",
"brewery data on attempt {}", attempt + 1, raw)});
attempt + 1)});
} }
return brewery; // Validate output: parse JSON and check required fields
}
// Validation failed: log error and prepare corrective feedback BreweryResult brewery;
const std::optional<std::string> validation_error =
ValidateBreweryJson(raw, brewery);
last_error = *validation_error; if (!validation_error.has_value()) {
if (logger_) { // Success: return parsed brewery data
logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::BreweryAndBeerGeneration,
.message = std::format(
"LlamaGenerator: malformed brewery JSON (attempt {}): {}",
attempt + 1, *validation_error)});
}
// Update prompt with error details to guide LLM toward correct output. if (logger_) {
user_prompt = std::format( logger_->Log({.level = LogLevel::Info,
"Your previous response was invalid. Error: {}\nReturn the thought " .phase = PipelinePhase::BreweryAndBeerGeneration,
"process before the JSON if needed, then return ONLY valid JSON " .message = std::format(
"with " "LlamaGenerator: successfully generated "
"exactly these keys, in this exact order: {{\"name_en\": " "brewery data on attempt {}",
"\"<English " attempt + 1)});
"brewery name>\", \"description_en\": \"<English single-paragraph " }
"description>\", \"name_local\": \"<local-language brewery "
"name>\", "
"\"description_local\": \"<local-language single-paragraph "
"description>\"}}.\nDo not include markdown, comments, extra keys, "
"or "
"literal placeholder values.\n\nKeep the JSON strings concise "
"enough "
"to fit within the token budget.\n\n{}",
*validation_error, retry_location);
}
// All retry attempts exhausted: log failure and throw exception return brewery;
if (logger_) { }
logger_->Log({.level = LogLevel::Error,
.phase = PipelinePhase::BreweryAndBeerGeneration, // Validation failed: log error and prepare corrective feedback
.message = std::format(
"LlamaGenerator: malformed brewery " last_error = *validation_error;
"response after {} attempts: {}", if (logger_) {
max_attempts, last_error.empty() ? raw : last_error)}); logger_->Log(
} {.level = LogLevel::Warn,
throw std::runtime_error("LlamaGenerator: malformed brewery response"); .phase = PipelinePhase::BreweryAndBeerGeneration,
.message = std::format(
"LlamaGenerator: malformed brewery JSON (attempt {}): {}",
attempt + 1, *validation_error)});
}
// Update prompt with error details to guide LLM toward correct output.
user_prompt = std::format(
"Your previous response was invalid. Error: {}\nReturn the thought "
"process before the JSON if needed, then return ONLY valid JSON "
"with "
"exactly these keys, in this exact order: {{\"name_en\": "
"\"<English "
"brewery name>\", \"description_en\": \"<English single-paragraph "
"description>\", \"name_local\": \"<local-language brewery "
"name>\", "
"\"description_local\": \"<local-language single-paragraph "
"description>\"}}.\nDo not include markdown, comments, extra keys, "
"or "
"literal placeholder values.\n\nKeep the JSON strings concise "
"enough "
"to fit within the token budget.\n\n{}",
*validation_error, retry_location);
}
// All retry attempts exhausted: log failure and throw exception
if (logger_) {
logger_->Log({.level = LogLevel::Error,
.phase = PipelinePhase::BreweryAndBeerGeneration,
.message = std::format(
"LlamaGenerator: malformed brewery "
"response after {} attempts: {}",
max_attempts, last_error.empty() ? raw : last_error)});
}
throw std::runtime_error("LlamaGenerator: malformed brewery response");
} }

166
tooling/pipeline/src/data_generation/llama/generate_user.cc
View File

@@ -19,97 +19,99 @@ static constexpr int kUserInitialMaxTokens = 1200;
UserResult LlamaGenerator::GenerateUser(const EnrichedCity& city, UserResult LlamaGenerator::GenerateUser(const EnrichedCity& city,
const UserPersona& persona, const UserPersona& persona,
const Name& name) { const Name& name) {
std::string style_affinities; std::string style_affinities;
for (const std::string& style : persona.style_affinities) { for (const std::string& style : persona.style_affinities) {
if (!style_affinities.empty()) { if (!style_affinities.empty()) {
style_affinities += ", "; style_affinities += ", ";
} }
style_affinities += style; style_affinities += style;
} }
const std::string system_prompt = prompt_directory_->Load("USER_GENERATION"); const std::string system_prompt = prompt_directory_->Load("USER_GENERATION");
std::string user_prompt = std::format( std::string user_prompt = std::format(
"## NAME:\n\n{} {}\n\n" "## NAME:\n\n{} {}\n\n"
"## GENDER:\n\n{}\n\n" "## GENDER:\n\n{}\n\n"
"## CITY:\n\n{}\n\n" "## CITY:\n\n{}\n\n"
"## COUNTRY:\n\n{}\n\n" "## COUNTRY:\n\n{}\n\n"
"## PERSONA:\n\n{}\n\n" "## PERSONA:\n\n{}\n\n"
"## PERSONA DESCRIPTION:\n\n{}\n\n" "## PERSONA DESCRIPTION:\n\n{}\n\n"
"## STYLE AFFINITIES:\n\n{}", "## STYLE AFFINITIES:\n\n{}",
name.first_name, name.last_name, name.gender, city.location.city, name.first_name, name.last_name, name.gender, city.location.city,
city.location.country, persona.name, persona.description, city.location.country, persona.name, persona.description,
style_affinities); style_affinities);
const std::string retry_context = std::format( const std::string retry_context = std::format(
"Name: {} {}\nCity: {}, {}\nPersona: {}", name.first_name, name.last_name, "Name: {} {}\nCity: {}, {}\nPersona: {}", name.first_name,
city.location.city, city.location.country, persona.name); name.last_name, city.location.city, city.location.country, persona.name);
constexpr int max_attempts = 3; constexpr int max_attempts = 3;
std::string raw; std::string raw;
std::string last_error; std::string last_error;
int max_tokens = kUserInitialMaxTokens; int max_tokens = kUserInitialMaxTokens;
for (int attempt = 0; attempt < max_attempts; ++attempt) { for (int attempt = 0; attempt < max_attempts; ++attempt) {
raw = this->Infer(system_prompt, user_prompt, max_tokens, kUserJsonGrammar); raw =
if (logger_) { this->Infer(system_prompt, user_prompt, max_tokens, kUserJsonGrammar);
logger_->Log(
{.level = LogLevel::Debug,
.phase = PipelinePhase::UserGeneration,
.message = std::format("LlamaGenerator: raw output (attempt {}): {}",
attempt + 1, raw)});
}
UserResult user;
const std::optional<std::string> validation_error =
ValidateUserJson(raw, user);
if (!validation_error.has_value()) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Debug,
{.level = LogLevel::Info, .phase = PipelinePhase::UserGeneration,
.phase = PipelinePhase::UserGeneration, .message = std::format(
.message = std::format("LlamaGenerator: successfully " "LlamaGenerator: raw output (attempt {}): {}",
"generated user data on attempt {}", attempt + 1, raw)});
attempt + 1)});
} }
user.first_name = name.first_name; UserResult user;
user.last_name = name.last_name; const std::optional<std::string> validation_error =
user.gender = name.gender; ValidateUserJson(raw, user);
return user;
}
last_error = *validation_error; if (!validation_error.has_value()) {
if (logger_) { if (logger_) {
logger_->Log({.level = LogLevel::Warn, logger_->Log(
{.level = LogLevel::Info,
.phase = PipelinePhase::UserGeneration,
.message = std::format("LlamaGenerator: successfully "
"generated user data on attempt {}",
attempt + 1)});
}
user.first_name = name.first_name;
user.last_name = name.last_name;
user.gender = name.gender;
return user;
}
last_error = *validation_error;
if (logger_) {
logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration,
.message = std::format(
"LlamaGenerator: malformed user JSON (attempt {}): {}",
attempt + 1, *validation_error)});
}
user_prompt = std::format(
"Your previous response was invalid. Error: {}\nReturn the thought "
"process before the JSON if needed, then return ONLY valid JSON "
"with "
"exactly these keys, in this exact order: {{\"username\": "
"\"<handle "
"derived from the given name>\", \"bio\": \"<first-person bio "
"grounded in the persona>\", \"activity_weight\": <number between "
"0 "
"and 1>}}.\nDo not include markdown, comments, extra keys, or "
"literal placeholder values.\n\n{}",
*validation_error, retry_context);
}
if (logger_) {
logger_->Log({.level = LogLevel::Error,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::UserGeneration,
.message = std::format( .message = std::format(
"LlamaGenerator: malformed user JSON (attempt {}): {}", "LlamaGenerator: malformed user response "
attempt + 1, *validation_error)}); "after {} attempts: {}",
} max_attempts, last_error.empty() ? raw : last_error)});
}
user_prompt = std::format( throw std::runtime_error("LlamaGenerator: malformed user response");
"Your previous response was invalid. Error: {}\nReturn the thought "
"process before the JSON if needed, then return ONLY valid JSON "
"with "
"exactly these keys, in this exact order: {{\"username\": "
"\"<handle "
"derived from the given name>\", \"bio\": \"<first-person bio "
"grounded in the persona>\", \"activity_weight\": <number between "
"0 "
"and 1>}}.\nDo not include markdown, comments, extra keys, or "
"literal placeholder values.\n\n{}",
*validation_error, retry_context);
}
if (logger_) {
logger_->Log({.level = LogLevel::Error,
.phase = PipelinePhase::UserGeneration,
.message = std::format(
"LlamaGenerator: malformed user response "
"after {} attempts: {}",
max_attempts, last_error.empty() ? raw : last_error)});
}
throw std::runtime_error("LlamaGenerator: malformed user response");
} }

329
tooling/pipeline/src/data_generation/llama/helpers.cc
View File

@@ -4,6 +4,8 @@
* parsing, token decoding, and JSON validation helpers for Llama modules. * parsing, token decoding, and JSON validation helpers for Llama modules.
*/ */
#include <llama.h>
#include <algorithm> #include <algorithm>
#include <array> #include <array>
#include <boost/json.hpp> #include <boost/json.hpp>
@@ -15,8 +17,6 @@
#include <string_view> #include <string_view>
#include <vector> #include <vector>
#include <llama.h>
#include "data_generation/llama_generator_helpers.h" #include "data_generation/llama_generator_helpers.h"
namespace { namespace {
@@ -24,14 +24,14 @@ namespace {
* String trimming: removes leading and trailing whitespace * String trimming: removes leading and trailing whitespace
*/ */
std::string Trim(std::string_view value) { std::string Trim(std::string_view value) {
constexpr std::string_view whitespace = " \t\n\r\f\v"; constexpr std::string_view whitespace = " \t\n\r\f\v";
const size_t first_index = value.find_first_not_of(whitespace); const size_t first_index = value.find_first_not_of(whitespace);
if (first_index == std::string_view::npos) { if (first_index == std::string_view::npos) {
return {}; return {};
} }
const size_t last_index = value.find_last_not_of(whitespace); const size_t last_index = value.find_last_not_of(whitespace);
return std::string(value.substr(first_index, last_index - first_index + 1)); return std::string(value.substr(first_index, last_index - first_index + 1));
} }
/** /**
@@ -39,26 +39,26 @@ std::string Trim(std::string_view value) {
* spaces * spaces
*/ */
std::string CondenseWhitespace(std::string_view text) { std::string CondenseWhitespace(std::string_view text) {
std::string out; std::string out;
out.reserve(text.size()); out.reserve(text.size());
bool pending_space = false; bool pending_space = false;
for (const char chr : text) { for (const char chr : text) {
if (std::isspace(static_cast<unsigned char>(chr)) != 0) { if (std::isspace(static_cast<unsigned char>(chr)) != 0) {
if (!out.empty()) { if (!out.empty()) {
pending_space = true; pending_space = true;
}
continue;
} }
continue;
}
if (pending_space) { if (pending_space) {
out.push_back(' '); out.push_back(' ');
pending_space = false; pending_space = false;
} }
out.push_back(chr); out.push_back(chr);
} }
return out; return out;
} }
// Guard against truncating in the first half of the string. // Guard against truncating in the first half of the string.
@@ -69,30 +69,30 @@ constexpr size_t kTruncationGuardDivisor = 2;
bool ReadRequiredTrimmedStringField(const boost::json::object& obj, bool ReadRequiredTrimmedStringField(const boost::json::object& obj,
std::string_view key, std::string& out, std::string_view key, std::string& out,
std::string* error_out) { std::string* error_out) {
const boost::json::value* field = obj.if_contains(key); const boost::json::value* field = obj.if_contains(key);
if (field == nullptr || !field->is_string()) { if (field == nullptr || !field->is_string()) {
return false; return false;
} }
const auto& string_value = field->as_string(); const auto& string_value = field->as_string();
out = Trim(std::string_view(string_value.data(), string_value.size())); out = Trim(std::string_view(string_value.data(), string_value.size()));
return !out.empty(); return !out.empty();
} }
bool HasSchemaPlaceholder(std::span<std::string* const> values) { bool HasSchemaPlaceholder(std::span<std::string* const> values) {
for (const std::string* value : values) { for (const std::string* value : values) {
std::string lowered = *value; std::string lowered = *value;
std::ranges::transform(lowered, lowered.begin(), std::ranges::transform(
[](const unsigned char character) { lowered, lowered.begin(), [](const unsigned char character) {
return static_cast<char>(std::tolower(character)); return static_cast<char>(std::tolower(character));
}); });
if (lowered == "string") { if (lowered == "string") {
return true; return true;
} }
} }
return false; return false;
} }
} // namespace } // namespace
@@ -102,166 +102,167 @@ bool HasSchemaPlaceholder(std::span<std::string* const> values) {
*/ */
std::string PrepareRegionContext(std::string_view region_context, std::string PrepareRegionContext(std::string_view region_context,
const size_t max_chars) { const size_t max_chars) {
std::string normalized = CondenseWhitespace(region_context); std::string normalized = CondenseWhitespace(region_context);
if (normalized.size() <= max_chars) { if (normalized.size() <= max_chars) {
return normalized; return normalized;
} }
normalized.resize(max_chars); normalized.resize(max_chars);
const size_t last_space = normalized.find_last_of(' '); const size_t last_space = normalized.find_last_of(' ');
if (last_space != std::string::npos && if (last_space != std::string::npos &&
last_space > max_chars / kTruncationGuardDivisor) { last_space > max_chars / kTruncationGuardDivisor) {
normalized.resize(last_space); normalized.resize(last_space);
} }
normalized += "..."; normalized += "...";
return normalized; return normalized;
} }
void AppendTokenPiece(const llama_vocab* vocab, llama_token token, void AppendTokenPiece(const llama_vocab* vocab, llama_token token,
std::string& output) { std::string& output) {
constexpr size_t initial_buffer_size = 256; constexpr size_t initial_buffer_size = 256;
std::array<char, initial_buffer_size> buffer{}; std::array<char, initial_buffer_size> buffer{};
// serialize the sampled token into UTF-8 bytes // serialize the sampled token into UTF-8 bytes
auto buffer_too_small = [](int32_t result) -> bool { return result < 0; }; auto buffer_too_small = [](int32_t result) -> bool { return result < 0; };
int32_t bytes = int32_t bytes = llama_token_to_piece(vocab, token, buffer.data(),
llama_token_to_piece(vocab, token, buffer.data(), buffer.size(), 0, true); buffer.size(), 0, true);
if (!buffer_too_small(bytes)) { if (!buffer_too_small(bytes)) {
// Append the decoded bytes from the stack buffer. // Append the decoded bytes from the stack buffer.
output.append(buffer.data(), static_cast<size_t>(bytes)); output.append(buffer.data(), static_cast<size_t>(bytes));
return; return;
} }
const int32_t required_size = -bytes; const int32_t required_size = -bytes;
std::vector<char> dynamic_buffer(static_cast<size_t>(required_size)); std::vector<char> dynamic_buffer(static_cast<size_t>(required_size));
// Retry token decoding against the larger heap buffer. // Retry token decoding against the larger heap buffer.
bytes = llama_token_to_piece(vocab, token, dynamic_buffer.data(), bytes = llama_token_to_piece(vocab, token, dynamic_buffer.data(),
static_cast<int32_t>(dynamic_buffer.size()), 0, static_cast<int32_t>(dynamic_buffer.size()), 0,
true); true);
if (!buffer_too_small(bytes)) { if (!buffer_too_small(bytes)) {
output.append(dynamic_buffer.data(), static_cast<size_t>(bytes)); output.append(dynamic_buffer.data(), static_cast<size_t>(bytes));
return; return;
} }
throw std::runtime_error( throw std::runtime_error(
"LlamaGenerator: failed to decode sampled token piece"); "LlamaGenerator: failed to decode sampled token piece");
} }
std::optional<std::string> ValidateBreweryJson(const std::string& raw, std::optional<std::string> ValidateBreweryJson(const std::string& raw,
BreweryResult& brewery_out) { BreweryResult& brewery_out) {
boost::system::error_code error_code; boost::system::error_code error_code;
const std::string_view raw_view(raw); const std::string_view raw_view(raw);
const size_t opening_brace = raw_view.find('{'); const size_t opening_brace = raw_view.find('{');
if (opening_brace == std::string_view::npos) { if (opening_brace == std::string_view::npos) {
return "JSON parse error: missing opening brace '{'"; return "JSON parse error: missing opening brace '{'";
} }
const std::string_view json_payload = raw_view.substr(opening_brace); const std::string_view json_payload = raw_view.substr(opening_brace);
boost::json::value json_value = boost::json::parse(json_payload, error_code); boost::json::value json_value = boost::json::parse(json_payload, error_code);
if (error_code) { if (error_code) {
return "JSON parse error: " + error_code.message(); return "JSON parse error: " + error_code.message();
} }
if (!json_value.is_object()) { if (!json_value.is_object()) {
return "JSON root must be an object"; return "JSON root must be an object";
} }
const auto& obj = json_value.get_object(); const auto& obj = json_value.get_object();
if (obj.size() != 4) { if (obj.size() != 4) {
return "JSON object must contain exactly four keys"; return "JSON object must contain exactly four keys";
} }
std::string validation_error; std::string validation_error;
if (!ReadRequiredTrimmedStringField(obj, "name_en", brewery_out.name_en, if (!ReadRequiredTrimmedStringField(obj, "name_en", brewery_out.name_en,
&validation_error)) { &validation_error)) {
return validation_error; return validation_error;
} }
if (!ReadRequiredTrimmedStringField(obj, "description_en", if (!ReadRequiredTrimmedStringField(obj, "description_en",
brewery_out.description_en, brewery_out.description_en,
&validation_error)) { &validation_error)) {
return validation_error; return validation_error;
} }
if (!ReadRequiredTrimmedStringField(obj, "name_local", brewery_out.name_local, if (!ReadRequiredTrimmedStringField(
&validation_error)) { obj, "name_local", brewery_out.name_local, &validation_error)) {
return validation_error; return validation_error;
} }
if (!ReadRequiredTrimmedStringField(obj, "description_local", if (!ReadRequiredTrimmedStringField(obj, "description_local",
brewery_out.description_local, brewery_out.description_local,
&validation_error)) { &validation_error)) {
return validation_error; return validation_error;
} }
const std::array schema_placeholders = { const std::array schema_placeholders = {
&brewery_out.name_en, &brewery_out.description_en, &brewery_out.name_en, &brewery_out.description_en,
&brewery_out.name_local, &brewery_out.description_local}; &brewery_out.name_local, &brewery_out.description_local};
if (HasSchemaPlaceholder(schema_placeholders)) { if (HasSchemaPlaceholder(schema_placeholders)) {
return "JSON appears to be a schema placeholder, not content"; return "JSON appears to be a schema placeholder, not content";
} }
return std::nullopt; return std::nullopt;
} }
std::optional<std::string> ValidateUserJson(const std::string& raw, std::optional<std::string> ValidateUserJson(const std::string& raw,
UserResult& user_out) { UserResult& user_out) {
boost::system::error_code error_code; boost::system::error_code error_code;
const std::string_view raw_view(raw); const std::string_view raw_view(raw);
const size_t opening_brace = raw_view.find('{'); const size_t opening_brace = raw_view.find('{');
if (opening_brace == std::string_view::npos) { if (opening_brace == std::string_view::npos) {
return "JSON parse error: missing opening brace '{'"; return "JSON parse error: missing opening brace '{'";
} }
const std::string_view json_payload = raw_view.substr(opening_brace); const std::string_view json_payload = raw_view.substr(opening_brace);
boost::json::value json_value = boost::json::parse(json_payload, error_code); boost::json::value json_value = boost::json::parse(json_payload, error_code);
if (error_code) { if (error_code) {
return "JSON parse error: " + error_code.message(); return "JSON parse error: " + error_code.message();
} }
if (!json_value.is_object()) { if (!json_value.is_object()) {
return "JSON root must be an object"; return "JSON root must be an object";
} }
const auto& obj = json_value.get_object(); const auto& obj = json_value.get_object();
if (obj.size() != 3) { if (obj.size() != 3) {
return "JSON object must contain exactly three keys"; return "JSON object must contain exactly three keys";
} }
std::string validation_error; std::string validation_error;
if (!ReadRequiredTrimmedStringField(obj, "username", user_out.username, if (!ReadRequiredTrimmedStringField(obj, "username", user_out.username,
&validation_error)) { &validation_error)) {
return validation_error; return validation_error;
} }
if (!ReadRequiredTrimmedStringField(obj, "bio", user_out.bio, if (!ReadRequiredTrimmedStringField(obj, "bio", user_out.bio,
&validation_error)) { &validation_error)) {
return validation_error; return validation_error;
} }
const boost::json::value* activity_weight_field = const boost::json::value* activity_weight_field =
obj.if_contains("activity_weight"); obj.if_contains("activity_weight");
if (activity_weight_field == nullptr || !activity_weight_field->is_number()) { if (activity_weight_field == nullptr ||
return "Missing or invalid numeric field: activity_weight"; !activity_weight_field->is_number()) {
} return "Missing or invalid numeric field: activity_weight";
}
const double activity_weight = activity_weight_field->to_number<double>(); const double activity_weight = activity_weight_field->to_number<double>();
if (activity_weight < 0.0 || activity_weight > 1.0) { if (activity_weight < 0.0 || activity_weight > 1.0) {
return "activity_weight must be between 0 and 1"; return "activity_weight must be between 0 and 1";
} }
user_out.activity_weight = static_cast<float>(activity_weight); user_out.activity_weight = static_cast<float>(activity_weight);
const std::array schema_placeholders = {&user_out.username, &user_out.bio}; const std::array schema_placeholders = {&user_out.username, &user_out.bio};
if (HasSchemaPlaceholder(schema_placeholders)) { if (HasSchemaPlaceholder(schema_placeholders)) {
return "JSON appears to be a schema placeholder, not content"; return "JSON appears to be a schema placeholder, not content";
} }
return std::nullopt; return std::nullopt;
} }

342
tooling/pipeline/src/data_generation/llama/infer.cc
View File

@@ -5,6 +5,8 @@
* output tokens back to text for system+user chat prompts. * output tokens back to text for system+user chat prompts.
*/ */
#include <llama.h>
#include <algorithm> #include <algorithm>
#include <chrono> #include <chrono>
#include <format> #include <format>
@@ -14,8 +16,6 @@
#include <string_view> #include <string_view>
#include <vector> #include <vector>
#include <llama.h>
#include "data_generation/llama_generator.h" #include "data_generation/llama_generator.h"
#include "data_generation/llama_generator_helpers.h" #include "data_generation/llama_generator_helpers.h"
@@ -30,48 +30,49 @@ using SamplerHandle =
std::unique_ptr<llama_sampler, decltype(&llama_sampler_free)>; std::unique_ptr<llama_sampler, decltype(&llama_sampler_free)>;
struct SamplerConfig { struct SamplerConfig {
float temperature; float temperature;
uint32_t top_k; uint32_t top_k;
float top_p; float top_p;
uint32_t seed; uint32_t seed;
}; };
SamplerHandle MakeSamplerChain(const llama_vocab* vocab, SamplerHandle MakeSamplerChain(const llama_vocab* vocab,
const SamplerConfig& config, const SamplerConfig& config,
std::string_view grammar) { std::string_view grammar) {
const llama_sampler_chain_params sampler_params = const llama_sampler_chain_params sampler_params =
llama_sampler_chain_default_params(); llama_sampler_chain_default_params();
SamplerHandle chain(llama_sampler_chain_init(sampler_params), SamplerHandle chain(llama_sampler_chain_init(sampler_params),
llama_sampler_free); llama_sampler_free);
if (!chain) { if (!chain) {
throw std::runtime_error("LlamaGenerator: failed to initialize sampler"); throw std::runtime_error("LlamaGenerator: failed to initialize sampler");
} }
auto add_sampler = [&](llama_sampler* sampler, const char* error_message) { auto add_sampler = [&](llama_sampler* sampler, const char* error_message) {
if (sampler == nullptr) { if (sampler == nullptr) {
throw std::runtime_error(error_message); throw std::runtime_error(error_message);
} }
llama_sampler_chain_add(chain.get(), sampler); llama_sampler_chain_add(chain.get(), sampler);
}; };
if (!grammar.empty()) { if (!grammar.empty()) {
const std::string grammar_text(grammar); const std::string grammar_text(grammar);
add_sampler(llama_sampler_init_grammar(vocab, grammar_text.c_str(), "root"), add_sampler(
"LlamaGenerator: failed to initialize grammar sampler"); llama_sampler_init_grammar(vocab, grammar_text.c_str(), "root"),
} "LlamaGenerator: failed to initialize grammar sampler");
}
add_sampler(llama_sampler_init_temp(config.temperature), add_sampler(llama_sampler_init_temp(config.temperature),
"LlamaGenerator: failed to initialize temperature sampler"); "LlamaGenerator: failed to initialize temperature sampler");
add_sampler(llama_sampler_init_top_k(static_cast<int32_t>(config.top_k)), add_sampler(llama_sampler_init_top_k(static_cast<int32_t>(config.top_k)),
"LlamaGenerator: failed to initialize top-k sampler"); "LlamaGenerator: failed to initialize top-k sampler");
add_sampler(llama_sampler_init_top_p(config.top_p, kTopPMinKeep), add_sampler(llama_sampler_init_top_p(config.top_p, kTopPMinKeep),
"LlamaGenerator: failed to initialize top-p sampler"); "LlamaGenerator: failed to initialize top-p sampler");
add_sampler(llama_sampler_init_dist(config.seed), add_sampler(llama_sampler_init_dist(config.seed),
"LlamaGenerator: failed to initialize distribution sampler"); "LlamaGenerator: failed to initialize distribution sampler");
return chain; return chain;
} }
} // namespace } // namespace
@@ -80,163 +81,164 @@ std::string LlamaGenerator::Infer(const std::string& system_prompt,
const std::string& prompt, const std::string& prompt,
const int max_tokens, const int max_tokens,
std::string_view grammar) { std::string_view grammar) {
return InferFormatted(prompt_formatter_->Format(system_prompt, prompt), return InferFormatted(prompt_formatter_->Format(system_prompt, prompt),
max_tokens, grammar); max_tokens, grammar);
} }
std::string LlamaGenerator::InferFormatted(const std::string& formatted_prompt, std::string LlamaGenerator::InferFormatted(const std::string& formatted_prompt,
const int max_tokens, const int max_tokens,
std::string_view grammar) { std::string_view grammar) {
/** /**
* Validate that model and context are loaded * Validate that model and context are loaded
*/ */
if (!model_ || !context_) { if (!model_ || !context_) {
throw std::runtime_error("LlamaGenerator: model not loaded"); throw std::runtime_error("LlamaGenerator: model not loaded");
} }
/** /**
* Get vocabulary for tokenization and token-to-text conversion * Get vocabulary for tokenization and token-to-text conversion
*/ */
const llama_vocab* vocab = llama_model_get_vocab(model_.get()); const llama_vocab* vocab = llama_model_get_vocab(model_.get());
if (vocab == nullptr) { if (vocab == nullptr) {
throw std::runtime_error("LlamaGenerator: vocab unavailable"); throw std::runtime_error("LlamaGenerator: vocab unavailable");
} }
const SamplerConfig sampler_config{ const SamplerConfig sampler_config{
.temperature = sampling_temperature_, .temperature = sampling_temperature_,
.top_k = sampling_top_k_, .top_k = sampling_top_k_,
.top_p = sampling_top_p_, .top_p = sampling_top_p_,
.seed = static_cast<uint32_t>(rng_()), .seed = static_cast<uint32_t>(rng_()),
}; };
const auto sampler = MakeSamplerChain(vocab, sampler_config, grammar); const auto sampler = MakeSamplerChain(vocab, sampler_config, grammar);
/** /**
* Clear KV cache to ensure clean inference state (no residual context) * Clear KV cache to ensure clean inference state (no residual context)
*/ */
llama_memory_clear(llama_get_memory(context_.get()), true); llama_memory_clear(llama_get_memory(context_.get()), true);
/** /**
* TOKENIZATION PHASE * TOKENIZATION PHASE
* Convert text prompt into token IDs (integers) that the model understands * Convert text prompt into token IDs (integers) that the model understands
*/ */
std::vector<llama_token> prompt_tokens(formatted_prompt.size() + std::vector<llama_token> prompt_tokens(formatted_prompt.size() +
kPromptTokenSlack); kPromptTokenSlack);
int32_t token_count = llama_tokenize( int32_t token_count = llama_tokenize(
vocab, formatted_prompt.c_str(), vocab, formatted_prompt.c_str(),
static_cast<int32_t>(formatted_prompt.size()), prompt_tokens.data(), static_cast<int32_t>(formatted_prompt.size()), prompt_tokens.data(),
static_cast<int32_t>(prompt_tokens.size()), true, true); static_cast<int32_t>(prompt_tokens.size()), true, true);
/** /**
* If buffer too small, negative return indicates required size * If buffer too small, negative return indicates required size
*/ */
if (token_count < 0) { if (token_count < 0) {
prompt_tokens.resize(static_cast<size_t>(-token_count)); prompt_tokens.resize(static_cast<size_t>(-token_count));
token_count = llama_tokenize( token_count = llama_tokenize(
vocab, formatted_prompt.c_str(), vocab, formatted_prompt.c_str(),
static_cast<int32_t>(formatted_prompt.size()), prompt_tokens.data(), static_cast<int32_t>(formatted_prompt.size()), prompt_tokens.data(),
static_cast<int32_t>(prompt_tokens.size()), true, true); static_cast<int32_t>(prompt_tokens.size()), true, true);
} }
if (token_count < 0) { if (token_count < 0) {
throw std::runtime_error("LlamaGenerator: prompt tokenization failed"); throw std::runtime_error("LlamaGenerator: prompt tokenization failed");
} }
/** /**
* CONTEXT SIZE VALIDATION * CONTEXT SIZE VALIDATION
* Validate and compute effective token budgets based on context window * Validate and compute effective token budgets based on context window
* constraints * constraints
*/ */
const auto n_ctx = static_cast<int32_t>(llama_n_ctx(context_.get())); const auto n_ctx = static_cast<int32_t>(llama_n_ctx(context_.get()));
const auto n_batch = static_cast<int32_t>(llama_n_batch(context_.get())); const auto n_batch = static_cast<int32_t>(llama_n_batch(context_.get()));
if (n_ctx <= 1 || n_batch <= 0) { if (n_ctx <= 1 || n_batch <= 0) {
throw std::runtime_error("LlamaGenerator: invalid context or batch size"); throw std::runtime_error("LlamaGenerator: invalid context or batch size");
} }
/** /**
* Clamp generation limit to available context window, reserve space for * Clamp generation limit to available context window, reserve space for
* output * output
*/ */
const int32_t effective_max_tokens = const int32_t effective_max_tokens =
std::max(1, std::min(max_tokens, n_ctx - 1)); std::max(1, std::min(max_tokens, n_ctx - 1));
/** /**
* Prompt can use remaining context after reserving space for generation * Prompt can use remaining context after reserving space for generation
*/ */
int32_t prompt_budget = std::min(n_batch, n_ctx - effective_max_tokens); int32_t prompt_budget = std::min(n_batch, n_ctx - effective_max_tokens);
prompt_budget = std::max<int32_t>(1, prompt_budget); prompt_budget = std::max<int32_t>(1, prompt_budget);
/** /**
* Truncate prompt if necessary to fit within constraints * Truncate prompt if necessary to fit within constraints
*/ */
prompt_tokens.resize(static_cast<size_t>(token_count)); prompt_tokens.resize(static_cast<size_t>(token_count));
if (token_count > prompt_budget) { if (token_count > prompt_budget) {
if (logger_) { if (logger_) {
logger_->Log({.level = LogLevel::Warn, logger_->Log(
.phase = PipelinePhase::BreweryAndBeerGeneration, {.level = LogLevel::Warn,
.message = std::format( .phase = PipelinePhase::BreweryAndBeerGeneration,
"LlamaGenerator: prompt too long ({} tokens), " .message = std::format(
"truncating to {} tokens to fit n_batch/n_ctx limits", "LlamaGenerator: prompt too long ({} tokens), "
token_count, prompt_budget)}); "truncating to {} tokens to fit n_batch/n_ctx limits",
} token_count, prompt_budget)});
prompt_tokens.resize(static_cast<size_t>(prompt_budget)); }
token_count = prompt_budget; prompt_tokens.resize(static_cast<size_t>(prompt_budget));
} token_count = prompt_budget;
}
/** /**
* PROMPT PROCESSING PHASE * PROMPT PROCESSING PHASE
* Create a batch containing all prompt tokens and feed through the model * Create a batch containing all prompt tokens and feed through the model
* This computes internal representations and fills the KV cache * This computes internal representations and fills the KV cache
*/ */
const llama_batch prompt_batch = llama_batch_get_one( const llama_batch prompt_batch = llama_batch_get_one(
prompt_tokens.data(), static_cast<int32_t>(prompt_tokens.size())); prompt_tokens.data(), static_cast<int32_t>(prompt_tokens.size()));
if (llama_decode(context_.get(), prompt_batch) != 0) { if (llama_decode(context_.get(), prompt_batch) != 0) {
throw std::runtime_error("LlamaGenerator: prompt decode failed"); throw std::runtime_error("LlamaGenerator: prompt decode failed");
} }
/** /**
* TOKEN GENERATION LOOP * TOKEN GENERATION LOOP
* Iteratively generate tokens one at a time until max_tokens or * Iteratively generate tokens one at a time until max_tokens or
* end-of-sequence * end-of-sequence
*/ */
std::vector<llama_token> generated_tokens; std::vector<llama_token> generated_tokens;
generated_tokens.reserve(static_cast<size_t>(effective_max_tokens)); generated_tokens.reserve(static_cast<size_t>(effective_max_tokens));
for (int i = 0; i < effective_max_tokens; ++i) { for (int i = 0; i < effective_max_tokens; ++i) {
/** /**
* Sample next token using configured sampler chain and model logits * Sample next token using configured sampler chain and model logits
* Index -1 means use the last output position from previous batch * Index -1 means use the last output position from previous batch
*/ */
const llama_token next = const llama_token next =
llama_sampler_sample(sampler.get(), context_.get(), -1); llama_sampler_sample(sampler.get(), context_.get(), -1);
/** /**
* Stop if model predicts end-of-generation token (EOS/EOT) * Stop if model predicts end-of-generation token (EOS/EOT)
*/ */
if (llama_vocab_is_eog(vocab, next)) { if (llama_vocab_is_eog(vocab, next)) {
break; break;
} }
generated_tokens.push_back(next); generated_tokens.push_back(next);
/** /**
* Feed the sampled token back into model for next iteration * Feed the sampled token back into model for next iteration
* (autoregressive) * (autoregressive)
*/ */
llama_token decode_token = next; llama_token decode_token = next;
const llama_batch one_token_batch = llama_batch_get_one(&decode_token, 1); const llama_batch one_token_batch = llama_batch_get_one(&decode_token, 1);
if (llama_decode(context_.get(), one_token_batch) != 0) { if (llama_decode(context_.get(), one_token_batch) != 0) {
throw std::runtime_error( throw std::runtime_error(
"LlamaGenerator: decode failed during generation"); "LlamaGenerator: decode failed during generation");
} }
} }
/** /**
* DETOKENIZATION PHASE * DETOKENIZATION PHASE
* Convert generated token IDs back to text using vocabulary * Convert generated token IDs back to text using vocabulary
*/ */
std::string output; std::string output;
for (const llama_token token : generated_tokens) { for (const llama_token token : generated_tokens) {
AppendTokenPiece(vocab, token, output); AppendTokenPiece(vocab, token, output);
} }
return output; return output;
} }

102
tooling/pipeline/src/data_generation/llama/llama_generator.cc
View File

@@ -5,30 +5,30 @@
#include "data_generation/llama_generator.h" #include "data_generation/llama_generator.h"
#include <llama.h>
#include <filesystem> #include <filesystem>
#include <memory> #include <memory>
#include <random> #include <random>
#include <stdexcept> #include <stdexcept>
#include <string> #include <string>
#include <llama.h>
#include "data_model/models.h" #include "data_model/models.h"
static constexpr uint32_t kMaxContextSize = 32768U; static constexpr uint32_t kMaxContextSize = 32768U;
void LlamaGenerator::ModelDeleter::operator()( void LlamaGenerator::ModelDeleter::operator()(
llama_model* model) const noexcept { llama_model* model) const noexcept {
if (model != nullptr) { if (model != nullptr) {
llama_model_free(model); llama_model_free(model);
} }
} }
void LlamaGenerator::ContextDeleter::operator()( void LlamaGenerator::ContextDeleter::operator()(
llama_context* context) const noexcept { llama_context* context) const noexcept {
if (context != nullptr) { if (context != nullptr) {
llama_free(context); llama_free(context);
} }
} }
LlamaGenerator::LlamaGenerator( LlamaGenerator::LlamaGenerator(
@@ -40,61 +40,61 @@ LlamaGenerator::LlamaGenerator(
logger_(std::move(logger)), logger_(std::move(logger)),
prompt_formatter_(std::move(prompt_formatter)), prompt_formatter_(std::move(prompt_formatter)),
prompt_directory_(std::move(prompt_directory)) { prompt_directory_(std::move(prompt_directory)) {
if (model_path.empty()) { if (model_path.empty()) {
throw std::runtime_error("LlamaGenerator: model path must not be empty"); throw std::runtime_error("LlamaGenerator: model path must not be empty");
} }
if (!prompt_formatter_) { if (!prompt_formatter_) {
throw std::runtime_error( throw std::runtime_error(
"LlamaGenerator: prompt formatter dependency must not be null"); "LlamaGenerator: prompt formatter dependency must not be null");
} }
if (!prompt_directory_) { if (!prompt_directory_) {
throw std::runtime_error( throw std::runtime_error(
"LlamaGenerator: prompt directory dependency must not be null"); "LlamaGenerator: prompt directory dependency must not be null");
} }
const auto sampling = options.generator.sampling.value_or(SamplingOptions{}); const auto sampling = options.generator.sampling.value_or(SamplingOptions{});
if (sampling.temperature < 0.0F) { if (sampling.temperature < 0.0F) {
throw std::runtime_error( throw std::runtime_error(
"LlamaGenerator: sampling temperature must be >= 0"); "LlamaGenerator: sampling temperature must be >= 0");
} }
if (sampling.top_p <= 0.0F || sampling.top_p > 1.0F) { if (sampling.top_p <= 0.0F || sampling.top_p > 1.0F) {
throw std::runtime_error( throw std::runtime_error(
"LlamaGenerator: sampling top-p must be in (0, 1]"); "LlamaGenerator: sampling top-p must be in (0, 1]");
} }
if (sampling.top_k == 0U) { if (sampling.top_k == 0U) {
throw std::runtime_error("LlamaGenerator: sampling top-k must be > 0"); throw std::runtime_error("LlamaGenerator: sampling top-k must be > 0");
} }
if (sampling.seed < -1) { if (sampling.seed < -1) {
throw std::runtime_error( throw std::runtime_error(
"LlamaGenerator: seed must be >= 0, or -1 for random"); "LlamaGenerator: seed must be >= 0, or -1 for random");
} }
if (sampling.n_ctx == 0 || sampling.n_ctx > kMaxContextSize) { if (sampling.n_ctx == 0 || sampling.n_ctx > kMaxContextSize) {
throw std::runtime_error( throw std::runtime_error(
"LlamaGenerator: context size must be in range [1, 32768]"); "LlamaGenerator: context size must be in range [1, 32768]");
} }
sampling_temperature_ = sampling.temperature; sampling_temperature_ = sampling.temperature;
sampling_top_p_ = sampling.top_p; sampling_top_p_ = sampling.top_p;
sampling_top_k_ = sampling.top_k; sampling_top_k_ = sampling.top_k;
if (sampling.seed == -1) { if (sampling.seed == -1) {
std::random_device random_device; std::random_device random_device;
rng_.seed(random_device()); rng_.seed(random_device());
} else { } else {
rng_.seed(static_cast<uint32_t>(sampling.seed)); rng_.seed(static_cast<uint32_t>(sampling.seed));
} }
n_ctx_ = sampling.n_ctx; n_ctx_ = sampling.n_ctx;
n_gpu_layers_ = sampling.n_gpu_layers; n_gpu_layers_ = sampling.n_gpu_layers;
this->Load(model_path); this->Load(model_path);
} }
LlamaGenerator::~LlamaGenerator() = default; LlamaGenerator::~LlamaGenerator() = default;

64
tooling/pipeline/src/data_generation/llama/load.cc
View File

@@ -4,15 +4,15 @@
* context, and resets prior resources during model initialization. * context, and resets prior resources during model initialization.
*/ */
#include <ggml-backend.h>
#include <llama.h>
#include <algorithm> #include <algorithm>
#include <chrono> #include <chrono>
#include <stdexcept> #include <stdexcept>
#include <string> #include <string>
#include <utility> #include <utility>
#include <ggml-backend.h>
#include <llama.h>
#include "data_generation/llama_generator.h" #include "data_generation/llama_generator.h"
// Maximum batch size for decode operations. Capping the batch prevents // Maximum batch size for decode operations. Capping the batch prevents
@@ -20,42 +20,42 @@
static constexpr uint32_t kMaxBatchSize = 5000U; static constexpr uint32_t kMaxBatchSize = 5000U;
void LlamaGenerator::Load(const std::string& model_path) { void LlamaGenerator::Load(const std::string& model_path) {
context_.reset(); context_.reset();
model_.reset(); model_.reset();
// Specifically load dynamic ggml backends (like CUDA) that are provided // Specifically load dynamic ggml backends (like CUDA) that are provided
// externally before attempting to load a model. // externally before attempting to load a model.
ggml_backend_load_all(); ggml_backend_load_all();
llama_model_params model_params = llama_model_default_params(); llama_model_params model_params = llama_model_default_params();
model_params.n_gpu_layers = n_gpu_layers_; model_params.n_gpu_layers = n_gpu_layers_;
ModelHandle loaded_model( ModelHandle loaded_model(
llama_model_load_from_file(model_path.c_str(), model_params)); llama_model_load_from_file(model_path.c_str(), model_params));
if (!loaded_model) { if (!loaded_model) {
throw std::runtime_error( throw std::runtime_error(
"LlamaGenerator: failed to load model from path: " + model_path); "LlamaGenerator: failed to load model from path: " + model_path);
} }
llama_context_params context_params = llama_context_default_params(); llama_context_params context_params = llama_context_default_params();
context_params.n_ctx = n_ctx_; context_params.n_ctx = n_ctx_;
context_params.n_batch = std::min(n_ctx_, kMaxBatchSize); context_params.n_batch = std::min(n_ctx_, kMaxBatchSize);
ContextHandle loaded_context( ContextHandle loaded_context(
llama_init_from_model(loaded_model.get(), context_params)); llama_init_from_model(loaded_model.get(), context_params));
if (!loaded_context) { if (!loaded_context) {
throw std::runtime_error("LlamaGenerator: failed to create context"); throw std::runtime_error("LlamaGenerator: failed to create context");
} }
model_ = std::move(loaded_model); model_ = std::move(loaded_model);
context_ = std::move(loaded_context); context_ = std::move(loaded_context);
if (logger_) { if (logger_) {
logger_->Log({.level = LogLevel::Info, logger_->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = std::format("[LlamaGenerator] Loaded model: {} ", .message = std::format("[LlamaGenerator] Loaded model: {} ",
model_path)}); model_path)});
} }
} }

18
tooling/pipeline/src/data_generation/mock/deterministic_hash.cc
View File

@@ -9,18 +9,18 @@
#include "data_generation/mock_generator.h" #include "data_generation/mock_generator.h"
size_t MockGenerator::DeterministicHash(const City& location) { size_t MockGenerator::DeterministicHash(const City& location) {
size_t seed = 0; size_t seed = 0;
boost::hash_combine(seed, location.city); boost::hash_combine(seed, location.city);
boost::hash_combine(seed, location.country); boost::hash_combine(seed, location.country);
return seed; return seed;
} }
size_t MockGenerator::DeterministicHash(const City& location, size_t MockGenerator::DeterministicHash(const City& location,
const UserPersona& persona, const UserPersona& persona,
const Name& name) { const Name& name) {
size_t seed = DeterministicHash(location); size_t seed = DeterministicHash(location);
boost::hash_combine(seed, persona.name); boost::hash_combine(seed, persona.name);
boost::hash_combine(seed, name.first_name); boost::hash_combine(seed, name.first_name);
boost::hash_combine(seed, name.last_name); boost::hash_combine(seed, name.last_name);
return seed; return seed;
} }

52
tooling/pipeline/src/data_generation/mock/generate_brewery.cc
View File

@@ -12,34 +12,34 @@
BreweryResult MockGenerator::GenerateBrewery( BreweryResult MockGenerator::GenerateBrewery(
const EnrichedCity& enriched_city) { const EnrichedCity& enriched_city) {
const City& location = enriched_city.location; const City& location = enriched_city.location;
const size_t hash = DeterministicHash(location); const size_t hash = DeterministicHash(location);
const std::string_view adjective = const std::string_view adjective =
kBreweryAdjectives.at(hash % kBreweryAdjectives.size()); kBreweryAdjectives.at(hash % kBreweryAdjectives.size());
const std::string_view noun = const std::string_view noun =
kBreweryNouns.at(hash / kNounHashStride % kBreweryNouns.size()); kBreweryNouns.at(hash / kNounHashStride % kBreweryNouns.size());
const std::string_view base_description = kBreweryDescriptions.at( const std::string_view base_description = kBreweryDescriptions.at(
(hash / kDescriptionHashStride) % kBreweryDescriptions.size()); (hash / kDescriptionHashStride) % kBreweryDescriptions.size());
const std::string name = const std::string name =
std::format("{} {} {}", location.city, adjective, noun); std::format("{} {} {}", location.city, adjective, noun);
const std::string state_suffix = const std::string state_suffix =
location.state_province.empty() location.state_province.empty()
? std::string{} ? std::string{}
: std::format(", {}", location.state_province); : std::format(", {}", location.state_province);
const std::string country_suffix = const std::string country_suffix =
location.country.empty() ? std::string{} location.country.empty() ? std::string{}
: std::format(", {}", location.country); : std::format(", {}", location.country);
const std::string description = const std::string description =
std::format("{} Located in {}{}{}.", base_description, location.city, std::format("{} Located in {}{}{}.", base_description, location.city,
state_suffix, country_suffix); state_suffix, country_suffix);
return { return {
.name_en = name, .name_en = name,
.description_en = description, .description_en = description,
.name_local = name, .name_local = name,
.description_local = description, .description_local = description,
}; };
} }

30
tooling/pipeline/src/data_generation/mock/generate_user.cc
View File

@@ -11,21 +11,21 @@
UserResult MockGenerator::GenerateUser(const EnrichedCity& city, UserResult MockGenerator::GenerateUser(const EnrichedCity& city,
const UserPersona& persona, const UserPersona& persona,
const Name& name) { const Name& name) {
const size_t hash = DeterministicHash(city.location, persona, name); const size_t hash = DeterministicHash(city.location, persona, name);
const std::string_view username = kUsernames[hash % kUsernames.size()]; const std::string_view username = kUsernames[hash % kUsernames.size()];
const std::string_view bio = kBios[hash / kBioHashStride % kBios.size()]; const std::string_view bio = kBios[hash / kBioHashStride % kBios.size()];
const float activity_weight = const float activity_weight =
static_cast<float>(hash / kActivityWeightHashStride % static_cast<float>(hash / kActivityWeightHashStride %
kActivityWeightHashRange) / kActivityWeightHashRange) /
static_cast<float>(kActivityWeightHashRange); static_cast<float>(kActivityWeightHashRange);
return { return {
.first_name = name.first_name, .first_name = name.first_name,
.last_name = name.last_name, .last_name = name.last_name,
.gender = name.gender, .gender = name.gender,
.username = std::string(username), .username = std::string(username),
.bio = std::string(bio), .bio = std::string(bio),
.activity_weight = activity_weight, .activity_weight = activity_weight,
}; };
} }

28
tooling/pipeline/src/data_generation/prompt_formatting/gemma4_jinja_prompt_formatter.cc
View File

@@ -8,25 +8,25 @@ static constexpr std::string_view kWhitespace = " \t\n\r\f\v";
// Strips leading and trailing whitespace to ensure clean prompt injection. // Strips leading and trailing whitespace to ensure clean prompt injection.
static std::string_view Trim(std::string_view value) { static std::string_view Trim(std::string_view value) {
const size_t first_index = value.find_first_not_of(kWhitespace); const size_t first_index = value.find_first_not_of(kWhitespace);
const bool is_all_whitespace = (first_index == std::string_view::npos); const bool is_all_whitespace = (first_index == std::string_view::npos);
if (is_all_whitespace) { if (is_all_whitespace) {
return ""; return "";
} }
const size_t last_index = value.find_last_not_of(kWhitespace); const size_t last_index = value.find_last_not_of(kWhitespace);
return value.substr(first_index, last_index - first_index + 1); return value.substr(first_index, last_index - first_index + 1);
} }
std::string Gemma4JinjaPromptFormatter::Format( std::string Gemma4JinjaPromptFormatter::Format(
std::string_view system_prompt, std::string_view user_prompt) const { std::string_view system_prompt, std::string_view user_prompt) const {
std::string_view trimmed_system = Trim(system_prompt); std::string_view trimmed_system = Trim(system_prompt);
std::string_view trimmed_user = Trim(user_prompt); std::string_view trimmed_user = Trim(user_prompt);
return std::format( return std::format(
"<|turn|>system\n<|think|>\n{}\n<|turn|>\n" "<|turn|>system\n<|think|>\n{}\n<|turn|>\n"
"<|turn|>user\n{}\n<|turn|>\n" "<|turn|>user\n{}\n<|turn|>\n"
"<|turn|>model\n<|channel>thought\n", "<|turn|>model\n<|channel>thought\n",
trimmed_system, trimmed_user); trimmed_system, trimmed_user);
} }

385
tooling/pipeline/src/main.cc
View File

@@ -25,223 +25,230 @@ namespace di = boost::di;
static constexpr size_t kLogMaxCount = 512; static constexpr size_t kLogMaxCount = 512;
int main(const int argc, char** argv) { int main(const int argc, char** argv) {
spdlog::set_level(spdlog::level::debug); spdlog::set_level(spdlog::level::debug);
spdlog::set_pattern("│ %Y-%m-%d %H:%M:%S.%e │ %^%-7l%$ │ %v"); spdlog::set_pattern("│ %Y-%m-%d %H:%M:%S.%e │ %^%-7l%$ │ %v");
BoundedChannel<LogEntry> log_channel(kLogMaxCount); BoundedChannel<LogEntry> log_channel(kLogMaxCount);
auto log_dispatcher = // auto log_dispatcher = //
std::make_unique<LogDispatcher>(log_channel); std::make_unique<LogDispatcher>(log_channel);
std::shared_ptr<ILogger> log_producer = std::shared_ptr<ILogger> log_producer =
std::make_shared<LogProducer>(log_channel); std::make_shared<LogProducer>(log_channel);
std::thread log_thread([&log_dispatcher] { log_dispatcher->Run(); }); std::thread log_thread([&log_dispatcher] { log_dispatcher->Run(); });
auto shutdown = [&](const int exit_code) { auto shutdown = [&](const int exit_code) {
log_channel.Close(); log_channel.Close();
log_thread.join(); log_thread.join();
return exit_code; return exit_code;
}; };
try { try {
Timer timer; Timer timer;
#ifndef BIERGARTEN_MOCK_ONLY #ifndef BIERGARTEN_MOCK_ONLY
const LlamaBackendState llama_backend_state; const LlamaBackendState llama_backend_state;
#endif #endif
log_producer->Log({.level = LogLevel::Info, log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = "STARTING PIPELINE"}); .message = "STARTING PIPELINE"});
const std::optional<ApplicationOptions> parsed_options = const std::optional<ApplicationOptions> parsed_options =
ParseArguments(argc, argv, log_producer); ParseArguments(argc, argv, log_producer);
if (!parsed_options.has_value()) { if (!parsed_options.has_value()) {
return shutdown(EXIT_FAILURE); return shutdown(EXIT_FAILURE);
}
const auto options = *parsed_options;
const std::string model_path = options.generator.model_path.string();
const auto sampling =
options.generator.sampling.value_or(SamplingOptions{});
std::unique_ptr<IPromptDirectory> prompt_directory;
if (!options.generator.use_mocked) {
try {
prompt_directory = std::make_unique<PromptDirectory>(
options.pipeline.prompt_dir, log_producer);
} catch (const std::exception& dir_error) {
log_producer->Log({.level = LogLevel::Error,
.phase = PipelinePhase::Startup,
.message = std::format("Invalid --prompt-dir: {}",
dir_error.what())});
return shutdown(EXIT_FAILURE);
} }
}
const auto injector = di::make_injector( const auto options = *parsed_options;
di::bind<ILogger>().to(log_producer), const std::string model_path = options.generator.model_path.string();
di::bind<ApplicationOptions>().to(options), const auto sampling =
di::bind<std::string>().to(model_path), options.generator.sampling.value_or(SamplingOptions{});
di::bind<IExportService>().to<SqliteExportService>(),
di::bind<IPostalCodeService>().to( std::unique_ptr<IPromptDirectory> prompt_directory;
[options, &log_producer]() -> std::unique_ptr<IPostalCodeService> {
if (options.generator.use_mocked) { if (!options.generator.use_mocked) {
try {
prompt_directory = std::make_unique<PromptDirectory>(
options.pipeline.prompt_dir, log_producer);
} catch (const std::exception& dir_error) {
log_producer->Log(
{.level = LogLevel::Error,
.phase = PipelinePhase::Startup,
.message = std::format("Invalid --prompt-dir: {}",
dir_error.what())});
return shutdown(EXIT_FAILURE);
}
}
const auto injector = di::make_injector(
di::bind<ILogger>().to(log_producer),
di::bind<ApplicationOptions>().to(options),
di::bind<std::string>().to(model_path),
di::bind<IExportService>().to<SqliteExportService>(),
di::bind<IPostalCodeService>().to(
[options,
&log_producer]() -> std::unique_ptr<IPostalCodeService> {
if (options.generator.use_mocked) {
log_producer->Log(
{.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "Postal code: mock (curated examples)"});
return std::make_unique<MockPostalCodeService>();
}
log_producer->Log(
{.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message =
"Postal code: Xeger (regex-based generation)"});
return std::make_unique<XegerPostalCodeService>();
}),
di::bind<ICuratedDataService>().to(
[options,
&log_producer]() -> std::unique_ptr<ICuratedDataService> {
if (options.generator.use_mocked) {
log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "Curated data: mock"});
return std::make_unique<MockCuratedDataService>();
}
log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "Curated data: JsonLoader"});
return std::make_unique<CuratedJsonDataService>(
CuratedDataFilePaths{
.locations_path = "locations.json",
.personas_path = "personas.json",
.forenames_path = "forenames-by-country.json",
.surnames_path = "surnames-by-country.json",
});
}),
di::bind<IPromptFormatter>().to([options, log_producer] {
if (options.generator.use_mocked) {
{
log_producer->Log(
{.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "Prompt formatter: none (mock mode)"});
}
return std::unique_ptr<IPromptFormatter>(nullptr);
}
{
log_producer->Log( log_producer->Log(
{.level = LogLevel::Info, {.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = "Postal code: mock (curated examples)"}); .message =
"Prompt formatter: Gemma4JinjaPromptFormatter"});
}
return std::unique_ptr<IPromptFormatter>(
std::make_unique<Gemma4JinjaPromptFormatter>());
}),
di::bind<WebClient>().to([options, log_producer] {
if (options.generator.use_mocked) {
{
log_producer->Log(
{.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "Web client: none (mock mode)"});
}
return std::unique_ptr<WebClient>(nullptr);
}
return std::make_unique<MockPostalCodeService>(); log_producer->Log({.level = LogLevel::Info,
} .phase = PipelinePhase::Startup,
.message = "Web client: HttpWebClient"});
log_producer->Log( return std::unique_ptr<WebClient>(
{.level = LogLevel::Info, std::make_unique<HttpWebClient>(log_producer));
.phase = PipelinePhase::Startup, }),
.message = "Postal code: Xeger (regex-based generation)"}); di::bind<IEnrichmentService>().to(
[options, &log_producer](
const auto& inj) -> std::unique_ptr<IEnrichmentService> {
if (options.generator.use_mocked) {
log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "Enrichment: mock"});
return std::make_unique<XegerPostalCodeService>(); return std::make_unique<MockEnrichmentService>();
}), }
di::bind<ICuratedDataService>().to(
[options, &log_producer]() -> std::unique_ptr<ICuratedDataService> {
if (options.generator.use_mocked) {
log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "Curated data: mock"});
return std::make_unique<MockCuratedDataService>(); log_producer->Log({.level = LogLevel::Info,
} .phase = PipelinePhase::Startup,
.message = "Enrichment: Wikipedia"});
log_producer->Log({.level = LogLevel::Info, return std::make_unique<WikipediaEnrichmentService>(
.phase = PipelinePhase::Startup, inj.template create<std::unique_ptr<WebClient>>(),
.message = "Curated data: JsonLoader"}); log_producer);
}),
di::bind<DataGenerator>().to(
[&options, &model_path, &sampling, &prompt_directory,
&log_producer](
const auto& inj) -> std::unique_ptr<DataGenerator> {
if (options.generator.use_mocked) {
log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "Generator: mock"});
return std::make_unique<CuratedJsonDataService>( return std::make_unique<MockGenerator>();
CuratedDataFilePaths{ }
.locations_path = "locations.json",
.personas_path = "personas.json",
.forenames_path = "forenames-by-country.json",
.surnames_path = "surnames-by-country.json",
});
}),
di::bind<IPromptFormatter>().to([options, log_producer] {
if (options.generator.use_mocked) {
{
log_producer->Log(
{.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "Prompt formatter: none (mock mode)"});
}
return std::unique_ptr<IPromptFormatter>(nullptr);
}
{
log_producer->Log(
{.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "Prompt formatter: Gemma4JinjaPromptFormatter"});
}
return std::unique_ptr<IPromptFormatter>(
std::make_unique<Gemma4JinjaPromptFormatter>());
}),
di::bind<WebClient>().to([options, log_producer] {
if (options.generator.use_mocked) {
{
log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "Web client: none (mock mode)"});
}
return std::unique_ptr<WebClient>(nullptr);
}
log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "Web client: HttpWebClient"});
return std::unique_ptr<WebClient>(
std::make_unique<HttpWebClient>(log_producer));
}),
di::bind<IEnrichmentService>().to(
[options, &log_producer](
const auto& inj) -> std::unique_ptr<IEnrichmentService> {
if (options.generator.use_mocked) {
log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "Enrichment: mock"});
return std::make_unique<MockEnrichmentService>();
}
log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "Enrichment: Wikipedia"});
return std::make_unique<WikipediaEnrichmentService>(
inj.template create<std::unique_ptr<WebClient>>(),
log_producer);
}),
di::bind<DataGenerator>().to(
[&options, &model_path, &sampling, &prompt_directory,
&log_producer](const auto& inj) -> std::unique_ptr<DataGenerator> {
if (options.generator.use_mocked) {
log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = "Generator: mock"});
return std::make_unique<MockGenerator>();
}
#ifdef BIERGARTEN_MOCK_ONLY #ifdef BIERGARTEN_MOCK_ONLY
// Unreachable: ParseArguments requires --mocked when // Unreachable: ParseArguments requires --mocked when
// BIERGARTEN_MOCK_ONLY is compiled in, so LlamaGenerator is // BIERGARTEN_MOCK_ONLY is compiled in, so LlamaGenerator is
// never constructed and is not linked into this binary. // never constructed and is not linked into this binary.
throw std::runtime_error( throw std::runtime_error(
"LlamaGenerator is unavailable in a BIERGARTEN_MOCK_ONLY " "LlamaGenerator is unavailable in a BIERGARTEN_MOCK_ONLY "
"build"); "build");
#else #else
log_producer->Log( log_producer->Log(
{.level = LogLevel::Info, {.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = std::format( .message = std::format(
"Generator: LlamaGenerator | model={} | " "Generator: LlamaGenerator | model={} | "
"temp={:.2f} top_p={:.2f} top_k={} n_ctx={} seed={}", "temp={:.2f} top_p={:.2f} top_k={} n_ctx={} seed={}",
model_path, sampling.temperature, sampling.top_p, model_path, sampling.temperature, sampling.top_p,
sampling.top_k, sampling.n_ctx, sampling.seed)}); sampling.top_k, sampling.n_ctx, sampling.seed)});
return std::make_unique<LlamaGenerator>( return std::make_unique<LlamaGenerator>(
options, model_path, log_producer, options, model_path, log_producer,
inj.template create<std::unique_ptr<IPromptFormatter>>(), inj.template create<std::unique_ptr<IPromptFormatter>>(),
std::move(prompt_directory)); std::move(prompt_directory));
#endif #endif
})); }));
const auto orchestrator = const auto orchestrator =
injector.create<std::unique_ptr<BiergartenPipelineOrchestrator>>(); injector.create<std::unique_ptr<BiergartenPipelineOrchestrator>>();
if (!orchestrator->Run()) { if (!orchestrator->Run()) {
log_producer->Log({.level = LogLevel::Error, log_producer->Log({.level = LogLevel::Error,
.phase = PipelinePhase::Teardown,
.message = "Pipeline execution failed"});
return shutdown(EXIT_FAILURE);
}
log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Teardown, .phase = PipelinePhase::Teardown,
.message = "Pipeline execution failed"}); .message = std::format("Pipeline complete in {} ms",
timer.Elapsed())});
return shutdown(EXIT_SUCCESS);
} catch (const std::exception& exception) {
const LogDTO log_entry{.level = LogLevel::Error,
.phase = PipelinePhase::Teardown,
.message = exception.what()};
if (log_producer) {
log_producer->Log(log_entry);
} else {
std::cerr << log_entry.message << std::endl;
}
return shutdown(EXIT_FAILURE); return shutdown(EXIT_FAILURE);
} }
log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Teardown,
.message = std::format("Pipeline complete in {} ms",
timer.Elapsed())});
return shutdown(EXIT_SUCCESS);
} catch (const std::exception& exception) {
const LogDTO log_entry{.level = LogLevel::Error,
.phase = PipelinePhase::Teardown,
.message = exception.what()};
if (log_producer) {
log_producer->Log(log_entry);
} else {
std::cerr << log_entry.message << std::endl;
}
return shutdown(EXIT_FAILURE);
}
} }

360
tooling/pipeline/src/services/curated_data/curated_json_data_service.cc
View File

@@ -21,71 +21,71 @@
static std::string ReadRequiredString(const boost::json::object& object, static std::string ReadRequiredString(const boost::json::object& object,
const char* key) { const char* key) {
const boost::json::value* value = object.if_contains(key); const boost::json::value* value = object.if_contains(key);
if (value == nullptr || !value->is_string()) { if (value == nullptr || !value->is_string()) {
throw std::runtime_error( throw std::runtime_error(
std::format("Missing or invalid string field: {}", key)); std::format("Missing or invalid string field: {}", key));
} }
const std::string_view text = value->as_string(); const std::string_view text = value->as_string();
return std::string(text); return std::string(text);
} }
static std::vector<std::string> ReadRequiredStringArray( static std::vector<std::string> ReadRequiredStringArray(
const boost::json::object& object, const char* key) { const boost::json::object& object, const char* key) {
const boost::json::value* value = object.if_contains(key); const boost::json::value* value = object.if_contains(key);
if (value == nullptr || !value->is_array()) { if (value == nullptr || !value->is_array()) {
throw std::runtime_error(
std::format("Missing or invalid string array field: {}", key));
}
const auto& array = value->as_array();
std::vector<std::string> items;
items.reserve(array.size());
for (const auto& item : array) {
if (!item.is_string()) {
throw std::runtime_error( throw std::runtime_error(
std::format("Missing or invalid string array field: {}", key)); std::format("Missing or invalid string array field: {}", key));
} }
items.emplace_back(item.as_string());
} const auto& array = value->as_array();
return items; std::vector<std::string> items;
items.reserve(array.size());
for (const auto& item : array) {
if (!item.is_string()) {
throw std::runtime_error(
std::format("Missing or invalid string array field: {}", key));
}
items.emplace_back(item.as_string());
}
return items;
} }
static PostalCodeSpec ReadPostalCodeSpec(const boost::json::object& object) { static PostalCodeSpec ReadPostalCodeSpec(const boost::json::object& object) {
const boost::json::value* value = object.if_contains("postal_code"); const boost::json::value* value = object.if_contains("postal_code");
if (value == nullptr || !value->is_object()) { if (value == nullptr || !value->is_object()) {
throw std::runtime_error("Missing or invalid object field: postal_code"); throw std::runtime_error("Missing or invalid object field: postal_code");
} }
const auto& postal_code = value->as_object(); const auto& postal_code = value->as_object();
return PostalCodeSpec{ return PostalCodeSpec{
.country_format_regex = .country_format_regex =
ReadRequiredString(postal_code, "country_format_regex"), ReadRequiredString(postal_code, "country_format_regex"),
.city_regexes = ReadRequiredStringArray(postal_code, "city_regex"), .city_regexes = ReadRequiredStringArray(postal_code, "city_regex"),
.examples = ReadRequiredStringArray(postal_code, "examples"), .examples = ReadRequiredStringArray(postal_code, "examples"),
}; };
} }
namespace { namespace {
boost::json::value ParseJsonFile(const std::filesystem::path& filepath, boost::json::value ParseJsonFile(const std::filesystem::path& filepath,
const char* what) { const char* what) {
std::ifstream input(filepath); std::ifstream input(filepath);
if (!input.is_open()) { if (!input.is_open()) {
throw std::runtime_error( throw std::runtime_error(
std::format("Failed to open {} file: {}", what, filepath.string())); std::format("Failed to open {} file: {}", what, filepath.string()));
} }
std::stringstream buffer; std::stringstream buffer;
buffer << input.rdbuf(); buffer << input.rdbuf();
boost::system::error_code error; boost::system::error_code error;
boost::json::value root = boost::json::parse(buffer.str(), error); boost::json::value root = boost::json::parse(buffer.str(), error);
if (error) { if (error) {
throw std::runtime_error( throw std::runtime_error(
std::format("Failed to parse {} JSON: {}", what, error.message())); std::format("Failed to parse {} JSON: {}", what, error.message()));
} }
return root; return root;
} }
/** /**
@@ -95,18 +95,18 @@ boost::json::value ParseJsonFile(const std::filesystem::path& filepath,
*/ */
std::string ReadFirstOfStringArray(const boost::json::object& object, std::string ReadFirstOfStringArray(const boost::json::object& object,
const char* key, const char* fallback_key) { const char* key, const char* fallback_key) {
for (const char* candidate_key : {key, fallback_key}) { for (const char* candidate_key : {key, fallback_key}) {
const boost::json::value* value = object.if_contains(candidate_key); const boost::json::value* value = object.if_contains(candidate_key);
if (value == nullptr || !value->is_array()) { if (value == nullptr || !value->is_array()) {
continue; continue;
} }
const auto& array = value->as_array(); const auto& array = value->as_array();
if (!array.empty() && array.front().is_string()) { if (!array.empty() && array.front().is_string()) {
return std::string(array.front().as_string()); return std::string(array.front().as_string());
} }
} }
throw std::runtime_error( throw std::runtime_error(
std::format("Missing or invalid string array field: {}", key)); std::format("Missing or invalid string array field: {}", key));
} }
} // namespace } // namespace
@@ -115,154 +115,156 @@ CuratedJsonDataService::CuratedJsonDataService(CuratedDataFilePaths filepaths)
: filepaths_(std::move(filepaths)) {} : filepaths_(std::move(filepaths)) {}
const LocationsList& CuratedJsonDataService::LoadLocations() { const LocationsList& CuratedJsonDataService::LoadLocations() {
if (!cache_.locations.empty()) { if (!cache_.locations.empty()) {
return cache_.locations; return cache_.locations;
} }
const boost::json::value root = const boost::json::value root =
ParseJsonFile(filepaths_.locations_path, "locations"); ParseJsonFile(filepaths_.locations_path, "locations");
if (!root.is_array()) { if (!root.is_array()) {
throw std::runtime_error(
"Invalid locations JSON: root element must be an array");
}
LocationsList locations;
const auto& items = root.as_array();
locations.reserve(items.size());
for (const auto& item : items) {
if (!item.is_object()) {
throw std::runtime_error( throw std::runtime_error(
"Invalid locations JSON: each entry must be an object"); "Invalid locations JSON: root element must be an array");
} }
const auto& object = item.as_object(); LocationsList locations;
locations.push_back(City{ const auto& items = root.as_array();
.city = ReadRequiredString(object, "city"), locations.reserve(items.size());
.state_province = ReadRequiredString(object, "state_province"),
.iso3166_2 = ReadRequiredString(object, "iso3166_2"), for (const auto& item : items) {
.country = ReadRequiredString(object, "country"), if (!item.is_object()) {
.iso3166_1 = ReadRequiredString(object, "iso3166_1"), throw std::runtime_error(
.local_languages = ReadRequiredStringArray(object, "local_languages"), "Invalid locations JSON: each entry must be an object");
.postal_code = ReadPostalCodeSpec(object), }
});
} const auto& object = item.as_object();
cache_.locations = std::move(locations); locations.push_back(City{
return cache_.locations; .city = ReadRequiredString(object, "city"),
.state_province = ReadRequiredString(object, "state_province"),
.iso3166_2 = ReadRequiredString(object, "iso3166_2"),
.country = ReadRequiredString(object, "country"),
.iso3166_1 = ReadRequiredString(object, "iso3166_1"),
.local_languages = ReadRequiredStringArray(object, "local_languages"),
.postal_code = ReadPostalCodeSpec(object),
});
}
cache_.locations = std::move(locations);
return cache_.locations;
} }
const PersonasList& CuratedJsonDataService::LoadPersonas() { const PersonasList& CuratedJsonDataService::LoadPersonas() {
if (!cache_.personas.empty()) { if (!cache_.personas.empty()) {
return cache_.personas; return cache_.personas;
} }
const boost::json::value root = const boost::json::value root =
ParseJsonFile(filepaths_.personas_path, "personas"); ParseJsonFile(filepaths_.personas_path, "personas");
if (!root.is_array()) { if (!root.is_array()) {
throw std::runtime_error(
"Invalid personas JSON: root element must be an array");
}
PersonasList personas;
const auto& items = root.as_array();
personas.reserve(items.size());
for (const auto& item : items) {
if (!item.is_object()) {
throw std::runtime_error( throw std::runtime_error(
"Invalid personas JSON: each entry must be an object"); "Invalid personas JSON: root element must be an array");
} }
const auto& object = item.as_object(); PersonasList personas;
personas.push_back(UserPersona{ const auto& items = root.as_array();
.name = ReadRequiredString(object, "name"), personas.reserve(items.size());
.description = ReadRequiredString(object, "description"),
.style_affinities = ReadRequiredStringArray(object, "style_affinities"),
});
}
cache_.personas = std::move(personas); for (const auto& item : items) {
return cache_.personas; if (!item.is_object()) {
throw std::runtime_error(
"Invalid personas JSON: each entry must be an object");
}
const auto& object = item.as_object();
personas.push_back(UserPersona{
.name = ReadRequiredString(object, "name"),
.description = ReadRequiredString(object, "description"),
.style_affinities =
ReadRequiredStringArray(object, "style_affinities"),
});
}
cache_.personas = std::move(personas);
return cache_.personas;
} }
const ForenamesByCountryMap& CuratedJsonDataService::LoadForenamesByCountry() { const ForenamesByCountryMap& CuratedJsonDataService::LoadForenamesByCountry() {
if (!cache_.forenames_by_country.empty()) { if (!cache_.forenames_by_country.empty()) {
return cache_.forenames_by_country; return cache_.forenames_by_country;
} }
const boost::json::value root = const boost::json::value root =
ParseJsonFile(filepaths_.forenames_path, "forenames-by-country"); ParseJsonFile(filepaths_.forenames_path, "forenames-by-country");
if (!root.is_object()) { if (!root.is_object()) {
throw std::runtime_error( throw std::runtime_error(
"Invalid forenames-by-country JSON: root element must be an object " "Invalid forenames-by-country JSON: root element must be an object "
"keyed by ISO 3166-1 country code"); "keyed by ISO 3166-1 country code");
} }
ForenamesByCountryMap forenames_by_country; ForenamesByCountryMap forenames_by_country;
for (const auto& [country_code, regions] : root.as_object()) { for (const auto& [country_code, regions] : root.as_object()) {
if (!regions.is_array()) { if (!regions.is_array()) {
continue; continue;
}
ForenameList entries;
for (const auto& region : regions.as_array()) {
if (!region.is_object()) {
continue;
} }
const boost::json::value* names = region.as_object().if_contains("names"); ForenameList entries;
if (names == nullptr || !names->is_array()) { for (const auto& region : regions.as_array()) {
continue; if (!region.is_object()) {
continue;
}
const boost::json::value* names =
region.as_object().if_contains("names");
if (names == nullptr || !names->is_array()) {
continue;
}
for (const auto& name_value : names->as_array()) {
if (!name_value.is_object()) {
continue;
}
const auto& name_object = name_value.as_object();
entries.emplace_back(ForenameEntry{
.name = ReadFirstOfStringArray(name_object, "romanized",
"localized"),
.gender = ReadRequiredString(name_object, "gender"),
});
}
} }
for (const auto& name_value : names->as_array()) { forenames_by_country.emplace(country_code, std::move(entries));
if (!name_value.is_object()) { }
continue;
}
const auto& name_object = name_value.as_object();
entries.emplace_back(ForenameEntry{
.name =
ReadFirstOfStringArray(name_object, "romanized", "localized"),
.gender = ReadRequiredString(name_object, "gender"),
});
}
}
forenames_by_country.emplace(country_code, std::move(entries));
}
cache_.forenames_by_country = std::move(forenames_by_country); cache_.forenames_by_country = std::move(forenames_by_country);
return cache_.forenames_by_country; return cache_.forenames_by_country;
} }
const SurnamesByCountryMap& CuratedJsonDataService::LoadSurnamesByCountry() { const SurnamesByCountryMap& CuratedJsonDataService::LoadSurnamesByCountry() {
if (!cache_.surnames_by_country.empty()) { if (!cache_.surnames_by_country.empty()) {
return cache_.surnames_by_country; return cache_.surnames_by_country;
} }
const boost::json::value root = const boost::json::value root =
ParseJsonFile(filepaths_.surnames_path, "surnames-by-country"); ParseJsonFile(filepaths_.surnames_path, "surnames-by-country");
if (!root.is_object()) { if (!root.is_object()) {
throw std::runtime_error( throw std::runtime_error(
"Invalid surnames-by-country JSON: root element must be an object " "Invalid surnames-by-country JSON: root element must be an object "
"keyed by ISO 3166-1 country code"); "keyed by ISO 3166-1 country code");
} }
SurnamesByCountryMap surnames_by_country; SurnamesByCountryMap surnames_by_country;
for (const auto& [country_code, name_entries] : root.as_object()) { for (const auto& [country_code, name_entries] : root.as_object()) {
if (!name_entries.is_array()) continue; if (!name_entries.is_array()) continue;
SurnameList surnames; SurnameList surnames;
for (const auto& name_value : name_entries.as_array()) { for (const auto& name_value : name_entries.as_array()) {
if (!name_value.is_object()) { if (!name_value.is_object()) {
continue; continue;
}
surnames.emplace_back(ReadFirstOfStringArray(
name_value.as_object(), "romanized", "localized"));
} }
surnames.emplace_back(ReadFirstOfStringArray(name_value.as_object(), surnames_by_country.emplace(country_code, std::move(surnames));
"romanized", "localized")); }
}
surnames_by_country.emplace(country_code, std::move(surnames));
}
cache_.surnames_by_country = std::move(surnames_by_country); cache_.surnames_by_country = std::move(surnames_by_country);
return cache_.surnames_by_country; return cache_.surnames_by_country;
} }

6
tooling/pipeline/src/services/curated_data/mock_curated_data_service.cc
View File

@@ -92,15 +92,15 @@ MockCuratedDataService::MockCuratedDataService()
} {} } {}
const LocationsList& MockCuratedDataService::LoadLocations() { const LocationsList& MockCuratedDataService::LoadLocations() {
return locations_; return locations_;
} }
const PersonasList& MockCuratedDataService::LoadPersonas() { return personas_; } const PersonasList& MockCuratedDataService::LoadPersonas() { return personas_; }
const ForenamesByCountryMap& MockCuratedDataService::LoadForenamesByCountry() { const ForenamesByCountryMap& MockCuratedDataService::LoadForenamesByCountry() {
return forenames_by_country_; return forenames_by_country_;
} }
const SurnamesByCountryMap& MockCuratedDataService::LoadSurnamesByCountry() { const SurnamesByCountryMap& MockCuratedDataService::LoadSurnamesByCountry() {
return surnames_by_country_; return surnames_by_country_;
} }

276
tooling/pipeline/src/services/enrichment/wikipedia/fetch_extract.cc
View File

@@ -15,147 +15,147 @@
using namespace boost; using namespace boost;
std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) { std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
const std::string cache_key(query); const std::string cache_key(query);
// 1. Cache Lookup // 1. Cache Lookup
if (const auto cache_it = this->extract_cache_.find(cache_key); if (const auto cache_it = this->extract_cache_.find(cache_key);
cache_it != this->extract_cache_.end()) { cache_it != this->extract_cache_.end()) {
if (logger_) { if (logger_) {
logger_->Log({.level = LogLevel::Debug,
.phase = PipelinePhase::Enrichment,
.message = std::format("Wikipedia: Cache hit for {}!",
cache_key)});
}
return cache_it->second;
}
const std::string encoded = this->client_->EncodeURL(cache_key);
const std::string url = std::format(
"https://en.wikipedia.org/w/"
"api.php?action=query&titles={}&prop=extracts&explaintext=1&format="
"json",
encoded);
const std::string body = this->client_->Get(url);
{
using namespace std::literals::chrono_literals;
std::this_thread::sleep_for(1s);
}
// 2. Parse JSON
system::error_code ec;
json::value doc = json::parse(body, ec);
if (ec) {
if (logger_) {
logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment,
.message = std::format(
"WikipediaService: JSON parse error for '{}': {}",
std::string(query), ec.message())});
}
return {};
}
// 3. Safe Extraction
const json::object* obj = doc.if_object();
if (obj == nullptr) {
if (logger_) {
logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment,
.message = std::format(
"WikipediaService: Expected root object for '{}'",
std::string(query))});
}
return {};
}
const json::value* query_ptr = obj->if_contains("query");
const json::value* pages_ptr =
((query_ptr != nullptr) && query_ptr->is_object())
? query_ptr->get_object().if_contains("pages")
: nullptr;
if ((pages_ptr == nullptr) || !pages_ptr->is_object()) {
if (logger_) {
logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment,
.message = std::format(
"WikipediaService: Missing query.pages for '{}'",
std::string(query))});
}
return {};
}
const json::object& pages = pages_ptr->get_object();
if (pages.empty()) {
if (logger_) {
logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment,
.message = std::format(
"WikipediaService: No pages returned for '{}'",
std::string(query))});
}
this->extract_cache_.emplace(cache_key, "");
return {};
}
// Wikipedia returns the page under a dynamic ID key; we just want the first
// one
const json::value& page_val = pages.begin()->value();
if (!page_val.is_object()) {
if (logger_) {
logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment,
.message = std::format(
"WikipediaService: Unexpected page format for '{}'",
std::string(query))});
}
return {};
}
const json::object& page = page_val.get_object();
// Handle 404/Missing status
if (page.contains("missing")) {
if (logger_) {
logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment,
.message = std::format(
"WikipediaService: Page '{}' does not exist",
std::string(query))});
}
this->extract_cache_.emplace(cache_key, "");
return {};
}
const json::value* extract_ptr = page.if_contains("extract");
if ((extract_ptr == nullptr) || !extract_ptr->is_string()) {
if (logger_) {
logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment,
.message = std::format(
"WikipediaService: No extract string found for '{}'",
std::string(query))});
}
this->extract_cache_.emplace(cache_key, "");
return {};
}
// 4. Success
std::string extract(extract_ptr->as_string());
if (logger_) {
logger_->Log( logger_->Log(
{.level = LogLevel::Debug, {.level = LogLevel::Info,
.phase = PipelinePhase::Enrichment, .phase = PipelinePhase::Enrichment,
.message = std::format("Wikipedia: Cache hit for {}!", cache_key)}); .message = std::format("WikipediaService: Fetched {} chars for '{}'",
} extract.size(), std::string(query))});
return cache_it->second; }
}
const std::string encoded = this->client_->EncodeURL(cache_key); this->extract_cache_.insert_or_assign(cache_key, extract);
const std::string url = std::format(
"https://en.wikipedia.org/w/"
"api.php?action=query&titles={}&prop=extracts&explaintext=1&format="
"json",
encoded);
const std::string body = this->client_->Get(url); return extract;
{
using namespace std::literals::chrono_literals;
std::this_thread::sleep_for(1s);
}
// 2. Parse JSON
system::error_code ec;
json::value doc = json::parse(body, ec);
if (ec) {
if (logger_) {
logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment,
.message = std::format(
"WikipediaService: JSON parse error for '{}': {}",
std::string(query), ec.message())});
}
return {};
}
// 3. Safe Extraction
const json::object* obj = doc.if_object();
if (obj == nullptr) {
if (logger_) {
logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment,
.message = std::format(
"WikipediaService: Expected root object for '{}'",
std::string(query))});
}
return {};
}
const json::value* query_ptr = obj->if_contains("query");
const json::value* pages_ptr =
((query_ptr != nullptr) && query_ptr->is_object())
? query_ptr->get_object().if_contains("pages")
: nullptr;
if ((pages_ptr == nullptr) || !pages_ptr->is_object()) {
if (logger_) {
logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment,
.message = std::format(
"WikipediaService: Missing query.pages for '{}'",
std::string(query))});
}
return {};
}
const json::object& pages = pages_ptr->get_object();
if (pages.empty()) {
if (logger_) {
logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment,
.message = std::format(
"WikipediaService: No pages returned for '{}'",
std::string(query))});
}
this->extract_cache_.emplace(cache_key, "");
return {};
}
// Wikipedia returns the page under a dynamic ID key; we just want the first
// one
const json::value& page_val = pages.begin()->value();
if (!page_val.is_object()) {
if (logger_) {
logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment,
.message = std::format(
"WikipediaService: Unexpected page format for '{}'",
std::string(query))});
}
return {};
}
const json::object& page = page_val.get_object();
// Handle 404/Missing status
if (page.contains("missing")) {
if (logger_) {
logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment,
.message = std::format("WikipediaService: Page '{}' does not exist",
std::string(query))});
}
this->extract_cache_.emplace(cache_key, "");
return {};
}
const json::value* extract_ptr = page.if_contains("extract");
if ((extract_ptr == nullptr) || !extract_ptr->is_string()) {
if (logger_) {
logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment,
.message = std::format(
"WikipediaService: No extract string found for '{}'",
std::string(query))});
}
this->extract_cache_.emplace(cache_key, "");
return {};
}
// 4. Success
std::string extract(extract_ptr->as_string());
if (logger_) {
logger_->Log(
{.level = LogLevel::Info,
.phase = PipelinePhase::Enrichment,
.message = std::format("WikipediaService: Fetched {} chars for '{}'",
extract.size(), std::string(query))});
}
this->extract_cache_.insert_or_assign(cache_key, extract);
return extract;
} }

91
tooling/pipeline/src/services/enrichment/wikipedia/get_summary.cc
View File

@@ -10,55 +10,54 @@
#include "services/enrichment/wikipedia_service.h" #include "services/enrichment/wikipedia_service.h"
std::string WikipediaEnrichmentService::GetLocationContext( std::string WikipediaEnrichmentService::GetLocationContext(const City& loc) {
const City& loc) { using namespace std::literals::chrono_literals;
using namespace std::literals::chrono_literals; if (!this->client_) {
if (!this->client_) { if (logger_) {
if (logger_) { logger_->Log({.level = LogLevel::Warn,
logger_->Log({.level = LogLevel::Warn, .phase = PipelinePhase::Enrichment,
.phase = PipelinePhase::Enrichment, .message = "Wikipedia client is nullptr."});
.message = "Wikipedia client is nullptr."}); }
} return {};
return {}; }
}
std::string result; std::string result;
constexpr std::string_view brewing_query = "brewing"; constexpr std::string_view brewing_query = "brewing";
const std::string location_query = const std::string location_query =
std::format("{}, {}", loc.city, loc.iso3166_2); std::format("{}, {}", loc.city, loc.iso3166_2);
const std::string beer_query = std::format("beer in {}", loc.country); const std::string beer_query = std::format("beer in {}", loc.country);
auto append_extract = [&result](const std::string& extract) -> void { auto append_extract = [&result](const std::string& extract) -> void {
if (extract.empty()) { if (extract.empty()) {
return; return;
} }
if (!result.empty()) { if (!result.empty()) {
result += "\n\n"; result += "\n\n";
} }
result += extract; result += extract;
}; };
try { try {
append_extract(FetchExtract(brewing_query)); append_extract(FetchExtract(brewing_query));
append_extract(FetchExtract(beer_query)); append_extract(FetchExtract(beer_query));
if (logger_) { if (logger_) {
logger_->Log({.level = LogLevel::Info, logger_->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Enrichment, .phase = PipelinePhase::Enrichment,
.message = std::format( .message = std::format(
"Done fetching for {}. Sleeping for 10 seconds.", "Done fetching for {}. Sleeping for 10 seconds.",
location_query)}); location_query)});
} }
std::this_thread::sleep_for(10s); std::this_thread::sleep_for(10s);
} catch (const std::runtime_error& e) { } catch (const std::runtime_error& e) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Debug,
{.level = LogLevel::Debug, .phase = PipelinePhase::Enrichment,
.phase = PipelinePhase::Enrichment, .message = std::format(
.message = std::format("WikipediaService lookup failed for '{}': {}", "WikipediaService lookup failed for '{}': {}",
location_query, e.what())}); location_query, e.what())});
} }
} }
return result; return result;
} }

88
tooling/pipeline/src/services/logging/log_dispatcher.cc
View File

@@ -16,25 +16,25 @@
namespace { namespace {
[[nodiscard]] constexpr std::string_view PipelinePhaseToString( [[nodiscard]] constexpr std::string_view PipelinePhaseToString(
PipelinePhase phase) { PipelinePhase phase) {
switch (phase) { switch (phase) {
case PipelinePhase::Startup: case PipelinePhase::Startup:
return "Startup"; return "Startup";
case PipelinePhase::Enrichment: case PipelinePhase::Enrichment:
return "Enrichment"; return "Enrichment";
case PipelinePhase::UserGeneration: case PipelinePhase::UserGeneration:
return "User Generation"; return "User Generation";
case PipelinePhase::BreweryAndBeerGeneration: case PipelinePhase::BreweryAndBeerGeneration:
return "Brewery & Beer Gen"; return "Brewery & Beer Gen";
case PipelinePhase::CheckinGeneration: case PipelinePhase::CheckinGeneration:
return "Checkin Gen"; return "Checkin Gen";
case PipelinePhase::RatingGeneration: case PipelinePhase::RatingGeneration:
return "Rating Gen"; return "Rating Gen";
case PipelinePhase::FollowGeneration: case PipelinePhase::FollowGeneration:
return "Follow Gen"; return "Follow Gen";
case PipelinePhase::Teardown: case PipelinePhase::Teardown:
return "Teardown"; return "Teardown";
} }
return "Unknown"; return "Unknown";
} }
} // namespace } // namespace
@@ -42,35 +42,35 @@ LogDispatcher::LogDispatcher(BoundedChannel<LogEntry>& channel)
: channel_(channel) {} : channel_(channel) {}
void LogDispatcher::Run() { void LogDispatcher::Run() {
auto logger = spdlog::default_logger(); auto logger = spdlog::default_logger();
while (true) { while (true) {
auto entry = channel_.Receive(); auto entry = channel_.Receive();
if (!entry.has_value()) { if (!entry.has_value()) {
// Channel is closed and drained. // Channel is closed and drained.
break; break;
} }
const auto& log = entry.value(); const auto& log = entry.value();
logger->log(ToSpdlogLevel(log.level), logger->log(ToSpdlogLevel(log.level),
"{:<20} │ thread: {:016x} │ [{}:{}] │ {}", "{:<20} │ thread: {:016x} │ [{}:{}] │ {}",
PipelinePhaseToString(log.phase), PipelinePhaseToString(log.phase),
std::hash<std::thread::id>{}(log.thread_id), std::hash<std::thread::id>{}(log.thread_id),
log.origin.file_name(), log.origin.line(), log.message); log.origin.file_name(), log.origin.line(), log.message);
} }
} }
spdlog::level::level_enum LogDispatcher::ToSpdlogLevel(LogLevel level) { spdlog::level::level_enum LogDispatcher::ToSpdlogLevel(LogLevel level) {
switch (level) { switch (level) {
case LogLevel::Debug: case LogLevel::Debug:
return spdlog::level::debug; return spdlog::level::debug;
case LogLevel::Info: case LogLevel::Info:
return spdlog::level::info; return spdlog::level::info;
case LogLevel::Warn: case LogLevel::Warn:
return spdlog::level::warn; return spdlog::level::warn;
case LogLevel::Error: case LogLevel::Error:
return spdlog::level::err; return spdlog::level::err;
} }
return spdlog::level::info; return spdlog::level::info;
} }

40
tooling/pipeline/src/services/postal_code/xeger_postal_code_service.cc
View File

@@ -18,27 +18,27 @@ XegerPostalCodeService::XegerPostalCodeService()
: rng_(std::random_device{}()) {} : rng_(std::random_device{}()) {}
std::string XegerPostalCodeService::GeneratePostalCode(const City& city) { std::string XegerPostalCodeService::GeneratePostalCode(const City& city) {
const PostalCodeSpec& spec = city.postal_code; const PostalCodeSpec& spec = city.postal_code;
std::vector<std::string_view> city_patterns; std::vector<std::string_view> city_patterns;
for (const std::string& pattern : spec.city_regexes) { for (const std::string& pattern : spec.city_regexes) {
if (!pattern.empty()) { if (!pattern.empty()) {
city_patterns.push_back(pattern); city_patterns.push_back(pattern);
} }
} }
std::string_view pattern; std::string_view pattern;
if (!city_patterns.empty()) { if (!city_patterns.empty()) {
std::uniform_int_distribution<size_t> dist(0, city_patterns.size() - 1); std::uniform_int_distribution<size_t> dist(0, city_patterns.size() - 1);
pattern = city_patterns[dist(rng_)]; pattern = city_patterns[dist(rng_)];
} else if (!spec.country_format_regex.empty()) { } else if (!spec.country_format_regex.empty()) {
pattern = spec.country_format_regex; pattern = spec.country_format_regex;
} else { } else {
throw std::runtime_error(std::format( throw std::runtime_error(std::format(
"XegerPostalCodeService: no postal-code pattern available for " "XegerPostalCodeService: no postal-code pattern available for "
"city '{}'", "city '{}'",
city.city)); city.city));
} }
return GenerateStringFromRegex(pattern, rng_); return GenerateStringFromRegex(pattern, rng_);
} }

124
tooling/pipeline/src/services/prompt_directory.cc
View File

@@ -23,80 +23,80 @@ PromptDirectory::PromptDirectory(const std::filesystem::path& prompt_dir)
PromptDirectory::PromptDirectory(const std::filesystem::path& prompt_dir, PromptDirectory::PromptDirectory(const std::filesystem::path& prompt_dir,
std::shared_ptr<ILogger> logger) std::shared_ptr<ILogger> logger)
: prompt_dir_(prompt_dir), logger_(std::move(logger)) { : prompt_dir_(prompt_dir), logger_(std::move(logger)) {
std::error_code ec; std::error_code ec;
// Directory must exist. // Directory must exist.
if (!std::filesystem::exists(prompt_dir_, ec) || ec) { if (!std::filesystem::exists(prompt_dir_, ec) || ec) {
throw std::runtime_error( throw std::runtime_error(
"PromptDirectory: prompt directory does not exist: " + "PromptDirectory: prompt directory does not exist: " +
prompt_dir_.string()); prompt_dir_.string());
} }
// Path must be a directory, not a file. // Path must be a directory, not a file.
if (!std::filesystem::is_directory(prompt_dir_, ec) || ec) { if (!std::filesystem::is_directory(prompt_dir_, ec) || ec) {
throw std::runtime_error( throw std::runtime_error(
"PromptDirectory: prompt directory path is not a directory: " + "PromptDirectory: prompt directory path is not a directory: " +
prompt_dir_.string()); prompt_dir_.string());
} }
// Directory must be readable (probe with directory_iterator). // Directory must be readable (probe with directory_iterator).
std::filesystem::directory_iterator probe(prompt_dir_, ec); std::filesystem::directory_iterator probe(prompt_dir_, ec);
if (ec) { if (ec) {
throw std::runtime_error(std::format( throw std::runtime_error(std::format(
"PromptDirectory: prompt directory is not readable: {} ({})", "PromptDirectory: prompt directory is not readable: {} ({})",
prompt_dir_.string(), ec.message())); prompt_dir_.string(), ec.message()));
} }
if (logger_) { if (logger_) {
logger_->Log( logger_->Log(
{.level = LogLevel::Info, {.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = .message =
std::string("[PromptDirectory] Resolved prompt directory: ") + std::string("[PromptDirectory] Resolved prompt directory: ") +
prompt_dir_.string()}); prompt_dir_.string()});
} }
} }
std::string PromptDirectory::Load(std::string_view key) { std::string PromptDirectory::Load(std::string_view key) {
const std::string key_str(key); const std::string key_str(key);
// Return cached content if already loaded during this run. // Return cached content if already loaded during this run.
const auto cache_it = cache_.find(key_str); const auto cache_it = cache_.find(key_str);
if (cache_it != cache_.end()) { if (cache_it != cache_.end()) {
return cache_it->second; return cache_it->second;
} }
// Resolve <prompt_dir>/<key>.md and require it to exist. // Resolve <prompt_dir>/<key>.md and require it to exist.
const std::filesystem::path file_path = const std::filesystem::path file_path =
prompt_dir_ / std::filesystem::path(std::format("{}.md", key_str)); prompt_dir_ / std::filesystem::path(std::format("{}.md", key_str));
std::ifstream file(file_path); std::ifstream file(file_path);
if (!file.is_open()) { if (!file.is_open()) {
throw std::runtime_error( throw std::runtime_error(
std::format("PromptDirectory: prompt file not found for key '{}': {}", std::format("PromptDirectory: prompt file not found for key '{}': {}",
key_str, file_path.string())); key_str, file_path.string()));
} }
std::string content((std::istreambuf_iterator<char>(file)), std::string content((std::istreambuf_iterator<char>(file)),
std::istreambuf_iterator<char>()); std::istreambuf_iterator<char>());
file.close(); file.close();
if (content.empty()) { if (content.empty()) {
throw std::runtime_error( throw std::runtime_error(
std::format("PromptDirectory: prompt file for key '{}' is empty: {}", std::format("PromptDirectory: prompt file for key '{}' is empty: {}",
key_str, file_path.string())); key_str, file_path.string()));
} }
if (logger_) { if (logger_) {
logger_->Log( logger_->Log(
{.level = LogLevel::Info, {.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = std::format( .message = std::format(
"[PromptDirectory] Loaded prompt '{}' from '{}' ({} chars)", "[PromptDirectory] Loaded prompt '{}' from '{}' ({} chars)",
key_str, file_path.string(), content.size())}); key_str, file_path.string(), content.size())});
} }
// Cache the loaded content for future calls. // Cache the loaded content for future calls.
cache_.emplace(key_str, content); cache_.emplace(key_str, content);
return content; return content;
} }

40
tooling/pipeline/src/services/sqlite/finalize.cc
View File

@@ -9,26 +9,26 @@
#include "services/database/sqlite_export_service_helpers.h" #include "services/database/sqlite_export_service_helpers.h"
void SqliteExportService::Finalize() { void SqliteExportService::Finalize() {
if (db_handle_ == nullptr) { if (db_handle_ == nullptr) {
return; return;
} }
try { try {
insert_user_address_stmt_.reset(); insert_user_address_stmt_.reset();
insert_user_stmt_.reset(); insert_user_stmt_.reset();
insert_brewery_address_stmt_.reset(); insert_brewery_address_stmt_.reset();
insert_brewery_stmt_.reset(); insert_brewery_stmt_.reset();
insert_city_stmt_.reset(); insert_city_stmt_.reset();
if (transaction_open_) { if (transaction_open_) {
sqlite_export_service_internal::ExecSql( sqlite_export_service_internal::ExecSql(
db_handle_, "COMMIT;", "Failed to commit SQLite transaction"); db_handle_, "COMMIT;", "Failed to commit SQLite transaction");
transaction_open_ = false; transaction_open_ = false;
} }
db_handle_.reset(); db_handle_.reset();
city_cache_.clear(); city_cache_.clear();
} catch (...) { } catch (...) {
RollbackAndCloseNoThrow(); RollbackAndCloseNoThrow();
throw; throw;
} }
} }

70
tooling/pipeline/src/services/sqlite/helpers/sqlite_connection_helpers.cc
View File

@@ -6,62 +6,62 @@
namespace sqlite_export_service_internal { namespace sqlite_export_service_internal {
void SqliteDatabaseDeleter::operator()(sqlite3* handle) const noexcept { void SqliteDatabaseDeleter::operator()(sqlite3* handle) const noexcept {
if (handle != nullptr) { if (handle != nullptr) {
sqlite3_close(handle); sqlite3_close(handle);
} }
} }
void SqliteStatementDeleter::operator()( void SqliteStatementDeleter::operator()(
sqlite3_stmt* statement) const noexcept { sqlite3_stmt* statement) const noexcept {
if (statement != nullptr) { if (statement != nullptr) {
sqlite3_finalize(statement); sqlite3_finalize(statement);
} }
} }
void ThrowSqliteError(sqlite3* db_handle, std::string_view action) { void ThrowSqliteError(sqlite3* db_handle, std::string_view action) {
const std::string message = const std::string message = db_handle != nullptr ? sqlite3_errmsg(db_handle)
db_handle != nullptr ? sqlite3_errmsg(db_handle) : "unknown SQLite error"; : "unknown SQLite error";
throw std::runtime_error(std::format("{}: {}", action, message)); throw std::runtime_error(std::format("{}: {}", action, message));
} }
SqliteDatabaseHandle OpenDatabase(const std::filesystem::path& path) { SqliteDatabaseHandle OpenDatabase(const std::filesystem::path& path) {
sqlite3* raw_handle = nullptr; sqlite3* raw_handle = nullptr;
const int result = sqlite3_open(path.string().c_str(), &raw_handle); const int result = sqlite3_open(path.string().c_str(), &raw_handle);
SqliteDatabaseHandle handle(raw_handle); SqliteDatabaseHandle handle(raw_handle);
if (result != SQLITE_OK) { if (result != SQLITE_OK) {
const std::string message = raw_handle != nullptr const std::string message = raw_handle != nullptr
? sqlite3_errmsg(raw_handle) ? sqlite3_errmsg(raw_handle)
: "unknown SQLite error"; : "unknown SQLite error";
throw std::runtime_error("Failed to open SQLite export database: " + throw std::runtime_error("Failed to open SQLite export database: " +
message); message);
} }
return handle; return handle;
} }
void ExecSql(const SqliteDatabaseHandle& db_handle, std::string_view sql, void ExecSql(const SqliteDatabaseHandle& db_handle, std::string_view sql,
const char* action) { const char* action) {
char* error_message = nullptr; char* error_message = nullptr;
const std::string sql_text(sql); const std::string sql_text(sql);
const int result = sqlite3_exec(db_handle.get(), sql_text.c_str(), nullptr, const int result = sqlite3_exec(db_handle.get(), sql_text.c_str(), nullptr,
nullptr, &error_message); nullptr, &error_message);
if (result != SQLITE_OK) { if (result != SQLITE_OK) {
const std::string message = error_message != nullptr const std::string message = error_message != nullptr
? error_message ? error_message
: sqlite3_errmsg(db_handle.get()); : sqlite3_errmsg(db_handle.get());
sqlite3_free(error_message); sqlite3_free(error_message);
throw std::runtime_error(std::format("{}: {}", action, message)); throw std::runtime_error(std::format("{}: {}", action, message));
} }
} }
void RollbackTransactionNoThrow( void RollbackTransactionNoThrow(
const SqliteDatabaseHandle& db_handle) noexcept { const SqliteDatabaseHandle& db_handle) noexcept {
if (!db_handle) { if (!db_handle) {
return; return;
} }
sqlite3_exec(db_handle.get(), "ROLLBACK;", nullptr, nullptr, nullptr); sqlite3_exec(db_handle.get(), "ROLLBACK;", nullptr, nullptr, nullptr);
} }
} // namespace sqlite_export_service_internal } // namespace sqlite_export_service_internal

100
tooling/pipeline/src/services/sqlite/helpers/sqlite_statement_helpers.cc
View File

@@ -13,87 +13,87 @@ namespace sqlite_export_service_internal {
SqliteStatementHandle PrepareStatement(const SqliteDatabaseHandle& db_handle, SqliteStatementHandle PrepareStatement(const SqliteDatabaseHandle& db_handle,
std::string_view sql, std::string_view sql,
const char* action) { const char* action) {
sqlite3_stmt* raw_statement = nullptr; sqlite3_stmt* raw_statement = nullptr;
const std::string sql_text(sql); const std::string sql_text(sql);
const int result = sqlite3_prepare_v2(db_handle.get(), sql_text.c_str(), -1, const int result = sqlite3_prepare_v2(db_handle.get(), sql_text.c_str(), -1,
&raw_statement, nullptr); &raw_statement, nullptr);
SqliteStatementHandle statement(raw_statement); SqliteStatementHandle statement(raw_statement);
if (result != SQLITE_OK) { if (result != SQLITE_OK) {
ThrowSqliteError(db_handle.get(), action); ThrowSqliteError(db_handle.get(), action);
} }
return statement; return statement;
} }
void ResetStatement(SqliteStatementHandle& statement) { void ResetStatement(SqliteStatementHandle& statement) {
if (statement != nullptr) { if (statement != nullptr) {
sqlite3_reset(statement.get()); sqlite3_reset(statement.get());
sqlite3_clear_bindings(statement.get()); sqlite3_clear_bindings(statement.get());
} }
} }
void Bind(const SqliteStatementHandle& statement, void Bind(const SqliteStatementHandle& statement,
const BoundParam<std::string_view>& param) { const BoundParam<std::string_view>& param) {
const auto byte_count = param.value.size(); const auto byte_count = param.value.size();
if (byte_count > static_cast<std::size_t>(std::numeric_limits<int>::max())) { if (byte_count > static_cast<std::size_t>(std::numeric_limits<int>::max())) {
ThrowSqliteError(sqlite3_db_handle(statement.get()), param.action); ThrowSqliteError(sqlite3_db_handle(statement.get()), param.action);
} }
auto delete_char_array = [](void* data) noexcept { auto delete_char_array = [](void* data) noexcept {
// NOLINT(cppcoreguidelines-owning-memory) // NOLINT(cppcoreguidelines-owning-memory)
delete[] static_cast<char*>(data); delete[] static_cast<char*>(data);
}; };
// NOLINT(cppcoreguidelines-avoid-c-arrays, modernize-avoid-c-arrays) // NOLINT(cppcoreguidelines-avoid-c-arrays, modernize-avoid-c-arrays)
auto buffer = std::make_unique<char[]>(byte_count + 1); auto buffer = std::make_unique<char[]>(byte_count + 1);
std::memcpy(buffer.get(), param.value.data(), byte_count); std::memcpy(buffer.get(), param.value.data(), byte_count);
buffer[byte_count] = '\0'; buffer[byte_count] = '\0';
char* raw_buffer = buffer.release(); char* raw_buffer = buffer.release();
if (sqlite3_bind_text(statement.get(), param.index, raw_buffer, if (sqlite3_bind_text(statement.get(), param.index, raw_buffer,
static_cast<int>(byte_count), static_cast<int>(byte_count),
delete_char_array) != SQLITE_OK) { delete_char_array) != SQLITE_OK) {
delete_char_array(raw_buffer); delete_char_array(raw_buffer);
ThrowSqliteError(sqlite3_db_handle(statement.get()), param.action); ThrowSqliteError(sqlite3_db_handle(statement.get()), param.action);
} }
} }
void Bind(const SqliteStatementHandle& statement, void Bind(const SqliteStatementHandle& statement,
const BoundParam<double>& param) { const BoundParam<double>& param) {
if (sqlite3_bind_double(statement.get(), param.index, param.value) != if (sqlite3_bind_double(statement.get(), param.index, param.value) !=
SQLITE_OK) { SQLITE_OK) {
ThrowSqliteError(sqlite3_db_handle(statement.get()), param.action); ThrowSqliteError(sqlite3_db_handle(statement.get()), param.action);
} }
} }
void Bind(const SqliteStatementHandle& statement, void Bind(const SqliteStatementHandle& statement,
const BoundParam<sqlite3_int64>& param) { const BoundParam<sqlite3_int64>& param) {
if (sqlite3_bind_int64(statement.get(), param.index, param.value) != if (sqlite3_bind_int64(statement.get(), param.index, param.value) !=
SQLITE_OK) { SQLITE_OK) {
ThrowSqliteError(sqlite3_db_handle(statement.get()), param.action); ThrowSqliteError(sqlite3_db_handle(statement.get()), param.action);
} }
} }
void StepStatement(const SqliteDatabaseHandle& db_handle, void StepStatement(const SqliteDatabaseHandle& db_handle,
const SqliteStatementHandle& statement, const SqliteStatementHandle& statement,
std::string_view action) { std::string_view action) {
if (sqlite3_step(statement.get()) != SQLITE_DONE) { if (sqlite3_step(statement.get()) != SQLITE_DONE) {
ThrowSqliteError(db_handle.get(), action); ThrowSqliteError(db_handle.get(), action);
} }
} }
sqlite3_int64 LastInsertRowId(const SqliteDatabaseHandle& db_handle) { sqlite3_int64 LastInsertRowId(const SqliteDatabaseHandle& db_handle) {
return sqlite3_last_insert_rowid(db_handle.get()); return sqlite3_last_insert_rowid(db_handle.get());
} }
std::string SerializeVector(const std::vector<std::string>& str_vec) { std::string SerializeVector(const std::vector<std::string>& str_vec) {
boost::json::array array; boost::json::array array;
array.reserve(str_vec.size()); array.reserve(str_vec.size());
for (const auto& s : str_vec) { for (const auto& s : str_vec) {
array.emplace_back(s); array.emplace_back(s);
} }
return boost::json::serialize(array); return boost::json::serialize(array);
} }
} // namespace sqlite_export_service_internal } // namespace sqlite_export_service_internal

152
tooling/pipeline/src/services/sqlite/initialize.cc
View File

@@ -12,99 +12,99 @@
#include "services/database/sqlite_export_service.h" #include "services/database/sqlite_export_service.h"
std::filesystem::path SqliteExportService::BuildDatabasePath() const { std::filesystem::path SqliteExportService::BuildDatabasePath() const {
std::filesystem::path base_filename("biergarten_seed_" + run_timestamp_utc_ + std::filesystem::path base_filename("biergarten_seed_" + run_timestamp_utc_ +
".sqlite"); ".sqlite");
std::filesystem::path candidate = output_path_ / base_filename; std::filesystem::path candidate = output_path_ / base_filename;
for (int suffix = 1; std::filesystem::exists(candidate); ++suffix) { for (int suffix = 1; std::filesystem::exists(candidate); ++suffix) {
candidate = output_path_ / std::filesystem::path( candidate = output_path_ / std::filesystem::path(
std::format("biergarten_seed_{}-{}.sqlite", std::format("biergarten_seed_{}-{}.sqlite",
run_timestamp_utc_, suffix)); run_timestamp_utc_, suffix));
} }
return candidate; return candidate;
} }
void SqliteExportService::InitializeSchema() const { void SqliteExportService::InitializeSchema() const {
sqlite_export_service_internal::ExecSql( sqlite_export_service_internal::ExecSql(
db_handle_, sqlite_export_service_internal::kCreateCitiesTableSql, db_handle_, sqlite_export_service_internal::kCreateCitiesTableSql,
"Failed to create SQLite cities table"); "Failed to create SQLite cities table");
sqlite_export_service_internal::ExecSql( sqlite_export_service_internal::ExecSql(
db_handle_, sqlite_export_service_internal::kCreateBreweriesTableSql, db_handle_, sqlite_export_service_internal::kCreateBreweriesTableSql,
"Failed to create SQLite breweries table"); "Failed to create SQLite breweries table");
sqlite_export_service_internal::ExecSql( sqlite_export_service_internal::ExecSql(
db_handle_, db_handle_,
sqlite_export_service_internal::kCreateBreweryAddressesTableSql, sqlite_export_service_internal::kCreateBreweryAddressesTableSql,
"Failed to create SQLite brewery_addresses table"); "Failed to create SQLite brewery_addresses table");
sqlite_export_service_internal::ExecSql( sqlite_export_service_internal::ExecSql(
db_handle_, sqlite_export_service_internal::kCreateUsersTableSql, db_handle_, sqlite_export_service_internal::kCreateUsersTableSql,
"Failed to create SQLite users table"); "Failed to create SQLite users table");
sqlite_export_service_internal::ExecSql( sqlite_export_service_internal::ExecSql(
db_handle_, sqlite_export_service_internal::kCreateUserAddressesTableSql, db_handle_, sqlite_export_service_internal::kCreateUserAddressesTableSql,
"Failed to create SQLite user_addresses table"); "Failed to create SQLite user_addresses table");
} }
void SqliteExportService::PrepareStatements() { void SqliteExportService::PrepareStatements() {
insert_city_stmt_ = sqlite_export_service_internal::PrepareStatement( insert_city_stmt_ = sqlite_export_service_internal::PrepareStatement(
db_handle_, sqlite_export_service_internal::kInsertCitySql, db_handle_, sqlite_export_service_internal::kInsertCitySql,
"Failed to prepare SQLite city insert statement"); "Failed to prepare SQLite city insert statement");
insert_brewery_stmt_ = sqlite_export_service_internal::PrepareStatement( insert_brewery_stmt_ = sqlite_export_service_internal::PrepareStatement(
db_handle_, sqlite_export_service_internal::kInsertBrewerySql, db_handle_, sqlite_export_service_internal::kInsertBrewerySql,
"Failed to prepare SQLite brewery insert statement"); "Failed to prepare SQLite brewery insert statement");
insert_brewery_address_stmt_ = insert_brewery_address_stmt_ =
sqlite_export_service_internal::PrepareStatement( sqlite_export_service_internal::PrepareStatement(
db_handle_, sqlite_export_service_internal::kInsertBreweryAddressSql, db_handle_, sqlite_export_service_internal::kInsertBreweryAddressSql,
"Failed to prepare SQLite brewery address insert statement"); "Failed to prepare SQLite brewery address insert statement");
insert_user_stmt_ = sqlite_export_service_internal::PrepareStatement( insert_user_stmt_ = sqlite_export_service_internal::PrepareStatement(
db_handle_, sqlite_export_service_internal::kInsertUserSql, db_handle_, sqlite_export_service_internal::kInsertUserSql,
"Failed to prepare SQLite user insert statement"); "Failed to prepare SQLite user insert statement");
insert_user_address_stmt_ = sqlite_export_service_internal::PrepareStatement( insert_user_address_stmt_ = sqlite_export_service_internal::PrepareStatement(
db_handle_, sqlite_export_service_internal::kInsertUserAddressSql, db_handle_, sqlite_export_service_internal::kInsertUserAddressSql,
"Failed to prepare SQLite user address insert statement"); "Failed to prepare SQLite user address insert statement");
} }
void SqliteExportService::RollbackAndCloseNoThrow() noexcept { void SqliteExportService::RollbackAndCloseNoThrow() noexcept {
if (db_handle_ == nullptr) { if (db_handle_ == nullptr) {
return; return;
} }
if (transaction_open_) { if (transaction_open_) {
sqlite_export_service_internal::RollbackTransactionNoThrow(db_handle_); sqlite_export_service_internal::RollbackTransactionNoThrow(db_handle_);
transaction_open_ = false; transaction_open_ = false;
} }
insert_user_address_stmt_.reset(); insert_user_address_stmt_.reset();
insert_user_stmt_.reset(); insert_user_stmt_.reset();
insert_brewery_address_stmt_.reset(); insert_brewery_address_stmt_.reset();
insert_brewery_stmt_.reset(); insert_brewery_stmt_.reset();
insert_city_stmt_.reset(); insert_city_stmt_.reset();
db_handle_.reset(); db_handle_.reset();
city_cache_.clear(); city_cache_.clear();
} }
void SqliteExportService::Initialize() { void SqliteExportService::Initialize() {
if (db_handle_ != nullptr) { if (db_handle_ != nullptr) {
throw std::runtime_error("SQLite export service is already initialized"); throw std::runtime_error("SQLite export service is already initialized");
} }
run_timestamp_utc_ = date_time_provider_->GetUtcTimestamp(); run_timestamp_utc_ = date_time_provider_->GetUtcTimestamp();
database_path_ = BuildDatabasePath(); database_path_ = BuildDatabasePath();
std::filesystem::create_directories(database_path_.parent_path()); std::filesystem::create_directories(database_path_.parent_path());
db_handle_ = sqlite_export_service_internal::OpenDatabase(database_path_); db_handle_ = sqlite_export_service_internal::OpenDatabase(database_path_);
try { try {
sqlite_export_service_internal::ExecSql( sqlite_export_service_internal::ExecSql(
db_handle_, "PRAGMA foreign_keys = ON;", db_handle_, "PRAGMA foreign_keys = ON;",
"Failed to enable SQLite foreign keys"); "Failed to enable SQLite foreign keys");
InitializeSchema(); InitializeSchema();
PrepareStatements(); PrepareStatements();
sqlite_export_service_internal::ExecSql( sqlite_export_service_internal::ExecSql(
db_handle_, "BEGIN IMMEDIATE TRANSACTION;", db_handle_, "BEGIN IMMEDIATE TRANSACTION;",
"Failed to begin SQLite transaction"); "Failed to begin SQLite transaction");
transaction_open_ = true; transaction_open_ = true;
} catch (...) { } catch (...) {
RollbackAndCloseNoThrow(); RollbackAndCloseNoThrow();
throw; throw;
} }
} }

275
tooling/pipeline/src/services/sqlite/process_record.cc
View File

@@ -12,164 +12,165 @@
#include "services/database/sqlite_export_service_helpers.h" #include "services/database/sqlite_export_service_helpers.h"
std::string SqliteExportService::BuildCityKey(const City& city) { std::string SqliteExportService::BuildCityKey(const City& city) {
std::ostringstream key_stream; std::ostringstream key_stream;
key_stream << city.city << '\n' key_stream << city.city << '\n'
<< city.state_province << '\n' << city.state_province << '\n'
<< city.iso3166_2 << '\n' << city.iso3166_2 << '\n'
<< city.country << '\n' << city.country << '\n'
<< city.iso3166_1 << '\n' << city.iso3166_1 << '\n'
<< sqlite_export_service_internal::SerializeVector( << sqlite_export_service_internal::SerializeVector(
city.local_languages); city.local_languages);
return key_stream.str(); return key_stream.str();
} }
sqlite3_int64 SqliteExportService::ResolveCityId(const City& city) { sqlite3_int64 SqliteExportService::ResolveCityId(const City& city) {
const std::string city_key = BuildCityKey(city); const std::string city_key = BuildCityKey(city);
const auto cached_city = city_cache_.find(city_key); const auto cached_city = city_cache_.find(city_key);
if (cached_city != city_cache_.end()) { if (cached_city != city_cache_.end()) {
return cached_city->second; return cached_city->second;
} }
const std::string local_languages_json = const std::string local_languages_json =
sqlite_export_service_internal::SerializeVector(city.local_languages); sqlite_export_service_internal::SerializeVector(city.local_languages);
const std::string city_regex_json = const std::string city_regex_json =
sqlite_export_service_internal::SerializeVector( sqlite_export_service_internal::SerializeVector(
city.postal_code.city_regexes); city.postal_code.city_regexes);
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_city_stmt_, insert_city_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kCityCityBindIndex, .index = sqlite_export_service_internal::kCityCityBindIndex,
.value = city.city, .value = city.city,
.action = "Failed to bind SQLite city name"}); .action = "Failed to bind SQLite city name"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_city_stmt_, insert_city_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kCityStateProvinceBindIndex, .index = sqlite_export_service_internal::kCityStateProvinceBindIndex,
.value = city.state_province, .value = city.state_province,
.action = "Failed to bind SQLite city state/province"}); .action = "Failed to bind SQLite city state/province"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_city_stmt_, insert_city_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kCityIso31662BindIndex, .index = sqlite_export_service_internal::kCityIso31662BindIndex,
.value = city.iso3166_2, .value = city.iso3166_2,
.action = "Failed to bind SQLite city ISO 3166-2 code"}); .action = "Failed to bind SQLite city ISO 3166-2 code"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_city_stmt_, insert_city_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kCityCountryBindIndex, .index = sqlite_export_service_internal::kCityCountryBindIndex,
.value = city.country, .value = city.country,
.action = "Failed to bind SQLite city country"}); .action = "Failed to bind SQLite city country"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_city_stmt_, insert_city_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kCityIso31661BindIndex, .index = sqlite_export_service_internal::kCityIso31661BindIndex,
.value = city.iso3166_1, .value = city.iso3166_1,
.action = "Failed to bind SQLite city ISO 3166-1 code"}); .action = "Failed to bind SQLite city ISO 3166-1 code"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_city_stmt_, insert_city_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kCityLanguagesBindIndex, .index = sqlite_export_service_internal::kCityLanguagesBindIndex,
.value = local_languages_json, .value = local_languages_json,
.action = "Failed to bind SQLite city languages"}); .action = "Failed to bind SQLite city languages"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_city_stmt_, insert_city_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal:: .index = sqlite_export_service_internal::
kCityPostalCodeCountryFormatRegexBindIndex, kCityPostalCodeCountryFormatRegexBindIndex,
.value = city.postal_code.country_format_regex, .value = city.postal_code.country_format_regex,
.action = "Failed to bind SQLite city postal-code format regex"}); .action = "Failed to bind SQLite city postal-code format regex"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_city_stmt_, insert_city_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal:: .index = sqlite_export_service_internal::
kCityPostalCodeCityRegexJsonBindIndex, kCityPostalCodeCityRegexJsonBindIndex,
.value = city_regex_json, .value = city_regex_json,
.action = "Failed to bind SQLite city postal-code city regexes"}); .action = "Failed to bind SQLite city postal-code city regexes"});
sqlite_export_service_internal::StepStatement( sqlite_export_service_internal::StepStatement(
db_handle_, insert_city_stmt_, "Failed to insert SQLite city row"); db_handle_, insert_city_stmt_, "Failed to insert SQLite city row");
const sqlite3_int64 city_id = const sqlite3_int64 city_id =
sqlite_export_service_internal::LastInsertRowId(db_handle_); sqlite_export_service_internal::LastInsertRowId(db_handle_);
city_cache_.emplace(city_key, city_id); city_cache_.emplace(city_key, city_id);
sqlite_export_service_internal::ResetStatement(insert_city_stmt_); sqlite_export_service_internal::ResetStatement(insert_city_stmt_);
return city_id; return city_id;
} }
uint64_t SqliteExportService::ProcessRecord(const BreweryRecord& brewery) { uint64_t SqliteExportService::ProcessRecord(const BreweryRecord& brewery) {
if (db_handle_ == nullptr || !transaction_open_) { if (db_handle_ == nullptr || !transaction_open_) {
throw std::runtime_error("SQLite export service is not initialized"); throw std::runtime_error("SQLite export service is not initialized");
} }
const sqlite3_int64 city_id = ResolveCityId(brewery.address.city); const sqlite3_int64 city_id = ResolveCityId(brewery.address.city);
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_stmt_, insert_brewery_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kBreweryEnglishNameBindIndex, .index =
.value = brewery.brewery.name_en, sqlite_export_service_internal::kBreweryEnglishNameBindIndex,
.action = "Failed to bind SQLite brewery English name"}); .value = brewery.brewery.name_en,
.action = "Failed to bind SQLite brewery English name"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_stmt_, insert_brewery_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal:: .index = sqlite_export_service_internal::
kBreweryEnglishDescriptionBindIndex, kBreweryEnglishDescriptionBindIndex,
.value = brewery.brewery.description_en, .value = brewery.brewery.description_en,
.action = "Failed to bind SQLite brewery English description"}); .action = "Failed to bind SQLite brewery English description"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_stmt_, insert_brewery_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kBreweryLocalNameBindIndex, .index = sqlite_export_service_internal::kBreweryLocalNameBindIndex,
.value = brewery.brewery.name_local, .value = brewery.brewery.name_local,
.action = "Failed to bind SQLite brewery local name"}); .action = "Failed to bind SQLite brewery local name"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_stmt_, insert_brewery_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = .index = sqlite_export_service_internal::
sqlite_export_service_internal::kBreweryLocalDescriptionBindIndex, kBreweryLocalDescriptionBindIndex,
.value = brewery.brewery.description_local, .value = brewery.brewery.description_local,
.action = "Failed to bind SQLite brewery local description"}); .action = "Failed to bind SQLite brewery local description"});
sqlite_export_service_internal::StepStatement( sqlite_export_service_internal::StepStatement(
db_handle_, insert_brewery_stmt_, "Failed to insert SQLite brewery row"); db_handle_, insert_brewery_stmt_, "Failed to insert SQLite brewery row");
const sqlite3_int64 brewery_id = const sqlite3_int64 brewery_id =
sqlite_export_service_internal::LastInsertRowId(db_handle_); sqlite_export_service_internal::LastInsertRowId(db_handle_);
sqlite_export_service_internal::ResetStatement(insert_brewery_stmt_); sqlite_export_service_internal::ResetStatement(insert_brewery_stmt_);
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_address_stmt_, insert_brewery_address_stmt_,
sqlite_export_service_internal::BoundParam<sqlite3_int64>{ sqlite_export_service_internal::BoundParam<sqlite3_int64>{
.index = .index = sqlite_export_service_internal::
sqlite_export_service_internal::kBreweryAddressBreweryIdBindIndex, kBreweryAddressBreweryIdBindIndex,
.value = brewery_id, .value = brewery_id,
.action = "Failed to bind SQLite brewery address brewery id"}); .action = "Failed to bind SQLite brewery address brewery id"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_address_stmt_, insert_brewery_address_stmt_,
sqlite_export_service_internal::BoundParam<sqlite3_int64>{ sqlite_export_service_internal::BoundParam<sqlite3_int64>{
.index = .index =
sqlite_export_service_internal::kBreweryAddressCityIdBindIndex, sqlite_export_service_internal::kBreweryAddressCityIdBindIndex,
.value = city_id, .value = city_id,
.action = "Failed to bind SQLite brewery address city id"}); .action = "Failed to bind SQLite brewery address city id"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_address_stmt_, insert_brewery_address_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal:: .index = sqlite_export_service_internal::
kBreweryAddressPostalCodeBindIndex, kBreweryAddressPostalCodeBindIndex,
.value = brewery.address.postal_code, .value = brewery.address.postal_code,
.action = "Failed to bind SQLite brewery address postal code"}); .action = "Failed to bind SQLite brewery address postal code"});
sqlite_export_service_internal::StepStatement( sqlite_export_service_internal::StepStatement(
db_handle_, insert_brewery_address_stmt_, db_handle_, insert_brewery_address_stmt_,
"Failed to insert SQLite brewery address row"); "Failed to insert SQLite brewery address row");
sqlite_export_service_internal::ResetStatement(insert_brewery_address_stmt_); sqlite_export_service_internal::ResetStatement(insert_brewery_address_stmt_);
return static_cast<uint64_t>(brewery_id); return static_cast<uint64_t>(brewery_id);
} }

163
tooling/pipeline/src/services/sqlite/process_user_record.cc
View File

@@ -9,94 +9,95 @@
#include "services/database/sqlite_export_service_helpers.h" #include "services/database/sqlite_export_service_helpers.h"
uint64_t SqliteExportService::ProcessRecord(const UserRecord& user) { uint64_t SqliteExportService::ProcessRecord(const UserRecord& user) {
if (db_handle_ == nullptr || !transaction_open_) { if (db_handle_ == nullptr || !transaction_open_) {
throw std::runtime_error("SQLite export service is not initialized"); throw std::runtime_error("SQLite export service is not initialized");
} }
const sqlite3_int64 city_id = ResolveCityId(user.address.city); const sqlite3_int64 city_id = ResolveCityId(user.address.city);
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_stmt_, insert_user_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kUserFirstNameBindIndex, .index = sqlite_export_service_internal::kUserFirstNameBindIndex,
.value = user.user.first_name, .value = user.user.first_name,
.action = "Failed to bind SQLite user first name"}); .action = "Failed to bind SQLite user first name"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_stmt_, insert_user_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kUserLastNameBindIndex, .index = sqlite_export_service_internal::kUserLastNameBindIndex,
.value = user.user.last_name, .value = user.user.last_name,
.action = "Failed to bind SQLite user last name"}); .action = "Failed to bind SQLite user last name"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_stmt_, insert_user_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kUserGenderBindIndex, .index = sqlite_export_service_internal::kUserGenderBindIndex,
.value = user.user.gender, .value = user.user.gender,
.action = "Failed to bind SQLite user gender"}); .action = "Failed to bind SQLite user gender"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_stmt_, insert_user_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kUserUsernameBindIndex, .index = sqlite_export_service_internal::kUserUsernameBindIndex,
.value = user.user.username, .value = user.user.username,
.action = "Failed to bind SQLite user username"}); .action = "Failed to bind SQLite user username"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_stmt_, insert_user_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kUserBioBindIndex, .index = sqlite_export_service_internal::kUserBioBindIndex,
.value = user.user.bio, .value = user.user.bio,
.action = "Failed to bind SQLite user bio"}); .action = "Failed to bind SQLite user bio"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_stmt_, insert_user_stmt_,
sqlite_export_service_internal::BoundParam<double>{ sqlite_export_service_internal::BoundParam<double>{
.index = sqlite_export_service_internal::kUserActivityWeightBindIndex, .index =
.value = static_cast<double>(user.user.activity_weight), sqlite_export_service_internal::kUserActivityWeightBindIndex,
.action = "Failed to bind SQLite user activity weight"}); .value = static_cast<double>(user.user.activity_weight),
sqlite_export_service_internal::Bind( .action = "Failed to bind SQLite user activity weight"});
insert_user_stmt_, sqlite_export_service_internal::Bind(
sqlite_export_service_internal::BoundParam<std::string_view>{ insert_user_stmt_,
.index = sqlite_export_service_internal::kUserEmailBindIndex, sqlite_export_service_internal::BoundParam<std::string_view>{
.value = user.email, .index = sqlite_export_service_internal::kUserEmailBindIndex,
.action = "Failed to bind SQLite user email"}); .value = user.email,
sqlite_export_service_internal::Bind( .action = "Failed to bind SQLite user email"});
insert_user_stmt_, sqlite_export_service_internal::Bind(
sqlite_export_service_internal::BoundParam<std::string_view>{ insert_user_stmt_,
.index = sqlite_export_service_internal::kUserDateOfBirthBindIndex, sqlite_export_service_internal::BoundParam<std::string_view>{
.value = user.date_of_birth, .index = sqlite_export_service_internal::kUserDateOfBirthBindIndex,
.action = "Failed to bind SQLite user date of birth"}); .value = user.date_of_birth,
.action = "Failed to bind SQLite user date of birth"});
sqlite_export_service_internal::StepStatement( sqlite_export_service_internal::StepStatement(
db_handle_, insert_user_stmt_, "Failed to insert SQLite user row"); db_handle_, insert_user_stmt_, "Failed to insert SQLite user row");
const sqlite3_int64 user_id = const sqlite3_int64 user_id =
sqlite_export_service_internal::LastInsertRowId(db_handle_); sqlite_export_service_internal::LastInsertRowId(db_handle_);
sqlite_export_service_internal::ResetStatement(insert_user_stmt_); sqlite_export_service_internal::ResetStatement(insert_user_stmt_);
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_address_stmt_, insert_user_address_stmt_,
sqlite_export_service_internal::BoundParam<sqlite3_int64>{ sqlite_export_service_internal::BoundParam<sqlite3_int64>{
.index = sqlite_export_service_internal::kUserAddressUserIdBindIndex, .index = sqlite_export_service_internal::kUserAddressUserIdBindIndex,
.value = user_id, .value = user_id,
.action = "Failed to bind SQLite user address user id"}); .action = "Failed to bind SQLite user address user id"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_address_stmt_, insert_user_address_stmt_,
sqlite_export_service_internal::BoundParam<sqlite3_int64>{ sqlite_export_service_internal::BoundParam<sqlite3_int64>{
.index = sqlite_export_service_internal::kUserAddressCityIdBindIndex, .index = sqlite_export_service_internal::kUserAddressCityIdBindIndex,
.value = city_id, .value = city_id,
.action = "Failed to bind SQLite user address city id"}); .action = "Failed to bind SQLite user address city id"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_address_stmt_, insert_user_address_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = .index =
sqlite_export_service_internal::kUserAddressPostalCodeBindIndex, sqlite_export_service_internal::kUserAddressPostalCodeBindIndex,
.value = user.address.postal_code, .value = user.address.postal_code,
.action = "Failed to bind SQLite user address postal code"}); .action = "Failed to bind SQLite user address postal code"});
sqlite_export_service_internal::StepStatement( sqlite_export_service_internal::StepStatement(
db_handle_, insert_user_address_stmt_, db_handle_, insert_user_address_stmt_,
"Failed to insert SQLite user address row"); "Failed to insert SQLite user address row");
sqlite_export_service_internal::ResetStatement(insert_user_address_stmt_); sqlite_export_service_internal::ResetStatement(insert_user_address_stmt_);
return static_cast<uint64_t>(user_id); return static_cast<uint64_t>(user_id);
} }

6
tooling/pipeline/src/services/sqlite/sqlite_export_service.cc
View File

@@ -12,7 +12,7 @@ SqliteExportService::SqliteExportService(const ApplicationOptions& options)
output_path_(options.pipeline.output_path) {} output_path_(options.pipeline.output_path) {}
SqliteExportService::~SqliteExportService() { SqliteExportService::~SqliteExportService() {
if (db_handle_ != nullptr) { if (db_handle_ != nullptr) {
RollbackAndCloseNoThrow(); RollbackAndCloseNoThrow();
} }
} }

8
tooling/pipeline/src/services/xeger/xeger.cc
View File

@@ -16,8 +16,8 @@
std::string GenerateStringFromRegex(std::string_view pattern, std::mt19937& rng, std::string GenerateStringFromRegex(std::string_view pattern, std::mt19937& rng,
const int unbounded_repeat_cap) { const int unbounded_repeat_cap) {
xeger_internal::Parser parser(pattern); xeger_internal::Parser parser(pattern);
const xeger_internal::NodePtr ast = parser.Parse(); const xeger_internal::NodePtr ast = parser.Parse();
xeger_internal::XegerGenerator generator(rng, unbounded_repeat_cap); xeger_internal::XegerGenerator generator(rng, unbounded_repeat_cap);
return generator.Run(*ast); return generator.Run(*ast);
} }

124
tooling/pipeline/src/services/xeger/xeger_generator.cc
View File

@@ -3,108 +3,108 @@
* @brief Visitor that walks a regex AST and emits a random matching string. * @brief Visitor that walks a regex AST and emits a random matching string.
*/ */
#include "services/xeger/xeger_internal.h"
#include <cassert> #include <cassert>
#include <limits> #include <limits>
#include <random> #include <random>
#include <stdexcept> #include <stdexcept>
#include <string> #include <string>
#include "services/xeger/xeger_internal.h"
namespace xeger_internal { namespace xeger_internal {
XegerGenerator::XegerGenerator(std::mt19937& rng, const int unbounded_cap) XegerGenerator::XegerGenerator(std::mt19937& rng, const int unbounded_cap)
: rng_(rng), unbounded_cap_(unbounded_cap) {} : rng_(rng), unbounded_cap_(unbounded_cap) {}
std::string XegerGenerator::Run(const Node& root) { std::string XegerGenerator::Run(const Node& root) {
out_.clear(); out_.clear();
root.Accept(*this); root.Accept(*this);
return out_; return out_;
} }
void XegerGenerator::Visit(const Literal& node) { out_ += node.ch; } void XegerGenerator::Visit(const Literal& node) { out_ += node.ch; }
void XegerGenerator::Visit(const AnyChar&) { void XegerGenerator::Visit(const AnyChar&) {
static constexpr int kPrintableLow = ' '; // empty space - character 32 static constexpr int kPrintableLow = ' '; // empty space - character 32
static constexpr int kPrintableHigh = '~'; // tilde - character 126 static constexpr int kPrintableHigh = '~'; // tilde - character 126
std::uniform_int_distribution<int> dist(kPrintableLow, kPrintableHigh); std::uniform_int_distribution<int> dist(kPrintableLow, kPrintableHigh);
// assert that the rand number is in the char range // assert that the rand number is in the char range
bool inCharRange = dist.min() >= std::numeric_limits<char>::min() && bool inCharRange = dist.min() >= std::numeric_limits<char>::min() &&
dist.max() <= std::numeric_limits<char>::max(); dist.max() <= std::numeric_limits<char>::max();
assert(inCharRange && "Random number is out of char range"); assert(inCharRange && "Random number is out of char range");
out_ += static_cast<char>(dist(rng_)); out_ += static_cast<char>(dist(rng_));
} }
void XegerGenerator::Visit(const CharClass& node) { void XegerGenerator::Visit(const CharClass& node) {
out_ += PickFromClass(node); out_ += PickFromClass(node);
} }
void XegerGenerator::Visit(const Concat& node) { void XegerGenerator::Visit(const Concat& node) {
for (const NodePtr& part : node.parts) { for (const NodePtr& part : node.parts) {
part->Accept(*this); part->Accept(*this);
} }
} }
void XegerGenerator::Visit(const Alternation& node) { void XegerGenerator::Visit(const Alternation& node) {
std::uniform_int_distribution<size_t> dist(0, node.options.size() - 1); std::uniform_int_distribution<size_t> dist(0, node.options.size() - 1);
node.options[dist(rng_)]->Accept(*this); node.options[dist(rng_)]->Accept(*this);
} }
void XegerGenerator::Visit(const Repeat& node) { void XegerGenerator::Visit(const Repeat& node) {
const int lo = node.min; const int lo = node.min;
const int hi = (node.max < 0) ? node.min + unbounded_cap_ : node.max; const int hi = (node.max < 0) ? node.min + unbounded_cap_ : node.max;
std::uniform_int_distribution<int> dist(lo, hi); std::uniform_int_distribution<int> dist(lo, hi);
const int count = dist(rng_); const int count = dist(rng_);
for (int i = 0; i < count; ++i) { for (int i = 0; i < count; ++i) {
node.child->Accept(*this); node.child->Accept(*this);
} }
} }
bool XegerGenerator::InRanges(const CharClass& node, const char c) { bool XegerGenerator::InRanges(const CharClass& node, const char c) {
for (const auto& [lo, hi] : node.ranges) { for (const auto& [lo, hi] : node.ranges) {
if (c >= lo && c <= hi) { if (c >= lo && c <= hi) {
return true; return true;
} }
} }
return false; return false;
} }
char XegerGenerator::PickFromClass(const CharClass& node) { char XegerGenerator::PickFromClass(const CharClass& node) {
if (!node.negated) { if (!node.negated) {
int total = 0; int total = 0;
for (const auto& [lo, hi] : node.ranges) { for (const auto& [lo, hi] : node.ranges) {
total += hi - lo + 1; total += hi - lo + 1;
}
std::uniform_int_distribution<int> dist(0, total - 1);
int k = dist(rng_);
for (const auto& [lo, hi] : node.ranges) {
const int width = hi - lo + 1;
if (k < width) {
return static_cast<char>(lo + k);
} }
k -= width; std::uniform_int_distribution<int> dist(0, total - 1);
} int k = dist(rng_);
return node.ranges.back().second; // unreachable for (const auto& [lo, hi] : node.ranges) {
} const int width = hi - lo + 1;
if (k < width) {
return static_cast<char>(lo + k);
}
k -= width;
}
return node.ranges.back().second; // unreachable
}
// Negated class: draw random printable characters until one falls // Negated class: draw random printable characters until one falls
// outside the forbidden set. // outside the forbidden set.
static constexpr int kPrintableLow = 32; static constexpr int kPrintableLow = 32;
static constexpr int kPrintableHigh = 126; static constexpr int kPrintableHigh = 126;
static constexpr int kMaxAttempts = 1000; static constexpr int kMaxAttempts = 1000;
std::uniform_int_distribution<int> dist(kPrintableLow, kPrintableHigh); std::uniform_int_distribution<int> dist(kPrintableLow, kPrintableHigh);
for (int attempt = 0; attempt < kMaxAttempts; ++attempt) { for (int attempt = 0; attempt < kMaxAttempts; ++attempt) {
const char c = static_cast<char>(dist(rng_)); const char c = static_cast<char>(dist(rng_));
if (!InRanges(node, c)) { if (!InRanges(node, c)) {
return c; return c;
} }
} }
throw std::runtime_error( throw std::runtime_error(
"GenerateStringFromRegex: negated class rules out every printable " "GenerateStringFromRegex: negated class rules out every printable "
"character"); "character");
} }
} // namespace xeger_internal } // namespace xeger_internal

479
tooling/pipeline/src/services/xeger/xeger_parser.cc
View File

@@ -3,8 +3,6 @@
* @brief Recursive-descent parser that turns a regex pattern into an AST. * @brief Recursive-descent parser that turns a regex pattern into an AST.
*/ */
#include "services/xeger/xeger_internal.h"
#include <cctype> #include <cctype>
#include <format> #include <format>
#include <memory> #include <memory>
@@ -14,31 +12,33 @@
#include <utility> #include <utility>
#include <vector> #include <vector>
#include "services/xeger/xeger_internal.h"
namespace xeger_internal { namespace xeger_internal {
Parser::Parser(std::string_view pattern) : src_(StripAnchors(pattern)) {} Parser::Parser(std::string_view pattern) : src_(StripAnchors(pattern)) {}
NodePtr Parser::Parse() { NodePtr Parser::Parse() {
NodePtr node = ParseAlternation(); NodePtr node = ParseAlternation();
if (!AtEnd()) { if (!AtEnd()) {
throw std::runtime_error( throw std::runtime_error(
std::format("GenerateStringFromRegex: unexpected '{}' at position {}", std::format("GenerateStringFromRegex: unexpected '{}' at position {}",
Peek(), pos_)); Peek(), pos_));
} }
return node; return node;
} }
std::string Parser::StripAnchors(std::string_view pattern) { std::string Parser::StripAnchors(std::string_view pattern) {
if (!pattern.empty() && pattern.front() == kAnchorStart) { if (!pattern.empty() && pattern.front() == kAnchorStart) {
pattern.remove_prefix(1); pattern.remove_prefix(1);
} }
const bool has_trailing_dollar = const bool has_trailing_dollar =
!pattern.empty() && pattern.back() == kAnchorEnd && !pattern.empty() && pattern.back() == kAnchorEnd &&
(pattern.size() < 2 || pattern[pattern.size() - 2] != kEscape); (pattern.size() < 2 || pattern[pattern.size() - 2] != kEscape);
if (has_trailing_dollar) { if (has_trailing_dollar) {
pattern.remove_suffix(1); pattern.remove_suffix(1);
} }
return std::string(pattern); return std::string(pattern);
} }
bool Parser::AtEnd() const { return pos_ >= src_.size(); } bool Parser::AtEnd() const { return pos_ >= src_.size(); }
@@ -46,262 +46,263 @@ char Parser::Peek() const { return src_[pos_]; }
char Parser::Advance() { return src_[pos_++]; } char Parser::Advance() { return src_[pos_++]; }
bool Parser::Match(const char c) { bool Parser::Match(const char c) {
if (!AtEnd() && Peek() == c) { if (!AtEnd() && Peek() == c) {
++pos_; ++pos_;
return true; return true;
} }
return false; return false;
} }
void Parser::Expect(const char c) { void Parser::Expect(const char c) {
if (!Match(c)) { if (!Match(c)) {
throw std::runtime_error(std::format( throw std::runtime_error(std::format(
"GenerateStringFromRegex: expected '{}' at position {}", c, pos_)); "GenerateStringFromRegex: expected '{}' at position {}", c, pos_));
} }
} }
NodePtr Parser::ParseAlternation() { NodePtr Parser::ParseAlternation() {
std::vector<NodePtr> options; std::vector<NodePtr> options;
options.push_back(ParseConcat()); options.push_back(ParseConcat());
while (Match(kAlternation)) { while (Match(kAlternation)) {
options.push_back(ParseConcat()); options.push_back(ParseConcat());
} }
if (options.size() == 1) { if (options.size() == 1) {
return std::move(options.front()); return std::move(options.front());
} }
auto alternation = std::make_unique<Alternation>(); auto alternation = std::make_unique<Alternation>();
alternation->options = std::move(options); alternation->options = std::move(options);
return alternation; return alternation;
} }
NodePtr Parser::ParseConcat() { NodePtr Parser::ParseConcat() {
auto concat = std::make_unique<Concat>(); auto concat = std::make_unique<Concat>();
while (!AtEnd() && Peek() != kAlternation && Peek() != kGroupClose) { while (!AtEnd() && Peek() != kAlternation && Peek() != kGroupClose) {
concat->parts.push_back(ParseRepeat()); concat->parts.push_back(ParseRepeat());
} }
if (concat->parts.size() == 1) { if (concat->parts.size() == 1) {
return std::move(concat->parts.front()); return std::move(concat->parts.front());
} }
return concat; return concat;
} }
NodePtr Parser::ParseRepeat() { NodePtr Parser::ParseRepeat() {
NodePtr atom = ParseAtom(); NodePtr atom = ParseAtom();
if (AtEnd()) { if (AtEnd()) {
return atom;
}
int lo = 0;
int hi = -1;
switch (Peek()) {
case kStar:
Advance();
lo = 0;
hi = -1;
break;
case kPlus:
Advance();
lo = 1;
hi = -1;
break;
case kOptional:
Advance();
lo = 0;
hi = 1;
break;
case kBraceOpen:
return ParseBrace(std::move(atom));
default:
return atom; return atom;
} }
auto repeat = std::make_unique<Repeat>();
repeat->child = std::move(atom); int lo = 0;
repeat->min = lo; int hi = -1;
repeat->max = hi; switch (Peek()) {
return repeat; case kStar:
Advance();
lo = 0;
hi = -1;
break;
case kPlus:
Advance();
lo = 1;
hi = -1;
break;
case kOptional:
Advance();
lo = 0;
hi = 1;
break;
case kBraceOpen:
return ParseBrace(std::move(atom));
default:
return atom;
}
auto repeat = std::make_unique<Repeat>();
repeat->child = std::move(atom);
repeat->min = lo;
repeat->max = hi;
return repeat;
} }
NodePtr Parser::ParseBrace(NodePtr atom) { NodePtr Parser::ParseBrace(NodePtr atom) {
Expect(kBraceOpen); Expect(kBraceOpen);
std::string lower_digits; std::string lower_digits;
std::string upper_digits; std::string upper_digits;
bool has_comma = false; bool has_comma = false;
while (!AtEnd() && std::isdigit(static_cast<unsigned char>(Peek())) != 0) { while (!AtEnd() && std::isdigit(static_cast<unsigned char>(Peek())) != 0) {
lower_digits += Advance(); lower_digits += Advance();
} }
if (Match(kBraceComma)) { if (Match(kBraceComma)) {
has_comma = true; has_comma = true;
while (!AtEnd() && while (!AtEnd() &&
std::isdigit(static_cast<unsigned char>(Peek())) != 0) { std::isdigit(static_cast<unsigned char>(Peek())) != 0) {
upper_digits += Advance(); upper_digits += Advance();
} }
} }
Expect(kBraceClose); Expect(kBraceClose);
auto repeat = std::make_unique<Repeat>(); auto repeat = std::make_unique<Repeat>();
repeat->child = std::move(atom); repeat->child = std::move(atom);
repeat->min = lower_digits.empty() ? 0 : std::stoi(lower_digits); repeat->min = lower_digits.empty() ? 0 : std::stoi(lower_digits);
if (!has_comma) { if (!has_comma) {
repeat->max = repeat->min; repeat->max = repeat->min;
} else if (upper_digits.empty()) { } else if (upper_digits.empty()) {
repeat->max = -1; repeat->max = -1;
} else { } else {
repeat->max = std::stoi(upper_digits); repeat->max = std::stoi(upper_digits);
} }
if (repeat->max != -1 && repeat->max < repeat->min) { if (repeat->max != -1 && repeat->max < repeat->min) {
throw std::runtime_error( throw std::runtime_error(
"GenerateStringFromRegex: repetition max is smaller than min"); "GenerateStringFromRegex: repetition max is smaller than min");
} }
return repeat; return repeat;
} }
NodePtr Parser::ParseAtom() { NodePtr Parser::ParseAtom() {
if (AtEnd()) { if (AtEnd()) {
throw std::runtime_error( throw std::runtime_error(
"GenerateStringFromRegex: pattern ended unexpectedly"); "GenerateStringFromRegex: pattern ended unexpectedly");
} }
const char c = Peek(); const char c = Peek();
switch (c) { switch (c) {
case kGroupOpen: { case kGroupOpen: {
Advance(); Advance();
// Non-capturing group "(?:...)" -- capture semantics don't apply // Non-capturing group "(?:...)" -- capture semantics don't apply
// here (nothing back-references a group), so it's treated exactly // here (nothing back-references a group), so it's treated exactly
// like a plain group once the "?:" marker is consumed. // like a plain group once the "?:" marker is consumed.
if (!AtEnd() && Peek() == kNonCapturingMarker) { if (!AtEnd() && Peek() == kNonCapturingMarker) {
if (pos_ + 1 < src_.size() && src_[pos_ + 1] == kNonCapturingColon) { if (pos_ + 1 < src_.size() &&
pos_ += 2; src_[pos_ + 1] == kNonCapturingColon) {
} else { pos_ += 2;
throw std::runtime_error(std::format( } else {
"GenerateStringFromRegex: unsupported group syntax at " throw std::runtime_error(std::format(
"position {} (only plain '(...)' and non-capturing " "GenerateStringFromRegex: unsupported group syntax at "
"'(?:...)' groups are supported)", "position {} (only plain '(...)' and non-capturing "
pos_)); "'(?:...)' groups are supported)",
} pos_));
}
}
NodePtr inner = ParseAlternation();
Expect(kGroupClose);
return inner;
} }
NodePtr inner = ParseAlternation(); case kClassOpen:
Expect(kGroupClose); return ParseCharClass();
return inner; case kAnyChar:
} Advance();
case kClassOpen: return std::make_unique<AnyChar>();
return ParseCharClass(); case kEscape:
case kAnyChar: Advance();
Advance(); return ParseEscape();
return std::make_unique<AnyChar>(); case kStar:
case kEscape: case kPlus:
Advance(); case kOptional:
return ParseEscape(); case kBraceOpen:
case kStar: throw std::runtime_error(std::format(
case kPlus: "GenerateStringFromRegex: quantifier with no preceding item at "
case kOptional: "position {}",
case kBraceOpen: pos_));
throw std::runtime_error(std::format( default:
"GenerateStringFromRegex: quantifier with no preceding item at " Advance();
"position {}", return std::make_unique<Literal>(c);
pos_)); }
default:
Advance();
return std::make_unique<Literal>(c);
}
} }
NodePtr Parser::MakeClass(std::vector<std::pair<char, char>> ranges, NodePtr Parser::MakeClass(std::vector<std::pair<char, char>> ranges,
const bool negated) { const bool negated) {
auto char_class = std::make_unique<CharClass>(); auto char_class = std::make_unique<CharClass>();
char_class->ranges = std::move(ranges); char_class->ranges = std::move(ranges);
char_class->negated = negated; char_class->negated = negated;
return char_class; return char_class;
} }
NodePtr Parser::ParseEscape() { NodePtr Parser::ParseEscape() {
if (AtEnd()) { if (AtEnd()) {
throw std::runtime_error( throw std::runtime_error(
"GenerateStringFromRegex: a backslash at the very end has nothing " "GenerateStringFromRegex: a backslash at the very end has nothing "
"to escape"); "to escape");
} }
const char c = Advance(); const char c = Advance();
switch (c) { switch (c) {
case 'd': case 'd':
return MakeClass({{'0', '9'}}, false); return MakeClass({{'0', '9'}}, false);
case 'D': case 'D':
return MakeClass({{'0', '9'}}, true); return MakeClass({{'0', '9'}}, true);
case 'w': case 'w':
return MakeClass({{'a', 'z'}, {'A', 'Z'}, {'0', '9'}, {'_', '_'}}, return MakeClass({{'a', 'z'}, {'A', 'Z'}, {'0', '9'}, {'_', '_'}},
false); false);
case 'W': case 'W':
return MakeClass({{'a', 'z'}, {'A', 'Z'}, {'0', '9'}, {'_', '_'}}, return MakeClass({{'a', 'z'}, {'A', 'Z'}, {'0', '9'}, {'_', '_'}},
true); true);
case 's': case 's':
return MakeClass({{' ', ' '}, {'\t', '\t'}, {'\n', '\n'}, {'\r', '\r'}}, return MakeClass(
false); {{' ', ' '}, {'\t', '\t'}, {'\n', '\n'}, {'\r', '\r'}}, false);
case 'S': case 'S':
return MakeClass({{' ', ' '}, {'\t', '\t'}, {'\n', '\n'}, {'\r', '\r'}}, return MakeClass(
true); {{' ', ' '}, {'\t', '\t'}, {'\n', '\n'}, {'\r', '\r'}}, true);
case 'n': case 'n':
return std::make_unique<Literal>('\n'); return std::make_unique<Literal>('\n');
case 't': case 't':
return std::make_unique<Literal>('\t'); return std::make_unique<Literal>('\t');
case 'r': case 'r':
return std::make_unique<Literal>('\r'); return std::make_unique<Literal>('\r');
default: default:
// \. \* \( \\ etc. -- the literal character itself. // \. \* \( \\ etc. -- the literal character itself.
return std::make_unique<Literal>(c); return std::make_unique<Literal>(c);
} }
} }
NodePtr Parser::ParseCharClass() { NodePtr Parser::ParseCharClass() {
Expect(kClassOpen); Expect(kClassOpen);
auto char_class = std::make_unique<CharClass>(); auto char_class = std::make_unique<CharClass>();
if (Match(kClassNegate)) { if (Match(kClassNegate)) {
char_class->negated = true; char_class->negated = true;
} }
if (AtEnd()) { if (AtEnd()) {
throw std::runtime_error( throw std::runtime_error(
"GenerateStringFromRegex: character class '[' is never closed"); "GenerateStringFromRegex: character class '[' is never closed");
} }
while (!AtEnd() && Peek() != kClassClose) { while (!AtEnd() && Peek() != kClassClose) {
char lo = ClassChar(); char lo = ClassChar();
const bool is_range = Peek() == kClassRange && pos_ + 1 < src_.size() && const bool is_range = Peek() == kClassRange && pos_ + 1 < src_.size() &&
src_[pos_ + 1] != kClassClose; src_[pos_ + 1] != kClassClose;
if (is_range) { if (is_range) {
Advance(); Advance();
char hi = ClassChar(); char hi = ClassChar();
if (hi < lo) { if (hi < lo) {
std::swap(lo, hi); std::swap(lo, hi);
}
char_class->ranges.emplace_back(lo, hi);
} else {
char_class->ranges.emplace_back(lo, lo);
} }
char_class->ranges.emplace_back(lo, hi); }
} else { Expect(kClassClose);
char_class->ranges.emplace_back(lo, lo); if (char_class->ranges.empty()) {
} throw std::runtime_error(
} "GenerateStringFromRegex: empty character class []");
Expect(kClassClose); }
if (char_class->ranges.empty()) { return char_class;
throw std::runtime_error(
"GenerateStringFromRegex: empty character class []");
}
return char_class;
} }
char Parser::ClassChar() { char Parser::ClassChar() {
const char c = Advance(); const char c = Advance();
if (c != kEscape) { if (c != kEscape) {
return c; return c;
} }
if (AtEnd()) { if (AtEnd()) {
throw std::runtime_error( throw std::runtime_error(
"GenerateStringFromRegex: backslash at end of character class"); "GenerateStringFromRegex: backslash at end of character class");
} }
const char escaped = Advance(); const char escaped = Advance();
switch (escaped) { switch (escaped) {
case 'n': case 'n':
return '\n'; return '\n';
case 't': case 't':
return '\t'; return '\t';
case 'r': case 'r':
return '\r'; return '\r';
default: default:
return escaped; return escaped;
} }
} }
} // namespace xeger_internal } // namespace xeger_internal

60
tooling/pipeline/src/web_client/http_web_client.cc
View File

@@ -26,47 +26,47 @@ const std::regex kUrlRegex(
R"(^(https?://[^/?#]+)(/[^?#]*(?:\?[^#]*)?(?:#.*)?)?)"); R"(^(https?://[^/?#]+)(/[^?#]*(?:\?[^#]*)?(?:#.*)?)?)");
std::pair<std::string, std::string> SplitUrl(const std::string& url) { std::pair<std::string, std::string> SplitUrl(const std::string& url) {
std::smatch match; std::smatch match;
if (!std::regex_match(url, match, kUrlRegex)) { if (!std::regex_match(url, match, kUrlRegex)) {
throw std::invalid_argument("[HttpWebClient] Malformed URL: " + url); throw std::invalid_argument("[HttpWebClient] Malformed URL: " + url);
} }
return {match[1].str(), match[2].matched ? match[2].str() : "/"}; return {match[1].str(), match[2].matched ? match[2].str() : "/"};
} }
} // namespace } // namespace
std::string HttpWebClient::Get(const std::string& url) { std::string HttpWebClient::Get(const std::string& url) {
const auto [origin, path] = SplitUrl(url); const auto [origin, path] = SplitUrl(url);
httplib::Client client(origin); httplib::Client client(origin);
client.set_follow_location(true); client.set_follow_location(true);
client.set_connection_timeout(kConnectionTimeoutSeconds); client.set_connection_timeout(kConnectionTimeoutSeconds);
client.set_read_timeout(kReadTimeoutSeconds); client.set_read_timeout(kReadTimeoutSeconds);
client.set_default_headers({{"Accept", "application/json"}, client.set_default_headers({{"Accept", "application/json"},
{"User-Agent", "biergarten-pipeline/1.0"}}); {"User-Agent", "biergarten-pipeline/1.0"}});
const httplib::Result result = client.Get(path); const httplib::Result result = client.Get(path);
if (!result) { if (!result) {
throw std::runtime_error( throw std::runtime_error(
std::format("[HttpWebClient] Request failed for URL: {} — {}", url, std::format("[HttpWebClient] Request failed for URL: {} — {}", url,
httplib::to_string(result.error()))); httplib::to_string(result.error())));
} }
if (result->status < kSuccessMin || result->status >= kSuccessMax) { if (result->status < kSuccessMin || result->status >= kSuccessMax) {
if (logger_) { if (logger_) {
logger_->Log({.level = LogLevel::Error, logger_->Log({.level = LogLevel::Error,
.phase = PipelinePhase::Enrichment, .phase = PipelinePhase::Enrichment,
.message = std::format( .message = std::format(
"[HttpWebClient] Request failed for URL: {}", url)}); "[HttpWebClient] Request failed for URL: {}", url)});
} }
throw std::runtime_error(std::format("[HttpWebClient] HTTP {} for URL: {}", throw std::runtime_error(std::format(
result->status, url)); "[HttpWebClient] HTTP {} for URL: {}", result->status, url));
} }
return result->body; return result->body;
} }
std::string HttpWebClient::EncodeURL(const std::string& value) { std::string HttpWebClient::EncodeURL(const std::string& value) {
return httplib::encode_uri_component(value); return httplib::encode_uri_component(value);
} }

18245
tooling/pipeline/surnames-by-country.json
View File

File diff suppressed because it is too large Load Diff