Compare commits

...

33 Commits

Author SHA1 Message Date
Aaron Po
3154c2feed Formatted src/web_client 2026-07-19 06:50:56 -04:00
Aaron Po
3df657dc54 Formatted src/services/xeger 2026-07-19 06:50:56 -04:00
Aaron Po
a238db1d37 Formatted src/services/sqlite/helpers 2026-07-19 06:50:56 -04:00
Aaron Po
5d48d6757a Formatted src/services/sqlite 2026-07-19 06:50:56 -04:00
Aaron Po
34a036d7b3 Formatted src/services/postal_code 2026-07-19 06:50:56 -04:00
Aaron Po
59053fcddd Formatted src/services/logging 2026-07-19 06:50:56 -04:00
Aaron Po
d4c7260127 Formatted src/services/enrichment/wikipedia 2026-07-19 06:50:56 -04:00
Aaron Po
be90522eae Formatted src/services/curated_data 2026-07-19 06:50:56 -04:00
Aaron Po
d055a634c2 Formatted src/services 2026-07-19 06:50:56 -04:00
Aaron Po
933b728135 Formatted src/data_generation/prompt_formatting 2026-07-19 06:50:56 -04:00
Aaron Po
ac98a2a22e Formatted src/data_generation/mock 2026-07-19 06:50:56 -04:00
Aaron Po
d723f53cba Formatted src/data_generation/llama 2026-07-19 06:50:56 -04:00
Aaron Po
1885d9859f Formatted src/biergarten_pipeline_orchestrator 2026-07-19 06:50:56 -04:00
Aaron Po
03467cbb65 Formatted src/application_options 2026-07-19 06:50:56 -04:00
Aaron Po
1f25d25dce Formatted src 2026-07-19 06:50:56 -04:00
Aaron Po
67bc5bfc2a Formatted includes/web_client 2026-07-19 06:50:56 -04:00
Aaron Po
eb32a55831 Formatted includes/services/xeger 2026-07-19 06:50:56 -04:00
Aaron Po
fad1611685 Formatted includes/services/prompting 2026-07-19 06:50:56 -04:00
Aaron Po
74f848d015 Formatted includes/services/postal_code 2026-07-19 06:50:56 -04:00
Aaron Po
86db1c2827 Formatted includes/services/logging 2026-07-19 06:50:56 -04:00
Aaron Po
9bcf4c6879 Formatted includes/services/enrichment 2026-07-19 06:50:56 -04:00
Aaron Po
d9b10df436 Formatted includes/services/datetime 2026-07-19 06:50:56 -04:00
Aaron Po
4f0f56d7d7 Formatted includes/services/database 2026-07-19 06:50:56 -04:00
Aaron Po
6c07edf845 Formatted includes/services/curated_data 2026-07-19 06:50:56 -04:00
Aaron Po
969125e3a5 Formatted includes/json_handling 2026-07-19 06:50:56 -04:00
Aaron Po
ffabd85b6d Formatted includes/data_model 2026-07-19 06:50:56 -04:00
Aaron Po
4fae5ead08 Formatted includes/data_generation/prompt_formatting 2026-07-19 06:50:56 -04:00
Aaron Po
efd0d4f2aa Formatted includes/data_generation 2026-07-19 06:50:56 -04:00
Aaron Po
522381357a Formatted includes/concurrency 2026-07-19 06:50:56 -04:00
Aaron Po
5f3706bb93 Formatted includes 2026-07-19 06:50:56 -04:00
Aaron Po
a52b14c49a update clang format 2026-07-19 06:50:56 -04:00
Aaron Po
4ebd28c06b format docs 2026-07-19 06:50:56 -04:00
Aaron Po
9593a07c13 Format curated data and prompts 2026-07-19 06:50:56 -04:00
78 changed files with 13388 additions and 32143 deletions

View File

@@ -102,21 +102,19 @@ vendored verbatim, unmodified, from
(the `common-forenames-by-country.json` / `common-surnames-by-country.json` (the `common-forenames-by-country.json` / `common-surnames-by-country.json`
release assets), released under **CC0** (public domain). That dataset's own release assets), released under **CC0** (public domain). That dataset's own
forename/surname lists are pulled from Wikipedia's "Lists of most common forename/surname lists are pulled from Wikipedia's "Lists of most common
surnames" and "List of most popular given names" as of the week of surnames" and "List of most popular given names" as of the week of 2023-07-08 —
2023-07-08 — see that project's README for full provenance. Names are not see that project's README for full provenance. Names are not LLM-generated; this
LLM-generated; this is curated fixture data per ROADMAP.md §2. Per-forename is curated fixture data per ROADMAP.md §2. Per-forename gender from the source
gender from the source data is preserved through to the sampled `Name` data is preserved through to the sampled `Name` (rather than discarded during
(rather than discarded during loading) so it's available for gender-aware loading) so it's available for gender-aware persona/bio generation later.
persona/bio generation later.
The full multinational dataset is kept as-is (106 countries for forenames, The full multinational dataset is kept as-is (106 countries for forenames, 75
75 for surnames) rather than trimmed to `locations.json`'s current country for surnames) rather than trimmed to `locations.json`'s current country list, so
list, so it doesn't need re-sourcing if more countries are added later. it doesn't need re-sourcing if more countries are added later. `SampleName()` (a
`SampleName()` (a free helper in `generate_users.cc`) returns no result for free helper in `generate_users.cc`) returns no result for a country present in
a country present in neither file; of the countries in `locations.json`, neither file; of the countries in `locations.json`, that's currently `KE`, `SE`,
that's currently `KE`, `SE`, `SG`, `TH`, `VN`, and `ZA` — `GenerateUsers` `SG`, `TH`, `VN`, and `ZA` — `GenerateUsers` skips cities in those countries the
skips cities in those countries the same way brewery generation skips same way brewery generation skips cities whose enrichment lookup fails.
cities whose enrichment lookup fails.
--- ---
@@ -273,8 +271,7 @@ For languages such as Welsh (Wales), Māori (Aotearoa/New Zealand), or Sicilian
but is semantically incoherent. This comes from limited training-data coverage but is semantically incoherent. This comes from limited training-data coverage
rather than prompt engineering. rather than prompt engineering.
Output sample: Output sample: [./french-cities.example](french-cities.example)
[./french-cities.example](french-cities.example)
#### Proposed Mitigations #### Proposed Mitigations
@@ -298,9 +295,9 @@ curated real-world examples.
The generated code is **format-conformant by construction** — it always The generated code is **format-conformant by construction** — it always
satisfies the regex it was generated from — but it is not looked up from any satisfies the regex it was generated from — but it is not looked up from any
real postal registry. It does not correspond to a real address, business, or real postal registry. It does not correspond to a real address, business, or
resident, and should not be treated as accurate geocoding or delivery data. resident, and should not be treated as accurate geocoding or delivery data. This
This is consistent with the rest of the dataset: fixture data for a is consistent with the rest of the dataset: fixture data for a proof-of-concept,
proof-of-concept, not a source of truth. not a source of truth.
--- ---

View File

@@ -1,9 +1,9 @@
# Biergarten Pipeline # Biergarten Pipeline
A C++20 command-line pipeline that samples city records from local JSON, A C++20 command-line pipeline that samples city records from local JSON,
enriches each with Wikipedia context, and generates bilingual brewery names enriches each with Wikipedia context, and generates bilingual brewery names and
and descriptions plus locale-grounded user profiles via a local GGUF model or descriptions plus locale-grounded user profiles via a local GGUF model or a
a deterministic mock. deterministic mock.
> **This pipeline produces AI-generated data.** It is not a source of truth for > **This pipeline produces AI-generated data.** It is not a source of truth for
> brewing techniques, cultural representation, or local-language accuracy. See > brewing techniques, cultural representation, or local-language accuracy. See
@@ -41,7 +41,7 @@ produces seed records the app imports at startup or during a dedicated seed
step. step.
| Planned app area | Pipeline contribution | | Planned app area | Pipeline contribution |
| -------------------------------- | ------------------------------------------------------------------ | | -------------------------------- | ------------------------------------------------------------------------------------------------- |
| Brewery discovery and management | Sampled city records, localized names, long-form descriptions | | Brewery discovery and management | Sampled city records, localized names, long-form descriptions |
| Beer reviews and ratings | Stable brewery fixtures with enough context to anchor review pages | | Beer reviews and ratings | Stable brewery fixtures with enough context to anchor review pages |
| Social follow relationships | Repeatable brewery entities for feeds, follows, and saved lists | | Social follow relationships | Repeatable brewery entities for feeds, follows, and saved lists |
@@ -65,10 +65,9 @@ cmake --build build
CMake automatically detects whether a compatible llama.cpp installation is CMake automatically detects whether a compatible llama.cpp installation is
present on the system (`libllama`, `libggml`, `libggml-base`, and `llama.h` present on the system (`libllama`, `libggml`, `libggml-base`, and `llama.h`
visible on the default search paths). If found, it links against those visible on the default search paths). If found, it links against those libraries
libraries and skips the FetchContent build. If not found, it fetches and builds and skips the FetchContent build. If not found, it fetches and builds llama.cpp
llama.cpp from source at tag `b9012`. No additional flags are required in from source at tag `b9012`. No additional flags are required in either case.
either case.
Metal is enabled automatically on Apple Silicon. CUDA or HIP/ROCm is detected Metal is enabled automatically on Apple Silicon. CUDA or HIP/ROCm is detected
automatically on Linux when the relevant toolkit is present. automatically on Linux when the relevant toolkit is present.
@@ -138,17 +137,18 @@ NVIDIA GPU.
The container uses a two-stage build. The builder stage installs CMake/Ninja, The container uses a two-stage build. The builder stage installs CMake/Ninja,
clones the matching llama.cpp release tag for its headers only (installed into clones the matching llama.cpp release tag for its headers only (installed into
`/usr/local/include`), and copies prebuilt shared libraries (`libllama`, `/usr/local/include`), and copies prebuilt shared libraries (`libllama`,
`libggml`, and CUDA/CPU backend plugins) from `ghcr.io/ggml-org/llama.cpp:full-cuda` `libggml`, and CUDA/CPU backend plugins) from
into `/usr/local/lib`. With both headers and libraries present, CMake's `ghcr.io/ggml-org/llama.cpp:full-cuda` into `/usr/local/lib`. With both headers
system-library detection (see [Build](#build) above) finds them and skips the and libraries present, CMake's system-library detection (see [Build](#build)
FetchContent source build, keeping image build times short. above) finds them and skips the FetchContent source build, keeping image build
times short.
The runtime stage copies the compiled binary, the same prebuilt shared The runtime stage copies the compiled binary, the same prebuilt shared
libraries, and config/prompt assets into a slim CUDA runtime image. It sets libraries, and config/prompt assets into a slim CUDA runtime image. It sets
`LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH` so the dynamic linker `LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH` so the dynamic linker resolves
resolves `libllama`/`libggml` at startup, and also co-locates `libllama`/`libggml` at startup, and also co-locates `libggml-cuda.so` and the
`libggml-cuda.so` and the CPU backend plugins next to the binary for CPU backend plugins next to the binary for `ggml_backend_load_all()`'s `dlopen`
`ggml_backend_load_all()`'s `dlopen` scan. scan.
### Build the image ### Build the image
@@ -177,8 +177,8 @@ output to confirm the fast path was taken.
The container always runs the model-backed path; there is no `--mocked` The container always runs the model-backed path; there is no `--mocked`
container mode (use a native build for that — see [Quick Start](#quick-start)). container mode (use a native build for that — see [Quick Start](#quick-start)).
The entrypoint, `runpod/start.sh`, downloads the GGUF model automatically if The entrypoint, `runpod/start.sh`, downloads the GGUF model automatically if it
it is not already present at the configured path. is not already present at the configured path.
```bash ```bash
docker run --rm \ docker run --rm \
@@ -189,14 +189,14 @@ docker run --rm \
biergarten-pipeline:latest biergarten-pipeline:latest
``` ```
By default this downloads `google_gemma-4-E4B-it-Q6_K.gguf` to By default this downloads `google_gemma-4-E4B-it-Q6_K.gguf` to `./models/` on
`./models/` on first run if it isn't already there. To use a pre-downloaded first run if it isn't already there. To use a pre-downloaded model, place it at
model, place it at that path first — see [Model](#model) above. that path first — see [Model](#model) above.
#### Environment variables #### Environment variables
| Variable | Purpose | | Variable | Purpose |
| ------------------------- | ----------------------------------------------------------------- | | ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| `BIERGARTEN_MODEL_PATH` | GGUF model path. Default: `/workspace/models/google_gemma-4-E4B-it-Q6_K.gguf`. | | `BIERGARTEN_MODEL_PATH` | GGUF model path. Default: `/workspace/models/google_gemma-4-E4B-it-Q6_K.gguf`. |
| `BIERGARTEN_OUTPUT_DIR` | SQLite output directory. Default: `/workspace/output`. | | `BIERGARTEN_OUTPUT_DIR` | SQLite output directory. Default: `/workspace/output`. |
| `BIERGARTEN_LOG_PATH` | Log file path. Default: `/workspace/logs/pipeline.log`. | | `BIERGARTEN_LOG_PATH` | Log file path. Default: `/workspace/logs/pipeline.log`. |
@@ -221,7 +221,7 @@ environment variables listed above to match your run.
### Pipeline Stages ### Pipeline Stages
| Stage | Implementation | | Stage | Implementation |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Load | `ICuratedDataService` (`CuratedJsonDataService`) reads `locations.json`, `personas.json`, `forenames-by-country.json`, and `surnames-by-country.json` (paths supplied via a `CuratedDataFilePaths` DTO at construction) into typed records, caching each after its first load. `--mocked` runs use `MockCuratedDataService`'s fixed in-memory dataset instead. | | Load | `ICuratedDataService` (`CuratedJsonDataService`) reads `locations.json`, `personas.json`, `forenames-by-country.json`, and `surnames-by-country.json` (paths supplied via a `CuratedDataFilePaths` DTO at construction) into typed records, caching each after its first load. `--mocked` runs use `MockCuratedDataService`'s fixed in-memory dataset instead. |
| Sample | `BiergartenPipelineOrchestrator::QueryCitiesWithCountries()` samples `--location-count` locations per run (default `10`). | | Sample | `BiergartenPipelineOrchestrator::QueryCitiesWithCountries()` samples `--location-count` locations per run (default `10`). |
| Enrich | `WikipediaEnrichmentService` fetches brewing and beer-related context. Keeps going when a lookup fails. `--mocked` runs use `MockEnrichmentService` instead and skip Wikipedia entirely. | | Enrich | `WikipediaEnrichmentService` fetches brewing and beer-related context. Keeps going when a lookup fails. `--mocked` runs use `MockEnrichmentService` instead and skip Wikipedia entirely. |
@@ -240,33 +240,32 @@ skipped and the pipeline continues. `GenerateUsers()` runs before
- `CuratedJsonDataService` — implements `ICuratedDataService`; takes a - `CuratedJsonDataService` — implements `ICuratedDataService`; takes a
`CuratedDataFilePaths` DTO (locations/personas/forenames/surnames paths) in `CuratedDataFilePaths` DTO (locations/personas/forenames/surnames paths) in
its constructor, then parses and validates curated location, persona, and its constructor, then parses and validates curated location, persona, and
forename/surname JSON, memoizing each result after its first load on a forename/surname JSON, memoizing each result after its first load on a given
given instance. `MockCuratedDataService` is the in-memory substitute (4 instance. `MockCuratedDataService` is the in-memory substitute (4 fixed
fixed locations, 3 personas, and name data for `US`/`DE`/`FR`/`BE`) used in locations, 3 personas, and name data for `US`/`DE`/`FR`/`BE`) used in
`--mocked` runs. `--mocked` runs.
- `WikipediaEnrichmentService` — queries Wikipedia extracts, caches results, - `WikipediaEnrichmentService` — queries Wikipedia extracts, caches results,
returns empty context on failure. `MockEnrichmentService` is the no-op returns empty context on failure. `MockEnrichmentService` is the no-op
substitute used in `--mocked` runs. substitute used in `--mocked` runs.
- `LlamaGenerator` — formats prompts for Gemma 4, validates JSON output for - `LlamaGenerator` — formats prompts for Gemma 4, validates JSON output for both
both `GenerateBrewery` and `GenerateUser`, retries malformed responses up `GenerateBrewery` and `GenerateUser`, retries malformed responses up to three
to three times with corrective feedback in the retry prompt. The token times with corrective feedback in the retry prompt. The token budget is fixed
budget is fixed across attempts; it is not raised automatically on across attempts; it is not raised automatically on truncation.
truncation. - `MockGenerator` — stable hash-based output so the same city/persona/name input
- `MockGenerator` — stable hash-based output so the same city/persona/name always produces the same brewery or user.
input always produces the same brewery or user. - `SqliteExportService` — creates a dated SQLite file per run and persists each
- `SqliteExportService` — creates a dated SQLite file per run and persists successful user and brewery into normalized tables.
each successful user and brewery into normalized tables.
- Brewery payloads include English and local-language name and description - Brewery payloads include English and local-language name and description
fields. User payloads carry a sampled first/last name and gender, an fields. User payloads carry a sampled first/last name and gender, an
LLM-generated username/bio/activity weight, and a programmatically LLM-generated username/bio/activity weight, and a programmatically generated
generated (not LLM-authored) unique email and date of birth. (not LLM-authored) unique email and date of birth.
### Runtime Behaviour ### Runtime Behaviour
`WikipediaEnrichmentService` fetches two Wikipedia extracts per city: a `WikipediaEnrichmentService` fetches two Wikipedia extracts per city: a generic
generic "brewing" extract and a "beer in `{country}`" extract. It does not "brewing" extract and a "beer in `{country}`" extract. It does not currently
currently query a city- or region-specific page. Each query string is cached query a city- or region-specific page. Each query string is cached after its
after its first successful (or empty) lookup. first successful (or empty) lookup.
`GetLocationContext()` returns an empty string when the web client is `GetLocationContext()` returns an empty string when the web client is
unavailable or when lookup/parsing fails. unavailable or when lookup/parsing fails.
@@ -281,15 +280,15 @@ runs.
`CuratedJsonDataService` memoizes each of `LoadLocations()`, `LoadPersonas()`, `CuratedJsonDataService` memoizes each of `LoadLocations()`, `LoadPersonas()`,
`LoadForenamesByCountry()`, and `LoadSurnamesByCountry()` independently the `LoadForenamesByCountry()`, and `LoadSurnamesByCountry()` independently the
first time each is called, since `BiergartenPipelineOrchestrator` owns a first time each is called, since `BiergartenPipelineOrchestrator` owns a single
single `ICuratedDataService` instance for the whole run — later calls return `ICuratedDataService` instance for the whole run — later calls return the cached
the cached result instead of re-parsing. result instead of re-parsing.
`GenerateUsers()` samples a forename/surname pair per city via `SampleName()`, `GenerateUsers()` samples a forename/surname pair per city via `SampleName()`,
keyed by the city's ISO 3166-1 code. Countries present in `locations.json` keyed by the city's ISO 3166-1 code. Countries present in `locations.json` but
but absent from either name fixture (currently `KE`, `SE`, `SG`, `TH`, `VN`, absent from either name fixture (currently `KE`, `SE`, `SG`, `TH`, `VN`, `ZA`)
`ZA`) are skipped, the same way a failed enrichment or generation call skips are skipped, the same way a failed enrichment or generation call skips a city —
a city — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section. see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section.
### Process Flow - Activity Diagram ### Process Flow - Activity Diagram
@@ -303,10 +302,10 @@ a city — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section.
## Generated Output ## Generated Output
Each successful run stores a `BreweryRecord` pair with the source location Each successful run stores a `BreweryRecord` pair with the source location and a
and a `BreweryResult` payload, and a `UserRecord` pair with the source `BreweryResult` payload, and a `UserRecord` pair with the source location and a
location and a `UserResult` payload. The same generated records are also `UserResult` payload. The same generated records are also written to a fresh
written to a fresh SQLite export file named with the current UTC timestamp. SQLite export file named with the current UTC timestamp.
| Field | Meaning | | Field | Meaning |
| ------------------- | ------------------------------------------ | | ------------------- | ------------------------------------------ |
@@ -316,7 +315,7 @@ written to a fresh SQLite export file named with the current UTC timestamp.
| `description_local` | Brewery description in the local language. | | `description_local` | Brewery description in the local language. |
| Field | Meaning | | Field | Meaning |
| ----------------- | ----------------------------------------------------------------- | | ----------------- | -------------------------------------------------------------------------------------- |
| `first_name` | Sampled forename, copied from the curated name data (not LLM-invented). | | `first_name` | Sampled forename, copied from the curated name data (not LLM-invented). |
| `last_name` | Sampled surname, copied from the curated name data (not LLM-invented). | | `last_name` | Sampled surname, copied from the curated name data (not LLM-invented). |
| `gender` | Gender associated with the sampled forename in the source dataset. | | `gender` | Gender associated with the sampled forename in the source dataset. |
@@ -358,11 +357,9 @@ The build fetches Boost.DI, spdlog, and SQLite via CMake. llama.cpp is fetched
only when a system installation is not detected. Metal is enabled on Apple only when a system installation is not detected. Metal is enabled on Apple
Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present. Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present.
> **Code Style:** Modern C++20 throughout — RAII for ownership, > **Code Style:** Code style generally follows the Google Style Guide and
> `std::unique_ptr` for injected dependencies, `std::optional` for parse > features modern C++20 throughout. Formatting follows the Google C++ Style
> outcomes, `std::span` for read-only views over generated city data, structured > Guide, with the exception of indentation width (3 spaces vs. Google's 2).
> bindings in pipeline loops. Formatting follows the Google C++ Style Guide via
> `.clang-format` with a narrow column limit and two-space indentation.
--- ---
@@ -404,24 +401,23 @@ Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present.
## Fixture Strategy ## Fixture Strategy
- `--mocked` for stable fixtures, repeatable screenshots, and Storybook runs. - `--mocked` for stable fixtures, repeatable screenshots, and Storybook runs.
`MockCuratedDataService` swaps in for `CuratedJsonDataService`, so no `MockCuratedDataService` swaps in for `CuratedJsonDataService`, so no fixture
fixture files need to be present on disk. files need to be present on disk.
- `--model` when geographically grounded content matters for demos. - `--model` when geographically grounded content matters for demos.
- Keep `locations.json` structured enough to support discovery and future - Keep `locations.json` structured enough to support discovery and future
filtering. filtering.
- `personas.json`, `forenames-by-country.json`, and - `personas.json`, `forenames-by-country.json`, and `surnames-by-country.json`
`surnames-by-country.json` are curated/vendored fixture data, not are curated/vendored fixture data, not LLM-generated — see
LLM-generated — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for provenance.
section for provenance. - Treat SQLite output as seed material for the app's brewery and user domains,
- Treat SQLite output as seed material for the app's brewery and user not production data.
domains, not production data.
--- ---
## Repo Layout ## Repo Layout
| Path | Purpose | | Path | Purpose |
| -------------------------------------- | -------------------------------------------------- | | -------------------------------------------- | ----------------------------------------------------------- |
| `tooling/pipeline/includes/` | Public headers and shared models. | | `tooling/pipeline/includes/` | Public headers and shared models. |
| `tooling/pipeline/src/` | Implementation files. | | `tooling/pipeline/src/` | Implementation files. |
| `tooling/pipeline/locations.json` | Curated city input copied into the build tree. | | `tooling/pipeline/locations.json` | Curated city input copied into the build tree. |
@@ -456,12 +452,11 @@ Paths below are relative to `tooling/pipeline/`.
## Next Steps ## Next Steps
The pipeline currently produces city-aware brewery and user records and The pipeline currently produces city-aware brewery and user records and dated
dated SQLite exports. The next passes add additional fixture types so the SQLite exports. The next passes add additional fixture types so the app can
app can exercise the full brewery and social domains without live data. For exercise the full brewery and social domains without live data. For the detailed
the detailed engineering breakdown of what's needed to reach the engineering breakdown of what's needed to reach the architecture in
architecture in [`diagrams/planned/`](./diagrams/planned/), see [`diagrams/planned/`](./diagrams/planned/), see [ROADMAP.md](./ROADMAP.md).
[ROADMAP.md](./ROADMAP.md).
### Testing — Very High Priority ### Testing — Very High Priority

View File

@@ -3,13 +3,12 @@
This is the engineering breakdown for closing the gap between the **current** This is the engineering breakdown for closing the gap between the **current**
pipeline and the **planned** architecture documented in pipeline and the **planned** architecture documented in
[`diagrams/planned/class.puml`](./diagrams/planned/class.puml) and [`diagrams/planned/class.puml`](./diagrams/planned/class.puml) and
[`diagrams/planned/activity.puml`](./diagrams/planned/activity.puml). Nothing [`diagrams/planned/activity.puml`](./diagrams/planned/activity.puml). Nothing in
in `diagrams/planned/` is implemented yet — this file tracks what it would `diagrams/planned/` is implemented yet — this file tracks what it would take to
take to get there. For the current implementation, see get there. For the current implementation, see [README.md](./README.md).
[README.md](./README.md).
Items are grouped by layer and roughly ordered by dependency: later groups Items are grouped by layer and roughly ordered by dependency: later groups build
build on types and services introduced earlier. on types and services introduced earlier.
The only concurrency in the current pipeline is the log dispatcher thread The only concurrency in the current pipeline is the log dispatcher thread
(`main.cc` spawns it to drain a `BoundedChannel<LogEntry>` into spdlog for the (`main.cc` spawns it to drain a `BoundedChannel<LogEntry>` into spdlog for the
@@ -25,31 +24,30 @@ architecture needs.
`tooling/pipeline/includes/data_model/generated_models.h` and `models.h`. `tooling/pipeline/includes/data_model/generated_models.h` and `models.h`.
- [ ] Add `Completeness` enum (`Full`, `Partial`, `Absent`) and a - [ ] Add `Completeness` enum (`Full`, `Partial`, `Absent`) and a
`LocationContext` struct (`text`, `completeness`, `char_count`). `LocationContext` struct (`text`, `completeness`, `char_count`). Replace
Replace `EnrichedCity::region_context` (currently a plain `EnrichedCity::region_context` (currently a plain `std::string`) with
`std::string`) with `context : LocationContext`. `context : LocationContext`.
- [ ] Add `BeerStyle` (`name`, `description`, `min_abv`, `max_abv`, - [ ] Add `BeerStyle` (`name`, `description`, `min_abv`, `max_abv`, `min_ibu`,
`min_ibu`, `max_ibu`). `max_ibu`).
- [ ] Add `BeerResult`, `CheckinResult`, `RatingResult` result payloads. - [ ] Add `BeerResult`, `CheckinResult`, `RatingResult` result payloads.
- [ ] Add `GenerationMetadata` (`generation_id`, `generated_time`, - [ ] Add `GenerationMetadata` (`generation_id`, `generated_time`,
`context_provided`, `generated_with`). `context_provided`, `generated_with`).
- [x] Add `first_name`, `last_name`, `gender`, and `activity_weight` to - [x] Add `first_name`, `last_name`, `gender`, and `activity_weight` to
`UserResult` (currently just `username`, `bio`). `first_name`/ `UserResult` (currently just `username`, `bio`). `first_name`/
`last_name`/`gender` are copied from the sampled `Name` (see below), `last_name`/`gender` are copied from the sampled `Name` (see below), not
not LLM-invented. LLM-invented.
- [x] Add `Name` (`first_name`, `last_name`, `gender`) — the sampled result - [x] Add `Name` (`first_name`, `last_name`, `gender`) — the sampled result
handed to `DataGenerator::GenerateUser`. Add `ForenameEntry` (`name`, handed to `DataGenerator::GenerateUser`. Add `ForenameEntry` (`name`,
`gender`). Landed as a flatter shape than originally planned here: no `gender`). Landed as a flatter shape than originally planned here: no
`NamesByCountry` wrapper class. `curated_data_service.h` instead `NamesByCountry` wrapper class. `curated_data_service.h` instead declares
declares `ForenameList = std::vector<ForenameEntry>` and `ForenameList = std::vector<ForenameEntry>` and
`SurnameList = std::vector<std::string>`, and `SurnameList = std::vector<std::string>`, and `ICuratedDataService`
`ICuratedDataService` exposes two maps keyed by ISO 3166-1 code — exposes two maps keyed by ISO 3166-1 code —
`ForenamesByCountryMap = unordered_map<string, ForenameList>` and `ForenamesByCountryMap = unordered_map<string, ForenameList>` and
`SurnamesByCountryMap = unordered_map<string, SurnameList>` — `SurnamesByCountryMap = unordered_map<string, SurnameList>` — directly
directly (see §2). Sampling is (see §2). Sampling is a free function,
a free function, `SampleName(forenames_by_country, `SampleName(forenames_by_country, surnames_by_country, iso3166_1, rng)`,
surnames_by_country, iso3166_1, rng)`, in in `generate_users.cc`'s anonymous namespace, not a method on a class.
`generate_users.cc`'s anonymous namespace, not a method on a class.
Pairing a forename with a surname happens at sample time so gender Pairing a forename with a surname happens at sample time so gender
(present per-forename in the source data) is never discarded the way a (present per-forename in the source data) is never discarded the way a
pre-flattened `{first_name, last_name}` list would lose it; see §2. pre-flattened `{first_name, last_name}` list would lose it; see §2.
@@ -58,14 +56,14 @@ architecture needs.
- [ ] Add `GeneratedBeer`, `GeneratedCheckin`, `GeneratedRating`, - [ ] Add `GeneratedBeer`, `GeneratedCheckin`, `GeneratedRating`,
`GeneratedFollow` aggregate structs. `GeneratedFollow` aggregate structs.
- [x] Add `UserRecord` (`location`, `user : UserResult`, `email`, - [x] Add `UserRecord` (`location`, `user : UserResult`, `email`,
`date_of_birth`) as the current stand-in for the planned `date_of_birth`) as the current stand-in for the planned `GeneratedUser` —
`GeneratedUser` — `email` and `date_of_birth` are programmatically `email` and `date_of_birth` are programmatically generated by the
generated by the orchestrator, never LLM-authored, so a downstream orchestrator, never LLM-authored, so a downstream auth-account seeding
auth-account seeding consumer can register real accounts from the consumer can register real accounts from the pipeline's SQLite export. No
pipeline's SQLite export. No `password`, `user_id`, or `metadata` `password`, `user_id`, or `metadata` field yet, matching `BreweryRecord`'s
field yet, matching `BreweryRecord`'s equally pre-`metadata` shape. equally pre-`metadata` shape. Already wired into
Already wired into `IExportService`/`SqliteExportService`: a `users` `IExportService`/`SqliteExportService`: a `users` table exists and
table exists and `GenerateUsers()` exports each successful record via `GenerateUsers()` exports each successful record via
`ProcessRecord(const UserRecord&)` (see §5) in addition to logging and `ProcessRecord(const UserRecord&)` (see §5) in addition to logging and
holding `generated_users_` in memory. Still missing vs. the planned holding `generated_users_` in memory. Still missing vs. the planned
`GeneratedUser`: `user_id`, `password`, and `metadata`. `GeneratedUser`: `user_id`, `password`, and `metadata`.
@@ -75,66 +73,62 @@ architecture needs.
`tooling/pipeline/includes/services/curated_data/`, fixture files. `tooling/pipeline/includes/services/curated_data/`, fixture files.
- [x] Extract an interface; have `CuratedJsonDataService` implement it. - [x] Extract an interface; have `CuratedJsonDataService` implement it. Landed
Landed as `ICuratedDataService` as `ICuratedDataService` (`services/curated_data/curated_data_service.h`),
(`services/curated_data/curated_data_service.h`), not `DataPreloader` not `DataPreloader` — `LoadLocations()`, `LoadPersonas()`,
— `LoadLocations()`, `LoadPersonas()`, `LoadForenamesByCountry()`, and `LoadForenamesByCountry()`, and `LoadSurnamesByCountry()` are all virtual
`LoadSurnamesByCountry()` are all virtual methods returning `const&` methods returning `const&` and take no arguments. `CuratedJsonDataService`
and take no arguments. `CuratedJsonDataService` (`services/curated_data/curated_json_data_service.h`, originally landed as
(`services/curated_data/curated_json_data_service.h`, originally `JsonLoader` under `json_handling/` before being renamed and moved) takes
landed as `JsonLoader` under `json_handling/` before being renamed and a `CuratedDataFilePaths` DTO (`locations_path`, `personas_path`,
moved) takes a `CuratedDataFilePaths` DTO (`locations_path`, `forenames_path`, `surnames_path`) in its constructor rather than a path
`personas_path`, `forenames_path`, `surnames_path`) in its per `Load*()` call, and memoizes each result in a private `cache` struct
constructor rather than a path per `Load*()` call, and memoizes each (`locations`, `personas`, `forenames_by_country`, `surnames_by_country`),
result in a private `cache` struct (`locations`, `personas`, so a second call on the same instance returns the cached result instead of
`forenames_by_country`, `surnames_by_country`), so a second call on re-parsing — safe because `CuratedJsonDataService` outlives every call
the same instance returns the cached result instead of re-parsing — site (owned by `BiergartenPipelineOrchestrator` via
safe because `CuratedJsonDataService` outlives every call site (owned
by `BiergartenPipelineOrchestrator` via
`unique_ptr<ICuratedDataService>` for the whole run). `main.cc`'s DI `unique_ptr<ICuratedDataService>` for the whole run). `main.cc`'s DI
injector also gained a `MockCuratedDataService` (fixed in-memory injector also gained a `MockCuratedDataService` (fixed in-memory data: 4
data: 4 locations across `US`/`DE`/`FR`/`BE`, 3 personas, and locations across `US`/`DE`/`FR`/`BE`, 3 personas, and matching
matching forename/surname sets for those four countries), bound in forename/surname sets for those four countries), bound in place of
place of `CuratedJsonDataService` under `--mocked`, mirroring the `CuratedJsonDataService` under `--mocked`, mirroring the existing
existing `MockEnrichmentService`/`MockGenerator` pattern. `MockEnrichmentService`/`MockGenerator` pattern.
- [ ] Add `LoadBeerStyles()`. `beer-styles.json` already exists in the repo - [ ] Add `LoadBeerStyles()`. `beer-styles.json` already exists in the repo and
and is already copied into the Docker image is already copied into the Docker image (`runpod/Dockerfile`), but no
(`runpod/Dockerfile`), but no loader reads it yet, and the native loader reads it yet, and the native CMake build doesn't copy it into
CMake build doesn't copy it into `build/` at all — `CMakeLists.txt`'s `build/` at all — `CMakeLists.txt`'s "Runtime Assets" step only copies
"Runtime Assets" step only copies `locations.json` and `prompts/`. `locations.json` and `prompts/`.
- [x] Add `LoadPersonas()` and author `personas.json` (doesn't exist yet). - [x] Add `LoadPersonas()` and author `personas.json` (doesn't exist yet).
- [x] Add name-by-country loading, parsing - [x] Add name-by-country loading, parsing
`tooling/pipeline/forenames-by-country.json` and `tooling/pipeline/forenames-by-country.json` and
`surnames-by-country.json`. Landed as two separate methods — `surnames-by-country.json`. Landed as two separate methods —
`LoadForenamesByCountry()` and `LoadSurnamesByCountry()` — each `LoadForenamesByCountry()` and `LoadSurnamesByCountry()` — each returning
returning a flat `ForenamesByCountryMap` / `SurnamesByCountryMap` a flat `ForenamesByCountryMap` / `SurnamesByCountryMap` (aliases for
(aliases for `unordered_map<string, ForenameList>` / `unordered_map<string, ForenameList>` /
`unordered_map<string, SurnameList>`) keyed by ISO 3166-1 code, rather `unordered_map<string, SurnameList>`) keyed by ISO 3166-1 code, rather
than one combined `LoadNamesByCountry() : NamesByCountry` call. Both than one combined `LoadNamesByCountry() : NamesByCountry` call. Both files
files are vendored verbatim (unmodified, are vendored verbatim (unmodified, full multinational coverage) from
full multinational coverage) from
`sigpwned/popular-names-by-country-dataset` (CC0) — see `sigpwned/popular-names-by-country-dataset` (CC0) — see
ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for
provenance. Deliberately not pre-paired or filtered down to just the provenance. Deliberately not pre-paired or filtered down to just the
countries in `locations.json`: keeping the source shape (including countries in `locations.json`: keeping the source shape (including
per-forename gender) intact means the loader can support more per-forename gender) intact means the loader can support more countries
countries later, or gender-aware persona/bio generation, without later, or gender-aware persona/bio generation, without re-sourcing data.
re-sourcing data.
## 3. Policy / Strategy Layer ## 3. Policy / Strategy Layer
Entirely new — no `includes/policy/` (or equivalent) directory exists today. Entirely new — no `includes/policy/` (or equivalent) directory exists today.
- [ ] `ContextStrategy` interface + `BreweryContextStrategy` / - [ ] `ContextStrategy` interface + `BreweryContextStrategy` /
`BeerContextStrategy`. Today `WikipediaEnrichmentService::GetLocationContext` `BeerContextStrategy`. Today
hardcodes a generic `"brewing"` query and a `"beer in {country}"` `WikipediaEnrichmentService::GetLocationContext` hardcodes a generic
query directly — no per-phase strategy selection. `"brewing"` query and a `"beer in {country}"` query directly — no
- [ ] `SamplingStrategy` interface + `UniformSamplingStrategy`, replacing per-phase strategy selection.
the inline `std::ranges::sample(...)` call in - [ ] `SamplingStrategy` interface + `UniformSamplingStrategy`, replacing the
inline `std::ranges::sample(...)` call in
`BiergartenPipelineOrchestrator::QueryCitiesWithCountries()`. `BiergartenPipelineOrchestrator::QueryCitiesWithCountries()`.
- [ ] `BeerSelectionStrategy` interface + `RandomBeerSelectionStrategy`, to - [ ] `BeerSelectionStrategy` interface + `RandomBeerSelectionStrategy`, to pick
pick styles per brewery from the `BeerStyle` palette (depends on §1 styles per brewery from the `BeerStyle` palette (depends on §1 and §2).
and §2).
- [ ] `CheckinDistributionStrategy` interface + `JCurveCheckinStrategy` / - [ ] `CheckinDistributionStrategy` interface + `JCurveCheckinStrategy` /
`RandomCheckinStrategy` — this is the "Check-In System" item already `RandomCheckinStrategy` — this is the "Check-In System" item already
called out in README.md's Next Steps, made concrete. called out in README.md's Next Steps, made concrete.
@@ -150,88 +144,87 @@ Entirely new — no `includes/policy/` (or equivalent) directory exists today.
`GenerateUser`). `GenerateUser`).
- [x] Implement `LlamaGenerator::GenerateUser` for real, with a retry loop - [x] Implement `LlamaGenerator::GenerateUser` for real, with a retry loop
mirroring `GenerateBrewery` (GBNF grammar, `ValidateUserJson`, up to 3 mirroring `GenerateBrewery` (GBNF grammar, `ValidateUserJson`, up to 3
attempts with corrective feedback) and a new attempts with corrective feedback) and a new `prompts/USER_GENERATION.md`.
`prompts/USER_GENERATION.md`. Changed the `DataGenerator::GenerateUser` Changed the `DataGenerator::GenerateUser` signature to
signature to
`(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult`, `(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult`,
matching what the activity diagram actually passes. `MockGenerator::GenerateUser` matching what the activity diagram actually passes.
updated to match the new signature too. `MockGenerator::GenerateUser` updated to match the new signature too.
- [ ] Implement `MockGenerator::GenerateBeer` / `GenerateCheckin` / - [ ] Implement `MockGenerator::GenerateBeer` / `GenerateCheckin` /
`GenerateRating`. `GenerateRating`.
- [ ] Add `IPromptFormatter::ExpectedArchitecture()` and - [ ] Add `IPromptFormatter::ExpectedArchitecture()` and
`LlamaGenerator::ValidateModelArchitecture()` so loading a GGUF that `LlamaGenerator::ValidateModelArchitecture()` so loading a GGUF that
doesn't match the configured chat template fails fast instead of doesn't match the configured chat template fails fast instead of silently
silently producing degraded output. producing degraded output.
- [ ] Add prompt template files for beer/checkin/rating generation next to - [ ] Add prompt template files for beer/checkin/rating generation next to the
the existing `prompts/BREWERY_GENERATION.md`. existing `prompts/BREWERY_GENERATION.md`.
## 5. Export Service ## 5. Export Service
`tooling/pipeline/includes/services/database/`, `src/services/sqlite/`. `tooling/pipeline/includes/services/database/`, `src/services/sqlite/`.
- [x] `IExportService::ProcessRecord(const UserRecord&)` and - [x] `IExportService::ProcessRecord(const UserRecord&)` and
`SqliteExportService`'s `users` table (see `SqliteExportService`'s `users` table (see `kCreateUsersTableSql` /
`kCreateUsersTableSql` / `kInsertUserSql` in `kInsertUserSql` in
`includes/services/database/sqlite_statement_helpers.h`) are `includes/services/database/sqlite_statement_helpers.h`) are implemented —
implemented — `GenerateUsers()` exports every successful user `GenerateUsers()` exports every successful user alongside its resolved
alongside its resolved `location_id`. `location_id`.
- [ ] Extend `IExportService` with `ProcessBeer`, `ProcessCheckin`, - [ ] Extend `IExportService` with `ProcessBeer`, `ProcessCheckin`,
`ProcessRating`, `ProcessFollow` (today `ProcessRecord(const `ProcessRating`, `ProcessFollow` (today
BreweryRecord&)` and `ProcessRecord(const UserRecord&)` exist). `ProcessRecord(const BreweryRecord&)` and
`ProcessRecord(const UserRecord&)` exist).
- [ ] Add `beers`, `checkins`, `ratings`, `follows` tables to - [ ] Add `beers`, `checkins`, `ratings`, `follows` tables to
`SqliteExportService::InitializeSchema()` (`locations`, `breweries`, `SqliteExportService::InitializeSchema()` (`locations`, `breweries`, and
and `users` already exist) — see `kCreateLocationsTableSql` / `users` already exist) — see `kCreateLocationsTableSql` /
`kCreateBreweriesTableSql` / `kCreateUsersTableSql` in `kCreateBreweriesTableSql` / `kCreateUsersTableSql` in
`includes/services/database/sqlite_statement_helpers.h` for the `includes/services/database/sqlite_statement_helpers.h` for the existing
existing pattern to follow. pattern to follow.
- [ ] Add a `brewery_cache_` alongside the existing `location_cache_`, and - [ ] Add a `brewery_cache_` alongside the existing `location_cache_`, and move
move from one open transaction for the whole run to per-phase batched from one open transaction for the whole run to per-phase batched commits
commits (`BEGIN` / `COMMIT & BEGIN` on a batch-size threshold), as (`BEGIN` / `COMMIT & BEGIN` on a batch-size threshold), as shown in the
shown in the planned activity diagram. planned activity diagram.
## 6. Concurrency & Orchestration ## 6. Concurrency & Orchestration
`tooling/pipeline/includes/concurrency/`, `tooling/pipeline/includes/concurrency/`,
`src/biergarten_pipeline_orchestrator/`. `src/biergarten_pipeline_orchestrator/`.
- [ ] `BoundedChannel<T>` already exists and is production-tested — but - [ ] `BoundedChannel<T>` already exists and is production-tested — but it's
it's only wired to the single log channel today. Stand up the only wired to the single log channel today. Stand up the per-phase
per-phase producer/consumer channels (`loc_ch`, `exp_ch`, etc.) the producer/consumer channels (`loc_ch`, `exp_ch`, etc.) the planned activity
planned activity diagram describes, each with a dedicated LLM worker diagram describes, each with a dedicated LLM worker thread and SQLite
thread and SQLite worker thread. worker thread.
- [ ] Rewrite `BiergartenPipelineOrchestrator::Run()` from one synchronous - [ ] Rewrite `BiergartenPipelineOrchestrator::Run()` from one synchronous pass
pass over `generated_breweries_` into phased methods — over `generated_breweries_` into phased methods — `RunUserPhase` →
`RunUserPhase` → `RunBreweryAndBeerPhase` (with a `RunBeerPhase` `RunBreweryAndBeerPhase` (with a `RunBeerPhase` sub-step once
sub-step once `brewery_pool_` is populated) → `RunCheckinPhase` / `brewery_pool_` is populated) → `RunCheckinPhase` / `RunFollowPhase`
`RunFollowPhase` (these two can run in parallel) → `RunRatingPhase` — (these two can run in parallel) → `RunRatingPhase` — backed by
backed by `user_pool_`, `brewery_pool_`, `beer_pool_`, `user_pool_`, `brewery_pool_`, `beer_pool_`, `checkin_pool_`,
`checkin_pool_`, `follow_pool_`. `follow_pool_`.
- [ ] Honor the structural-concurrency requirement already called out in a - [ ] Honor the structural-concurrency requirement already called out in a
comment on `BiergartenPipelineOrchestrator::Run()` in comment on `BiergartenPipelineOrchestrator::Run()` in
`biergarten_pipeline_orchestrator.h`: once real worker threads exist, `biergarten_pipeline_orchestrator.h`: once real worker threads exist, they
they must be structurally joined (e.g. via `std::jthread`) before must be structurally joined (e.g. via `std::jthread`) before `Run()`
`Run()` returns, so no worker logs to a closed channel during returns, so no worker logs to a closed channel during teardown.
teardown.
## 7. Enrichment ## 7. Enrichment
- [ ] Decide whether to restore the city/region-specific Wikipedia query - [ ] Decide whether to restore the city/region-specific Wikipedia query that's
that's currently commented out in currently commented out in
`src/services/enrichment/wikipedia/get_summary.cc` `src/services/enrichment/wikipedia/get_summary.cc` (`GetLocationContext`).
(`GetLocationContext`). The `ContextStrategy` work in §3 is a natural The `ContextStrategy` work in §3 is a natural place to reintroduce it via
place to reintroduce it via `BreweryContextStrategy::QueriesFor()`. `BreweryContextStrategy::QueriesFor()`.
- [ ] Pre-warm caches at startup (`PreWarmBeerStyleCache`, - [ ] Pre-warm caches at startup (`PreWarmBeerStyleCache`,
`PreWarmLocationCache` in the planned activity diagram) instead of `PreWarmLocationCache` in the planned activity diagram) instead of
fetching lazily per record, so the streaming phases never block on a fetching lazily per record, so the streaming phases never block on a cold
cold cache mid-run. cache mid-run.
## 8. Fixtures / Build ## 8. Fixtures / Build
- [x] Author `personas.json`, `forenames-by-country.json`, and - [x] Author `personas.json`, `forenames-by-country.json`, and
`surnames-by-country.json` (see §2). `surnames-by-country.json` (see §2).
- [ ] Fix the `beer-styles.json` build-tree gap: add it to the "Runtime - [ ] Fix the `beer-styles.json` build-tree gap: add it to the "Runtime Assets"
Assets" `configure_file` step in `CMakeLists.txt` so native builds and `configure_file` step in `CMakeLists.txt` so native builds and the Docker
the Docker image agree on what's available at runtime. image agree on what's available at runtime.
--- ---
@@ -240,13 +233,12 @@ Entirely new — no `includes/policy/` (or equivalent) directory exists today.
The planned activity diagram's own phase notes ("brewery*pool* is now fully The planned activity diagram's own phase notes ("brewery*pool* is now fully
populated. Phase 1b may begin.", etc.) imply a dependency order. Roughly: populated. Phase 1b may begin.", etc.) imply a dependency order. Roughly:
1. Domain models (§1) and data preloading (§2) — nothing else compiles 1. Domain models (§1) and data preloading (§2) — nothing else compiles without
without these. these.
2. Export schema (§5) — so every later generation phase has somewhere to 2. Export schema (§5) — so every later generation phase has somewhere to land
land its output. its output.
3. Policy/strategy layer (§3) and generator interface (§4). 3. Policy/strategy layer (§3) and generator interface (§4).
4. Concurrency/orchestration rewrite (§6), which is the only piece that 4. Concurrency/orchestration rewrite (§6), which is the only piece that actually
actually wires §1–§5 into the phased, parallel pipeline shown in the wires §1–§5 into the phased, parallel pipeline shown in the planned diagrams.
planned diagrams. 5. Enrichment cache pre-warming and the city/region query decision (§7) — useful
5. Enrichment cache pre-warming and the city/region query decision (§7) — at any point, but most valuable once phases run concurrently.
useful at any point, but most valuable once phases run concurrently.

View File

@@ -1,5 +1,5 @@
--- ---
BasedOnStyle: Google BasedOnStyle: Google
ColumnLimit: 80 ColumnLimit: 80
IndentWidth: 2 IndentWidth: 3
... ...

View File

@@ -1,12 +1,6 @@
Checks: > Checks: >
-*, -*, bugprone-*, google-*, modernize-*, readability-*, cppcoreguidelines-*,
bugprone-*, -modernize-use-trailing-return-type, -google-runtime-references
google-*,
modernize-*,
readability-*,
cppcoreguidelines-*,
-modernize-use-trailing-return-type,
-google-runtime-references
CheckOptions: CheckOptions:
# Enforce Google Naming Conventions with valid clang-tidy strings # Enforce Google Naming Conventions with valid clang-tidy strings

File diff suppressed because it is too large Load Diff

View File

@@ -37,7 +37,8 @@ class LlamaGenerator final : public DataGenerator {
* @param prompt_directory Directory service for loading named prompt files. * @param prompt_directory Directory service for loading named prompt files.
*/ */
LlamaGenerator(const ApplicationOptions& options, LlamaGenerator(const ApplicationOptions& options,
const std::string& model_path, std::shared_ptr<ILogger> logger, const std::string& model_path,
std::shared_ptr<ILogger> logger,
std::unique_ptr<IPromptFormatter> prompt_formatter, std::unique_ptr<IPromptFormatter> prompt_formatter,
std::unique_ptr<IPromptDirectory> prompt_directory); std::unique_ptr<IPromptDirectory> prompt_directory);
@@ -103,7 +104,8 @@ class LlamaGenerator final : public DataGenerator {
* @param grammar Optional GBNF grammar constraining generated output. * @param grammar Optional GBNF grammar constraining generated output.
* @return Generated text. * @return Generated text.
*/ */
std::string Infer(const std::string& system_prompt, const std::string& prompt, std::string Infer(const std::string& system_prompt,
const std::string& prompt,
int max_tokens = kDefaultMaxTokens, int max_tokens = kDefaultMaxTokens,
std::string_view grammar = {}); std::string_view grammar = {});

View File

@@ -52,7 +52,8 @@ class MockGenerator final : public DataGenerator {
* @return Deterministic hash value. * @return Deterministic hash value.
*/ */
static size_t DeterministicHash(const City& location, static size_t DeterministicHash(const City& location,
const UserPersona& persona, const Name& name); const UserPersona& persona,
const Name& name);
// Hash stride constants for deterministic distribution across fixed-size // Hash stride constants for deterministic distribution across fixed-size
// arrays. These coprime strides spread hash values uniformly without // arrays. These coprime strides spread hash values uniformly without

View File

@@ -11,7 +11,8 @@ class Gemma4JinjaPromptFormatter final : public IPromptFormatter {
Gemma4JinjaPromptFormatter() = default; Gemma4JinjaPromptFormatter() = default;
~Gemma4JinjaPromptFormatter() override = default; ~Gemma4JinjaPromptFormatter() override = default;
[[nodiscard]] std::string Format(std::string_view system_prompt, [[nodiscard]] std::string Format(
std::string_view system_prompt,
std::string_view user_prompt) const override; std::string_view user_prompt) const override;
}; };

View File

@@ -39,7 +39,8 @@ inline void PrettyPrint(std::ostream& outstream,
if (!obj.empty()) { if (!obj.empty()) {
const auto* iter = obj.begin(); const auto* iter = obj.begin();
for (;;) { for (;;) {
outstream << *indent << boost::json::serialize(iter->key()) << " : "; outstream << *indent << boost::json::serialize(iter->key())
<< " : ";
PrettyPrint(outstream, iter->value(), indent); PrettyPrint(outstream, iter->value(), indent);
iter = std::next(iter); iter = std::next(iter);
if (iter == obj.end()) { if (iter == obj.end()) {

View File

@@ -15,8 +15,6 @@
*/ */
class MockEnrichmentService final : public IEnrichmentService { class MockEnrichmentService final : public IEnrichmentService {
public: public:
std::string GetLocationContext(const City& /*loc*/) override { std::string GetLocationContext(const City& /*loc*/) override { return {}; }
return {};
}
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_MOCK_ENRICHMENT_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_MOCK_ENRICHMENT_H_

View File

@@ -67,7 +67,8 @@ class MockPostalCodeService final : public IPostalCodeService {
if (!spec.country_format_regex.empty()) { if (!spec.country_format_regex.empty()) {
has_pattern = true; has_pattern = true;
if (std::regex_match(candidate, std::regex(spec.country_format_regex))) { if (std::regex_match(candidate,
std::regex(spec.country_format_regex))) {
return true; return true;
} }
} }

View File

@@ -2,7 +2,12 @@
{ {
"name": "Hophead Explorer", "name": "Hophead Explorer",
"description": "Chases the newest hop varieties and double-digit IBUs, tracking hazy releases and limited tap takeovers across town.", "description": "Chases the newest hop varieties and double-digit IBUs, tracking hazy releases and limited tap takeovers across town.",
"style_affinities": ["American IPA", "Double IPA", "New England IPA", "Black IPA"] "style_affinities": [
"American IPA",
"Double IPA",
"New England IPA",
"Black IPA"
]
}, },
{ {
"name": "Malt & Tradition Loyalist", "name": "Malt & Tradition Loyalist",
@@ -12,7 +17,12 @@
{ {
"name": "Dark & Roasty Devotee", "name": "Dark & Roasty Devotee",
"description": "Orders the darkest beer on the board year-round, drawn to roasted malt, coffee, and chocolate notes over carbonated lightness.", "description": "Orders the darkest beer on the board year-round, drawn to roasted malt, coffee, and chocolate notes over carbonated lightness.",
"style_affinities": ["Imperial Stout", "Robust Porter", "Baltic Porter", "Oatmeal Stout"] "style_affinities": [
"Imperial Stout",
"Robust Porter",
"Baltic Porter",
"Oatmeal Stout"
]
}, },
{ {
"name": "Sour & Wild Forager", "name": "Sour & Wild Forager",
@@ -22,21 +32,41 @@
{ {
"name": "Session Sipper", "name": "Session Sipper",
"description": "Values an easy-drinking pint that holds up over a long afternoon at the taproom more than raw intensity or novelty.", "description": "Values an easy-drinking pint that holds up over a long afternoon at the taproom more than raw intensity or novelty.",
"style_affinities": ["Session IPA", "Ordinary Bitter", "Cream Ale", "Blonde Ale"] "style_affinities": [
"Session IPA",
"Ordinary Bitter",
"Cream Ale",
"Blonde Ale"
]
}, },
{ {
"name": "Belgian Abbey Pilgrim", "name": "Belgian Abbey Pilgrim",
"description": "Collects abbey and trappist releases, drawn to the dried-fruit esters and warming strength of classic Belgian ales.", "description": "Collects abbey and trappist releases, drawn to the dried-fruit esters and warming strength of classic Belgian ales.",
"style_affinities": ["Belgian Tripel", "Belgian Dubbel", "Belgian Quadrupel", "Trappist Single"] "style_affinities": [
"Belgian Tripel",
"Belgian Dubbel",
"Belgian Quadrupel",
"Trappist Single"
]
}, },
{ {
"name": "Wheat & Citrus Casual", "name": "Wheat & Citrus Casual",
"description": "Reaches for something light, cloudy, and refreshing, usually with a citrus or coriander note, especially on a patio in summer.", "description": "Reaches for something light, cloudy, and refreshing, usually with a citrus or coriander note, especially on a patio in summer.",
"style_affinities": ["Hefeweizen", "Witbier", "American Wheat Beer", "Berliner Weisse"] "style_affinities": [
"Hefeweizen",
"Witbier",
"American Wheat Beer",
"Berliner Weisse"
]
}, },
{ {
"name": "Big & Boozy Connoisseur", "name": "Big & Boozy Connoisseur",
"description": "Sips slowly from a snifter, seeking out high-ABV releases meant for aging and savoring rather than quick refreshment.", "description": "Sips slowly from a snifter, seeking out high-ABV releases meant for aging and savoring rather than quick refreshment.",
"style_affinities": ["English Barleywine", "Wee Heavy", "Doppelbock", "Old Ale"] "style_affinities": [
"English Barleywine",
"Wee Heavy",
"Doppelbock",
"Old Ale"
]
} }
] ]

View File

@@ -41,11 +41,11 @@ You will receive the inputs like this:
ABSOLUTELY NO MARKDOWN FORMATTING. Do NOT wrap your response in json or ``` ABSOLUTELY NO MARKDOWN FORMATTING. Do NOT wrap your response in json or ```
blocks. blocks.
Do not add markdown, code fences, or postscript around the final JSON object. Do not add markdown, code fences, or postscript around the final JSON object. Do
Do not say "Here is the JSON" or "Enjoy!". not say "Here is the JSON" or "Enjoy!".
The JSON must contain exactly three keys ("username", "bio", The JSON must contain exactly three keys ("username", "bio", "activity_weight")
"activity_weight") in that order. Do not rename or add any other keys. in that order. Do not rename or add any other keys.
ESCAPE ALL QUOTES inside the bio field using \", or use single quotes (' ') ESCAPE ALL QUOTES inside the bio field using \", or use single quotes (' ')
instead. instead.
@@ -53,15 +53,15 @@ instead.
DO NOT use actual line breaks (\n) inside any string. Keep the bio as one DO NOT use actual line breaks (\n) inside any string. Keep the bio as one
continuous string. continuous string.
The bio must be between 25 and 60 words, written in first person, and must The bio must be between 25 and 60 words, written in first person, and must read
read as something this specific person would write about themselves -- not a as something this specific person would write about themselves -- not a generic
generic profile blurb. profile blurb.
`activity_weight` is a number between 0 and 1 (inclusive) representing how `activity_weight` is a number between 0 and 1 (inclusive) representing how often
often this persona checks in at breweries relative to other users. Persona this persona checks in at breweries relative to other users. Persona archetypes
archetypes implying frequent, habitual brewery visits should skew higher; implying frequent, habitual brewery visits should skew higher; casual or
casual or occasional-visitor personas should skew lower. Do not default to occasional-visitor personas should skew lower. Do not default to 0.5 -- vary it
0.5 -- vary it meaningfully based on the persona description. meaningfully based on the persona description.
Expected JSON format: Expected JSON format:
@@ -78,8 +78,8 @@ Expected JSON format:
### THE USERNAME: ### THE USERNAME:
Derive the username from the given first and/or last name (e.g. lowercase, Derive the username from the given first and/or last name (e.g. lowercase,
abbreviated, or combined with a short word or number tied to the persona's abbreviated, or combined with a short word or number tied to the persona's style
style affinities). Do not invent an unrelated handle. Do not include spaces. affinities). Do not invent an unrelated handle. Do not include spaces.
### THE BIO: ### THE BIO:
@@ -90,14 +90,14 @@ city or country naturally if it fits, but do not force it.
### VOICE & PERSPECTIVE: ### VOICE & PERSPECTIVE:
Write in the first person ("I"/"my"). The tone should match the persona: Write in the first person ("I"/"my"). The tone should match the persona: an
an enthusiastic explorer sounds different from a relaxed, easy-drinking enthusiastic explorer sounds different from a relaxed, easy-drinking regular. Do
regular. Do not write in second or third person. not write in second or third person.
### THE BLOCKLIST (FORBIDDEN CONCEPTS): ### THE BLOCKLIST (FORBIDDEN CONCEPTS):
You absolutely cannot use the following words and phrases. Make sure your You absolutely cannot use the following words and phrases. Make sure your final
final output doesn't have any of these: output doesn't have any of these:
- "hidden gem" - "hidden gem"
- "passion" - "passion"
@@ -108,14 +108,14 @@ final output doesn't have any of these:
#### FORBIDDEN WRITING PATTERNS #### FORBIDDEN WRITING PATTERNS
The following patterns are common AI writing pitfalls and must not appear in The following patterns are common AI writing pitfalls and must not appear in the
the bio: bio:
- Negative parallelism constructions: "It's not X, it's Y" - Negative parallelism constructions: "It's not X, it's Y"
- Inflated significance phrases: "stands as a testament," "plays a vital - Inflated significance phrases: "stands as a testament," "plays a vital role,"
role," "deeply rooted" "deeply rooted"
- Superficial trailing analyses: sentences ending in -ing words that add - Superficial trailing analyses: sentences ending in -ing words that add opinion
opinion without content without content
- Promotional travel-copy tone: "breathtaking," "must-visit," "vibrant" - Promotional travel-copy tone: "breathtaking," "must-visit," "vibrant"
- Overused conjunctive transitions used as sentence openers: "Moreover," - Overused conjunctive transitions used as sentence openers: "Moreover,"
"Furthermore," "In addition" "Furthermore," "In addition"

View File

@@ -18,7 +18,8 @@ std::optional<ApplicationOptions> ParseArguments(
auto add_sampling_options = [&]() -> void { auto add_sampling_options = [&]() -> void {
const SamplingOptions sampling_defaults{}; const SamplingOptions sampling_defaults{};
opt("temperature", opt("temperature",
prog_opts::value<float>()->default_value(sampling_defaults.temperature), prog_opts::value<float>()->default_value(
sampling_defaults.temperature),
"Sampling temperature (higher = more random)"); "Sampling temperature (higher = more random)");
opt("top-p", opt("top-p",
prog_opts::value<float>()->default_value(sampling_defaults.top_p), prog_opts::value<float>()->default_value(sampling_defaults.top_p),
@@ -29,7 +30,8 @@ std::optional<ApplicationOptions> ParseArguments(
opt("n-ctx", opt("n-ctx",
prog_opts::value<uint32_t>()->default_value(sampling_defaults.n_ctx), prog_opts::value<uint32_t>()->default_value(sampling_defaults.n_ctx),
"Context window size in tokens"); "Context window size in tokens");
opt("seed", prog_opts::value<int>()->default_value(sampling_defaults.seed), opt("seed",
prog_opts::value<int>()->default_value(sampling_defaults.seed),
"Sampler seed: -1 for random, otherwise non-negative integer"); "Sampler seed: -1 for random, otherwise non-negative integer");
opt("n-gpu-layers", prog_opts::value<int>()->default_value(0), opt("n-gpu-layers", prog_opts::value<int>()->default_value(0),
"Number of layers to offload to GPU"); "Number of layers to offload to GPU");
@@ -82,7 +84,8 @@ std::optional<ApplicationOptions> ParseArguments(
try { try {
prog_opts::variables_map var_map; prog_opts::variables_map var_map;
prog_opts::store(prog_opts::parse_command_line(argc, argv, desc), var_map); prog_opts::store(prog_opts::parse_command_line(argc, argv, desc),
var_map);
prog_opts::notify(var_map); prog_opts::notify(var_map);
if (var_map.contains("help")) { if (var_map.contains("help")) {
@@ -101,7 +104,8 @@ std::optional<ApplicationOptions> ParseArguments(
options.pipeline.output_path = var_map["output"].as<std::string>(); options.pipeline.output_path = var_map["output"].as<std::string>();
options.pipeline.log_path = var_map["log-path"].as<std::string>(); options.pipeline.log_path = var_map["log-path"].as<std::string>();
options.pipeline.prompt_dir = var_map["prompt-dir"].as<std::string>(); options.pipeline.prompt_dir = var_map["prompt-dir"].as<std::string>();
options.pipeline.location_count = var_map["location-count"].as<uint32_t>(); options.pipeline.location_count =
var_map["location-count"].as<uint32_t>();
const bool use_mocked = var_map["mocked"].as<bool>(); const bool use_mocked = var_map["mocked"].as<bool>();
const std::string model_path = var_map["model"].as<std::string>(); const std::string model_path = var_map["model"].as<std::string>();
@@ -161,9 +165,10 @@ std::optional<ApplicationOptions> ParseArguments(
// back to its own SamplingOptions{} defaults, keeping the two paths // back to its own SamplingOptions{} defaults, keeping the two paths
// consistent without redundant copies. // consistent without redundant copies.
const bool user_provided_sampling = const bool user_provided_sampling =
!var_map["temperature"].defaulted() || !var_map["top-p"].defaulted() || !var_map["temperature"].defaulted() ||
!var_map["top-k"].defaulted() || !var_map["n-ctx"].defaulted() || !var_map["top-p"].defaulted() || !var_map["top-k"].defaulted() ||
!var_map["seed"].defaulted() || !var_map["n_gpu_layers"].defaulted(); !var_map["n-ctx"].defaulted() || !var_map["seed"].defaulted() ||
!var_map["n_gpu_layers"].defaulted();
if (user_provided_sampling) { if (user_provided_sampling) {
// Warn but do not fail — the run is still valid, the flags are just // Warn but do not fail — the run is still valid, the flags are just

View File

@@ -24,7 +24,8 @@ void BiergartenPipelineOrchestrator::GenerateBreweries(
[this, &skipped_count]( [this, &skipped_count](
const EnrichedCity& enriched_city) -> std::optional<BreweryRecord> { const EnrichedCity& enriched_city) -> std::optional<BreweryRecord> {
try { try {
const BreweryResult brewery = generator_->GenerateBrewery(enriched_city); const BreweryResult brewery =
generator_->GenerateBrewery(enriched_city);
const std::string postal_code = const std::string postal_code =
postal_code_service_->GeneratePostalCode(enriched_city.location); postal_code_service_->GeneratePostalCode(enriched_city.location);
return BreweryRecord{ return BreweryRecord{
@@ -34,10 +35,10 @@ void BiergartenPipelineOrchestrator::GenerateBreweries(
} catch (const std::exception& e) { } catch (const std::exception& e) {
++skipped_count; ++skipped_count;
logger_->Log( logger_->Log({.level = LogLevel::Warn,
{.level = LogLevel::Warn,
.phase = PipelinePhase::BreweryAndBeerGeneration, .phase = PipelinePhase::BreweryAndBeerGeneration,
.message = std::format("[Pipeline] Skipping city '{}' ({}): brewery " .message = std::format(
"[Pipeline] Skipping city '{}' ({}): brewery "
"generation failed: {}", "generation failed: {}",
enriched_city.location.city, enriched_city.location.city,
enriched_city.location.country, e.what())}); enriched_city.location.country, e.what())});
@@ -45,13 +46,14 @@ void BiergartenPipelineOrchestrator::GenerateBreweries(
} }
}; };
const auto export_record = [this, &export_failed_count]( const auto export_record =
const BreweryRecord& record) { [this, &export_failed_count](const BreweryRecord& record) {
try { try {
exporter_->ProcessRecord(record); exporter_->ProcessRecord(record);
} catch (const std::exception& export_exception) { } catch (const std::exception& export_exception) {
++export_failed_count; ++export_failed_count;
logger_->Log({.level = LogLevel::Warn, logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::BreweryAndBeerGeneration, .phase = PipelinePhase::BreweryAndBeerGeneration,
.message = std::format( .message = std::format(
"[Pipeline] Generated brewery for '{}' ({}) " "[Pipeline] Generated brewery for '{}' ({}) "
@@ -62,7 +64,8 @@ void BiergartenPipelineOrchestrator::GenerateBreweries(
}; };
for (const EnrichedCity& enriched_city : cities) { for (const EnrichedCity& enriched_city : cities) {
const std::optional<BreweryRecord> record = generate_record(enriched_city); const std::optional<BreweryRecord> record =
generate_record(enriched_city);
if (!record.has_value()) { if (!record.has_value()) {
continue; continue;
} }

View File

@@ -33,8 +33,8 @@ std::string Sanitize(std::string_view value) {
std::string BuildEmail(const Name& name, std::string BuildEmail(const Name& name,
std::unordered_set<std::string>& used_local_parts) { std::unordered_set<std::string>& used_local_parts) {
const std::string base = const std::string base = std::format("{}.{}", Sanitize(name.first_name),
std::format("{}.{}", Sanitize(name.first_name), Sanitize(name.last_name)); Sanitize(name.last_name));
std::string local_part = base; std::string local_part = base;
uint32_t suffix = 1; uint32_t suffix = 1;
@@ -147,12 +147,13 @@ void BiergartenPipelineOrchestrator::GenerateUsers(
}; };
} catch (const std::exception& e) { } catch (const std::exception& e) {
++skipped_count; ++skipped_count;
logger_->Log({.level = LogLevel::Warn, logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::UserGeneration,
.message = std::format( .message = std::format("[Pipeline] Skipping city '{}' ({}): "
"[Pipeline] Skipping city '{}' ({}): "
"user generation failed: {}", "user generation failed: {}",
city.location.city, city.location.country, e.what())}); city.location.city, city.location.country,
e.what())});
return std::nullopt; return std::nullopt;
} }
}; };
@@ -164,7 +165,8 @@ void BiergartenPipelineOrchestrator::GenerateUsers(
} catch (const std::exception& export_exception) { } catch (const std::exception& export_exception) {
++export_failed_count; ++export_failed_count;
logger_->Log({.level = LogLevel::Warn, logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::UserGeneration,
.message = std::format( .message = std::format(
"[Pipeline] Generated user for '{}' ({}) but " "[Pipeline] Generated user for '{}' ({}) but "

View File

@@ -14,8 +14,7 @@
#include "services/curated_data/curated_json_data_service.h" #include "services/curated_data/curated_json_data_service.h"
#include "services/logging/logger.h" #include "services/logging/logger.h"
std::vector<City> std::vector<City> BiergartenPipelineOrchestrator::QueryLocations() {
BiergartenPipelineOrchestrator::QueryLocations() {
logger_->Log({.level = LogLevel::Info, logger_->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = "=== GEOGRAPHIC DATA OVERVIEW ==="}); .message = "=== GEOGRAPHIC DATA OVERVIEW ==="});

View File

@@ -21,7 +21,8 @@ bool BiergartenPipelineOrchestrator::Run() {
size_t skipped_count = 0; size_t skipped_count = 0;
for (auto& city : cities) { for (auto& city : cities) {
try { try {
std::string region_context = context_service_->GetLocationContext(city); std::string region_context =
context_service_->GetLocationContext(city);
enriched.push_back( enriched.push_back(
EnrichedCity{.location = std::move(city), EnrichedCity{.location = std::move(city),

View File

@@ -89,10 +89,10 @@ BreweryResult LlamaGenerator::GenerateBrewery(
raw = this->Infer(system_prompt, user_prompt, max_tokens, raw = this->Infer(system_prompt, user_prompt, max_tokens,
kBreweryJsonGrammar); kBreweryJsonGrammar);
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Debug,
{.level = LogLevel::Debug,
.phase = PipelinePhase::BreweryAndBeerGeneration, .phase = PipelinePhase::BreweryAndBeerGeneration,
.message = std::format("LlamaGenerator: raw output (attempt {}): {}", .message = std::format(
"LlamaGenerator: raw output (attempt {}): {}",
attempt + 1, raw)}); attempt + 1, raw)});
} }
@@ -106,10 +106,10 @@ BreweryResult LlamaGenerator::GenerateBrewery(
// Success: return parsed brewery data // Success: return parsed brewery data
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Info,
{.level = LogLevel::Info,
.phase = PipelinePhase::BreweryAndBeerGeneration, .phase = PipelinePhase::BreweryAndBeerGeneration,
.message = std::format("LlamaGenerator: successfully generated " .message = std::format(
"LlamaGenerator: successfully generated "
"brewery data on attempt {}", "brewery data on attempt {}",
attempt + 1)}); attempt + 1)});
} }

View File

@@ -42,8 +42,8 @@ UserResult LlamaGenerator::GenerateUser(const EnrichedCity& city,
style_affinities); style_affinities);
const std::string retry_context = std::format( const std::string retry_context = std::format(
"Name: {} {}\nCity: {}, {}\nPersona: {}", name.first_name, name.last_name, "Name: {} {}\nCity: {}, {}\nPersona: {}", name.first_name,
city.location.city, city.location.country, persona.name); name.last_name, city.location.city, city.location.country, persona.name);
constexpr int max_attempts = 3; constexpr int max_attempts = 3;
std::string raw; std::string raw;
@@ -51,12 +51,13 @@ UserResult LlamaGenerator::GenerateUser(const EnrichedCity& city,
int max_tokens = kUserInitialMaxTokens; int max_tokens = kUserInitialMaxTokens;
for (int attempt = 0; attempt < max_attempts; ++attempt) { for (int attempt = 0; attempt < max_attempts; ++attempt) {
raw = this->Infer(system_prompt, user_prompt, max_tokens, kUserJsonGrammar); raw =
this->Infer(system_prompt, user_prompt, max_tokens, kUserJsonGrammar);
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Debug,
{.level = LogLevel::Debug,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::UserGeneration,
.message = std::format("LlamaGenerator: raw output (attempt {}): {}", .message = std::format(
"LlamaGenerator: raw output (attempt {}): {}",
attempt + 1, raw)}); attempt + 1, raw)});
} }
@@ -82,7 +83,8 @@ UserResult LlamaGenerator::GenerateUser(const EnrichedCity& city,
last_error = *validation_error; last_error = *validation_error;
if (logger_) { if (logger_) {
logger_->Log({.level = LogLevel::Warn, logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::UserGeneration,
.message = std::format( .message = std::format(
"LlamaGenerator: malformed user JSON (attempt {}): {}", "LlamaGenerator: malformed user JSON (attempt {}): {}",

View File

@@ -4,6 +4,8 @@
* parsing, token decoding, and JSON validation helpers for Llama modules. * parsing, token decoding, and JSON validation helpers for Llama modules.
*/ */
#include <llama.h>
#include <algorithm> #include <algorithm>
#include <array> #include <array>
#include <boost/json.hpp> #include <boost/json.hpp>
@@ -15,8 +17,6 @@
#include <string_view> #include <string_view>
#include <vector> #include <vector>
#include <llama.h>
#include "data_generation/llama_generator_helpers.h" #include "data_generation/llama_generator_helpers.h"
namespace { namespace {
@@ -82,8 +82,8 @@ bool ReadRequiredTrimmedStringField(const boost::json::object& obj,
bool HasSchemaPlaceholder(std::span<std::string* const> values) { bool HasSchemaPlaceholder(std::span<std::string* const> values) {
for (const std::string* value : values) { for (const std::string* value : values) {
std::string lowered = *value; std::string lowered = *value;
std::ranges::transform(lowered, lowered.begin(), std::ranges::transform(
[](const unsigned char character) { lowered, lowered.begin(), [](const unsigned char character) {
return static_cast<char>(std::tolower(character)); return static_cast<char>(std::tolower(character));
}); });
@@ -128,8 +128,8 @@ void AppendTokenPiece(const llama_vocab* vocab, llama_token token,
auto buffer_too_small = [](int32_t result) -> bool { return result < 0; }; auto buffer_too_small = [](int32_t result) -> bool { return result < 0; };
int32_t bytes = int32_t bytes = llama_token_to_piece(vocab, token, buffer.data(),
llama_token_to_piece(vocab, token, buffer.data(), buffer.size(), 0, true); buffer.size(), 0, true);
if (!buffer_too_small(bytes)) { if (!buffer_too_small(bytes)) {
// Append the decoded bytes from the stack buffer. // Append the decoded bytes from the stack buffer.
@@ -190,8 +190,8 @@ std::optional<std::string> ValidateBreweryJson(const std::string& raw,
return validation_error; return validation_error;
} }
if (!ReadRequiredTrimmedStringField(obj, "name_local", brewery_out.name_local, if (!ReadRequiredTrimmedStringField(
&validation_error)) { obj, "name_local", brewery_out.name_local, &validation_error)) {
return validation_error; return validation_error;
} }
@@ -248,7 +248,8 @@ std::optional<std::string> ValidateUserJson(const std::string& raw,
const boost::json::value* activity_weight_field = const boost::json::value* activity_weight_field =
obj.if_contains("activity_weight"); obj.if_contains("activity_weight");
if (activity_weight_field == nullptr || !activity_weight_field->is_number()) { if (activity_weight_field == nullptr ||
!activity_weight_field->is_number()) {
return "Missing or invalid numeric field: activity_weight"; return "Missing or invalid numeric field: activity_weight";
} }

View File

@@ -5,6 +5,8 @@
* output tokens back to text for system+user chat prompts. * output tokens back to text for system+user chat prompts.
*/ */
#include <llama.h>
#include <algorithm> #include <algorithm>
#include <chrono> #include <chrono>
#include <format> #include <format>
@@ -14,8 +16,6 @@
#include <string_view> #include <string_view>
#include <vector> #include <vector>
#include <llama.h>
#include "data_generation/llama_generator.h" #include "data_generation/llama_generator.h"
#include "data_generation/llama_generator_helpers.h" #include "data_generation/llama_generator_helpers.h"
@@ -58,7 +58,8 @@ SamplerHandle MakeSamplerChain(const llama_vocab* vocab,
if (!grammar.empty()) { if (!grammar.empty()) {
const std::string grammar_text(grammar); const std::string grammar_text(grammar);
add_sampler(llama_sampler_init_grammar(vocab, grammar_text.c_str(), "root"), add_sampler(
llama_sampler_init_grammar(vocab, grammar_text.c_str(), "root"),
"LlamaGenerator: failed to initialize grammar sampler"); "LlamaGenerator: failed to initialize grammar sampler");
} }
@@ -173,7 +174,8 @@ std::string LlamaGenerator::InferFormatted(const std::string& formatted_prompt,
prompt_tokens.resize(static_cast<size_t>(token_count)); prompt_tokens.resize(static_cast<size_t>(token_count));
if (token_count > prompt_budget) { if (token_count > prompt_budget) {
if (logger_) { if (logger_) {
logger_->Log({.level = LogLevel::Warn, logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::BreweryAndBeerGeneration, .phase = PipelinePhase::BreweryAndBeerGeneration,
.message = std::format( .message = std::format(
"LlamaGenerator: prompt too long ({} tokens), " "LlamaGenerator: prompt too long ({} tokens), "

View File

@@ -5,14 +5,14 @@
#include "data_generation/llama_generator.h" #include "data_generation/llama_generator.h"
#include <llama.h>
#include <filesystem> #include <filesystem>
#include <memory> #include <memory>
#include <random> #include <random>
#include <stdexcept> #include <stdexcept>
#include <string> #include <string>
#include <llama.h>
#include "data_model/models.h" #include "data_model/models.h"
static constexpr uint32_t kMaxContextSize = 32768U; static constexpr uint32_t kMaxContextSize = 32768U;

View File

@@ -4,15 +4,15 @@
* context, and resets prior resources during model initialization. * context, and resets prior resources during model initialization.
*/ */
#include <ggml-backend.h>
#include <llama.h>
#include <algorithm> #include <algorithm>
#include <chrono> #include <chrono>
#include <stdexcept> #include <stdexcept>
#include <string> #include <string>
#include <utility> #include <utility>
#include <ggml-backend.h>
#include <llama.h>
#include "data_generation/llama_generator.h" #include "data_generation/llama_generator.h"
// Maximum batch size for decode operations. Capping the batch prevents // Maximum batch size for decode operations. Capping the batch prevents

View File

@@ -72,7 +72,8 @@ int main(const int argc, char** argv) {
prompt_directory = std::make_unique<PromptDirectory>( prompt_directory = std::make_unique<PromptDirectory>(
options.pipeline.prompt_dir, log_producer); options.pipeline.prompt_dir, log_producer);
} catch (const std::exception& dir_error) { } catch (const std::exception& dir_error) {
log_producer->Log({.level = LogLevel::Error, log_producer->Log(
{.level = LogLevel::Error,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = std::format("Invalid --prompt-dir: {}", .message = std::format("Invalid --prompt-dir: {}",
dir_error.what())}); dir_error.what())});
@@ -87,7 +88,8 @@ int main(const int argc, char** argv) {
di::bind<std::string>().to(model_path), di::bind<std::string>().to(model_path),
di::bind<IExportService>().to<SqliteExportService>(), di::bind<IExportService>().to<SqliteExportService>(),
di::bind<IPostalCodeService>().to( di::bind<IPostalCodeService>().to(
[options, &log_producer]() -> std::unique_ptr<IPostalCodeService> { [options,
&log_producer]() -> std::unique_ptr<IPostalCodeService> {
if (options.generator.use_mocked) { if (options.generator.use_mocked) {
log_producer->Log( log_producer->Log(
{.level = LogLevel::Info, {.level = LogLevel::Info,
@@ -100,12 +102,14 @@ int main(const int argc, char** argv) {
log_producer->Log( log_producer->Log(
{.level = LogLevel::Info, {.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = "Postal code: Xeger (regex-based generation)"}); .message =
"Postal code: Xeger (regex-based generation)"});
return std::make_unique<XegerPostalCodeService>(); return std::make_unique<XegerPostalCodeService>();
}), }),
di::bind<ICuratedDataService>().to( di::bind<ICuratedDataService>().to(
[options, &log_producer]() -> std::unique_ptr<ICuratedDataService> { [options,
&log_producer]() -> std::unique_ptr<ICuratedDataService> {
if (options.generator.use_mocked) { if (options.generator.use_mocked) {
log_producer->Log({.level = LogLevel::Info, log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
@@ -140,7 +144,8 @@ int main(const int argc, char** argv) {
log_producer->Log( log_producer->Log(
{.level = LogLevel::Info, {.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = "Prompt formatter: Gemma4JinjaPromptFormatter"}); .message =
"Prompt formatter: Gemma4JinjaPromptFormatter"});
} }
return std::unique_ptr<IPromptFormatter>( return std::unique_ptr<IPromptFormatter>(
std::make_unique<Gemma4JinjaPromptFormatter>()); std::make_unique<Gemma4JinjaPromptFormatter>());
@@ -148,7 +153,8 @@ int main(const int argc, char** argv) {
di::bind<WebClient>().to([options, log_producer] { di::bind<WebClient>().to([options, log_producer] {
if (options.generator.use_mocked) { if (options.generator.use_mocked) {
{ {
log_producer->Log({.level = LogLevel::Info, log_producer->Log(
{.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = "Web client: none (mock mode)"}); .message = "Web client: none (mock mode)"});
} }
@@ -183,7 +189,8 @@ int main(const int argc, char** argv) {
}), }),
di::bind<DataGenerator>().to( di::bind<DataGenerator>().to(
[&options, &model_path, &sampling, &prompt_directory, [&options, &model_path, &sampling, &prompt_directory,
&log_producer](const auto& inj) -> std::unique_ptr<DataGenerator> { &log_producer](
const auto& inj) -> std::unique_ptr<DataGenerator> {
if (options.generator.use_mocked) { if (options.generator.use_mocked) {
log_producer->Log({.level = LogLevel::Info, log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,

View File

@@ -179,7 +179,8 @@ const PersonasList& CuratedJsonDataService::LoadPersonas() {
personas.push_back(UserPersona{ personas.push_back(UserPersona{
.name = ReadRequiredString(object, "name"), .name = ReadRequiredString(object, "name"),
.description = ReadRequiredString(object, "description"), .description = ReadRequiredString(object, "description"),
.style_affinities = ReadRequiredStringArray(object, "style_affinities"), .style_affinities =
ReadRequiredStringArray(object, "style_affinities"),
}); });
} }
@@ -211,7 +212,8 @@ const ForenamesByCountryMap& CuratedJsonDataService::LoadForenamesByCountry() {
if (!region.is_object()) { if (!region.is_object()) {
continue; continue;
} }
const boost::json::value* names = region.as_object().if_contains("names"); const boost::json::value* names =
region.as_object().if_contains("names");
if (names == nullptr || !names->is_array()) { if (names == nullptr || !names->is_array()) {
continue; continue;
} }
@@ -221,8 +223,8 @@ const ForenamesByCountryMap& CuratedJsonDataService::LoadForenamesByCountry() {
} }
const auto& name_object = name_value.as_object(); const auto& name_object = name_value.as_object();
entries.emplace_back(ForenameEntry{ entries.emplace_back(ForenameEntry{
.name = .name = ReadFirstOfStringArray(name_object, "romanized",
ReadFirstOfStringArray(name_object, "romanized", "localized"), "localized"),
.gender = ReadRequiredString(name_object, "gender"), .gender = ReadRequiredString(name_object, "gender"),
}); });
} }
@@ -257,8 +259,8 @@ const SurnamesByCountryMap& CuratedJsonDataService::LoadSurnamesByCountry() {
if (!name_value.is_object()) { if (!name_value.is_object()) {
continue; continue;
} }
surnames.emplace_back(ReadFirstOfStringArray(name_value.as_object(), surnames.emplace_back(ReadFirstOfStringArray(
"romanized", "localized")); name_value.as_object(), "romanized", "localized"));
} }
surnames_by_country.emplace(country_code, std::move(surnames)); surnames_by_country.emplace(country_code, std::move(surnames));
} }

View File

@@ -21,10 +21,10 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
if (const auto cache_it = this->extract_cache_.find(cache_key); if (const auto cache_it = this->extract_cache_.find(cache_key);
cache_it != this->extract_cache_.end()) { cache_it != this->extract_cache_.end()) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Debug,
{.level = LogLevel::Debug,
.phase = PipelinePhase::Enrichment, .phase = PipelinePhase::Enrichment,
.message = std::format("Wikipedia: Cache hit for {}!", cache_key)}); .message = std::format("Wikipedia: Cache hit for {}!",
cache_key)});
} }
return cache_it->second; return cache_it->second;
} }
@@ -121,10 +121,10 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
// Handle 404/Missing status // Handle 404/Missing status
if (page.contains("missing")) { if (page.contains("missing")) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Warn,
{.level = LogLevel::Warn,
.phase = PipelinePhase::Enrichment, .phase = PipelinePhase::Enrichment,
.message = std::format("WikipediaService: Page '{}' does not exist", .message = std::format(
"WikipediaService: Page '{}' does not exist",
std::string(query))}); std::string(query))});
} }
this->extract_cache_.emplace(cache_key, ""); this->extract_cache_.emplace(cache_key, "");

View File

@@ -10,8 +10,7 @@
#include "services/enrichment/wikipedia_service.h" #include "services/enrichment/wikipedia_service.h"
std::string WikipediaEnrichmentService::GetLocationContext( std::string WikipediaEnrichmentService::GetLocationContext(const City& loc) {
const City& loc) {
using namespace std::literals::chrono_literals; using namespace std::literals::chrono_literals;
if (!this->client_) { if (!this->client_) {
if (logger_) { if (logger_) {
@@ -53,10 +52,10 @@ std::string WikipediaEnrichmentService::GetLocationContext(
} catch (const std::runtime_error& e) { } catch (const std::runtime_error& e) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Debug,
{.level = LogLevel::Debug,
.phase = PipelinePhase::Enrichment, .phase = PipelinePhase::Enrichment,
.message = std::format("WikipediaService lookup failed for '{}': {}", .message = std::format(
"WikipediaService lookup failed for '{}': {}",
location_query, e.what())}); location_query, e.what())});
} }
} }

View File

@@ -19,8 +19,8 @@ void SqliteStatementDeleter::operator()(
} }
void ThrowSqliteError(sqlite3* db_handle, std::string_view action) { void ThrowSqliteError(sqlite3* db_handle, std::string_view action) {
const std::string message = const std::string message = db_handle != nullptr ? sqlite3_errmsg(db_handle)
db_handle != nullptr ? sqlite3_errmsg(db_handle) : "unknown SQLite error"; : "unknown SQLite error";
throw std::runtime_error(std::format("{}: {}", action, message)); throw std::runtime_error(std::format("{}: {}", action, message));
} }

View File

@@ -108,7 +108,8 @@ uint64_t SqliteExportService::ProcessRecord(const BreweryRecord& brewery) {
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_stmt_, insert_brewery_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = sqlite_export_service_internal::kBreweryEnglishNameBindIndex, .index =
sqlite_export_service_internal::kBreweryEnglishNameBindIndex,
.value = brewery.brewery.name_en, .value = brewery.brewery.name_en,
.action = "Failed to bind SQLite brewery English name"}); .action = "Failed to bind SQLite brewery English name"});
@@ -130,8 +131,8 @@ uint64_t SqliteExportService::ProcessRecord(const BreweryRecord& brewery) {
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_stmt_, insert_brewery_stmt_,
sqlite_export_service_internal::BoundParam<std::string_view>{ sqlite_export_service_internal::BoundParam<std::string_view>{
.index = .index = sqlite_export_service_internal::
sqlite_export_service_internal::kBreweryLocalDescriptionBindIndex, kBreweryLocalDescriptionBindIndex,
.value = brewery.brewery.description_local, .value = brewery.brewery.description_local,
.action = "Failed to bind SQLite brewery local description"}); .action = "Failed to bind SQLite brewery local description"});
@@ -146,8 +147,8 @@ uint64_t SqliteExportService::ProcessRecord(const BreweryRecord& brewery) {
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_address_stmt_, insert_brewery_address_stmt_,
sqlite_export_service_internal::BoundParam<sqlite3_int64>{ sqlite_export_service_internal::BoundParam<sqlite3_int64>{
.index = .index = sqlite_export_service_internal::
sqlite_export_service_internal::kBreweryAddressBreweryIdBindIndex, kBreweryAddressBreweryIdBindIndex,
.value = brewery_id, .value = brewery_id,
.action = "Failed to bind SQLite brewery address brewery id"}); .action = "Failed to bind SQLite brewery address brewery id"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(

View File

@@ -48,7 +48,8 @@ uint64_t SqliteExportService::ProcessRecord(const UserRecord& user) {
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_user_stmt_, insert_user_stmt_,
sqlite_export_service_internal::BoundParam<double>{ sqlite_export_service_internal::BoundParam<double>{
.index = sqlite_export_service_internal::kUserActivityWeightBindIndex, .index =
sqlite_export_service_internal::kUserActivityWeightBindIndex,
.value = static_cast<double>(user.user.activity_weight), .value = static_cast<double>(user.user.activity_weight),
.action = "Failed to bind SQLite user activity weight"}); .action = "Failed to bind SQLite user activity weight"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(

View File

@@ -3,14 +3,14 @@
* @brief Visitor that walks a regex AST and emits a random matching string. * @brief Visitor that walks a regex AST and emits a random matching string.
*/ */
#include "services/xeger/xeger_internal.h"
#include <cassert> #include <cassert>
#include <limits> #include <limits>
#include <random> #include <random>
#include <stdexcept> #include <stdexcept>
#include <string> #include <string>
#include "services/xeger/xeger_internal.h"
namespace xeger_internal { namespace xeger_internal {
XegerGenerator::XegerGenerator(std::mt19937& rng, const int unbounded_cap) XegerGenerator::XegerGenerator(std::mt19937& rng, const int unbounded_cap)

View File

@@ -3,8 +3,6 @@
* @brief Recursive-descent parser that turns a regex pattern into an AST. * @brief Recursive-descent parser that turns a regex pattern into an AST.
*/ */
#include "services/xeger/xeger_internal.h"
#include <cctype> #include <cctype>
#include <format> #include <format>
#include <memory> #include <memory>
@@ -14,6 +12,8 @@
#include <utility> #include <utility>
#include <vector> #include <vector>
#include "services/xeger/xeger_internal.h"
namespace xeger_internal { namespace xeger_internal {
Parser::Parser(std::string_view pattern) : src_(StripAnchors(pattern)) {} Parser::Parser(std::string_view pattern) : src_(StripAnchors(pattern)) {}
@@ -168,7 +168,8 @@ NodePtr Parser::ParseAtom() {
// here (nothing back-references a group), so it's treated exactly // here (nothing back-references a group), so it's treated exactly
// like a plain group once the "?:" marker is consumed. // like a plain group once the "?:" marker is consumed.
if (!AtEnd() && Peek() == kNonCapturingMarker) { if (!AtEnd() && Peek() == kNonCapturingMarker) {
if (pos_ + 1 < src_.size() && src_[pos_ + 1] == kNonCapturingColon) { if (pos_ + 1 < src_.size() &&
src_[pos_ + 1] == kNonCapturingColon) {
pos_ += 2; pos_ += 2;
} else { } else {
throw std::runtime_error(std::format( throw std::runtime_error(std::format(
@@ -231,11 +232,11 @@ NodePtr Parser::ParseEscape() {
return MakeClass({{'a', 'z'}, {'A', 'Z'}, {'0', '9'}, {'_', '_'}}, return MakeClass({{'a', 'z'}, {'A', 'Z'}, {'0', '9'}, {'_', '_'}},
true); true);
case 's': case 's':
return MakeClass({{' ', ' '}, {'\t', '\t'}, {'\n', '\n'}, {'\r', '\r'}}, return MakeClass(
false); {{' ', ' '}, {'\t', '\t'}, {'\n', '\n'}, {'\r', '\r'}}, false);
case 'S': case 'S':
return MakeClass({{' ', ' '}, {'\t', '\t'}, {'\n', '\n'}, {'\r', '\r'}}, return MakeClass(
true); {{' ', ' '}, {'\t', '\t'}, {'\n', '\n'}, {'\r', '\r'}}, true);
case 'n': case 'n':
return std::make_unique<Literal>('\n'); return std::make_unique<Literal>('\n');
case 't': case 't':

View File

@@ -60,8 +60,8 @@ std::string HttpWebClient::Get(const std::string& url) {
.message = std::format( .message = std::format(
"[HttpWebClient] Request failed for URL: {}", url)}); "[HttpWebClient] Request failed for URL: {}", url)});
} }
throw std::runtime_error(std::format("[HttpWebClient] HTTP {} for URL: {}", throw std::runtime_error(std::format(
result->status, url)); "[HttpWebClient] HTTP {} for URL: {}", result->status, url));
} }
return result->body; return result->body;

File diff suppressed because it is too large Load Diff