AaronPo/the-biergarten-app
1
0
Fork 0
mirror of https://github.com/aaronpo97/the-biergarten-app.git synced 2026-07-16 17:47:22 +00:00
Code Issues Packages Projects Releases Wiki Activity

Pipeline: rename Location to City, wire postal code into breweries

Browse Source
This commit is contained in:
Aaron Po
2026-07-13 02:26:44 -04:00
parent fbcf438381
commit d52dba904c
48 changed files with 2261 additions and 1227 deletions
Download Patch File Download Diff File Expand all files Collapse all files

49
docs/pipeline/README.md
View File

@@ -86,7 +86,7 @@ curl -L \
### Run
Run from `build/` so the copied `locations.json` and `prompts/` are available.
Run from `build/` so the copied `cities.json` and `prompts/` are available.
Each run writes a fresh dated SQLite file such as
`biergarten_seed_2026-04-19T15-30-45.123456Z.sqlite` into the working directory.
@@ -109,7 +109,7 @@ Each run writes a fresh dated SQLite file such as
| `--prompt-dir` | Directory containing prompt files (e.g. `BREWERY_GENERATION.md`). Required unless `--mocked` is set. |
| `--output, -o` | Directory for generated SQLite artifacts. Default: `output`. |
| `--log-path` | Path for application logs. Default: `pipeline.log`. |
| `--location-count` | Number of cities to sample from `locations.json` per run. Default: `10`. |
| `--location-count` | Number of cities to sample from `cities.json` per run. Default: `10`. |
| `--temperature` | Sampling temperature. Default: `1.0`. |
| `--top-p` | Nucleus sampling. Default: `0.95`. |
| `--top-k` | Top-k sampling. Default: `64`. |
@@ -222,12 +222,12 @@ environment variables listed above to match your run.
| Stage | Implementation |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| Load | `ICuratedDataService` (`CuratedJsonDataService`) reads `locations.json`, `personas.json`, `forenames-by-country.json`, and `surnames-by-country.json` (paths supplied via a `CuratedDataFilePaths` DTO at construction) into typed records, caching each after its first load. `--mocked` runs use `MockCuratedDataService`'s fixed in-memory dataset instead. |
| Load | `ICuratedDataService` (`CuratedJsonDataService`) reads `cities.json`, `personas.json`, `forenames-by-country.json`, and `surnames-by-country.json` (paths supplied via a `CuratedDataFilePaths` DTO at construction) into typed records, caching each after its first load. `--mocked` runs use `MockCuratedDataService`'s fixed in-memory dataset instead. |
| Sample | `BiergartenPipelineOrchestrator::QueryCitiesWithCountries()` samples `--location-count` locations per run (default `10`). |
| Enrich | `WikipediaEnrichmentService` fetches brewing and beer-related context. Keeps going when a lookup fails. `--mocked` runs use `MockEnrichmentService` instead and skip Wikipedia entirely. |
| Generate Users | `GenerateUsers()` samples a persona and a forename/surname pair per enriched city (skipping countries with no name data), then `MockGenerator` or `LlamaGenerator` produces a username, bio, and activity weight around the sampled name. |
| Generate Breweries | `MockGenerator` or `LlamaGenerator` produces brewery names and descriptions in English and the local language. |
| Store | `SqliteExportService` writes each successful user and brewery into a fresh dated `.sqlite` database with normalized `locations`, `users`, and `breweries` tables. |
| Store | `SqliteExportService` writes each successful user and brewery into a fresh dated `.sqlite` database with normalized `cities`, `users`, and `breweries` tables. |
| Log | `spdlog` writes results and warnings to the console. |
If name sampling, enrichment, or generation fails for a city, that city is
@@ -241,12 +241,27 @@ skipped and the pipeline continues. `GenerateUsers()` runs before
`CuratedDataFilePaths` DTO (locations/personas/forenames/surnames paths) in
its constructor, then parses and validates curated location, persona, and
forename/surname JSON, memoizing each result after its first load on a
given instance. `MockCuratedDataService` is the in-memory substitute (4
fixed locations, 3 personas, and name data for `US`/`DE`/`FR`/`BE`) used in
`--mocked` runs.
given instance. Each `cities.json` entry's `postal_code.city_regex` and
`postal_code.examples` are parsed into `City::postal_regex` and
`City::postal_code_examples`. `MockCuratedDataService` is the in-memory
substitute (4 fixed locations, 3 personas, and name data for
`US`/`DE`/`FR`/`BE`) used in `--mocked` runs, and carries matching
`postal_regex`/`postal_code_examples` values for its 4 locations.
- `WikipediaEnrichmentService` — queries Wikipedia extracts, caches results,
returns empty context on failure. `MockEnrichmentService` is the no-op
substitute used in `--mocked` runs.
- `IPostalCodeService` — generates a postal code for a `City`, consumed by
`GenerateBreweries()` and stored on `BreweryRecord::address` (an `Address`
struct, currently just `postal_code`, mirroring the web backend's
`BreweryPostLocation`). Only `MockPostalCodeService` exists today, which
ignores `postal_regex` and returns `postal_code_examples.front()` — it's
wired into the Boost.DI graph unconditionally (no `--mocked` branch yet,
since there's no real implementation to switch to). A real implementation
still needs a **xeger**-style generator — turning a `postal_regex` pattern
into a random matching string — instead of always replaying a fixed
example; see [ROADMAP.md §9](./ROADMAP.md#9-postal-code-generation).
Street-address generation (`Address::address_line1`) has no fixture data or
service yet and remains future work.
- `LlamaGenerator` — formats prompts for Gemma 4, validates JSON output for
both `GenerateBrewery` and `GenerateUser`, retries malformed responses up
to three times with corrective feedback in the retry prompt. The token
@@ -279,14 +294,14 @@ valid output on the first pass; the retry path is kept for resilience.
`MockGenerator` uses stable hashes for repeatable output in demos and Storybook
runs.
`CuratedJsonDataService` memoizes each of `LoadLocations()`, `LoadPersonas()`,
`CuratedJsonDataService` memoizes each of `LoadCities()`, `LoadPersonas()`,
`LoadForenamesByCountry()`, and `LoadSurnamesByCountry()` independently the
first time each is called, since `BiergartenPipelineOrchestrator` owns a
single `ICuratedDataService` instance for the whole run — later calls return
the cached result instead of re-parsing.
`GenerateUsers()` samples a forename/surname pair per city via `SampleName()`,
keyed by the city's ISO 3166-1 code. Countries present in `locations.json`
keyed by the city's ISO 3166-1 code. Countries present in `cities.json`
but absent from either name fixture (currently `KE`, `SE`, `SG`, `TH`, `VN`,
`ZA`) are skipped, the same way a failed enrichment or generation call skips
a city — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section.
@@ -303,9 +318,9 @@ a city — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section.
## Generated Output
Each successful run stores a `BreweryRecord` pair with the source location
and a `BreweryResult` payload, and a `UserRecord` pair with the source
location and a `UserResult` payload. The same generated records are also
Each successful run stores a `BreweryRecord` (source `City`, an `Address`,
and a `BreweryResult` payload), and a `UserRecord` pair with the source
`City` and a `UserResult` payload. The same generated records are also
written to a fresh SQLite export file named with the current UTC timestamp.
| Field | Meaning |
@@ -314,6 +329,7 @@ written to a fresh SQLite export file named with the current UTC timestamp.
| `description_en` | Brewery description in English. |
| `name_local` | Brewery name in the local language. |
| `description_local` | Brewery description in the local language. |
| `postal_code` | Postal code generated for the brewery's city (see `IPostalCodeService`, above). |
| Field | Meaning |
| ----------------- | ----------------------------------------------------------------- |
@@ -339,6 +355,7 @@ code, latitude, and longitude for each entry.
| `local_languages` | Locale-aware copy selection |
| `name_en`, `description_en` | Default English display content |
| `name_local`, `description_local` | Local-language display content |
| `postal_code` | Brewery address matching, mirrors the web backend's `BreweryPostLocation.PostalCode` |
---
@@ -407,7 +424,7 @@ Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present.
`MockCuratedDataService` swaps in for `CuratedJsonDataService`, so no
fixture files need to be present on disk.
- `--model` when geographically grounded content matters for demos.
- Keep `locations.json` structured enough to support discovery and future
- Keep `cities.json` structured enough to support discovery and future
filtering.
- `personas.json`, `forenames-by-country.json`, and
`surnames-by-country.json` are curated/vendored fixture data, not
@@ -424,7 +441,7 @@ Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present.
| -------------------------------------- | -------------------------------------------------- |
| `tooling/pipeline/includes/` | Public headers and shared models. |
| `tooling/pipeline/src/` | Implementation files. |
| `tooling/pipeline/locations.json` | Curated city input copied into the build tree. |
| `tooling/pipeline/cities.json` | Curated city input copied into the build tree. |
| `tooling/pipeline/personas.json` | Curated user persona archetypes copied into the build tree. |
| `tooling/pipeline/forenames-by-country.json` | Vendored (CC0) forename data by ISO 3166-1 country code. |
| `tooling/pipeline/surnames-by-country.json` | Vendored (CC0) surname data by ISO 3166-1 country code. |
@@ -446,6 +463,10 @@ Paths below are relative to `tooling/pipeline/`.
`ICuratedDataService`, and `MockCuratedDataService`, the in-memory
`ICuratedDataService` used in `--mocked` runs.
- `src/services/enrichment/wikipedia/` — enrichment service and cache.
- `includes/services/postal_code/` — `IPostalCodeService` and
`MockPostalCodeService` (header-only), consumed by `GenerateBreweries()`.
The real xeger-based implementation and its `--mocked`-aware DI wiring are
still to come.
- `src/services/sqlite/` — SQLite export implementation.
- `src/data_generation/llama/` — local inference, prompt loading, output
validation.