AaronPo/the-biergarten-app
1
0
Fork 0
mirror of https://github.com/aaronpo97/the-biergarten-app.git synced 2026-07-16 17:47:22 +00:00
Code Issues Packages Projects Releases Wiki Activity

Pipeline: rename Location to City, wire postal code into breweries

Browse Source
This commit is contained in:
Aaron Po
2026-07-13 02:26:44 -04:00
parent fbcf438381
commit d52dba904c
48 changed files with 2261 additions and 1227 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
docs/pipeline/ETHICS-AND-KNOWN-ISSUES.md
View File

@@ -109,10 +109,10 @@ gender from the source data is preserved through to the sampled `Name`
persona/bio generation later.
The full multinational dataset is kept as-is (106 countries for forenames,
75 for surnames) rather than trimmed to `locations.json`'s current country
75 for surnames) rather than trimmed to `cities.json`'s current country
list, so it doesn't need re-sourcing if more countries are added later.
`SampleName()` (a free helper in `generate_users.cc`) returns no result for
a country present in neither file; of the countries in `locations.json`,
a country present in neither file; of the countries in `cities.json`,
that's currently `KE`, `SE`, `SG`, `TH`, `VN`, and `ZA` — `GenerateUsers`
skips cities in those countries the same way brewery generation skips
cities whose enrichment lookup fails.
@@ -281,7 +281,7 @@ Output sample:
a location's code is unlisted, skip `description_local` generation and fall
back to English.
- **Upstream sanitization:** strip known low-resource language codes from the
`locations.json` payload before generation.
`cities.json` payload before generation.
- **Downstream flagging:** add a `description_local_confidence` column to the
SQLite schema so downstream applications can filter or flag potentially
hallucinated text by language tier.

49
docs/pipeline/README.md
View File

@@ -86,7 +86,7 @@ curl -L \
### Run
Run from `build/` so the copied `locations.json` and `prompts/` are available.
Run from `build/` so the copied `cities.json` and `prompts/` are available.
Each run writes a fresh dated SQLite file such as
`biergarten_seed_2026-04-19T15-30-45.123456Z.sqlite` into the working directory.
@@ -109,7 +109,7 @@ Each run writes a fresh dated SQLite file such as
| `--prompt-dir` | Directory containing prompt files (e.g. `BREWERY_GENERATION.md`). Required unless `--mocked` is set. |
| `--output, -o` | Directory for generated SQLite artifacts. Default: `output`. |
| `--log-path` | Path for application logs. Default: `pipeline.log`. |
| `--location-count` | Number of cities to sample from `locations.json` per run. Default: `10`. |
| `--location-count` | Number of cities to sample from `cities.json` per run. Default: `10`. |
| `--temperature` | Sampling temperature. Default: `1.0`. |
| `--top-p` | Nucleus sampling. Default: `0.95`. |
| `--top-k` | Top-k sampling. Default: `64`. |
@@ -222,12 +222,12 @@ environment variables listed above to match your run.
| Stage | Implementation |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| Load | `ICuratedDataService` (`CuratedJsonDataService`) reads `locations.json`, `personas.json`, `forenames-by-country.json`, and `surnames-by-country.json` (paths supplied via a `CuratedDataFilePaths` DTO at construction) into typed records, caching each after its first load. `--mocked` runs use `MockCuratedDataService`'s fixed in-memory dataset instead. |
| Load | `ICuratedDataService` (`CuratedJsonDataService`) reads `cities.json`, `personas.json`, `forenames-by-country.json`, and `surnames-by-country.json` (paths supplied via a `CuratedDataFilePaths` DTO at construction) into typed records, caching each after its first load. `--mocked` runs use `MockCuratedDataService`'s fixed in-memory dataset instead. |
| Sample | `BiergartenPipelineOrchestrator::QueryCitiesWithCountries()` samples `--location-count` locations per run (default `10`). |
| Enrich | `WikipediaEnrichmentService` fetches brewing and beer-related context. Keeps going when a lookup fails. `--mocked` runs use `MockEnrichmentService` instead and skip Wikipedia entirely. |
| Generate Users | `GenerateUsers()` samples a persona and a forename/surname pair per enriched city (skipping countries with no name data), then `MockGenerator` or `LlamaGenerator` produces a username, bio, and activity weight around the sampled name. |
| Generate Breweries | `MockGenerator` or `LlamaGenerator` produces brewery names and descriptions in English and the local language. |
| Store | `SqliteExportService` writes each successful user and brewery into a fresh dated `.sqlite` database with normalized `locations`, `users`, and `breweries` tables. |
| Store | `SqliteExportService` writes each successful user and brewery into a fresh dated `.sqlite` database with normalized `cities`, `users`, and `breweries` tables. |
| Log | `spdlog` writes results and warnings to the console. |
If name sampling, enrichment, or generation fails for a city, that city is
@@ -241,12 +241,27 @@ skipped and the pipeline continues. `GenerateUsers()` runs before
`CuratedDataFilePaths` DTO (locations/personas/forenames/surnames paths) in
its constructor, then parses and validates curated location, persona, and
forename/surname JSON, memoizing each result after its first load on a
given instance. `MockCuratedDataService` is the in-memory substitute (4
fixed locations, 3 personas, and name data for `US`/`DE`/`FR`/`BE`) used in
`--mocked` runs.
given instance. Each `cities.json` entry's `postal_code.city_regex` and
`postal_code.examples` are parsed into `City::postal_regex` and
`City::postal_code_examples`. `MockCuratedDataService` is the in-memory
substitute (4 fixed locations, 3 personas, and name data for
`US`/`DE`/`FR`/`BE`) used in `--mocked` runs, and carries matching
`postal_regex`/`postal_code_examples` values for its 4 locations.
- `WikipediaEnrichmentService` — queries Wikipedia extracts, caches results,
returns empty context on failure. `MockEnrichmentService` is the no-op
substitute used in `--mocked` runs.
- `IPostalCodeService` — generates a postal code for a `City`, consumed by
`GenerateBreweries()` and stored on `BreweryRecord::address` (an `Address`
struct, currently just `postal_code`, mirroring the web backend's
`BreweryPostLocation`). Only `MockPostalCodeService` exists today, which
ignores `postal_regex` and returns `postal_code_examples.front()` — it's
wired into the Boost.DI graph unconditionally (no `--mocked` branch yet,
since there's no real implementation to switch to). A real implementation
still needs a **xeger**-style generator — turning a `postal_regex` pattern
into a random matching string — instead of always replaying a fixed
example; see [ROADMAP.md §9](./ROADMAP.md#9-postal-code-generation).
Street-address generation (`Address::address_line1`) has no fixture data or
service yet and remains future work.
- `LlamaGenerator` — formats prompts for Gemma 4, validates JSON output for
both `GenerateBrewery` and `GenerateUser`, retries malformed responses up
to three times with corrective feedback in the retry prompt. The token
@@ -279,14 +294,14 @@ valid output on the first pass; the retry path is kept for resilience.
`MockGenerator` uses stable hashes for repeatable output in demos and Storybook
runs.
`CuratedJsonDataService` memoizes each of `LoadLocations()`, `LoadPersonas()`,
`CuratedJsonDataService` memoizes each of `LoadCities()`, `LoadPersonas()`,
`LoadForenamesByCountry()`, and `LoadSurnamesByCountry()` independently the
first time each is called, since `BiergartenPipelineOrchestrator` owns a
single `ICuratedDataService` instance for the whole run — later calls return
the cached result instead of re-parsing.
`GenerateUsers()` samples a forename/surname pair per city via `SampleName()`,
keyed by the city's ISO 3166-1 code. Countries present in `locations.json`
keyed by the city's ISO 3166-1 code. Countries present in `cities.json`
but absent from either name fixture (currently `KE`, `SE`, `SG`, `TH`, `VN`,
`ZA`) are skipped, the same way a failed enrichment or generation call skips
a city — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section.
@@ -303,9 +318,9 @@ a city — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section.
## Generated Output
Each successful run stores a `BreweryRecord` pair with the source location
and a `BreweryResult` payload, and a `UserRecord` pair with the source
location and a `UserResult` payload. The same generated records are also
Each successful run stores a `BreweryRecord` (source `City`, an `Address`,
and a `BreweryResult` payload), and a `UserRecord` pair with the source
`City` and a `UserResult` payload. The same generated records are also
written to a fresh SQLite export file named with the current UTC timestamp.
| Field | Meaning |
@@ -314,6 +329,7 @@ written to a fresh SQLite export file named with the current UTC timestamp.
| `description_en` | Brewery description in English. |
| `name_local` | Brewery name in the local language. |
| `description_local` | Brewery description in the local language. |
| `postal_code` | Postal code generated for the brewery's city (see `IPostalCodeService`, above). |
| Field | Meaning |
| ----------------- | ----------------------------------------------------------------- |
@@ -339,6 +355,7 @@ code, latitude, and longitude for each entry.
| `local_languages` | Locale-aware copy selection |
| `name_en`, `description_en` | Default English display content |
| `name_local`, `description_local` | Local-language display content |
| `postal_code` | Brewery address matching, mirrors the web backend's `BreweryPostLocation.PostalCode` |
---
@@ -407,7 +424,7 @@ Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present.
`MockCuratedDataService` swaps in for `CuratedJsonDataService`, so no
fixture files need to be present on disk.
- `--model` when geographically grounded content matters for demos.
- Keep `locations.json` structured enough to support discovery and future
- Keep `cities.json` structured enough to support discovery and future
filtering.
- `personas.json`, `forenames-by-country.json`, and
`surnames-by-country.json` are curated/vendored fixture data, not
@@ -424,7 +441,7 @@ Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present.
| -------------------------------------- | -------------------------------------------------- |
| `tooling/pipeline/includes/` | Public headers and shared models. |
| `tooling/pipeline/src/` | Implementation files. |
| `tooling/pipeline/locations.json` | Curated city input copied into the build tree. |
| `tooling/pipeline/cities.json` | Curated city input copied into the build tree. |
| `tooling/pipeline/personas.json` | Curated user persona archetypes copied into the build tree. |
| `tooling/pipeline/forenames-by-country.json` | Vendored (CC0) forename data by ISO 3166-1 country code. |
| `tooling/pipeline/surnames-by-country.json` | Vendored (CC0) surname data by ISO 3166-1 country code. |
@@ -446,6 +463,10 @@ Paths below are relative to `tooling/pipeline/`.
`ICuratedDataService`, and `MockCuratedDataService`, the in-memory
`ICuratedDataService` used in `--mocked` runs.
- `src/services/enrichment/wikipedia/` — enrichment service and cache.
- `includes/services/postal_code/` — `IPostalCodeService` and
`MockPostalCodeService` (header-only), consumed by `GenerateBreweries()`.
The real xeger-based implementation and its `--mocked`-aware DI wiring are
still to come.
- `src/services/sqlite/` — SQLite export implementation.
- `src/data_generation/llama/` — local inference, prompt loading, output
validation.

85
docs/pipeline/ROADMAP.md
View File

@@ -54,7 +54,9 @@ architecture needs.
(present per-forename in the source data) is never discarded the way a
pre-flattened `{first_name, last_name}` list would lose it; see §2.
- [ ] Extend `GeneratedBrewery` with `brewery_id`, `context_completeness`,
`metadata` (currently just `{location, brewery}`).
`metadata` (currently `{location, address, brewery}` — see the
`Location`→`City` rename and `Address` struct added in the
`postal_regex`/§9 entries below).
- [ ] Add `GeneratedBeer`, `GeneratedCheckin`, `GeneratedRating`,
`GeneratedFollow` aggregate structs.
- [x] Add `UserRecord` (`location`, `user : UserResult`, `email`,
@@ -70,6 +72,30 @@ architecture needs.
holding `generated_users_` in memory. Still missing vs. the planned
`GeneratedUser`: `user_id`, `password`, and `metadata`.
- [x] Add `UserPersona` (`name`, `description`, `style_affinities`).
- [x] Add `postal_regex` (`std::vector<std::string>`) and
`postal_code_examples` (`std::vector<std::string>`) to `Location`
(now `City` — see below), sourced from each `cities.json` entry's
`postal_code.city_regex` / `postal_code.examples`. Not part of the
originally planned `Location` shape in `diagrams/planned/class.puml` —
added ahead of postal code generation work; see §9.
- [x] Renamed `Location` → `City` throughout the pipeline (struct, all
parameter/return types, `LocationsList`→`CityList`,
`LoadLocations()`→`LoadCities()`) so the domain vocabulary matches the
web backend's `City`/`CityId`. Field/variable names that merely hold a
`City` instance (e.g. `EnrichedCity::location`, `BreweryRecord::location`)
were left as-is — `City` itself has a `city` field (the city name
string), so renaming the container field too would read as
`enriched_city.city.city`. The SQLite export layer's own vocabulary
(`locations` table, `location_id` column, `location_cache_`, etc.) was
renamed to `cities`/`city_id` to match, since it's this pipeline's own
throwaway export, not the backend's SQL Server schema (see §5).
- [x] Added a new `Address` struct (`postal_code` only, for now) and a
`BreweryRecord::address` member, mirroring the web backend's
`BreweryPostLocation` (`AddressLine1`, `AddressLine2`, `PostalCode`,
`CityId`). Only `postal_code` is populated — `address_line1`/
`address_line2` generation has no fixture data or service yet. Users
do not get an `Address`: the backend's `UserAccount` has no address
fields, only `BreweryPostLocation` does.
## 2. Data Preloading
@@ -78,7 +104,7 @@ architecture needs.
- [x] Extract an interface; have `CuratedJsonDataService` implement it.
Landed as `ICuratedDataService`
(`services/curated_data/curated_data_service.h`), not `DataPreloader`
— `LoadLocations()`, `LoadPersonas()`, `LoadForenamesByCountry()`, and
— `LoadCities()`, `LoadPersonas()`, `LoadForenamesByCountry()`, and
`LoadSurnamesByCountry()` are all virtual methods returning `const&`
and take no arguments. `CuratedJsonDataService`
(`services/curated_data/curated_json_data_service.h`, originally
@@ -101,7 +127,7 @@ architecture needs.
and is already copied into the Docker image
(`runpod/Dockerfile`), but no loader reads it yet, and the native
CMake build doesn't copy it into `build/` at all — `CMakeLists.txt`'s
"Runtime Assets" step only copies `locations.json` and `prompts/`.
"Runtime Assets" step only copies `cities.json` and `prompts/`.
- [x] Add `LoadPersonas()` and author `personas.json` (doesn't exist yet).
- [x] Add name-by-country loading, parsing
`tooling/pipeline/forenames-by-country.json` and
@@ -116,10 +142,16 @@ architecture needs.
`sigpwned/popular-names-by-country-dataset` (CC0) — see
ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for
provenance. Deliberately not pre-paired or filtered down to just the
countries in `locations.json`: keeping the source shape (including
countries in `cities.json`: keeping the source shape (including
per-forename gender) intact means the loader can support more
countries later, or gender-aware persona/bio generation, without
re-sourcing data.
- [x] Parse `postal_code.city_regex` and `postal_code.examples` out of each
`cities.json` entry's nested `postal_code` object in
`CuratedJsonDataService::LoadCities()`, into `City::postal_regex`
/ `City::postal_code_examples` (see §1). `MockCuratedDataService`'s
4 fixed locations carry matching values pulled from the corresponding
`cities.json` entries.
## 3. Policy / Strategy Layer
@@ -174,17 +206,20 @@ Entirely new — no `includes/policy/` (or equivalent) directory exists today.
`kCreateUsersTableSql` / `kInsertUserSql` in
`includes/services/database/sqlite_statement_helpers.h`) are
implemented — `GenerateUsers()` exports every successful user
alongside its resolved `location_id`.
alongside its resolved `city_id`.
- [x] `ProcessRecord(const BreweryRecord&)` now also binds
`BreweryRecord::address.postal_code` into a `postal_code` column on
`breweries` (see §9).
- [ ] Extend `IExportService` with `ProcessBeer`, `ProcessCheckin`,
`ProcessRating`, `ProcessFollow` (today `ProcessRecord(const
BreweryRecord&)` and `ProcessRecord(const UserRecord&)` exist).
- [ ] Add `beers`, `checkins`, `ratings`, `follows` tables to
`SqliteExportService::InitializeSchema()` (`locations`, `breweries`,
and `users` already exist) — see `kCreateLocationsTableSql` /
`SqliteExportService::InitializeSchema()` (`cities`, `breweries`,
and `users` already exist) — see `kCreateCitiesTableSql` /
`kCreateBreweriesTableSql` / `kCreateUsersTableSql` in
`includes/services/database/sqlite_statement_helpers.h` for the
existing pattern to follow.
- [ ] Add a `brewery_cache_` alongside the existing `location_cache_`, and
- [ ] Add a `brewery_cache_` alongside the existing `city_cache_`, and
move from one open transaction for the whole run to per-phase batched
commits (`BEGIN` / `COMMIT & BEGIN` on a batch-size threshold), as
shown in the planned activity diagram.
@@ -233,6 +268,36 @@ Entirely new — no `includes/policy/` (or equivalent) directory exists today.
Assets" `configure_file` step in `CMakeLists.txt` so native builds and
the Docker image agree on what's available at runtime.
## 9. Postal Code Generation
`tooling/pipeline/includes/services/postal_code/`. Not part of the original
`diagrams/planned/class.puml` sketch — added once `Location` (now `City`)
gained `postal_regex`/`postal_code_examples` (§1, §2).
- [x] Add the `IPostalCodeService` interface —
`GeneratePostalCode(location : const City&) : std::string`.
- [x] Add `MockPostalCodeService`: ignores `location.postal_regex` entirely
and returns `location.postal_code_examples.front()`.
- [ ] **Implement a real, xeger-based `IPostalCodeService`.** A xeger
generator turns a regex pattern (here, one of
`City::postal_regex`) into a random string that matches it — the
inverse of matching a string against a regex. Without it, every
generated postal code is either a hardcoded mock value or absent
entirely; a real implementation is required before postal codes can
vary per generated brewery instead of always reusing the same
curated example.
- [x] Wire `IPostalCodeService` into `main.cc`'s Boost.DI injector — bound
unconditionally to `MockPostalCodeService` (no `--mocked` branch yet,
since a real implementation doesn't exist to switch to; add one once
the xeger-based generator above lands, mirroring `IEnrichmentService`)
— and into `BiergartenPipelineOrchestrator::GenerateBreweries()`,
which calls `GeneratePostalCode(location)` per brewery and stores the
result on `BreweryRecord::address` (a new `Address` struct — see §1),
exported to the `breweries.postal_code` SQLite column (§5).
Postal-code generation is per-brewery, not per-city or per-user: users
don't have an address concept in the web backend, only breweries do
(`BreweryPostLocation`), so `UserRecord` was deliberately not extended.
---
## Suggested build order
@@ -250,3 +315,7 @@ populated. Phase 1b may begin.", etc.) imply a dependency order. Roughly:
planned diagrams.
5. Enrichment cache pre-warming and the city/region query decision (§7) —
useful at any point, but most valuable once phases run concurrently.
6. Postal code generation (§9) is independent of the phases above — the
xeger-based generator just needs `City::postal_regex`, already in
place. Wiring the mock into brewery output has already landed (§9); the
real generator can be swapped in once built.

22
docs/pipeline/diagrams/current/activity.puml
View File

@@ -69,14 +69,14 @@ end note
:QueryCitiesWithCountries();
|#E2EBDC|JsonLoader|
:JsonLoader::LoadLocations("locations.json");
:JsonLoader::LoadCities("cities.json");
:std::ranges::sample(all_locations, --location-count);
note right
--location-count defaults to 10.
end note
|#EAF0E8|BiergartenPipelineOrchestrator|
while (For each sampled Location?) is (Remaining cities)
while (For each sampled City?) is (Remaining cities)
|#DCE8D8|IEnrichmentService|
:GetLocationContext(loc);
note right
@@ -89,7 +89,7 @@ while (For each sampled Location?) is (Remaining cities)
if (Lookup failed?) then (yes)
:Log warning, skip city;
else (no)
:Store EnrichedCity{Location, region_context};
:Store EnrichedCity{City, region_context};
endif
endwhile (Done)
@@ -146,12 +146,12 @@ while (For each EnrichedCity?) is (Remaining cities)
|#E0EAE0|SqliteExportService|
:ProcessRecord(UserRecord);
if (Location in cache?) then (yes)
:Reuse location_id;
if (City in cache?) then (yes)
:Reuse city_id;
else (no)
:Insert Location & Cache ID;
:Insert City & Cache ID;
endif
:Insert User (FK: location_id);
:Insert User (FK: city_id);
if (Exception caught during insert?) then (yes)
|#EAF0E8|BiergartenPipelineOrchestrator|
@@ -197,12 +197,12 @@ while (For each EnrichedCity?) is (Remaining cities)
if (Generation successful?) then (yes)
|#E0EAE0|SqliteExportService|
:ProcessRecord(GeneratedBrewery);
if (Location in cache?) then (yes)
:Reuse location_id;
if (City in cache?) then (yes)
:Reuse city_id;
else (no)
:Insert Location & Cache ID;
:Insert City & Cache ID;
endif
:Insert Brewery (FK: location_id);
:Insert Brewery (FK: city_id);
if (Exception caught during insert?) then (yes)
|#EAF0E8|BiergartenPipelineOrchestrator|

55
docs/pipeline/diagrams/current/class.puml
View File

@@ -31,11 +31,12 @@ class BiergartenPipelineOrchestrator {
- generator_ : std::unique_ptr<DataGenerator>
- exporter_ : std::unique_ptr<IExportService>
- curated_data_service_ : std::unique_ptr<ICuratedDataService>
- postal_code_service_ : std::unique_ptr<IPostalCodeService>
- application_options_ : ApplicationOptions
- generated_breweries_ : std::vector<BreweryRecord>
- generated_users_ : std::vector<UserRecord>
+ Run() : bool
- QueryCitiesWithCountries() : std::vector<Location>
- QueryCitiesWithCountries() : std::vector<City>
- GenerateBreweries(cities : std::span<const EnrichedCity>) : void
- GenerateUsers(cities : std::span<const EnrichedCity>) : void
- LogResults() : void
@@ -90,17 +91,17 @@ class LogDispatcher {
}
interface IEnrichmentService <<interface>> {
+ GetLocationContext(loc : const Location&) : std::string
+ GetLocationContext(loc : const City&) : std::string
}
class MockEnrichmentService {
+ GetLocationContext(loc : const Location&) : std::string
+ GetLocationContext(loc : const City&) : std::string
}
class WikipediaEnrichmentService {
- client_ : std::unique_ptr<WebClient>
- extract_cache_ : std::unordered_map<std::string, std::string>
+ GetLocationContext(loc : const Location&) : std::string
+ GetLocationContext(loc : const City&) : std::string
- FetchExtract(query : std::string_view) : std::string
}
@@ -115,15 +116,15 @@ class HttpWebClient {
}
interface DataGenerator <<interface>> {
+ GenerateBrewery(location : const Location&, region_context : const std::string&) : BreweryResult
+ GenerateBrewery(location : const City&, region_context : const std::string&) : BreweryResult
+ GenerateUser(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult
}
class MockGenerator {
+ GenerateBrewery(location : const Location&, region_context : const std::string&) : BreweryResult
+ GenerateBrewery(location : const City&, region_context : const std::string&) : BreweryResult
+ GenerateUser(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult
- DeterministicHash(location : const Location&) : size_t
- DeterministicHash(location : const Location&, persona : const UserPersona&, name : const Name&) : size_t
- DeterministicHash(location : const City&) : size_t
- DeterministicHash(location : const City&, persona : const UserPersona&, name : const Name&) : size_t
}
class LlamaGenerator {
@@ -158,7 +159,7 @@ class PromptDirectory {
}
interface ICuratedDataService <<interface>> {
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
+ LoadCities(filepath : const std::filesystem::path&) : const std::vector<City>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
@@ -166,7 +167,7 @@ interface ICuratedDataService <<interface>> {
class JsonLoader {
- cache_ : cache
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
+ LoadCities(filepath : const std::filesystem::path&) : const std::vector<City>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
@@ -180,11 +181,11 @@ note right of JsonLoader
end note
class MockCuratedDataService {
- locations_ : std::vector<Location>
- locations_ : std::vector<City>
- personas_ : std::vector<UserPersona>
- forenames_by_country_ : std::unordered_map<std::string, forename_list>
- surnames_by_country_ : std::unordered_map<std::string, surname_list>
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
+ LoadCities(filepath : const std::filesystem::path&) : const std::vector<City>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
@@ -196,6 +197,27 @@ note right of MockCuratedDataService
filepath arguments are ignored.
end note
interface IPostalCodeService <<interface>> {
+ GeneratePostalCode(location : const City&) : std::string
}
class MockPostalCodeService {
+ GeneratePostalCode(location : const City&) : std::string
}
note right of MockPostalCodeService
Ignores location.postal_regex and returns
location.postal_code_examples.front(). Bound
unconditionally in main.cc's DI graph (no
--mocked branch yet -- no real implementation
exists to switch to) and called from
GenerateBreweries() per brewery, stored on
BreweryRecord::address. A real implementation
needs a xeger-style generator to turn
postal_regex into a synthesized matching
string. See ROADMAP.md §9.
end note
interface IExportService <<interface>> {
+ Initialize() : void
+ ProcessRecord(brewery : const BreweryRecord&) : uint64_t
@@ -208,17 +230,17 @@ class SqliteExportService {
- run_timestamp_utc_ : std::string
- database_path_ : std::filesystem::path
- db_handle_ : SqliteDatabaseHandle
- insert_location_stmt_ : SqliteStatementHandle
- insert_city_stmt_ : SqliteStatementHandle
- insert_brewery_stmt_ : SqliteStatementHandle
- insert_user_stmt_ : SqliteStatementHandle
- transaction_open_ : bool
- location_cache_ : std::unordered_map<std::string, sqlite3_int64>
- city_cache_ : std::unordered_map<std::string, sqlite3_int64>
+ Initialize() : void
+ ProcessRecord(brewery : const BreweryRecord&) : uint64_t
+ ProcessRecord(user : const UserRecord&) : uint64_t
+ Finalize() : void
- InitializeSchema() : void
- ResolveLocationId(location : const Location&) : sqlite3_int64
- ResolveCityId(location : const City&) : sqlite3_int64
}
interface IDateTimeProvider <<interface>> {
@@ -235,6 +257,7 @@ BiergartenPipelineOrchestrator *-- IEnrichmentService : owns
BiergartenPipelineOrchestrator *-- DataGenerator : owns
BiergartenPipelineOrchestrator *-- IExportService : owns
BiergartenPipelineOrchestrator *-- ICuratedDataService : owns
BiergartenPipelineOrchestrator *-- IPostalCodeService : owns
LogEntry *-- LogLevel
LogEntry *-- PipelinePhase
@@ -262,6 +285,8 @@ IPromptDirectory <|.. PromptDirectory : implements
ICuratedDataService <|.. JsonLoader : implements
ICuratedDataService <|.. MockCuratedDataService : implements
IPostalCodeService <|.. MockPostalCodeService : implements
IExportService <|.. SqliteExportService : implements
SqliteExportService *-- IDateTimeProvider : owns
IDateTimeProvider <|.. SystemDateTimeProvider : implements

2
docs/pipeline/diagrams/current/output/activity.svg
View File

File diff suppressed because one or more lines are too long
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 86 KiB

After

Width:  |  Height:  |  Size: 86 KiB

2
docs/pipeline/diagrams/current/output/class.svg
View File

File diff suppressed because one or more lines are too long
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 87 KiB

After

Width:  |  Height:  |  Size: 94 KiB

10
docs/pipeline/diagrams/planned/activity.puml
View File

@@ -39,7 +39,7 @@ fork
:JsonLoader::LoadBeerStyles("beer-styles.json");
:EnrichmentService::PreWarmBeerStyleCache(beer_styles);
fork again
:JsonLoader::LoadLocations("locations.json");
:JsonLoader::LoadCities("cities.json");
:EnrichmentService::PreWarmLocationCache(sampled_locations);
end fork
fork
@@ -75,7 +75,7 @@ fork
fork again
|LLM Worker|
while (loc_ch has items?) is (yes)
:Receive Location;
:Receive City;
:GetLocationContextFromCache(location);
note right
@@ -146,13 +146,13 @@ end fork
fork
|Orchestrator|
:Loop: Sample User from user_pool_
and pair with Location;
:Send BreweryTask(Location, User) -> loc_ch;
and pair with City;
:Send BreweryTask(City, User) -> loc_ch;
:Close loc_ch;
fork again
|LLM Worker|
while (loc_ch has items?) is (yes)
:Receive BreweryTask(Location, User);
:Receive BreweryTask(City, User);
:GetLocationContextFromCache(task.location);
note right

106
docs/pipeline/diagrams/planned/class.puml
View File

@@ -12,17 +12,40 @@ title Biergarten Data Pipeline — Class Diagram
package "Domain: Models" {
class Location {
class City {
+ city : std::string
+ state_province : std::string
+ iso3166_2 : std::string
+ country : std::string
+ iso3166_1 : std::string
+ postal_regex : std::vector<std::string>
+ postal_code_examples : std::vector<std::string>
+ local_languages : std::vector<std::string>
+ latitude : double
+ longitude : double
}
note right of City
postal_regex / postal_code_examples are
sourced from each cities.json entry's
postal_code.city_regex / postal_code.examples
(implemented today, ahead of the rest of this
diagram -- see ROADMAP.md §1/§2). Renamed from
Location to City to match the web backend's
City/CityId vocabulary -- see ROADMAP.md §1.
end note
class Address {
+ postal_code : std::string
}
note right of Address
Mirrors the web backend's BreweryPostLocation.
Only postal_code is populated today --
address_line1/address_line2 have no fixture
data or generator yet. See ROADMAP.md §9.
end note
class LocationContext {
+ text : std::string
+ completeness : Completeness
@@ -36,7 +59,7 @@ package "Domain: Models" {
}
class EnrichedCity {
+ location : Location
+ location : City
+ context : LocationContext
}
@@ -130,7 +153,8 @@ package "Domain: Models" {
class GeneratedBrewery {
+ brewery_id : uint64_t
+ location : Location
+ location : City
+ address : Address
+ brewery : BreweryResult
+ context_completeness : LocationContext::Completeness
+ metadata : GenerationMetadata
@@ -139,7 +163,7 @@ package "Domain: Models" {
class GeneratedBeer {
+ beer_id : uint64_t
+ brewery_id : uint64_t
+ location : Location
+ location : City
+ style : BeerStyle
+ beer : BeerResult
+ metadata : GenerationMetadata
@@ -147,7 +171,7 @@ package "Domain: Models" {
class GeneratedUser {
+ user_id : uint64_t
+ location : Location
+ location : City
+ user : UserResult
+ email : std::string
+ date_of_birth : std::string
@@ -227,27 +251,27 @@ package "Domain: Application Configuration" {
package "Domain: Policy" {
interface ContextStrategy <<interface>> {
+ QueriesFor(loc : const Location&) : std::vector<std::string>
+ QueriesFor(loc : const City&) : std::vector<std::string>
+ MaxContextChars() : size_t
}
class BreweryContextStrategy {
+ QueriesFor(loc : const Location&) : std::vector<std::string>
+ QueriesFor(loc : const City&) : std::vector<std::string>
+ MaxContextChars() : size_t
}
class BeerContextStrategy {
+ QueriesFor(loc : const Location&) : std::vector<std::string>
+ QueriesFor(loc : const City&) : std::vector<std::string>
+ MaxContextChars() : size_t
}
interface SamplingStrategy <<interface>> {
+ Sample(locations : const std::vector<Location>&) : std::vector<Location>
+ Sample(locations : const std::vector<City>&) : std::vector<City>
}
class UniformSamplingStrategy {
- sample_size_ : size_t
+ Sample(locations : const std::vector<Location>&) : std::vector<Location>
+ Sample(locations : const std::vector<City>&) : std::vector<City>
}
interface BeerSelectionStrategy <<interface>> {
@@ -369,7 +393,7 @@ package "Infrastructure: Pipeline Channel" {
package "Infrastructure: Data Preloading" {
interface ICuratedDataService <<interface>> {
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
+ LoadCities(filepath : const std::filesystem::path&) : const std::vector<City>&
+ LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
@@ -377,7 +401,7 @@ package "Infrastructure: Data Preloading" {
}
class JsonLoader {
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
+ LoadCities(filepath : const std::filesystem::path&) : const std::vector<City>&
+ LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
@@ -391,7 +415,7 @@ package "Infrastructure: Data Preloading" {
end note
class MockCuratedDataService {
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
+ LoadCities(filepath : const std::filesystem::path&) : const std::vector<City>&
+ LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
@@ -410,13 +434,13 @@ package "Infrastructure: Data Preloading" {
package "Infrastructure: Enrichment" {
interface EnrichmentService <<interface>> {
+ GetLocationContext(loc : const Location&,\n strategy : const ContextStrategy&) : LocationContext
+ GetLocationContext(loc : const City&,\n strategy : const ContextStrategy&) : LocationContext
}
class WikipediaService {
- client_ : std::unique_ptr<WebClient>
- extract_cache_ : std::unordered_map<std::string, std::string>
+ GetLocationContext(loc : const Location&,\n strategy : const ContextStrategy&) : LocationContext
+ GetLocationContext(loc : const City&,\n strategy : const ContextStrategy&) : LocationContext
- FetchExtract(query : std::string_view) : std::string
}
@@ -432,6 +456,39 @@ package "Infrastructure: Enrichment" {
}
package "Infrastructure: Postal Code Generation" {
interface IPostalCodeService <<interface>> {
+ GeneratePostalCode(location : const City&) : std::string
}
class MockPostalCodeService {
+ GeneratePostalCode(location : const City&) : std::string
}
class XegerPostalCodeService {
+ GeneratePostalCode(location : const City&) : std::string
}
note right of MockPostalCodeService
Implemented today: ignores postal_regex and
returns postal_code_examples.front(). Not yet
bound in main.cc's DI graph or used by
BiergartenPipelineOrchestrator.
end note
note right of XegerPostalCodeService
Not implemented yet. Picks one of
location.postal_regex and generates a random
string matching it (a "xeger" -- the inverse of
regex matching) instead of always replaying a
fixed example. See ROADMAP.md §9.
end note
IPostalCodeService <|.. MockPostalCodeService
IPostalCodeService <|.. XegerPostalCodeService
}
package "Infrastructure: Prompting" {
interface IPromptDirectory <<interface>> {
@@ -451,8 +508,8 @@ package "Infrastructure: Prompting" {
package "Infrastructure: Data Generation" {
interface DataGenerator <<interface>> {
+ GenerateBrewery(location : const Location&,\n context : const LocationContext&) : BreweryResult
+ GenerateBeer(brewery_id : uint64_t,\n location : const Location&,\n context : const LocationContext&,\n style : const BeerStyle&) : BeerResult
+ GenerateBrewery(location : const City&,\n context : const LocationContext&) : BreweryResult
+ GenerateBeer(brewery_id : uint64_t,\n location : const City&,\n context : const LocationContext&,\n style : const BeerStyle&) : BeerResult
+ GenerateUser(city : const EnrichedCity&,\n persona : const UserPersona&,\n name : const Name&) : UserResult
+ GenerateCheckin(user : const GeneratedUser&,\n brewery : const GeneratedBrewery&,\n timestamp : const std::string&) : CheckinResult
+ GenerateRating(user : const GeneratedUser&,\n beer : const GeneratedBeer&,\n checkin_id : uint64_t) : RatingResult
@@ -464,7 +521,7 @@ package "Infrastructure: Data Generation" {
+ GenerateUser(...) : UserResult
+ GenerateCheckin(...) : CheckinResult
+ GenerateRating(...) : RatingResult
- DeterministicHash(location : const Location&) : size_t
- DeterministicHash(location : const City&) : size_t
}
class LlamaGenerator {
@@ -566,8 +623,8 @@ class BiergartenPipelineOrchestrator {
- follow_pool_ : std::vector<GeneratedFollow>
--
+ Run() : bool
- RunUserPhase(locations : const std::vector<Location>&) : void
- RunBreweryAndBeerPhase(locations : const std::vector<Location>&) : void
- RunUserPhase(locations : const std::vector<City>&) : void
- RunBreweryAndBeerPhase(locations : const std::vector<City>&) : void
- RunCheckinPhase() : void
- RunRatingPhase() : void
- RunFollowPhase() : void
@@ -625,16 +682,17 @@ PipelineLogger o-- BoundedChannel : logs to
LogWorker o-- BoundedChannel : drains from
' --- Domain Containment ---
EnrichedCity *-- Location
EnrichedCity *-- City
EnrichedCity *-- LocationContext
GeneratedBrewery *-- Location
GeneratedBrewery *-- City
GeneratedBrewery *-- Address
GeneratedBrewery *-- BreweryResult
GeneratedBrewery *-- GenerationMetadata
GeneratedBeer *-- Location
GeneratedBeer *-- City
GeneratedBeer *-- BeerStyle
GeneratedBeer *-- BeerResult
GeneratedBeer *-- GenerationMetadata
GeneratedUser *-- Location
GeneratedUser *-- City
GeneratedUser *-- UserResult
GeneratedUser *-- GenerationMetadata
GeneratedCheckin *-- CheckinResult

2
docs/pipeline/diagrams/planned/output/biergarten_activity.svg
View File

File diff suppressed because one or more lines are too long
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 142 KiB

After

Width:  |  Height:  |  Size: 142 KiB

2
docs/pipeline/diagrams/planned/output/class.svg
View File

File diff suppressed because one or more lines are too long
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 240 KiB

After

Width:  |  Height:  |  Size: 257 KiB