diff --git a/docs/pipeline/ETHICS-AND-KNOWN-ISSUES.md b/docs/pipeline/ETHICS-AND-KNOWN-ISSUES.md index febf030..e400f29 100644 --- a/docs/pipeline/ETHICS-AND-KNOWN-ISSUES.md +++ b/docs/pipeline/ETHICS-AND-KNOWN-ISSUES.md @@ -109,10 +109,10 @@ gender from the source data is preserved through to the sampled `Name` persona/bio generation later. The full multinational dataset is kept as-is (106 countries for forenames, -75 for surnames) rather than trimmed to `locations.json`'s current country +75 for surnames) rather than trimmed to `cities.json`'s current country list, so it doesn't need re-sourcing if more countries are added later. `SampleName()` (a free helper in `generate_users.cc`) returns no result for -a country present in neither file; of the countries in `locations.json`, +a country present in neither file; of the countries in `cities.json`, that's currently `KE`, `SE`, `SG`, `TH`, `VN`, and `ZA` — `GenerateUsers` skips cities in those countries the same way brewery generation skips cities whose enrichment lookup fails. @@ -281,7 +281,7 @@ Output sample: a location's code is unlisted, skip `description_local` generation and fall back to English. - **Upstream sanitization:** strip known low-resource language codes from the - `locations.json` payload before generation. + `cities.json` payload before generation. - **Downstream flagging:** add a `description_local_confidence` column to the SQLite schema so downstream applications can filter or flag potentially hallucinated text by language tier. diff --git a/docs/pipeline/README.md b/docs/pipeline/README.md index eed2936..d4bd162 100644 --- a/docs/pipeline/README.md +++ b/docs/pipeline/README.md @@ -86,7 +86,7 @@ curl -L \ ### Run -Run from `build/` so the copied `locations.json` and `prompts/` are available. +Run from `build/` so the copied `cities.json` and `prompts/` are available. Each run writes a fresh dated SQLite file such as `biergarten_seed_2026-04-19T15-30-45.123456Z.sqlite` into the working directory. @@ -109,7 +109,7 @@ Each run writes a fresh dated SQLite file such as | `--prompt-dir` | Directory containing prompt files (e.g. `BREWERY_GENERATION.md`). Required unless `--mocked` is set. | | `--output, -o` | Directory for generated SQLite artifacts. Default: `output`. | | `--log-path` | Path for application logs. Default: `pipeline.log`. | -| `--location-count` | Number of cities to sample from `locations.json` per run. Default: `10`. | +| `--location-count` | Number of cities to sample from `cities.json` per run. Default: `10`. | | `--temperature` | Sampling temperature. Default: `1.0`. | | `--top-p` | Nucleus sampling. Default: `0.95`. | | `--top-k` | Top-k sampling. Default: `64`. | @@ -222,12 +222,12 @@ environment variables listed above to match your run. | Stage | Implementation | | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | -| Load | `ICuratedDataService` (`CuratedJsonDataService`) reads `locations.json`, `personas.json`, `forenames-by-country.json`, and `surnames-by-country.json` (paths supplied via a `CuratedDataFilePaths` DTO at construction) into typed records, caching each after its first load. `--mocked` runs use `MockCuratedDataService`'s fixed in-memory dataset instead. | +| Load | `ICuratedDataService` (`CuratedJsonDataService`) reads `cities.json`, `personas.json`, `forenames-by-country.json`, and `surnames-by-country.json` (paths supplied via a `CuratedDataFilePaths` DTO at construction) into typed records, caching each after its first load. `--mocked` runs use `MockCuratedDataService`'s fixed in-memory dataset instead. | | Sample | `BiergartenPipelineOrchestrator::QueryCitiesWithCountries()` samples `--location-count` locations per run (default `10`). | | Enrich | `WikipediaEnrichmentService` fetches brewing and beer-related context. Keeps going when a lookup fails. `--mocked` runs use `MockEnrichmentService` instead and skip Wikipedia entirely. | | Generate Users | `GenerateUsers()` samples a persona and a forename/surname pair per enriched city (skipping countries with no name data), then `MockGenerator` or `LlamaGenerator` produces a username, bio, and activity weight around the sampled name. | | Generate Breweries | `MockGenerator` or `LlamaGenerator` produces brewery names and descriptions in English and the local language. | -| Store | `SqliteExportService` writes each successful user and brewery into a fresh dated `.sqlite` database with normalized `locations`, `users`, and `breweries` tables. | +| Store | `SqliteExportService` writes each successful user and brewery into a fresh dated `.sqlite` database with normalized `cities`, `users`, and `breweries` tables. | | Log | `spdlog` writes results and warnings to the console. | If name sampling, enrichment, or generation fails for a city, that city is @@ -241,12 +241,27 @@ skipped and the pipeline continues. `GenerateUsers()` runs before `CuratedDataFilePaths` DTO (locations/personas/forenames/surnames paths) in its constructor, then parses and validates curated location, persona, and forename/surname JSON, memoizing each result after its first load on a - given instance. `MockCuratedDataService` is the in-memory substitute (4 - fixed locations, 3 personas, and name data for `US`/`DE`/`FR`/`BE`) used in - `--mocked` runs. + given instance. Each `cities.json` entry's `postal_code.city_regex` and + `postal_code.examples` are parsed into `City::postal_regex` and + `City::postal_code_examples`. `MockCuratedDataService` is the in-memory + substitute (4 fixed locations, 3 personas, and name data for + `US`/`DE`/`FR`/`BE`) used in `--mocked` runs, and carries matching + `postal_regex`/`postal_code_examples` values for its 4 locations. - `WikipediaEnrichmentService` — queries Wikipedia extracts, caches results, returns empty context on failure. `MockEnrichmentService` is the no-op substitute used in `--mocked` runs. +- `IPostalCodeService` — generates a postal code for a `City`, consumed by + `GenerateBreweries()` and stored on `BreweryRecord::address` (an `Address` + struct, currently just `postal_code`, mirroring the web backend's + `BreweryPostLocation`). Only `MockPostalCodeService` exists today, which + ignores `postal_regex` and returns `postal_code_examples.front()` — it's + wired into the Boost.DI graph unconditionally (no `--mocked` branch yet, + since there's no real implementation to switch to). A real implementation + still needs a **xeger**-style generator — turning a `postal_regex` pattern + into a random matching string — instead of always replaying a fixed + example; see [ROADMAP.md §9](./ROADMAP.md#9-postal-code-generation). + Street-address generation (`Address::address_line1`) has no fixture data or + service yet and remains future work. - `LlamaGenerator` — formats prompts for Gemma 4, validates JSON output for both `GenerateBrewery` and `GenerateUser`, retries malformed responses up to three times with corrective feedback in the retry prompt. The token @@ -279,14 +294,14 @@ valid output on the first pass; the retry path is kept for resilience. `MockGenerator` uses stable hashes for repeatable output in demos and Storybook runs. -`CuratedJsonDataService` memoizes each of `LoadLocations()`, `LoadPersonas()`, +`CuratedJsonDataService` memoizes each of `LoadCities()`, `LoadPersonas()`, `LoadForenamesByCountry()`, and `LoadSurnamesByCountry()` independently the first time each is called, since `BiergartenPipelineOrchestrator` owns a single `ICuratedDataService` instance for the whole run — later calls return the cached result instead of re-parsing. `GenerateUsers()` samples a forename/surname pair per city via `SampleName()`, -keyed by the city's ISO 3166-1 code. Countries present in `locations.json` +keyed by the city's ISO 3166-1 code. Countries present in `cities.json` but absent from either name fixture (currently `KE`, `SE`, `SG`, `TH`, `VN`, `ZA`) are skipped, the same way a failed enrichment or generation call skips a city — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section. @@ -303,9 +318,9 @@ a city — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section. ## Generated Output -Each successful run stores a `BreweryRecord` pair with the source location -and a `BreweryResult` payload, and a `UserRecord` pair with the source -location and a `UserResult` payload. The same generated records are also +Each successful run stores a `BreweryRecord` (source `City`, an `Address`, +and a `BreweryResult` payload), and a `UserRecord` pair with the source +`City` and a `UserResult` payload. The same generated records are also written to a fresh SQLite export file named with the current UTC timestamp. | Field | Meaning | @@ -314,6 +329,7 @@ written to a fresh SQLite export file named with the current UTC timestamp. | `description_en` | Brewery description in English. | | `name_local` | Brewery name in the local language. | | `description_local` | Brewery description in the local language. | +| `postal_code` | Postal code generated for the brewery's city (see `IPostalCodeService`, above). | | Field | Meaning | | ----------------- | ----------------------------------------------------------------- | @@ -339,6 +355,7 @@ code, latitude, and longitude for each entry. | `local_languages` | Locale-aware copy selection | | `name_en`, `description_en` | Default English display content | | `name_local`, `description_local` | Local-language display content | +| `postal_code` | Brewery address matching, mirrors the web backend's `BreweryPostLocation.PostalCode` | --- @@ -407,7 +424,7 @@ Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present. `MockCuratedDataService` swaps in for `CuratedJsonDataService`, so no fixture files need to be present on disk. - `--model` when geographically grounded content matters for demos. -- Keep `locations.json` structured enough to support discovery and future +- Keep `cities.json` structured enough to support discovery and future filtering. - `personas.json`, `forenames-by-country.json`, and `surnames-by-country.json` are curated/vendored fixture data, not @@ -424,7 +441,7 @@ Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present. | -------------------------------------- | -------------------------------------------------- | | `tooling/pipeline/includes/` | Public headers and shared models. | | `tooling/pipeline/src/` | Implementation files. | -| `tooling/pipeline/locations.json` | Curated city input copied into the build tree. | +| `tooling/pipeline/cities.json` | Curated city input copied into the build tree. | | `tooling/pipeline/personas.json` | Curated user persona archetypes copied into the build tree. | | `tooling/pipeline/forenames-by-country.json` | Vendored (CC0) forename data by ISO 3166-1 country code. | | `tooling/pipeline/surnames-by-country.json` | Vendored (CC0) surname data by ISO 3166-1 country code. | @@ -446,6 +463,10 @@ Paths below are relative to `tooling/pipeline/`. `ICuratedDataService`, and `MockCuratedDataService`, the in-memory `ICuratedDataService` used in `--mocked` runs. - `src/services/enrichment/wikipedia/` — enrichment service and cache. +- `includes/services/postal_code/` — `IPostalCodeService` and + `MockPostalCodeService` (header-only), consumed by `GenerateBreweries()`. + The real xeger-based implementation and its `--mocked`-aware DI wiring are + still to come. - `src/services/sqlite/` — SQLite export implementation. - `src/data_generation/llama/` — local inference, prompt loading, output validation. diff --git a/docs/pipeline/ROADMAP.md b/docs/pipeline/ROADMAP.md index 0f2876f..e880e6f 100644 --- a/docs/pipeline/ROADMAP.md +++ b/docs/pipeline/ROADMAP.md @@ -54,7 +54,9 @@ architecture needs. (present per-forename in the source data) is never discarded the way a pre-flattened `{first_name, last_name}` list would lose it; see §2. - [ ] Extend `GeneratedBrewery` with `brewery_id`, `context_completeness`, - `metadata` (currently just `{location, brewery}`). + `metadata` (currently `{location, address, brewery}` — see the + `Location`→`City` rename and `Address` struct added in the + `postal_regex`/§9 entries below). - [ ] Add `GeneratedBeer`, `GeneratedCheckin`, `GeneratedRating`, `GeneratedFollow` aggregate structs. - [x] Add `UserRecord` (`location`, `user : UserResult`, `email`, @@ -70,6 +72,30 @@ architecture needs. holding `generated_users_` in memory. Still missing vs. the planned `GeneratedUser`: `user_id`, `password`, and `metadata`. - [x] Add `UserPersona` (`name`, `description`, `style_affinities`). +- [x] Add `postal_regex` (`std::vector`) and + `postal_code_examples` (`std::vector`) to `Location` + (now `City` — see below), sourced from each `cities.json` entry's + `postal_code.city_regex` / `postal_code.examples`. Not part of the + originally planned `Location` shape in `diagrams/planned/class.puml` — + added ahead of postal code generation work; see §9. +- [x] Renamed `Location` → `City` throughout the pipeline (struct, all + parameter/return types, `LocationsList`→`CityList`, + `LoadLocations()`→`LoadCities()`) so the domain vocabulary matches the + web backend's `City`/`CityId`. Field/variable names that merely hold a + `City` instance (e.g. `EnrichedCity::location`, `BreweryRecord::location`) + were left as-is — `City` itself has a `city` field (the city name + string), so renaming the container field too would read as + `enriched_city.city.city`. The SQLite export layer's own vocabulary + (`locations` table, `location_id` column, `location_cache_`, etc.) was + renamed to `cities`/`city_id` to match, since it's this pipeline's own + throwaway export, not the backend's SQL Server schema (see §5). +- [x] Added a new `Address` struct (`postal_code` only, for now) and a + `BreweryRecord::address` member, mirroring the web backend's + `BreweryPostLocation` (`AddressLine1`, `AddressLine2`, `PostalCode`, + `CityId`). Only `postal_code` is populated — `address_line1`/ + `address_line2` generation has no fixture data or service yet. Users + do not get an `Address`: the backend's `UserAccount` has no address + fields, only `BreweryPostLocation` does. ## 2. Data Preloading @@ -78,7 +104,7 @@ architecture needs. - [x] Extract an interface; have `CuratedJsonDataService` implement it. Landed as `ICuratedDataService` (`services/curated_data/curated_data_service.h`), not `DataPreloader` - — `LoadLocations()`, `LoadPersonas()`, `LoadForenamesByCountry()`, and + — `LoadCities()`, `LoadPersonas()`, `LoadForenamesByCountry()`, and `LoadSurnamesByCountry()` are all virtual methods returning `const&` and take no arguments. `CuratedJsonDataService` (`services/curated_data/curated_json_data_service.h`, originally @@ -101,7 +127,7 @@ architecture needs. and is already copied into the Docker image (`runpod/Dockerfile`), but no loader reads it yet, and the native CMake build doesn't copy it into `build/` at all — `CMakeLists.txt`'s - "Runtime Assets" step only copies `locations.json` and `prompts/`. + "Runtime Assets" step only copies `cities.json` and `prompts/`. - [x] Add `LoadPersonas()` and author `personas.json` (doesn't exist yet). - [x] Add name-by-country loading, parsing `tooling/pipeline/forenames-by-country.json` and @@ -116,10 +142,16 @@ architecture needs. `sigpwned/popular-names-by-country-dataset` (CC0) — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for provenance. Deliberately not pre-paired or filtered down to just the - countries in `locations.json`: keeping the source shape (including + countries in `cities.json`: keeping the source shape (including per-forename gender) intact means the loader can support more countries later, or gender-aware persona/bio generation, without re-sourcing data. +- [x] Parse `postal_code.city_regex` and `postal_code.examples` out of each + `cities.json` entry's nested `postal_code` object in + `CuratedJsonDataService::LoadCities()`, into `City::postal_regex` + / `City::postal_code_examples` (see §1). `MockCuratedDataService`'s + 4 fixed locations carry matching values pulled from the corresponding + `cities.json` entries. ## 3. Policy / Strategy Layer @@ -174,17 +206,20 @@ Entirely new — no `includes/policy/` (or equivalent) directory exists today. `kCreateUsersTableSql` / `kInsertUserSql` in `includes/services/database/sqlite_statement_helpers.h`) are implemented — `GenerateUsers()` exports every successful user - alongside its resolved `location_id`. + alongside its resolved `city_id`. +- [x] `ProcessRecord(const BreweryRecord&)` now also binds + `BreweryRecord::address.postal_code` into a `postal_code` column on + `breweries` (see §9). - [ ] Extend `IExportService` with `ProcessBeer`, `ProcessCheckin`, `ProcessRating`, `ProcessFollow` (today `ProcessRecord(const BreweryRecord&)` and `ProcessRecord(const UserRecord&)` exist). - [ ] Add `beers`, `checkins`, `ratings`, `follows` tables to - `SqliteExportService::InitializeSchema()` (`locations`, `breweries`, - and `users` already exist) — see `kCreateLocationsTableSql` / + `SqliteExportService::InitializeSchema()` (`cities`, `breweries`, + and `users` already exist) — see `kCreateCitiesTableSql` / `kCreateBreweriesTableSql` / `kCreateUsersTableSql` in `includes/services/database/sqlite_statement_helpers.h` for the existing pattern to follow. -- [ ] Add a `brewery_cache_` alongside the existing `location_cache_`, and +- [ ] Add a `brewery_cache_` alongside the existing `city_cache_`, and move from one open transaction for the whole run to per-phase batched commits (`BEGIN` / `COMMIT & BEGIN` on a batch-size threshold), as shown in the planned activity diagram. @@ -233,6 +268,36 @@ Entirely new — no `includes/policy/` (or equivalent) directory exists today. Assets" `configure_file` step in `CMakeLists.txt` so native builds and the Docker image agree on what's available at runtime. +## 9. Postal Code Generation + +`tooling/pipeline/includes/services/postal_code/`. Not part of the original +`diagrams/planned/class.puml` sketch — added once `Location` (now `City`) +gained `postal_regex`/`postal_code_examples` (§1, §2). + +- [x] Add the `IPostalCodeService` interface — + `GeneratePostalCode(location : const City&) : std::string`. +- [x] Add `MockPostalCodeService`: ignores `location.postal_regex` entirely + and returns `location.postal_code_examples.front()`. +- [ ] **Implement a real, xeger-based `IPostalCodeService`.** A xeger + generator turns a regex pattern (here, one of + `City::postal_regex`) into a random string that matches it — the + inverse of matching a string against a regex. Without it, every + generated postal code is either a hardcoded mock value or absent + entirely; a real implementation is required before postal codes can + vary per generated brewery instead of always reusing the same + curated example. +- [x] Wire `IPostalCodeService` into `main.cc`'s Boost.DI injector — bound + unconditionally to `MockPostalCodeService` (no `--mocked` branch yet, + since a real implementation doesn't exist to switch to; add one once + the xeger-based generator above lands, mirroring `IEnrichmentService`) + — and into `BiergartenPipelineOrchestrator::GenerateBreweries()`, + which calls `GeneratePostalCode(location)` per brewery and stores the + result on `BreweryRecord::address` (a new `Address` struct — see §1), + exported to the `breweries.postal_code` SQLite column (§5). + Postal-code generation is per-brewery, not per-city or per-user: users + don't have an address concept in the web backend, only breweries do + (`BreweryPostLocation`), so `UserRecord` was deliberately not extended. + --- ## Suggested build order @@ -250,3 +315,7 @@ populated. Phase 1b may begin.", etc.) imply a dependency order. Roughly: planned diagrams. 5. Enrichment cache pre-warming and the city/region query decision (§7) — useful at any point, but most valuable once phases run concurrently. +6. Postal code generation (§9) is independent of the phases above — the + xeger-based generator just needs `City::postal_regex`, already in + place. Wiring the mock into brewery output has already landed (§9); the + real generator can be swapped in once built. diff --git a/docs/pipeline/diagrams/current/activity.puml b/docs/pipeline/diagrams/current/activity.puml index cd3e7b2..d0cf5e9 100644 --- a/docs/pipeline/diagrams/current/activity.puml +++ b/docs/pipeline/diagrams/current/activity.puml @@ -69,14 +69,14 @@ end note :QueryCitiesWithCountries(); |#E2EBDC|JsonLoader| -:JsonLoader::LoadLocations("locations.json"); +:JsonLoader::LoadCities("cities.json"); :std::ranges::sample(all_locations, --location-count); note right --location-count defaults to 10. end note |#EAF0E8|BiergartenPipelineOrchestrator| -while (For each sampled Location?) is (Remaining cities) +while (For each sampled City?) is (Remaining cities) |#DCE8D8|IEnrichmentService| :GetLocationContext(loc); note right @@ -89,7 +89,7 @@ while (For each sampled Location?) is (Remaining cities) if (Lookup failed?) then (yes) :Log warning, skip city; else (no) - :Store EnrichedCity{Location, region_context}; + :Store EnrichedCity{City, region_context}; endif endwhile (Done) @@ -146,12 +146,12 @@ while (For each EnrichedCity?) is (Remaining cities) |#E0EAE0|SqliteExportService| :ProcessRecord(UserRecord); - if (Location in cache?) then (yes) - :Reuse location_id; + if (City in cache?) then (yes) + :Reuse city_id; else (no) - :Insert Location & Cache ID; + :Insert City & Cache ID; endif - :Insert User (FK: location_id); + :Insert User (FK: city_id); if (Exception caught during insert?) then (yes) |#EAF0E8|BiergartenPipelineOrchestrator| @@ -197,12 +197,12 @@ while (For each EnrichedCity?) is (Remaining cities) if (Generation successful?) then (yes) |#E0EAE0|SqliteExportService| :ProcessRecord(GeneratedBrewery); - if (Location in cache?) then (yes) - :Reuse location_id; + if (City in cache?) then (yes) + :Reuse city_id; else (no) - :Insert Location & Cache ID; + :Insert City & Cache ID; endif - :Insert Brewery (FK: location_id); + :Insert Brewery (FK: city_id); if (Exception caught during insert?) then (yes) |#EAF0E8|BiergartenPipelineOrchestrator| diff --git a/docs/pipeline/diagrams/current/class.puml b/docs/pipeline/diagrams/current/class.puml index 4eef014..b714c39 100644 --- a/docs/pipeline/diagrams/current/class.puml +++ b/docs/pipeline/diagrams/current/class.puml @@ -31,11 +31,12 @@ class BiergartenPipelineOrchestrator { - generator_ : std::unique_ptr - exporter_ : std::unique_ptr - curated_data_service_ : std::unique_ptr + - postal_code_service_ : std::unique_ptr - application_options_ : ApplicationOptions - generated_breweries_ : std::vector - generated_users_ : std::vector + Run() : bool - - QueryCitiesWithCountries() : std::vector + - QueryCitiesWithCountries() : std::vector - GenerateBreweries(cities : std::span) : void - GenerateUsers(cities : std::span) : void - LogResults() : void @@ -90,17 +91,17 @@ class LogDispatcher { } interface IEnrichmentService <> { - + GetLocationContext(loc : const Location&) : std::string + + GetLocationContext(loc : const City&) : std::string } class MockEnrichmentService { - + GetLocationContext(loc : const Location&) : std::string + + GetLocationContext(loc : const City&) : std::string } class WikipediaEnrichmentService { - client_ : std::unique_ptr - extract_cache_ : std::unordered_map - + GetLocationContext(loc : const Location&) : std::string + + GetLocationContext(loc : const City&) : std::string - FetchExtract(query : std::string_view) : std::string } @@ -115,15 +116,15 @@ class HttpWebClient { } interface DataGenerator <> { - + GenerateBrewery(location : const Location&, region_context : const std::string&) : BreweryResult + + GenerateBrewery(location : const City&, region_context : const std::string&) : BreweryResult + GenerateUser(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult } class MockGenerator { - + GenerateBrewery(location : const Location&, region_context : const std::string&) : BreweryResult + + GenerateBrewery(location : const City&, region_context : const std::string&) : BreweryResult + GenerateUser(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult - - DeterministicHash(location : const Location&) : size_t - - DeterministicHash(location : const Location&, persona : const UserPersona&, name : const Name&) : size_t + - DeterministicHash(location : const City&) : size_t + - DeterministicHash(location : const City&, persona : const UserPersona&, name : const Name&) : size_t } class LlamaGenerator { @@ -158,7 +159,7 @@ class PromptDirectory { } interface ICuratedDataService <> { - + LoadLocations(filepath : const std::filesystem::path&) : const std::vector& + + LoadCities(filepath : const std::filesystem::path&) : const std::vector& + LoadPersonas(filepath : const std::filesystem::path&) : const std::vector& + LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map& + LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map& @@ -166,7 +167,7 @@ interface ICuratedDataService <> { class JsonLoader { - cache_ : cache - + LoadLocations(filepath : const std::filesystem::path&) : const std::vector& + + LoadCities(filepath : const std::filesystem::path&) : const std::vector& + LoadPersonas(filepath : const std::filesystem::path&) : const std::vector& + LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map& + LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map& @@ -180,11 +181,11 @@ note right of JsonLoader end note class MockCuratedDataService { - - locations_ : std::vector + - locations_ : std::vector - personas_ : std::vector - forenames_by_country_ : std::unordered_map - surnames_by_country_ : std::unordered_map - + LoadLocations(filepath : const std::filesystem::path&) : const std::vector& + + LoadCities(filepath : const std::filesystem::path&) : const std::vector& + LoadPersonas(filepath : const std::filesystem::path&) : const std::vector& + LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map& + LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map& @@ -196,6 +197,27 @@ note right of MockCuratedDataService filepath arguments are ignored. end note +interface IPostalCodeService <> { + + GeneratePostalCode(location : const City&) : std::string +} + +class MockPostalCodeService { + + GeneratePostalCode(location : const City&) : std::string +} + +note right of MockPostalCodeService + Ignores location.postal_regex and returns + location.postal_code_examples.front(). Bound + unconditionally in main.cc's DI graph (no + --mocked branch yet -- no real implementation + exists to switch to) and called from + GenerateBreweries() per brewery, stored on + BreweryRecord::address. A real implementation + needs a xeger-style generator to turn + postal_regex into a synthesized matching + string. See ROADMAP.md §9. +end note + interface IExportService <> { + Initialize() : void + ProcessRecord(brewery : const BreweryRecord&) : uint64_t @@ -208,17 +230,17 @@ class SqliteExportService { - run_timestamp_utc_ : std::string - database_path_ : std::filesystem::path - db_handle_ : SqliteDatabaseHandle - - insert_location_stmt_ : SqliteStatementHandle + - insert_city_stmt_ : SqliteStatementHandle - insert_brewery_stmt_ : SqliteStatementHandle - insert_user_stmt_ : SqliteStatementHandle - transaction_open_ : bool - - location_cache_ : std::unordered_map + - city_cache_ : std::unordered_map + Initialize() : void + ProcessRecord(brewery : const BreweryRecord&) : uint64_t + ProcessRecord(user : const UserRecord&) : uint64_t + Finalize() : void - InitializeSchema() : void - - ResolveLocationId(location : const Location&) : sqlite3_int64 + - ResolveCityId(location : const City&) : sqlite3_int64 } interface IDateTimeProvider <> { @@ -235,6 +257,7 @@ BiergartenPipelineOrchestrator *-- IEnrichmentService : owns BiergartenPipelineOrchestrator *-- DataGenerator : owns BiergartenPipelineOrchestrator *-- IExportService : owns BiergartenPipelineOrchestrator *-- ICuratedDataService : owns +BiergartenPipelineOrchestrator *-- IPostalCodeService : owns LogEntry *-- LogLevel LogEntry *-- PipelinePhase @@ -262,6 +285,8 @@ IPromptDirectory <|.. PromptDirectory : implements ICuratedDataService <|.. JsonLoader : implements ICuratedDataService <|.. MockCuratedDataService : implements +IPostalCodeService <|.. MockPostalCodeService : implements + IExportService <|.. SqliteExportService : implements SqliteExportService *-- IDateTimeProvider : owns IDateTimeProvider <|.. SystemDateTimeProvider : implements diff --git a/docs/pipeline/diagrams/current/output/activity.svg b/docs/pipeline/diagrams/current/output/activity.svg index b45b201..4a0b015 100644 --- a/docs/pipeline/diagrams/current/output/activity.svg +++ b/docs/pipeline/diagrams/current/output/activity.svg @@ -1 +1 @@ -The Biergarten Data Pipeline (Current — Synchronous Data Path)Create BoundedChannel<LogEntry> log_channelThe only concurrency in the current pipeline:a dedicated thread drains log_channel andforwards entries to spdlog for the entire run.Joined during shutdown after log_channel closes.Spawn log dispatcher thread(LogDispatcher::Run())ParseArguments(argc, argv)Log error / usage infonoAre arguments valid?yesInit OpenSSL global state & LlamaBackendStateBinds ILogger, IEnrichmentService, DataGenerator,IExportService, IPromptFormatter, WebClient basedon --mocked vs. model-backed mode.di::make_injector(...)injector.create<std::unique_ptr<BiergartenPipelineOrchestrator>>()Run() itself is synchronous — sampling, enrichment,generation, and export all happen on this thread.BiergartenPipelineOrchestrator::Run()Close log_channelJoin log dispatcher threadReturn 0Initialize SQLite exportQueryCitiesWithCountries()Lookup failed?yesnoLog warning, skip cityStore EnrichedCity{Location, region_context}Remaining citiesFor each sampled Location?DoneGenerateUsers(enriched_cities)SampleName(forenames_by_country,surnames_by_country, location.iso3166_1)Names available for country?noyesLog warning, skip citySample UserPersona (uniform random)Generation successful?yesnoBuildEmail(name, used_email_local_parts)email and date_of_birth are generatedprogrammatically, never LLM-authored.GenerateDateOfBirth(rng)Log warning "SQLite export failed"Log warning "Generation failed, skipping..."Remaining citiesFor each EnrichedCity?DoneGenerateBreweries(enriched_cities)Generation successful?yesnoData loss is prevented per-record.The pipeline continues running.Log warning "SQLite export failed"Log warning "Generation failed, skipping..."GetUtcTimestamp() from SystemDateTimeProviderBuilds a fresh biergarten_seed_<UTC datetime>.sqlite filenameAppends a numeric suffix if the timestamp already existsOpens DB ConnectionExecutes Schema DDLBegins TransactionInitialize()ProcessRecord(UserRecord)Location in cache?yesnoReuse location_idInsert Location & Cache IDInsert User (FK: location_id)yesException caught during insert?noProcessRecord(GeneratedBrewery)Location in cache?yesnoReuse location_idInsert Location & Cache IDInsert Brewery (FK: location_id)yesException caught during insert?noCommits TransactionCloses Database ConnectionFinalize()JsonLoader::LoadLocations("locations.json")--location-count defaults to 10.std::ranges::sample(all_locations, --location-count)LoadPersonas("personas.json")LoadForenamesByCountry("forenames-by-country.json")Each call is cached after its first parse.MockCuratedDataService (--mocked) returns afixed in-memory dataset instead and skipsthese three JSON files entirely.LoadSurnamesByCountry("surnames-by-country.json")WikipediaEnrichmentService fetches a generic"brewing" extract and a "beer in {country}" extract(not a city/region-specific page). MockEnrichmentService(--mocked) returns an empty string instead.GetLocationContext(loc)Generator ModeMockGeneratorLlamaGeneratorDeterministicHash & FormatResolves to USER_GENERATION.md inside --prompt-dir.prompt_directory_->Load("USER_GENERATION")Infer(system_prompt, user_prompt, max_tokens, kUserJsonGrammar)ValidateUserJson(raw, user)Is JSON Valid?yesnoAttempt++Append validation error to retry promptAttempt < 3?yesGenerator ModeMockGeneratorLlamaGeneratorDeterministicHash & FormatPrepareRegionContextResolves to BREWERY_GENERATION.md inside --prompt-dir.prompt_directory_->Load("BREWERY_GENERATION")Infer(system_prompt, user_prompt, max_tokens, kBreweryJsonGrammar)ValidateBreweryJson(raw, brewery)max_tokens is fixed across retries —it is not raised on truncation.Is JSON Valid?yesnoAttempt++Append validation error to retry promptAttempt < 3?yesRemaining citiesFor each EnrichedCity?Donemain.ccBiergartenPipelineOrchestratorSqliteExportServiceJsonLoaderIEnrichmentServiceDataGenerator \ No newline at end of file +The Biergarten Data Pipeline (Current — Synchronous Data Path)Create BoundedChannel<LogEntry> log_channelThe only concurrency in the current pipeline:a dedicated thread drains log_channel andforwards entries to spdlog for the entire run.Joined during shutdown after log_channel closes.Spawn log dispatcher thread(LogDispatcher::Run())ParseArguments(argc, argv)Log error / usage infonoAre arguments valid?yesInit OpenSSL global state & LlamaBackendStateBinds ILogger, IEnrichmentService, DataGenerator,IExportService, IPromptFormatter, WebClient basedon --mocked vs. model-backed mode.di::make_injector(...)injector.create<std::unique_ptr<BiergartenPipelineOrchestrator>>()Run() itself is synchronous — sampling, enrichment,generation, and export all happen on this thread.BiergartenPipelineOrchestrator::Run()Close log_channelJoin log dispatcher threadReturn 0Initialize SQLite exportQueryCitiesWithCountries()Lookup failed?yesnoLog warning, skip cityStore EnrichedCity{City, region_context}Remaining citiesFor each sampled City?DoneGenerateUsers(enriched_cities)SampleName(forenames_by_country,surnames_by_country, location.iso3166_1)Names available for country?noyesLog warning, skip citySample UserPersona (uniform random)Generation successful?yesnoBuildEmail(name, used_email_local_parts)email and date_of_birth are generatedprogrammatically, never LLM-authored.GenerateDateOfBirth(rng)Log warning "SQLite export failed"Log warning "Generation failed, skipping..."Remaining citiesFor each EnrichedCity?DoneGenerateBreweries(enriched_cities)Generation successful?yesnoData loss is prevented per-record.The pipeline continues running.Log warning "SQLite export failed"Log warning "Generation failed, skipping..."GetUtcTimestamp() from SystemDateTimeProviderBuilds a fresh biergarten_seed_<UTC datetime>.sqlite filenameAppends a numeric suffix if the timestamp already existsOpens DB ConnectionExecutes Schema DDLBegins TransactionInitialize()ProcessRecord(UserRecord)City in cache?yesnoReuse city_idInsert City & Cache IDInsert User (FK: city_id)yesException caught during insert?noProcessRecord(GeneratedBrewery)City in cache?yesnoReuse city_idInsert City & Cache IDInsert Brewery (FK: city_id)yesException caught during insert?noCommits TransactionCloses Database ConnectionFinalize()JsonLoader::LoadCities("cities.json")--location-count defaults to 10.std::ranges::sample(all_locations, --location-count)LoadPersonas("personas.json")LoadForenamesByCountry("forenames-by-country.json")Each call is cached after its first parse.MockCuratedDataService (--mocked) returns afixed in-memory dataset instead and skipsthese three JSON files entirely.LoadSurnamesByCountry("surnames-by-country.json")WikipediaEnrichmentService fetches a generic"brewing" extract and a "beer in {country}" extract(not a city/region-specific page). MockEnrichmentService(--mocked) returns an empty string instead.GetLocationContext(loc)Generator ModeMockGeneratorLlamaGeneratorDeterministicHash & FormatResolves to USER_GENERATION.md inside --prompt-dir.prompt_directory_->Load("USER_GENERATION")Infer(system_prompt, user_prompt, max_tokens, kUserJsonGrammar)ValidateUserJson(raw, user)Is JSON Valid?yesnoAttempt++Append validation error to retry promptAttempt < 3?yesGenerator ModeMockGeneratorLlamaGeneratorDeterministicHash & FormatPrepareRegionContextResolves to BREWERY_GENERATION.md inside --prompt-dir.prompt_directory_->Load("BREWERY_GENERATION")Infer(system_prompt, user_prompt, max_tokens, kBreweryJsonGrammar)ValidateBreweryJson(raw, brewery)max_tokens is fixed across retries —it is not raised on truncation.Is JSON Valid?yesnoAttempt++Append validation error to retry promptAttempt < 3?yesRemaining citiesFor each EnrichedCity?Donemain.ccBiergartenPipelineOrchestratorSqliteExportServiceJsonLoaderIEnrichmentServiceDataGenerator \ No newline at end of file diff --git a/docs/pipeline/diagrams/current/output/class.svg b/docs/pipeline/diagrams/current/output/class.svg index 53d928a..df87edd 100644 --- a/docs/pipeline/diagrams/current/output/class.svg +++ b/docs/pipeline/diagrams/current/output/class.svg @@ -1 +1 @@ -The Biergarten Data Pipeline - Class DiagramBiergartenPipelineOrchestratorlogger_ : std::shared_ptr<ILogger>context_service_ : std::unique_ptr<IEnrichmentService>generator_ : std::unique_ptr<DataGenerator>exporter_ : std::unique_ptr<IExportService>curated_data_service_ : std::unique_ptr<ICuratedDataService>application_options_ : ApplicationOptionsgenerated_breweries_ : std::vector<BreweryRecord>generated_users_ : std::vector<UserRecord>Run() : boolQueryCitiesWithCountries() : std::vector<Location>GenerateBreweries(cities : std::span<const EnrichedCity>) : voidGenerateUsers(cities : std::span<const EnrichedCity>) : voidLogResults() : void«enumeration»LogLevelDebugInfoWarnError«enumeration»PipelinePhaseStartupUserGenerationBreweryAndBeerGenerationCheckinGenerationRatingGenerationFollowGenerationTeardownLogDTOlevel : LogLevelphase : PipelinePhasemessage : std::stringLogEntrytimestamp : std::chrono::system_clock::time_pointorigin : std::source_locationthread_id : std::thread::idlevel : LogLevelphase : PipelinePhasemessage : std::string«interface»ILoggerLog(payload : LogDTO) : voidDoLog(entry : LogEntry) : voidLogProducerchannel_ : BoundedChannel<LogEntry>&DoLog(entry : LogEntry) : voidLogDispatcherchannel_ : BoundedChannel<LogEntry>&Run() : voidToSpdlogLevel(level) : spdlog::level::level_enum«interface»IEnrichmentServiceGetLocationContext(loc : const Location&) : std::stringMockEnrichmentServiceGetLocationContext(loc : const Location&) : std::stringWikipediaEnrichmentServiceclient_ : std::unique_ptr<WebClient>extract_cache_ : std::unordered_map<std::string, std::string>GetLocationContext(loc : const Location&) : std::stringFetchExtract(query : std::string_view) : std::string«interface»WebClientGet(url : const std::string&) : std::stringUrlEncode(value : const std::string&) : std::stringHttpWebClientGet(url : const std::string&) : std::stringUrlEncode(value : const std::string&) : std::string«interface»DataGeneratorGenerateBrewery(location : const Location&, region_context : const std::string&) : BreweryResultGenerateUser(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResultMockGeneratorGenerateBrewery(location : const Location&, region_context : const std::string&) : BreweryResultGenerateUser(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResultDeterministicHash(location : const Location&) : size_tDeterministicHash(location : const Location&, persona : const UserPersona&, name : const Name&) : size_tLlamaGeneratormodel_ : ModelHandlecontext_ : ContextHandleprompt_formatter_ : std::unique_ptr<IPromptFormatter>prompt_directory_ : std::unique_ptr<IPromptDirectory>rng_ : std::mt19937GenerateBrewery(...) : BreweryResultGenerateUser(...) : UserResultLoad(model_path : const std::string&) : voidInfer(...) : std::stringInferFormatted(...) : std::string«interface»IPromptFormatterFormat(system_prompt : std::string_view, user_prompt : std::string_view) : std::stringGemma4JinjaPromptFormatterFormat(system_prompt : std::string_view, user_prompt : std::string_view) : std::string«interface»IPromptDirectoryLoad(key : std::string_view) : std::stringPromptDirectoryprompt_dir_ : std::filesystem::pathcache_ : std::unordered_map<std::string, std::string>Load(key : std::string_view) : std::string«interface»ICuratedDataServiceLoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&JsonLoadercache_ : cacheLoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&Each Load* method memoizes its result incache_ on first call; later calls on thesame instance return the cached valuewithout re-parsing.MockCuratedDataServicelocations_ : std::vector<Location>personas_ : std::vector<UserPersona>forenames_by_country_ : std::unordered_map<std::string, forename_list>surnames_by_country_ : std::unordered_map<std::string, surname_list>LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&Fixed 4-location / 3-persona dataset(US, DE, FR, BE) used in --mocked runs;filepath arguments are ignored.«interface»IExportServiceInitialize() : voidProcessRecord(brewery : const BreweryRecord&) : uint64_tProcessRecord(user : const UserRecord&) : uint64_tFinalize() : voidSqliteExportServicedate_time_provider_ : std::unique_ptr<IDateTimeProvider>run_timestamp_utc_ : std::stringdatabase_path_ : std::filesystem::pathdb_handle_ : SqliteDatabaseHandleinsert_location_stmt_ : SqliteStatementHandleinsert_brewery_stmt_ : SqliteStatementHandleinsert_user_stmt_ : SqliteStatementHandletransaction_open_ : boollocation_cache_ : std::unordered_map<std::string, sqlite3_int64>Initialize() : voidProcessRecord(brewery : const BreweryRecord&) : uint64_tProcessRecord(user : const UserRecord&) : uint64_tFinalize() : voidInitializeSchema() : voidResolveLocationId(location : const Location&) : sqlite3_int64«interface»IDateTimeProviderGetUtcTimestamp() : std::stringSystemDateTimeProviderGetUtcTimestamp() : std::stringownsownsownsownsownsimplementsemitsconsumesimplementsimplementsownsimplementsimplementsimplementsusesusesimplementsimplementsimplementsimplementsimplementsownsimplements \ No newline at end of file +The Biergarten Data Pipeline - Class DiagramBiergartenPipelineOrchestratorlogger_ : std::shared_ptr<ILogger>context_service_ : std::unique_ptr<IEnrichmentService>generator_ : std::unique_ptr<DataGenerator>exporter_ : std::unique_ptr<IExportService>curated_data_service_ : std::unique_ptr<ICuratedDataService>postal_code_service_ : std::unique_ptr<IPostalCodeService>application_options_ : ApplicationOptionsgenerated_breweries_ : std::vector<BreweryRecord>generated_users_ : std::vector<UserRecord>Run() : boolQueryCitiesWithCountries() : std::vector<City>GenerateBreweries(cities : std::span<const EnrichedCity>) : voidGenerateUsers(cities : std::span<const EnrichedCity>) : voidLogResults() : void«enumeration»LogLevelDebugInfoWarnError«enumeration»PipelinePhaseStartupUserGenerationBreweryAndBeerGenerationCheckinGenerationRatingGenerationFollowGenerationTeardownLogDTOlevel : LogLevelphase : PipelinePhasemessage : std::stringLogEntrytimestamp : std::chrono::system_clock::time_pointorigin : std::source_locationthread_id : std::thread::idlevel : LogLevelphase : PipelinePhasemessage : std::string«interface»ILoggerLog(payload : LogDTO) : voidDoLog(entry : LogEntry) : voidLogProducerchannel_ : BoundedChannel<LogEntry>&DoLog(entry : LogEntry) : voidLogDispatcherchannel_ : BoundedChannel<LogEntry>&Run() : voidToSpdlogLevel(level) : spdlog::level::level_enum«interface»IEnrichmentServiceGetLocationContext(loc : const City&) : std::stringMockEnrichmentServiceGetLocationContext(loc : const City&) : std::stringWikipediaEnrichmentServiceclient_ : std::unique_ptr<WebClient>extract_cache_ : std::unordered_map<std::string, std::string>GetLocationContext(loc : const City&) : std::stringFetchExtract(query : std::string_view) : std::string«interface»WebClientGet(url : const std::string&) : std::stringUrlEncode(value : const std::string&) : std::stringHttpWebClientGet(url : const std::string&) : std::stringUrlEncode(value : const std::string&) : std::string«interface»DataGeneratorGenerateBrewery(location : const City&, region_context : const std::string&) : BreweryResultGenerateUser(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResultMockGeneratorGenerateBrewery(location : const City&, region_context : const std::string&) : BreweryResultGenerateUser(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResultDeterministicHash(location : const City&) : size_tDeterministicHash(location : const City&, persona : const UserPersona&, name : const Name&) : size_tLlamaGeneratormodel_ : ModelHandlecontext_ : ContextHandleprompt_formatter_ : std::unique_ptr<IPromptFormatter>prompt_directory_ : std::unique_ptr<IPromptDirectory>rng_ : std::mt19937GenerateBrewery(...) : BreweryResultGenerateUser(...) : UserResultLoad(model_path : const std::string&) : voidInfer(...) : std::stringInferFormatted(...) : std::string«interface»IPromptFormatterFormat(system_prompt : std::string_view, user_prompt : std::string_view) : std::stringGemma4JinjaPromptFormatterFormat(system_prompt : std::string_view, user_prompt : std::string_view) : std::string«interface»IPromptDirectoryLoad(key : std::string_view) : std::stringPromptDirectoryprompt_dir_ : std::filesystem::pathcache_ : std::unordered_map<std::string, std::string>Load(key : std::string_view) : std::string«interface»ICuratedDataServiceLoadCities(filepath : const std::filesystem::path&) : const std::vector<City>&LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&JsonLoadercache_ : cacheLoadCities(filepath : const std::filesystem::path&) : const std::vector<City>&LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&Each Load* method memoizes its result incache_ on first call; later calls on thesame instance return the cached valuewithout re-parsing.MockCuratedDataServicelocations_ : std::vector<City>personas_ : std::vector<UserPersona>forenames_by_country_ : std::unordered_map<std::string, forename_list>surnames_by_country_ : std::unordered_map<std::string, surname_list>LoadCities(filepath : const std::filesystem::path&) : const std::vector<City>&LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&Fixed 4-location / 3-persona dataset(US, DE, FR, BE) used in --mocked runs;filepath arguments are ignored.«interface»IPostalCodeServiceGeneratePostalCode(location : const City&) : std::stringMockPostalCodeServiceGeneratePostalCode(location : const City&) : std::stringIgnores location.postal_regex and returnslocation.postal_code_examples.front(). Boundunconditionally in main.cc's DI graph (nomocked branch yetno real implementationexists to switch to) and called fromGenerateBreweries() per brewery, stored onBreweryRecord::address. A real implementationneeds a xeger-style generator to turnpostal_regex into a synthesized matchingstring. See ROADMAP.md §9.«interface»IExportServiceInitialize() : voidProcessRecord(brewery : const BreweryRecord&) : uint64_tProcessRecord(user : const UserRecord&) : uint64_tFinalize() : voidSqliteExportServicedate_time_provider_ : std::unique_ptr<IDateTimeProvider>run_timestamp_utc_ : std::stringdatabase_path_ : std::filesystem::pathdb_handle_ : SqliteDatabaseHandleinsert_city_stmt_ : SqliteStatementHandleinsert_brewery_stmt_ : SqliteStatementHandleinsert_user_stmt_ : SqliteStatementHandletransaction_open_ : boolcity_cache_ : std::unordered_map<std::string, sqlite3_int64>Initialize() : voidProcessRecord(brewery : const BreweryRecord&) : uint64_tProcessRecord(user : const UserRecord&) : uint64_tFinalize() : voidInitializeSchema() : voidResolveCityId(location : const City&) : sqlite3_int64«interface»IDateTimeProviderGetUtcTimestamp() : std::stringSystemDateTimeProviderGetUtcTimestamp() : std::stringownsownsownsownsownsownsimplementsemitsconsumesimplementsimplementsownsimplementsimplementsimplementsusesusesimplementsimplementsimplementsimplementsimplementsimplementsownsimplements \ No newline at end of file diff --git a/docs/pipeline/diagrams/planned/activity.puml b/docs/pipeline/diagrams/planned/activity.puml index 67cabae..2e4362f 100644 --- a/docs/pipeline/diagrams/planned/activity.puml +++ b/docs/pipeline/diagrams/planned/activity.puml @@ -39,7 +39,7 @@ fork :JsonLoader::LoadBeerStyles("beer-styles.json"); :EnrichmentService::PreWarmBeerStyleCache(beer_styles); fork again - :JsonLoader::LoadLocations("locations.json"); + :JsonLoader::LoadCities("cities.json"); :EnrichmentService::PreWarmLocationCache(sampled_locations); end fork fork @@ -75,7 +75,7 @@ fork fork again |LLM Worker| while (loc_ch has items?) is (yes) - :Receive Location; + :Receive City; :GetLocationContextFromCache(location); note right @@ -146,13 +146,13 @@ end fork fork |Orchestrator| :Loop: Sample User from user_pool_ - and pair with Location; - :Send BreweryTask(Location, User) -> loc_ch; + and pair with City; + :Send BreweryTask(City, User) -> loc_ch; :Close loc_ch; fork again |LLM Worker| while (loc_ch has items?) is (yes) - :Receive BreweryTask(Location, User); + :Receive BreweryTask(City, User); :GetLocationContextFromCache(task.location); note right diff --git a/docs/pipeline/diagrams/planned/class.puml b/docs/pipeline/diagrams/planned/class.puml index 68876ef..8cd578e 100644 --- a/docs/pipeline/diagrams/planned/class.puml +++ b/docs/pipeline/diagrams/planned/class.puml @@ -12,17 +12,40 @@ title Biergarten Data Pipeline — Class Diagram package "Domain: Models" { - class Location { + class City { + city : std::string + state_province : std::string + iso3166_2 : std::string + country : std::string + iso3166_1 : std::string + + postal_regex : std::vector + + postal_code_examples : std::vector + local_languages : std::vector + latitude : double + longitude : double } + note right of City + postal_regex / postal_code_examples are + sourced from each cities.json entry's + postal_code.city_regex / postal_code.examples + (implemented today, ahead of the rest of this + diagram -- see ROADMAP.md §1/§2). Renamed from + Location to City to match the web backend's + City/CityId vocabulary -- see ROADMAP.md §1. + end note + + class Address { + + postal_code : std::string + } + + note right of Address + Mirrors the web backend's BreweryPostLocation. + Only postal_code is populated today -- + address_line1/address_line2 have no fixture + data or generator yet. See ROADMAP.md §9. + end note + class LocationContext { + text : std::string + completeness : Completeness @@ -36,7 +59,7 @@ package "Domain: Models" { } class EnrichedCity { - + location : Location + + location : City + context : LocationContext } @@ -130,7 +153,8 @@ package "Domain: Models" { class GeneratedBrewery { + brewery_id : uint64_t - + location : Location + + location : City + + address : Address + brewery : BreweryResult + context_completeness : LocationContext::Completeness + metadata : GenerationMetadata @@ -139,7 +163,7 @@ package "Domain: Models" { class GeneratedBeer { + beer_id : uint64_t + brewery_id : uint64_t - + location : Location + + location : City + style : BeerStyle + beer : BeerResult + metadata : GenerationMetadata @@ -147,7 +171,7 @@ package "Domain: Models" { class GeneratedUser { + user_id : uint64_t - + location : Location + + location : City + user : UserResult + email : std::string + date_of_birth : std::string @@ -227,27 +251,27 @@ package "Domain: Application Configuration" { package "Domain: Policy" { interface ContextStrategy <> { - + QueriesFor(loc : const Location&) : std::vector + + QueriesFor(loc : const City&) : std::vector + MaxContextChars() : size_t } class BreweryContextStrategy { - + QueriesFor(loc : const Location&) : std::vector + + QueriesFor(loc : const City&) : std::vector + MaxContextChars() : size_t } class BeerContextStrategy { - + QueriesFor(loc : const Location&) : std::vector + + QueriesFor(loc : const City&) : std::vector + MaxContextChars() : size_t } interface SamplingStrategy <> { - + Sample(locations : const std::vector&) : std::vector + + Sample(locations : const std::vector&) : std::vector } class UniformSamplingStrategy { - sample_size_ : size_t - + Sample(locations : const std::vector&) : std::vector + + Sample(locations : const std::vector&) : std::vector } interface BeerSelectionStrategy <> { @@ -369,7 +393,7 @@ package "Infrastructure: Pipeline Channel" { package "Infrastructure: Data Preloading" { interface ICuratedDataService <> { - + LoadLocations(filepath : const std::filesystem::path&) : const std::vector& + + LoadCities(filepath : const std::filesystem::path&) : const std::vector& + LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector& + LoadPersonas(filepath : const std::filesystem::path&) : const std::vector& + LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map& @@ -377,7 +401,7 @@ package "Infrastructure: Data Preloading" { } class JsonLoader { - + LoadLocations(filepath : const std::filesystem::path&) : const std::vector& + + LoadCities(filepath : const std::filesystem::path&) : const std::vector& + LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector& + LoadPersonas(filepath : const std::filesystem::path&) : const std::vector& + LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map& @@ -391,7 +415,7 @@ package "Infrastructure: Data Preloading" { end note class MockCuratedDataService { - + LoadLocations(filepath : const std::filesystem::path&) : const std::vector& + + LoadCities(filepath : const std::filesystem::path&) : const std::vector& + LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector& + LoadPersonas(filepath : const std::filesystem::path&) : const std::vector& + LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map& @@ -410,13 +434,13 @@ package "Infrastructure: Data Preloading" { package "Infrastructure: Enrichment" { interface EnrichmentService <> { - + GetLocationContext(loc : const Location&,\n strategy : const ContextStrategy&) : LocationContext + + GetLocationContext(loc : const City&,\n strategy : const ContextStrategy&) : LocationContext } class WikipediaService { - client_ : std::unique_ptr - extract_cache_ : std::unordered_map - + GetLocationContext(loc : const Location&,\n strategy : const ContextStrategy&) : LocationContext + + GetLocationContext(loc : const City&,\n strategy : const ContextStrategy&) : LocationContext - FetchExtract(query : std::string_view) : std::string } @@ -432,6 +456,39 @@ package "Infrastructure: Enrichment" { } +package "Infrastructure: Postal Code Generation" { + + interface IPostalCodeService <> { + + GeneratePostalCode(location : const City&) : std::string + } + + class MockPostalCodeService { + + GeneratePostalCode(location : const City&) : std::string + } + + class XegerPostalCodeService { + + GeneratePostalCode(location : const City&) : std::string + } + + note right of MockPostalCodeService + Implemented today: ignores postal_regex and + returns postal_code_examples.front(). Not yet + bound in main.cc's DI graph or used by + BiergartenPipelineOrchestrator. + end note + + note right of XegerPostalCodeService + Not implemented yet. Picks one of + location.postal_regex and generates a random + string matching it (a "xeger" -- the inverse of + regex matching) instead of always replaying a + fixed example. See ROADMAP.md §9. + end note + + IPostalCodeService <|.. MockPostalCodeService + IPostalCodeService <|.. XegerPostalCodeService +} + package "Infrastructure: Prompting" { interface IPromptDirectory <> { @@ -451,8 +508,8 @@ package "Infrastructure: Prompting" { package "Infrastructure: Data Generation" { interface DataGenerator <> { - + GenerateBrewery(location : const Location&,\n context : const LocationContext&) : BreweryResult - + GenerateBeer(brewery_id : uint64_t,\n location : const Location&,\n context : const LocationContext&,\n style : const BeerStyle&) : BeerResult + + GenerateBrewery(location : const City&,\n context : const LocationContext&) : BreweryResult + + GenerateBeer(brewery_id : uint64_t,\n location : const City&,\n context : const LocationContext&,\n style : const BeerStyle&) : BeerResult + GenerateUser(city : const EnrichedCity&,\n persona : const UserPersona&,\n name : const Name&) : UserResult + GenerateCheckin(user : const GeneratedUser&,\n brewery : const GeneratedBrewery&,\n timestamp : const std::string&) : CheckinResult + GenerateRating(user : const GeneratedUser&,\n beer : const GeneratedBeer&,\n checkin_id : uint64_t) : RatingResult @@ -464,7 +521,7 @@ package "Infrastructure: Data Generation" { + GenerateUser(...) : UserResult + GenerateCheckin(...) : CheckinResult + GenerateRating(...) : RatingResult - - DeterministicHash(location : const Location&) : size_t + - DeterministicHash(location : const City&) : size_t } class LlamaGenerator { @@ -566,8 +623,8 @@ class BiergartenPipelineOrchestrator { - follow_pool_ : std::vector -- + Run() : bool - - RunUserPhase(locations : const std::vector&) : void - - RunBreweryAndBeerPhase(locations : const std::vector&) : void + - RunUserPhase(locations : const std::vector&) : void + - RunBreweryAndBeerPhase(locations : const std::vector&) : void - RunCheckinPhase() : void - RunRatingPhase() : void - RunFollowPhase() : void @@ -625,16 +682,17 @@ PipelineLogger o-- BoundedChannel : logs to LogWorker o-- BoundedChannel : drains from ' --- Domain Containment --- -EnrichedCity *-- Location +EnrichedCity *-- City EnrichedCity *-- LocationContext -GeneratedBrewery *-- Location +GeneratedBrewery *-- City +GeneratedBrewery *-- Address GeneratedBrewery *-- BreweryResult GeneratedBrewery *-- GenerationMetadata -GeneratedBeer *-- Location +GeneratedBeer *-- City GeneratedBeer *-- BeerStyle GeneratedBeer *-- BeerResult GeneratedBeer *-- GenerationMetadata -GeneratedUser *-- Location +GeneratedUser *-- City GeneratedUser *-- UserResult GeneratedUser *-- GenerationMetadata GeneratedCheckin *-- CheckinResult diff --git a/docs/pipeline/diagrams/planned/output/biergarten_activity.svg b/docs/pipeline/diagrams/planned/output/biergarten_activity.svg index e565241..001e4fd 100644 --- a/docs/pipeline/diagrams/planned/output/biergarten_activity.svg +++ b/docs/pipeline/diagrams/planned/output/biergarten_activity.svg @@ -1 +1 @@ -The Biergarten Data Pipeline — Activity DiagramParseArguments(argc, argv)spdlog::erroryesInvalid args?noInit OpenSSL global state & LlamaBackendStateBuild DI injectorOpens SQLite connection.(Transactions are now managedper-phase via batching).Initialize SqliteExportServiceCreate BoundedChannel<LogEntry> log_chLog worker drains log_ch for theentire pipeline lifetime.All workers emit LogEntry structsvia PipelineLogger -- never spdlog directly.Spawn Log Worker threadBiergartenPipelineOrchestrator::Run()spdlog::info "Pipeline complete in X ms"Drain guarantees no LogEntry isdropped at shutdown.Join Log WorkerJsonLoader::LoadBeerStyles("beer-styles.json")EnrichmentService::PreWarmBeerStyleCache(beer_styles)JsonLoader::LoadLocations("locations.json")EnrichmentService::PreWarmLocationCache(sampled_locations)Each call is memoized after its first parse(implemented today). MockCuratedDataServicereturns a fixed in-memory dataset insteadunder --mocked, skipping these JSON files.JsonLoader::LoadForenamesByCountry("forenames-by-country.json")JsonLoader::LoadSurnamesByCountry("surnames-by-country.json")JsonLoader::LoadPersonas("personas.json")RunUserPhase(sampled_locations)Create BoundedChannels(loc_ch, exp_ch)Loop: Send Locations -> loc_chProducer closes loc_ch.LLM Worker while loopterminates on empty + closed.Close loc_chJoin LLM Worker, SQLite WorkerRunBreweryPhase(sampled_locations)Create BoundedChannels(loc_ch, exp_ch)Loop: Sample User from user_pool_and pair with LocationSend BreweryTask(Location, User) -> loc_chClose loc_chbrewery_pool_ is now fully populated.Phase 1b may begin.Join LLM Worker, SQLite WorkerRunBeerPhase()Create BoundedChannels(brew_ch, exp_ch)Loop: Send Breweries -> brew_chClose brew_chBoth brewery_pool_ and beer_pool_are now completely populated.Checkin and Follow phases maynow run in parallel.Join LLM Worker, SQLite WorkerRunCheckinPhase()Weights seeded from each user'spersona.checkin_weight. J-curve profileemerges from persona distribution.ICheckinDistributionStrategy::AssignActivityWeights(user_pool_)BEGIN TRANSACTIONCheckinsForUser(user, brewery_pool_.size())TimestampFor(user, index)Select brewery from brewery_pool_GenerateCheckin(user, brewery, timestamp)via DataGeneratorProcessCheckin(checkin)PipelineLogger::Log(Info, CheckinGeneration,nullopt, checkin_id, "sqlite")Append -> checkin_pool_COMMIT & BEGINyesBatch size reached?noremainingFor each checkin index?doneremainingFor each GeneratedUser in user_pool_?doneCOMMIT (Final)RunFollowPhase()For RandomFollowStrategy, weightsare uniform. For ActivityWeightedFollowStrategy,weights derived from user.activity_weightso high-activity users attract more followers.IFollowGenerationStrategy::AssignFollowWeights(user_pool_)BEGIN TRANSACTIONSelf-follow constraint (follower_id != followed_id)enforced here and at the DB schema level.IFollowGenerationStrategy::GenerateFollows(user_pool_)ProcessFollow(follow)PipelineLogger::Log(Info, FollowGeneration,nullopt, follower_id, "sqlite")Append -> follow_pool_COMMIT & BEGINyesBatch size reached?noremainingFor each GeneratedFollow?doneCOMMIT (Final)checkin_pool_ and follow_pool_are now fully populated.Rating phase may begin.Join CheckinPhase, FollowPhaseBeer selection biased byuser.persona.style_affinities and abv_range.Rating skew modulated per persona.RunRatingPhase()BEGIN TRANSACTIONMatch brewery_id, select beer from beer_pool_(same brewery_id, biased by persona affinities)Beer exists for brewery?yesnoGenerateRating(user, beer, checkin_id)via DataGeneratorProcessRating(rating)PipelineLogger::Log(Info, RatingGeneration,nullopt, rating_id, "sqlite")COMMIT & BEGINyesBatch size reached?noPipelineLogger::Log(Warn, RatingGeneration,nullopt, brewery_id, "sqlite")Skip -- brewery has no beersremainingFor each GeneratedCheckin in checkin_pool_?doneCOMMIT (Final)Safely closes the DB connection.Finalize SqliteExportServiceClose log_chReceive LocationGuaranteed cache hit from startup.GetLocationContextFromCache(location)Guaranteed cache hit from startup.Returns a Persona struct carryingstyle_affinities, abv_range,ibu_preference, checkin_weight.IPersonaSelectionStrategy::SelectPersona(personas_palette_)Deterministic lookup -- no LLM involved.Free function (implemented today ingenerate_users.cc), not a class method.Name selected from pre-keyed mapsand passed into the generation prompt.Skips the city if either map has noentry for iso3166_1.SampleName(forenames_by_country,surnames_by_country, location.iso3166_1, rng)LLM receives: EnrichedCity context + personadescription + sampled name. Generatesbio and preference signals groundedin locale and persona.GenerateUser(enriched_city, persona, sampled_name)via DataGeneratorPipelineLogger::Log(Info, UserGeneration,city, user_id, "llm")Send GeneratedUser -> exp_chyesloc_ch has items?noProducer closes exp_ch.SQLite Worker while loopterminates on empty + closed.Close exp_chReceive BreweryTask(Location, User)Guaranteed cache hit from startup.GetLocationContextFromCache(task.location)KV cache stays warm.Brewery is linked to the sampled owner_user_id.GenerateBrewery(enriched_city, context, task.user)via DataGeneratorPipelineLogger::Log(Info,BreweryGeneration,city, brewery_id, "llm")Send GeneratedBrewery -> exp_chyesloc_ch has items?noClose exp_chReceive GeneratedBreweryIBeerSelectionStrategy::SelectStyles(brewery, beer_style_palette_)Guaranteed cache hit from startup.KV cache stays warm across allbeer generations -- system promptdoes not change within this phase.GetStyleContextFromCache(style)GenerateBeer(brewery, style_context)via DataGeneratorAttach GeneratedBeer to bundleremainingFor each selected BeerStyle?donePipelineLogger::Log(Info,BeerGeneration,city, brewery_id, "llm")Send BeersBundle -> exp_chyesbrew_ch has items?noClose exp_chBEGIN TRANSACTIONReceive GeneratedUserProcessUser(user)PipelineLogger::Log(Info, UserGeneration,city, user_id, "sqlite")Append -> user_pool_COMMIT & BEGINyesBatch size reached?noyesexp_ch has items?noCOMMIT (Final)BEGIN TRANSACTIONReceive GeneratedBreweryProcessBrewery(brewery)PipelineLogger::Log(Info,BreweryGeneration,city, brewery_id, "sqlite")Append -> brewery_pool_COMMIT & BEGINyesBatch size reached?noyesexp_ch has items?noCOMMIT (Final)BEGIN TRANSACTIONReceive BeersBundleSet beer.brewery_id from bundleProcessBeer(beer)Append -> beer_pool_remainingFor each beer in bundle?donePipelineLogger::Log(Info,BeerGeneration,city, brewery_id, "sqlite")COMMIT & BEGINyesBatch size reached?noyesexp_ch has items?noCOMMIT (Final)MainBiergartenPipelineOrchestrator::Run()OrchestratorLLM WorkerSQLite Worker \ No newline at end of file +The Biergarten Data Pipeline — Activity DiagramParseArguments(argc, argv)spdlog::erroryesInvalid args?noInit OpenSSL global state & LlamaBackendStateBuild DI injectorOpens SQLite connection.(Transactions are now managedper-phase via batching).Initialize SqliteExportServiceCreate BoundedChannel<LogEntry> log_chLog worker drains log_ch for theentire pipeline lifetime.All workers emit LogEntry structsvia PipelineLogger -- never spdlog directly.Spawn Log Worker threadBiergartenPipelineOrchestrator::Run()spdlog::info "Pipeline complete in X ms"Drain guarantees no LogEntry isdropped at shutdown.Join Log WorkerJsonLoader::LoadBeerStyles("beer-styles.json")EnrichmentService::PreWarmBeerStyleCache(beer_styles)JsonLoader::LoadCities("cities.json")EnrichmentService::PreWarmLocationCache(sampled_locations)Each call is memoized after its first parse(implemented today). MockCuratedDataServicereturns a fixed in-memory dataset insteadunder --mocked, skipping these JSON files.JsonLoader::LoadForenamesByCountry("forenames-by-country.json")JsonLoader::LoadSurnamesByCountry("surnames-by-country.json")JsonLoader::LoadPersonas("personas.json")RunUserPhase(sampled_locations)Create BoundedChannels(loc_ch, exp_ch)Loop: Send Locations -> loc_chProducer closes loc_ch.LLM Worker while loopterminates on empty + closed.Close loc_chJoin LLM Worker, SQLite WorkerRunBreweryPhase(sampled_locations)Create BoundedChannels(loc_ch, exp_ch)Loop: Sample User from user_pool_and pair with CitySend BreweryTask(City, User) -> loc_chClose loc_chbrewery_pool_ is now fully populated.Phase 1b may begin.Join LLM Worker, SQLite WorkerRunBeerPhase()Create BoundedChannels(brew_ch, exp_ch)Loop: Send Breweries -> brew_chClose brew_chBoth brewery_pool_ and beer_pool_are now completely populated.Checkin and Follow phases maynow run in parallel.Join LLM Worker, SQLite WorkerRunCheckinPhase()Weights seeded from each user'spersona.checkin_weight. J-curve profileemerges from persona distribution.ICheckinDistributionStrategy::AssignActivityWeights(user_pool_)BEGIN TRANSACTIONCheckinsForUser(user, brewery_pool_.size())TimestampFor(user, index)Select brewery from brewery_pool_GenerateCheckin(user, brewery, timestamp)via DataGeneratorProcessCheckin(checkin)PipelineLogger::Log(Info, CheckinGeneration,nullopt, checkin_id, "sqlite")Append -> checkin_pool_COMMIT & BEGINyesBatch size reached?noremainingFor each checkin index?doneremainingFor each GeneratedUser in user_pool_?doneCOMMIT (Final)RunFollowPhase()For RandomFollowStrategy, weightsare uniform. For ActivityWeightedFollowStrategy,weights derived from user.activity_weightso high-activity users attract more followers.IFollowGenerationStrategy::AssignFollowWeights(user_pool_)BEGIN TRANSACTIONSelf-follow constraint (follower_id != followed_id)enforced here and at the DB schema level.IFollowGenerationStrategy::GenerateFollows(user_pool_)ProcessFollow(follow)PipelineLogger::Log(Info, FollowGeneration,nullopt, follower_id, "sqlite")Append -> follow_pool_COMMIT & BEGINyesBatch size reached?noremainingFor each GeneratedFollow?doneCOMMIT (Final)checkin_pool_ and follow_pool_are now fully populated.Rating phase may begin.Join CheckinPhase, FollowPhaseBeer selection biased byuser.persona.style_affinities and abv_range.Rating skew modulated per persona.RunRatingPhase()BEGIN TRANSACTIONMatch brewery_id, select beer from beer_pool_(same brewery_id, biased by persona affinities)Beer exists for brewery?yesnoGenerateRating(user, beer, checkin_id)via DataGeneratorProcessRating(rating)PipelineLogger::Log(Info, RatingGeneration,nullopt, rating_id, "sqlite")COMMIT & BEGINyesBatch size reached?noPipelineLogger::Log(Warn, RatingGeneration,nullopt, brewery_id, "sqlite")Skip -- brewery has no beersremainingFor each GeneratedCheckin in checkin_pool_?doneCOMMIT (Final)Safely closes the DB connection.Finalize SqliteExportServiceClose log_chReceive CityGuaranteed cache hit from startup.GetLocationContextFromCache(location)Guaranteed cache hit from startup.Returns a Persona struct carryingstyle_affinities, abv_range,ibu_preference, checkin_weight.IPersonaSelectionStrategy::SelectPersona(personas_palette_)Deterministic lookup -- no LLM involved.Free function (implemented today ingenerate_users.cc), not a class method.Name selected from pre-keyed mapsand passed into the generation prompt.Skips the city if either map has noentry for iso3166_1.SampleName(forenames_by_country,surnames_by_country, location.iso3166_1, rng)LLM receives: EnrichedCity context + personadescription + sampled name. Generatesbio and preference signals groundedin locale and persona.GenerateUser(enriched_city, persona, sampled_name)via DataGeneratorPipelineLogger::Log(Info, UserGeneration,city, user_id, "llm")Send GeneratedUser -> exp_chyesloc_ch has items?noProducer closes exp_ch.SQLite Worker while loopterminates on empty + closed.Close exp_chReceive BreweryTask(City, User)Guaranteed cache hit from startup.GetLocationContextFromCache(task.location)KV cache stays warm.Brewery is linked to the sampled owner_user_id.GenerateBrewery(enriched_city, context, task.user)via DataGeneratorPipelineLogger::Log(Info,BreweryGeneration,city, brewery_id, "llm")Send GeneratedBrewery -> exp_chyesloc_ch has items?noClose exp_chReceive GeneratedBreweryIBeerSelectionStrategy::SelectStyles(brewery, beer_style_palette_)Guaranteed cache hit from startup.KV cache stays warm across allbeer generations -- system promptdoes not change within this phase.GetStyleContextFromCache(style)GenerateBeer(brewery, style_context)via DataGeneratorAttach GeneratedBeer to bundleremainingFor each selected BeerStyle?donePipelineLogger::Log(Info,BeerGeneration,city, brewery_id, "llm")Send BeersBundle -> exp_chyesbrew_ch has items?noClose exp_chBEGIN TRANSACTIONReceive GeneratedUserProcessUser(user)PipelineLogger::Log(Info, UserGeneration,city, user_id, "sqlite")Append -> user_pool_COMMIT & BEGINyesBatch size reached?noyesexp_ch has items?noCOMMIT (Final)BEGIN TRANSACTIONReceive GeneratedBreweryProcessBrewery(brewery)PipelineLogger::Log(Info,BreweryGeneration,city, brewery_id, "sqlite")Append -> brewery_pool_COMMIT & BEGINyesBatch size reached?noyesexp_ch has items?noCOMMIT (Final)BEGIN TRANSACTIONReceive BeersBundleSet beer.brewery_id from bundleProcessBeer(beer)Append -> beer_pool_remainingFor each beer in bundle?donePipelineLogger::Log(Info,BeerGeneration,city, brewery_id, "sqlite")COMMIT & BEGINyesBatch size reached?noyesexp_ch has items?noCOMMIT (Final)MainBiergartenPipelineOrchestrator::Run()OrchestratorLLM WorkerSQLite Worker \ No newline at end of file diff --git a/docs/pipeline/diagrams/planned/output/class.svg b/docs/pipeline/diagrams/planned/output/class.svg index f796ee6..cbd42f0 100644 --- a/docs/pipeline/diagrams/planned/output/class.svg +++ b/docs/pipeline/diagrams/planned/output/class.svg @@ -1 +1 @@ -Biergarten Data Pipeline — Class DiagramDomain: ModelsDomain: Application ConfigurationDomain: PolicyInfrastructure: LoggingInfrastructure: Pipeline ChannelInfrastructure: Data PreloadingInfrastructure: EnrichmentInfrastructure: PromptingInfrastructure: Data GenerationInfrastructure: Data ExportLocationcity : std::stringstate_province : std::stringiso3166_2 : std::stringcountry : std::stringiso3166_1 : std::stringlocal_languages : std::vector<std::string>latitude : doublelongitude : doubleLocationContexttext : std::stringcompleteness : Completenesschar_count : size_tCompletenessFullPartialAbsentEnrichedCitylocation : Locationcontext : LocationContextBeerStylename : std::stringdescription : std::stringmin_abv : floatmax_abv : floatmin_ibu : intmax_ibu : intBreweryResultname_en : std::stringdescription_en : std::stringname_local : std::stringdescription_local : std::stringBeerResultname_en : std::stringdescription_en : std::stringname_local : std::stringdescription_local : std::stringstyle : std::stringabv : floatibu : intUserResultfirst_name : std::stringlast_name : std::stringgender : std::stringusername : std::stringbio : std::stringactivity_weight : floatNamefirst_name : std::stringlast_name : std::stringgender : std::stringForenameEntryname : std::stringgender : std::stringforename_list = std::unordered_set<ForenameEntry>surname_list = std::unordered_set<std::string>(aliases declared in curated_data_service.h).ICuratedDataService::LoadForenamesByCountry() /LoadSurnamesByCountry() each return a flatstd::unordered_map<std::string, forename_list>or <std::string, surname_list> keyed by ISO3166-1 country code -- no NamesByCountrywrapper class. SampleName(forenames_by_country,surnames_by_country, iso3166_1, rng) is a freefunction, not a method, so it can be called byany per-city worker without owning the maps.Sourced from popular-names-by-country-dataset(CC0) -- see ETHICS-AND-KNOWN-ISSUES.md. Genderis preserved per forename so it can be threadedthrough to persona/bio generation later; surnamescarry no gender in the source data. Pairinghappens at sample time, not at fixture-authoringtime, so no information from the source datasetis discarded up front.CheckinResultchecked_in_at : std::stringnote : std::stringRatingResultscore : floatnote : std::stringGenerationMetadatageneration_id : uint64_tgenerated_time : std::stringcontext_provided : boolgenerated_with : std::stringGeneratedBrewerybrewery_id : uint64_tlocation : Locationbrewery : BreweryResultcontext_completeness : LocationContext::Completenessmetadata : GenerationMetadataGeneratedBeerbeer_id : uint64_tbrewery_id : uint64_tlocation : Locationstyle : BeerStylebeer : BeerResultmetadata : GenerationMetadataGeneratedUseruser_id : uint64_tlocation : Locationuser : UserResultemail : std::stringdate_of_birth : std::stringpassword : std::stringmetadata : GenerationMetadataemail, date_of_birth, and password areprogrammatically generated by the orchestrator(not LLM-authored) so a downstream auth-accountseeding consumer can register real accounts fromthis export. See ROADMAP.md §1.GeneratedCheckincheckin_id : uint64_tuser_id : uint64_tbrewery_id : uint64_tcheckin : CheckinResultmetadata : GenerationMetadataGeneratedRatinguser_id : uint64_tbeer_id : uint64_tcheckin_id : uint64_trating : RatingResultmetadata : GenerationMetadataGeneratedFollowfollower_id : uint64_tfollowed_id : uint64_tmetadata : GenerationMetadataUserPersonaname: std::stringdescription: std::stringstyle_affinities: std::vector<std::string>SamplingOptionstemperature: float = 1.0Ftop_p: float = 0.95Ftop_k: uint32_t = 64n_ctx: uint32_t = 8192seed: int = -1GeneratorOptionsmodel_path: std::filesystem::pathuse_mocked: bool = falsesampling: std::optional<SamplingOptions>PipelineOptionsoutput_path: std::filesystem::pathlog_path: std::filesystem::pathApplicationOptionsgenerator: GeneratorOptionspipeline: PipelineOptions«interface»ContextStrategyQueriesFor(loc : const Location&) : std::vector<std::string>MaxContextChars() : size_tBreweryContextStrategyQueriesFor(loc : const Location&) : std::vector<std::string>MaxContextChars() : size_tBeerContextStrategyQueriesFor(loc : const Location&) : std::vector<std::string>MaxContextChars() : size_t«interface»SamplingStrategySample(locations : const std::vector<Location>&) : std::vector<Location>UniformSamplingStrategysample_size_ : size_tSample(locations : const std::vector<Location>&) : std::vector<Location>«interface»BeerSelectionStrategySelectStyles(brewery : const GeneratedBrewery&,palette : std::span<const BeerStyle>) : std::vector<BeerStyle>RandomBeerSelectionStrategyrng_ : std::mt19937min_beers_ : size_tmax_beers_ : size_tSelectStyles(brewery : const GeneratedBrewery&,palette : std::span<const BeerStyle>) : std::vector<BeerStyle>«interface»CheckinDistributionStrategyAssignActivityWeights(users : std::vector<GeneratedUser>&) : voidCheckinsForUser(user : const GeneratedUser&,brewery_count : size_t) : size_tTimestampFor(user : const GeneratedUser&,index : size_t) : std::stringJCurveCheckinStrategyrng_ : std::mt19937AssignActivityWeights(users : std::vector<GeneratedUser>&) : voidCheckinsForUser(user : const GeneratedUser&,brewery_count : size_t) : size_tTimestampFor(user : const GeneratedUser&,index : size_t) : std::stringRandomCheckinStrategyrng_ : std::mt19937min_checkins_ : size_tmax_checkins_ : size_tAssignActivityWeights(users : std::vector<GeneratedUser>&) : voidCheckinsForUser(user : const GeneratedUser&,brewery_count : size_t) : size_tTimestampFor(user : const GeneratedUser&,index : size_t) : std::string«interface»FollowGenerationStrategyGenerateFollows(users : const std::vector<GeneratedUser>&) : std::vector<GeneratedFollow>RandomFollowStrategyrng_ : std::mt19937min_follows_ : size_tmax_follows_ : size_tGenerateFollows(users : const std::vector<GeneratedUser>&) : std::vector<GeneratedFollow>ActivityWeightedFollowStrategyrng_ : std::mt19937min_follows_ : size_tmax_follows_ : size_tGenerateFollows(users : const std::vector<GeneratedUser>&) : std::vector<GeneratedFollow>LogLevelDebugInfoWarnErrorPipelinePhaseStartupUserGenerationBreweryAndBeerGenerationCheckinGenerationRatingGenerationFollowGenerationTeardownLogEntrytimestamp : std::chrono::system_clock::time_pointlevel : LogLevelphase : PipelinePhasemessage : std::stringworker : std::optional<std::string>«interface»ILoggerLog(entry : const LogEntry&) : voidLogProducerchannel_ : BoundedChannel<LogEntry>&Log(entry : const LogEntry&) : voidLogDispatcherchannel_ : BoundedChannel<LogEntry>&Run() : voidToSpdlogLevel(level) : spdlog::level::level_enumBoundedChannelTqueue_ : std::queue<T>mutex_ : std::mutexnot_full_ : std::condition_variablenot_empty_ : std::condition_variablecapacity_ : size_tclosed_ : boolSend(item : T) : voidReceive() : std::optional<T>Close() : void«interface»ICuratedDataServiceLoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&JsonLoaderLoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&Each Load* call is memoized after its firstparse -- implemented today, ahead of the restof this package (LoadBeerStyles is still planned).MockCuratedDataServiceLoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&Fixed in-memory dataset used in --mocked mode,implemented today for Locations/Personas/Forenames/Surnames; would need a LoadBeerStylesfixture too once that method is implemented.«interface»EnrichmentServiceGetLocationContext(loc : const Location&,strategy : const ContextStrategy&) : LocationContextWikipediaServiceclient_ : std::unique_ptr<WebClient>extract_cache_ : std::unordered_map<std::string, std::string>GetLocationContext(loc : const Location&,strategy : const ContextStrategy&) : LocationContextFetchExtract(query : std::string_view) : std::string«interface»WebClientGet(url : const std::string&) : std::stringUrlEncode(value : const std::string&) : std::stringHttpWebClientGet(url : const std::string&) : std::stringUrlEncode(value : const std::string&) : std::string«interface»IPromptDirectoryLoad(key : std::string_view) : std::stringPromptDirectoryprompt_dir_ : std::filesystem::pathcache_ : std::unordered_map<std::string, std::string>PromptDirectory(prompt_dir : const std::filesystem::path&)Load(key : std::string_view) : std::string«interface»DataGeneratorGenerateBrewery(location : const Location&,context : const LocationContext&) : BreweryResultGenerateBeer(brewery_id : uint64_t,location : const Location&,context : const LocationContext&,style : const BeerStyle&) : BeerResultGenerateUser(city : const EnrichedCity&,persona : const UserPersona&,name : const Name&) : UserResultGenerateCheckin(user : const GeneratedUser&,brewery : const GeneratedBrewery&,timestamp : const std::string&) : CheckinResultGenerateRating(user : const GeneratedUser&,beer : const GeneratedBeer&,checkin_id : uint64_t) : RatingResultMockGeneratorGenerateBrewery(...) : BreweryResultGenerateBeer(...) : BeerResultGenerateUser(...) : UserResultGenerateCheckin(...) : CheckinResultGenerateRating(...) : RatingResultDeterministicHash(location : const Location&) : size_tLlamaGeneratormodel_ : ModelHandlecontext_ : ContextHandleprompt_formatter_ : std::unique_ptr<PromptFormatter>prompt_directory_ : std::unique_ptr<IPromptDirectory>rng_ : std::mt19937GenerateBrewery(...) : BreweryResultGenerateBeer(...) : BeerResultGenerateUser(...) : UserResultGenerateCheckin(...) : CheckinResultGenerateRating(...) : RatingResultLoad(opts : const GeneratorOptions&) : voidInfer(system_prompt, user_prompt,max_tokens, grammar) : std::stringValidateModelArchitecture() : void«interface»PromptFormatterFormat(system_prompt : std::string_view,user_prompt : std::string_view) : std::stringExpectedArchitecture() : std::string_viewGemma4JinjaPromptFormatterFormat(...) : std::stringExpectedArchitecture() : std::string_view«interface»ExportServiceInitialize() : voidProcessBrewery(brewery : const GeneratedBrewery&) : uint64_tProcessBeer(beer : const GeneratedBeer&) : uint64_tProcessUser(user : const GeneratedUser&) : uint64_tProcessCheckin(checkin : const GeneratedCheckin&) : uint64_tProcessRating(rating : const GeneratedRating&) : voidProcessFollow(follow : const GeneratedFollow&) : voidFinalize() : voidSqliteExportServicedate_time_provider_ : std::unique_ptr<DateTimeProvider>db_handle_ : SqliteDatabaseHandleinsert_location_stmt_ : SqliteStatementHandleinsert_brewery_stmt_ : SqliteStatementHandleinsert_beer_stmt_ : SqliteStatementHandleinsert_user_stmt_ : SqliteStatementHandleinsert_checkin_stmt_ : SqliteStatementHandleinsert_rating_stmt_ : SqliteStatementHandleinsert_follow_stmt_ : SqliteStatementHandletransaction_open_ : boollocation_cache_ : std::unordered_map<std::string, uint64_t>brewery_cache_ : std::unordered_map<std::string, uint64_t>Initialize() : voidProcessRecord(brewery : const GeneratedBrewery&) : uint64_tProcessRecord(beer : const GeneratedBeer&) : uint64_tProcessRecord(user : const GeneratedUser&) : uint64_tProcessRecord(checkin : const GeneratedCheckin&) : uint64_tProcessRecord(rating : const GeneratedRating&) : voidProcessRecord(follow : const GeneratedFollow&) : voidFinalize() : voidInitializeSchema() : voidPrepareStatements() : voidRollbackAndCloseNoThrow() : voidFinalizeStatements() : void«interface»DateTimeProviderGetUtcTimestamp() : std::stringSystemDateTimeProviderGetUtcTimestamp() : std::stringBiergartenPipelineOrchestratorcurated_data_service_ : std::unique_ptr<ICuratedDataService>enrichment_service_ : std::unique_ptr<EnrichmentService>generator_ : std::unique_ptr<DataGenerator>logger_ : std::unique_ptr<Logger>exporter_ : std::unique_ptr<ExportService>brewery_context_strategy_ : std::unique_ptr<ContextStrategy>sampling_strategy_ : std::unique_ptr<SamplingStrategy>beer_selection_strategy_ : std::unique_ptr<BeerSelectionStrategy>checkin_strategy_ : std::unique_ptr<CheckinDistributionStrategy>follow_strategy_ : std::unique_ptr<FollowGenerationStrategy>beer_style_palette_ : std::vector<BeerStyle>options_ : ApplicationOptionsuser_pool_ : std::vector<GeneratedUser>brewery_pool_ : std::vector<GeneratedBrewery>beer_pool_ : std::vector<GeneratedBeer>checkin_pool_ : std::vector<GeneratedCheckin>follow_pool_ : std::vector<GeneratedFollow>Run() : boolRunUserPhase(locations : const std::vector<Location>&) : voidRunBreweryAndBeerPhase(locations : const std::vector<Location>&) : voidRunCheckinPhase() : voidRunRatingPhase() : voidRunFollowPhase() : voidLoggerPipelineLoggerLogWorkerSampleName() producesemitsconsumesuser_pool_0..*brewery_pool_0..*beer_pool_0..*checkin_pool_0..*follow_pool_0..*logs todrains from \ No newline at end of file +Biergarten Data Pipeline — Class DiagramDomain: ModelsDomain: Application ConfigurationDomain: PolicyInfrastructure: LoggingInfrastructure: Pipeline ChannelInfrastructure: Data PreloadingInfrastructure: EnrichmentInfrastructure: Postal Code GenerationInfrastructure: PromptingInfrastructure: Data GenerationInfrastructure: Data ExportCitycity : std::stringstate_province : std::stringiso3166_2 : std::stringcountry : std::stringiso3166_1 : std::stringpostal_regex : std::vector<std::string>postal_code_examples : std::vector<std::string>local_languages : std::vector<std::string>latitude : doublelongitude : doublepostal_regex / postal_code_examples aresourced from each cities.json entry'spostal_code.city_regex / postal_code.examples(implemented today, ahead of the rest of thisdiagram -- see ROADMAP.md §1/§2). Renamed fromLocation to City to match the web backend'sCity/CityId vocabulary -- see ROADMAP.md §1.Addresspostal_code : std::stringMirrors the web backend's BreweryPostLocation.Only postal_code is populated today --address_line1/address_line2 have no fixturedata or generator yet. See ROADMAP.md §9.LocationContexttext : std::stringcompleteness : Completenesschar_count : size_tCompletenessFullPartialAbsentEnrichedCitylocation : Citycontext : LocationContextBeerStylename : std::stringdescription : std::stringmin_abv : floatmax_abv : floatmin_ibu : intmax_ibu : intBreweryResultname_en : std::stringdescription_en : std::stringname_local : std::stringdescription_local : std::stringBeerResultname_en : std::stringdescription_en : std::stringname_local : std::stringdescription_local : std::stringstyle : std::stringabv : floatibu : intUserResultfirst_name : std::stringlast_name : std::stringgender : std::stringusername : std::stringbio : std::stringactivity_weight : floatNamefirst_name : std::stringlast_name : std::stringgender : std::stringForenameEntryname : std::stringgender : std::stringforename_list = std::unordered_set<ForenameEntry>surname_list = std::unordered_set<std::string>(aliases declared in curated_data_service.h).ICuratedDataService::LoadForenamesByCountry() /LoadSurnamesByCountry() each return a flatstd::unordered_map<std::string, forename_list>or <std::string, surname_list> keyed by ISO3166-1 country code -- no NamesByCountrywrapper class. SampleName(forenames_by_country,surnames_by_country, iso3166_1, rng) is a freefunction, not a method, so it can be called byany per-city worker without owning the maps.Sourced from popular-names-by-country-dataset(CC0) -- see ETHICS-AND-KNOWN-ISSUES.md. Genderis preserved per forename so it can be threadedthrough to persona/bio generation later; surnamescarry no gender in the source data. Pairinghappens at sample time, not at fixture-authoringtime, so no information from the source datasetis discarded up front.CheckinResultchecked_in_at : std::stringnote : std::stringRatingResultscore : floatnote : std::stringGenerationMetadatageneration_id : uint64_tgenerated_time : std::stringcontext_provided : boolgenerated_with : std::stringGeneratedBrewerybrewery_id : uint64_tlocation : Cityaddress : Addressbrewery : BreweryResultcontext_completeness : LocationContext::Completenessmetadata : GenerationMetadataGeneratedBeerbeer_id : uint64_tbrewery_id : uint64_tlocation : Citystyle : BeerStylebeer : BeerResultmetadata : GenerationMetadataGeneratedUseruser_id : uint64_tlocation : Cityuser : UserResultemail : std::stringdate_of_birth : std::stringpassword : std::stringmetadata : GenerationMetadataemail, date_of_birth, and password areprogrammatically generated by the orchestrator(not LLM-authored) so a downstream auth-accountseeding consumer can register real accounts fromthis export. See ROADMAP.md §1.GeneratedCheckincheckin_id : uint64_tuser_id : uint64_tbrewery_id : uint64_tcheckin : CheckinResultmetadata : GenerationMetadataGeneratedRatinguser_id : uint64_tbeer_id : uint64_tcheckin_id : uint64_trating : RatingResultmetadata : GenerationMetadataGeneratedFollowfollower_id : uint64_tfollowed_id : uint64_tmetadata : GenerationMetadataUserPersonaname: std::stringdescription: std::stringstyle_affinities: std::vector<std::string>SamplingOptionstemperature: float = 1.0Ftop_p: float = 0.95Ftop_k: uint32_t = 64n_ctx: uint32_t = 8192seed: int = -1GeneratorOptionsmodel_path: std::filesystem::pathuse_mocked: bool = falsesampling: std::optional<SamplingOptions>PipelineOptionsoutput_path: std::filesystem::pathlog_path: std::filesystem::pathApplicationOptionsgenerator: GeneratorOptionspipeline: PipelineOptions«interface»ContextStrategyQueriesFor(loc : const City&) : std::vector<std::string>MaxContextChars() : size_tBreweryContextStrategyQueriesFor(loc : const City&) : std::vector<std::string>MaxContextChars() : size_tBeerContextStrategyQueriesFor(loc : const City&) : std::vector<std::string>MaxContextChars() : size_t«interface»SamplingStrategySample(locations : const std::vector<City>&) : std::vector<City>UniformSamplingStrategysample_size_ : size_tSample(locations : const std::vector<City>&) : std::vector<City>«interface»BeerSelectionStrategySelectStyles(brewery : const GeneratedBrewery&,palette : std::span<const BeerStyle>) : std::vector<BeerStyle>RandomBeerSelectionStrategyrng_ : std::mt19937min_beers_ : size_tmax_beers_ : size_tSelectStyles(brewery : const GeneratedBrewery&,palette : std::span<const BeerStyle>) : std::vector<BeerStyle>«interface»CheckinDistributionStrategyAssignActivityWeights(users : std::vector<GeneratedUser>&) : voidCheckinsForUser(user : const GeneratedUser&,brewery_count : size_t) : size_tTimestampFor(user : const GeneratedUser&,index : size_t) : std::stringJCurveCheckinStrategyrng_ : std::mt19937AssignActivityWeights(users : std::vector<GeneratedUser>&) : voidCheckinsForUser(user : const GeneratedUser&,brewery_count : size_t) : size_tTimestampFor(user : const GeneratedUser&,index : size_t) : std::stringRandomCheckinStrategyrng_ : std::mt19937min_checkins_ : size_tmax_checkins_ : size_tAssignActivityWeights(users : std::vector<GeneratedUser>&) : voidCheckinsForUser(user : const GeneratedUser&,brewery_count : size_t) : size_tTimestampFor(user : const GeneratedUser&,index : size_t) : std::string«interface»FollowGenerationStrategyGenerateFollows(users : const std::vector<GeneratedUser>&) : std::vector<GeneratedFollow>RandomFollowStrategyrng_ : std::mt19937min_follows_ : size_tmax_follows_ : size_tGenerateFollows(users : const std::vector<GeneratedUser>&) : std::vector<GeneratedFollow>ActivityWeightedFollowStrategyrng_ : std::mt19937min_follows_ : size_tmax_follows_ : size_tGenerateFollows(users : const std::vector<GeneratedUser>&) : std::vector<GeneratedFollow>LogLevelDebugInfoWarnErrorPipelinePhaseStartupUserGenerationBreweryAndBeerGenerationCheckinGenerationRatingGenerationFollowGenerationTeardownLogEntrytimestamp : std::chrono::system_clock::time_pointlevel : LogLevelphase : PipelinePhasemessage : std::stringworker : std::optional<std::string>«interface»ILoggerLog(entry : const LogEntry&) : voidLogProducerchannel_ : BoundedChannel<LogEntry>&Log(entry : const LogEntry&) : voidLogDispatcherchannel_ : BoundedChannel<LogEntry>&Run() : voidToSpdlogLevel(level) : spdlog::level::level_enumBoundedChannelTqueue_ : std::queue<T>mutex_ : std::mutexnot_full_ : std::condition_variablenot_empty_ : std::condition_variablecapacity_ : size_tclosed_ : boolSend(item : T) : voidReceive() : std::optional<T>Close() : void«interface»ICuratedDataServiceLoadCities(filepath : const std::filesystem::path&) : const std::vector<City>&LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&JsonLoaderLoadCities(filepath : const std::filesystem::path&) : const std::vector<City>&LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&Each Load* call is memoized after its firstparse -- implemented today, ahead of the restof this package (LoadBeerStyles is still planned).MockCuratedDataServiceLoadCities(filepath : const std::filesystem::path&) : const std::vector<City>&LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&Fixed in-memory dataset used in --mocked mode,implemented today for Locations/Personas/Forenames/Surnames; would need a LoadBeerStylesfixture too once that method is implemented.«interface»EnrichmentServiceGetLocationContext(loc : const City&,strategy : const ContextStrategy&) : LocationContextWikipediaServiceclient_ : std::unique_ptr<WebClient>extract_cache_ : std::unordered_map<std::string, std::string>GetLocationContext(loc : const City&,strategy : const ContextStrategy&) : LocationContextFetchExtract(query : std::string_view) : std::string«interface»WebClientGet(url : const std::string&) : std::stringUrlEncode(value : const std::string&) : std::stringHttpWebClientGet(url : const std::string&) : std::stringUrlEncode(value : const std::string&) : std::string«interface»IPostalCodeServiceGeneratePostalCode(location : const City&) : std::stringMockPostalCodeServiceGeneratePostalCode(location : const City&) : std::stringXegerPostalCodeServiceGeneratePostalCode(location : const City&) : std::stringImplemented today: ignores postal_regex andreturns postal_code_examples.front(). Not yetbound in main.cc's DI graph or used byBiergartenPipelineOrchestrator.Not implemented yet. Picks one oflocation.postal_regex and generates a randomstring matching it (a "xeger" -- the inverse ofregex matching) instead of always replaying afixed example. See ROADMAP.md §9.«interface»IPromptDirectoryLoad(key : std::string_view) : std::stringPromptDirectoryprompt_dir_ : std::filesystem::pathcache_ : std::unordered_map<std::string, std::string>PromptDirectory(prompt_dir : const std::filesystem::path&)Load(key : std::string_view) : std::string«interface»DataGeneratorGenerateBrewery(location : const City&,context : const LocationContext&) : BreweryResultGenerateBeer(brewery_id : uint64_t,location : const City&,context : const LocationContext&,style : const BeerStyle&) : BeerResultGenerateUser(city : const EnrichedCity&,persona : const UserPersona&,name : const Name&) : UserResultGenerateCheckin(user : const GeneratedUser&,brewery : const GeneratedBrewery&,timestamp : const std::string&) : CheckinResultGenerateRating(user : const GeneratedUser&,beer : const GeneratedBeer&,checkin_id : uint64_t) : RatingResultMockGeneratorGenerateBrewery(...) : BreweryResultGenerateBeer(...) : BeerResultGenerateUser(...) : UserResultGenerateCheckin(...) : CheckinResultGenerateRating(...) : RatingResultDeterministicHash(location : const City&) : size_tLlamaGeneratormodel_ : ModelHandlecontext_ : ContextHandleprompt_formatter_ : std::unique_ptr<PromptFormatter>prompt_directory_ : std::unique_ptr<IPromptDirectory>rng_ : std::mt19937GenerateBrewery(...) : BreweryResultGenerateBeer(...) : BeerResultGenerateUser(...) : UserResultGenerateCheckin(...) : CheckinResultGenerateRating(...) : RatingResultLoad(opts : const GeneratorOptions&) : voidInfer(system_prompt, user_prompt,max_tokens, grammar) : std::stringValidateModelArchitecture() : void«interface»PromptFormatterFormat(system_prompt : std::string_view,user_prompt : std::string_view) : std::stringExpectedArchitecture() : std::string_viewGemma4JinjaPromptFormatterFormat(...) : std::stringExpectedArchitecture() : std::string_view«interface»ExportServiceInitialize() : voidProcessBrewery(brewery : const GeneratedBrewery&) : uint64_tProcessBeer(beer : const GeneratedBeer&) : uint64_tProcessUser(user : const GeneratedUser&) : uint64_tProcessCheckin(checkin : const GeneratedCheckin&) : uint64_tProcessRating(rating : const GeneratedRating&) : voidProcessFollow(follow : const GeneratedFollow&) : voidFinalize() : voidSqliteExportServicedate_time_provider_ : std::unique_ptr<DateTimeProvider>db_handle_ : SqliteDatabaseHandleinsert_location_stmt_ : SqliteStatementHandleinsert_brewery_stmt_ : SqliteStatementHandleinsert_beer_stmt_ : SqliteStatementHandleinsert_user_stmt_ : SqliteStatementHandleinsert_checkin_stmt_ : SqliteStatementHandleinsert_rating_stmt_ : SqliteStatementHandleinsert_follow_stmt_ : SqliteStatementHandletransaction_open_ : boollocation_cache_ : std::unordered_map<std::string, uint64_t>brewery_cache_ : std::unordered_map<std::string, uint64_t>Initialize() : voidProcessRecord(brewery : const GeneratedBrewery&) : uint64_tProcessRecord(beer : const GeneratedBeer&) : uint64_tProcessRecord(user : const GeneratedUser&) : uint64_tProcessRecord(checkin : const GeneratedCheckin&) : uint64_tProcessRecord(rating : const GeneratedRating&) : voidProcessRecord(follow : const GeneratedFollow&) : voidFinalize() : voidInitializeSchema() : voidPrepareStatements() : voidRollbackAndCloseNoThrow() : voidFinalizeStatements() : void«interface»DateTimeProviderGetUtcTimestamp() : std::stringSystemDateTimeProviderGetUtcTimestamp() : std::stringBiergartenPipelineOrchestratorcurated_data_service_ : std::unique_ptr<ICuratedDataService>enrichment_service_ : std::unique_ptr<EnrichmentService>generator_ : std::unique_ptr<DataGenerator>logger_ : std::unique_ptr<Logger>exporter_ : std::unique_ptr<ExportService>brewery_context_strategy_ : std::unique_ptr<ContextStrategy>sampling_strategy_ : std::unique_ptr<SamplingStrategy>beer_selection_strategy_ : std::unique_ptr<BeerSelectionStrategy>checkin_strategy_ : std::unique_ptr<CheckinDistributionStrategy>follow_strategy_ : std::unique_ptr<FollowGenerationStrategy>beer_style_palette_ : std::vector<BeerStyle>options_ : ApplicationOptionsuser_pool_ : std::vector<GeneratedUser>brewery_pool_ : std::vector<GeneratedBrewery>beer_pool_ : std::vector<GeneratedBeer>checkin_pool_ : std::vector<GeneratedCheckin>follow_pool_ : std::vector<GeneratedFollow>Run() : boolRunUserPhase(locations : const std::vector<City>&) : voidRunBreweryAndBeerPhase(locations : const std::vector<City>&) : voidRunCheckinPhase() : voidRunRatingPhase() : voidRunFollowPhase() : voidLoggerPipelineLoggerLogWorkerSampleName() producesemitsconsumesuser_pool_0..*brewery_pool_0..*beer_pool_0..*checkin_pool_0..*follow_pool_0..*logs todrains from \ No newline at end of file diff --git a/tooling/pipeline/CMakeLists.txt b/tooling/pipeline/CMakeLists.txt index d37bfea..4a25f15 100644 --- a/tooling/pipeline/CMakeLists.txt +++ b/tooling/pipeline/CMakeLists.txt @@ -140,6 +140,8 @@ FetchContent_MakeAvailable(cpp-httplib) add_executable(${PROJECT_NAME} includes/services/enrichment/mock_enrichment.h includes/services/curated_data/mock_curated_data_service.h + includes/services/postal_code/postal_code_service.h + includes/services/postal_code/mock_postal_code_service.h includes/json_handling/pretty_print.h) # --- Entry point --- @@ -264,8 +266,8 @@ target_compile_options(biergarten-pipeline PRIVATE # 7. Runtime Assets configure_file( - ${CMAKE_SOURCE_DIR}/locations.json - ${CMAKE_BINARY_DIR}/locations.json + ${CMAKE_SOURCE_DIR}/cities.json + ${CMAKE_BINARY_DIR}/cities.json COPYONLY ) configure_file( @@ -288,4 +290,3 @@ add_custom_command(TARGET ${PROJECT_NAME} POST_BUILD ${CMAKE_SOURCE_DIR}/prompts ${CMAKE_BINARY_DIR}/prompts ) - diff --git a/tooling/pipeline/cities.json b/tooling/pipeline/cities.json new file mode 100644 index 0000000..f65b804 --- /dev/null +++ b/tooling/pipeline/cities.json @@ -0,0 +1,1718 @@ +[ + { + "city": "Cape Town", + "state_province": "Western Cape", + "iso3166_2": "ZA-WC", + "country": "South Africa", + "iso3166_1": "ZA", + "latitude": -33.9249, + "longitude": 18.4241, + "local_languages": ["af", "en", "xh"], + "postal_code": { + "country_format_regex": "^\\d{4}$", + "city_regex": ["^(?:7[0-9]{3}|80[0-9]{2})$"], + "examples": ["8001", "7700", "7441"], + "confidence": "medium", + "notes": "Cape Town metro spans roughly 7000-8099; 7xxx also covers parts of the wider Western Cape." + } + }, + { + "city": "Johannesburg", + "state_province": "Gauteng", + "iso3166_2": "ZA-GT", + "country": "South Africa", + "iso3166_1": "ZA", + "latitude": -26.2041, + "longitude": 28.0473, + "local_languages": ["en", "zu", "st", "af"], + "postal_code": { + "country_format_regex": "^\\d{4}$", + "city_regex": ["^2[01][0-9]{2}$"], + "examples": ["2001", "2196"], + "confidence": "medium", + "notes": "Johannesburg street codes cluster in 2000-2199; PO Box codes differ." + } + }, + { + "city": "Durban", + "state_province": "KwaZulu-Natal", + "iso3166_2": "ZA-NL", + "country": "South Africa", + "iso3166_1": "ZA", + "latitude": -29.8587, + "longitude": 31.0218, + "local_languages": ["zu", "en"], + "postal_code": { + "country_format_regex": "^\\d{4}$", + "city_regex": ["^4[01][0-9]{2}$"], + "examples": ["4001", "4051"], + "confidence": "medium", + "notes": "eThekwini/Durban street codes cluster in 4000-4199." + } + }, + { + "city": "Franschhoek", + "state_province": "Western Cape", + "iso3166_2": "ZA-WC", + "country": "South Africa", + "iso3166_1": "ZA", + "latitude": -33.9146, + "longitude": 19.1198, + "local_languages": ["af", "en"], + "postal_code": { + "country_format_regex": "^\\d{4}$", + "city_regex": ["^7690$"], + "examples": ["7690"], + "confidence": "high", + "notes": "Single town code." + } + }, + { + "city": "Nairobi", + "state_province": "Nairobi", + "iso3166_2": "KE-30", + "country": "Kenya", + "iso3166_1": "KE", + "latitude": -1.2921, + "longitude": 36.8219, + "local_languages": ["sw", "en"], + "postal_code": { + "country_format_regex": "^\\d{5}$", + "city_regex": ["^00[1-9][0-9]{2}$"], + "examples": ["00100", "00200", "00800"], + "confidence": "high", + "notes": "All Nairobi County codes begin 00; GPO is 00100. Range 00100-00999." + } + }, + { + "city": "Buenos Aires", + "state_province": "Buenos Aires City", + "iso3166_2": "AR-C", + "country": "Argentina", + "iso3166_1": "AR", + "latitude": -34.6037, + "longitude": -58.3816, + "local_languages": ["es-AR"], + "postal_code": { + "country_format_regex": "^(?:[A-HJ-NP-TV-Z]\\d{4}[A-Z]{3}|\\d{4})$", + "city_regex": ["^C1[0-4][0-9]{2}[A-Z]{3}$", "^1[0-4][0-9]{2}$"], + "examples": ["C1425DKE", "1425"], + "confidence": "high", + "notes": "CABA uses CPA prefix 'C'. Legacy 4-digit form 1000-1499 still widely accepted." + } + }, + { + "city": "Bariloche", + "state_province": "Río Negro", + "iso3166_2": "AR-R", + "country": "Argentina", + "iso3166_1": "AR", + "latitude": -41.1335, + "longitude": -71.3103, + "local_languages": ["es-AR"], + "postal_code": { + "country_format_regex": "^(?:[A-HJ-NP-TV-Z]\\d{4}[A-Z]{3}|\\d{4})$", + "city_regex": ["^R8400[A-Z]{3}$", "^8400$"], + "examples": ["R8400AGC", "8400"], + "confidence": "high", + "notes": "San Carlos de Bariloche legacy code 8400; CPA prefix 'R' (Rio Negro)." + } + }, + { + "city": "Bogotá", + "state_province": "Bogotá D.C.", + "iso3166_2": "CO-DC", + "country": "Colombia", + "iso3166_1": "CO", + "latitude": 4.711, + "longitude": -74.0721, + "local_languages": ["es-CO"], + "postal_code": { + "country_format_regex": "^\\d{6}$", + "city_regex": ["^11[0-9]{4}$"], + "examples": ["110111", "111321"], + "confidence": "high", + "notes": "Bogota D.C. = DANE department code 11." + } + }, + { + "city": "Medellín", + "state_province": "Antioquia", + "iso3166_2": "CO-ANT", + "country": "Colombia", + "iso3166_1": "CO", + "latitude": 6.2442, + "longitude": -75.5812, + "local_languages": ["es-CO"], + "postal_code": { + "country_format_regex": "^\\d{6}$", + "city_regex": ["^050[0-9]{3}$"], + "examples": ["050001", "050022"], + "confidence": "high", + "notes": "Antioquia = 05; Medellin zone = 050xxx. Wider metro uses 051xxx-055xxx." + } + }, + { + "city": "São Paulo", + "state_province": "São Paulo", + "iso3166_2": "BR-SP", + "country": "Brazil", + "iso3166_1": "BR", + "latitude": -23.5505, + "longitude": -46.6333, + "local_languages": ["pt-BR"], + "postal_code": { + "country_format_regex": "^\\d{5}-?\\d{3}$", + "city_regex": [ + "^0[1-5][0-9]{3}-?[0-9]{3}$", + "^08[0-4][0-9]{2}-?[0-9]{3}$" + ], + "examples": ["01310-100", "05407-002"], + "confidence": "high", + "notes": "Sao Paulo city CEP: 01000-000 to 05999-999, plus 08000-000 to 08499-999." + } + }, + { + "city": "Curitiba", + "state_province": "Paraná", + "iso3166_2": "BR-PR", + "country": "Brazil", + "iso3166_1": "BR", + "latitude": -25.4284, + "longitude": -49.2733, + "local_languages": ["pt-BR"], + "postal_code": { + "country_format_regex": "^\\d{5}-?\\d{3}$", + "city_regex": ["^8[0-2][0-9]{3}-?[0-9]{3}$"], + "examples": ["80010-010", "82590-300"], + "confidence": "high", + "notes": "Curitiba CEP: 80000-000 to 82999-999." + } + }, + { + "city": "Rio de Janeiro", + "state_province": "Rio de Janeiro", + "iso3166_2": "BR-RJ", + "country": "Brazil", + "iso3166_1": "BR", + "latitude": -22.9068, + "longitude": -43.1729, + "local_languages": ["pt-BR"], + "postal_code": { + "country_format_regex": "^\\d{5}-?\\d{3}$", + "city_regex": ["^(?:2[0-2][0-9]{3}|23[0-7][0-9]{2})-?[0-9]{3}$"], + "examples": ["20040-020", "22071-900"], + "confidence": "high", + "notes": "Rio city CEP: 20000-000 to 23799-999." + } + }, + { + "city": "Santiago", + "state_province": "Santiago Metropolitan", + "iso3166_2": "CL-RM", + "country": "Chile", + "iso3166_1": "CL", + "latitude": -33.4489, + "longitude": -70.6693, + "local_languages": ["es-CL"], + "postal_code": { + "country_format_regex": "^\\d{7}$", + "city_regex": ["^(?:7[5-9][0-9]|8[0-9]{2}|9[0-2][0-9])[0-9]{4}$"], + "examples": ["8320000", "7500000"], + "confidence": "medium", + "notes": "Santiago Province spans approx. 7500000-9250000; 7xx = eastern communes, 8xx = central." + } + }, + { + "city": "Valdivia", + "state_province": "Los Ríos", + "iso3166_2": "CL-LR", + "country": "Chile", + "iso3166_1": "CL", + "latitude": -39.8142, + "longitude": -73.2459, + "local_languages": ["es-CL"], + "postal_code": { + "country_format_regex": "^\\d{7}$", + "city_regex": ["^(?:509|51[0-9]|52[0-3])[0-9]{4}$"], + "examples": ["5090000", "5110000"], + "confidence": "low", + "notes": "Los Rios region range approx. 5090000-5230000; Valdivia commune base code 5090000." + } + }, + { + "city": "Lima", + "state_province": "Lima", + "iso3166_2": "PE-LMA", + "country": "Peru", + "iso3166_1": "PE", + "latitude": -12.0464, + "longitude": -77.0428, + "local_languages": ["es-PE"], + "postal_code": { + "country_format_regex": "^\\d{5}$", + "city_regex": ["^15[0-9]{3}$"], + "examples": ["15001", "15074"], + "confidence": "medium", + "notes": "Lima Region = prefix 15; metropolitan Lima districts sit in 15001-15099. Callao is 07xxx." + } + }, + { + "city": "Tokyo", + "state_province": "Tokyo", + "iso3166_2": "JP-13", + "country": "Japan", + "iso3166_1": "JP", + "latitude": 35.6762, + "longitude": 139.6503, + "local_languages": ["ja"], + "postal_code": { + "country_format_regex": "^\\d{3}-?\\d{4}$", + "city_regex": ["^(?:1[0-9]{2}|20[0-8])-?[0-9]{4}$"], + "examples": ["100-0001", "150-0002"], + "confidence": "high", + "notes": "Tokyo Metropolis: 100-xxxx to 208-xxxx. The 23 wards are 100-179." + } + }, + { + "city": "Osaka", + "state_province": "Osaka", + "iso3166_2": "JP-27", + "country": "Japan", + "iso3166_1": "JP", + "latitude": 34.6937, + "longitude": 135.5023, + "local_languages": ["ja"], + "postal_code": { + "country_format_regex": "^\\d{3}-?\\d{4}$", + "city_regex": ["^5[3-5][0-9]-?[0-9]{4}$"], + "examples": ["530-0001", "559-0034"], + "confidence": "high", + "notes": "Osaka City: 530-xxxx to 559-xxxx. Wider Osaka Prefecture is 5xx." + } + }, + { + "city": "Kyoto", + "state_province": "Kyoto", + "iso3166_2": "JP-26", + "country": "Japan", + "iso3166_1": "JP", + "latitude": 35.0116, + "longitude": 135.7681, + "local_languages": ["ja"], + "postal_code": { + "country_format_regex": "^\\d{3}-?\\d{4}$", + "city_regex": ["^6(?:0[0-9]|1[0-6])-?[0-9]{4}$"], + "examples": ["600-8216", "606-8501"], + "confidence": "high", + "notes": "Kyoto City: 600-xxxx to 616-xxxx." + } + }, + { + "city": "Sapporo", + "state_province": "Hokkaido", + "iso3166_2": "JP-01", + "country": "Japan", + "iso3166_1": "JP", + "latitude": 43.0618, + "longitude": 141.3545, + "local_languages": ["ja"], + "postal_code": { + "country_format_regex": "^\\d{3}-?\\d{4}$", + "city_regex": ["^(?:00[1-7]|06[0-5])-?[0-9]{4}$"], + "examples": ["060-0001", "001-0010"], + "confidence": "high", + "notes": "Sapporo: 001-xxxx to 007-xxxx (north/west) and 060-xxxx to 065-xxxx (central/east)." + } + }, + { + "city": "Seoul", + "state_province": "Seoul", + "iso3166_2": "KR-11", + "country": "South Korea", + "iso3166_1": "KR", + "latitude": 37.5665, + "longitude": 126.978, + "local_languages": ["ko"], + "postal_code": { + "country_format_regex": "^\\d{5}$", + "city_regex": ["^0[1-9][0-9]{3}$"], + "examples": ["04524", "06236"], + "confidence": "high", + "notes": "Post-2015 5-digit system: Seoul = 01000-09999." + } + }, + { + "city": "Busan", + "state_province": "Busan", + "iso3166_2": "KR-26", + "country": "South Korea", + "iso3166_1": "KR", + "latitude": 35.1796, + "longitude": 129.0756, + "local_languages": ["ko"], + "postal_code": { + "country_format_regex": "^\\d{5}$", + "city_regex": ["^4[6-9][0-9]{3}$"], + "examples": ["48058", "46241"], + "confidence": "high", + "notes": "Busan = 46000-49999." + } + }, + { + "city": "Ho Chi Minh City", + "state_province": "Ho Chi Minh", + "iso3166_2": "VN-SG", + "country": "Vietnam", + "iso3166_1": "VN", + "latitude": 10.8231, + "longitude": 106.6297, + "local_languages": ["vi"], + "postal_code": { + "country_format_regex": "^\\d{5,6}$", + "city_regex": ["^7[0-4][0-9]{3}$", "^7[0-4][0-9]{4}$"], + "examples": ["70000", "71000", "700000"], + "confidence": "medium", + "notes": "Current 5-digit system: HCMC = 70000-74999. Legacy/6-digit form (700000) still appears in some systems." + } + }, + { + "city": "Hanoi", + "state_province": "Hanoi", + "iso3166_2": "VN-HN", + "country": "Vietnam", + "iso3166_1": "VN", + "latitude": 21.0285, + "longitude": 105.8542, + "local_languages": ["vi"], + "postal_code": { + "country_format_regex": "^\\d{5,6}$", + "city_regex": ["^1[0-4][0-9]{3}$", "^1[0-4][0-9]{4}$"], + "examples": ["10000", "11100", "100000"], + "confidence": "medium", + "notes": "Hanoi = 10000-14999 (5-digit). Legacy 6-digit form 100000 still in circulation." + } + }, + { + "city": "Da Nang", + "state_province": "Da Nang", + "iso3166_2": "VN-DN", + "country": "Vietnam", + "iso3166_1": "VN", + "latitude": 16.0544, + "longitude": 108.2022, + "local_languages": ["vi"], + "postal_code": { + "country_format_regex": "^\\d{5,6}$", + "city_regex": ["^5[0-5][0-9]{3}$", "^5[0-5][0-9]{4}$"], + "examples": ["50000", "55000", "550000"], + "confidence": "low", + "notes": "Sources conflict: Da Nang is cited as both 50000 and 55000. Pattern accepts 50000-55999 and the 6-digit legacy form." + } + }, + { + "city": "Bangkok", + "state_province": "Bangkok", + "iso3166_2": "TH-10", + "country": "Thailand", + "iso3166_1": "TH", + "latitude": 13.7563, + "longitude": 100.5018, + "local_languages": ["th"], + "postal_code": { + "country_format_regex": "^\\d{5}$", + "city_regex": ["^10[0-9]{3}$"], + "examples": ["10100", "10330"], + "confidence": "high", + "notes": "Bangkok = province code 10; codes run 10100-10900." + } + }, + { + "city": "Taipei", + "state_province": "Taipei", + "iso3166_2": "TW-TPE", + "country": "Taiwan", + "iso3166_1": "TW", + "latitude": 25.033, + "longitude": 121.5654, + "local_languages": ["zh-TW"], + "postal_code": { + "country_format_regex": "^\\d{3}(?:\\d{2}|\\d{3})?$", + "city_regex": ["^1(?:0[0-9]|1[0-6])(?:[0-9]{2}|[0-9]{3})?$"], + "examples": ["100", "10058", "100013"], + "confidence": "high", + "notes": "Taipei City districts = 100-116. 3-digit is valid alone; 3+2 is legacy, 3+3 is current." + } + }, + { + "city": "Beijing", + "state_province": "Beijing", + "iso3166_2": "CN-BJ", + "country": "China", + "iso3166_1": "CN", + "latitude": 39.9042, + "longitude": 116.4074, + "local_languages": ["zh-CN"], + "postal_code": { + "country_format_regex": "^\\d{6}$", + "city_regex": ["^10[0-2][0-9]{3}$"], + "examples": ["100000", "100010"], + "confidence": "high", + "notes": "Beijing municipality = 100000-102629." + } + }, + { + "city": "Shanghai", + "state_province": "Shanghai", + "iso3166_2": "CN-SH", + "country": "China", + "iso3166_1": "CN", + "latitude": 31.2304, + "longitude": 121.4737, + "local_languages": ["zh-CN"], + "postal_code": { + "country_format_regex": "^\\d{6}$", + "city_regex": ["^20[0-2][0-9]{3}$"], + "examples": ["200000", "200080"], + "confidence": "high", + "notes": "Shanghai municipality = 200000-202699." + } + }, + { + "city": "Bengaluru", + "state_province": "Karnataka", + "iso3166_2": "IN-KA", + "country": "India", + "iso3166_1": "IN", + "latitude": 12.9716, + "longitude": 77.5946, + "local_languages": ["kn", "en"], + "postal_code": { + "country_format_regex": "^[1-9]\\d{5}$", + "city_regex": ["^560[0-9]{3}$"], + "examples": ["560001", "560100"], + "confidence": "high", + "notes": "Bengaluru urban PIN codes = 560xxx. Bengaluru Rural uses 561xxx/562xxx." + } + }, + { + "city": "Singapore", + "state_province": "Central Singapore", + "iso3166_2": "SG-01", + "country": "Singapore", + "iso3166_1": "SG", + "latitude": 1.3521, + "longitude": 103.8198, + "local_languages": ["en", "zh", "ms", "ta"], + "postal_code": { + "country_format_regex": "^\\d{6}$", + "city_regex": ["^[0-8][0-9]{5}$"], + "examples": ["018956", "238823"], + "confidence": "high", + "notes": "Singapore is a single postal district; sectors 01-82 cover the whole island." + } + }, + { + "city": "Melbourne", + "state_province": "Victoria", + "iso3166_2": "AU-VIC", + "country": "Australia", + "iso3166_1": "AU", + "latitude": -37.8136, + "longitude": 144.9631, + "local_languages": ["en-AU"], + "postal_code": { + "country_format_regex": "^\\d{4}$", + "city_regex": ["^3(?:0[0-9]{2}|1[0-9]{2}|20[0-7])$"], + "examples": ["3000", "3121"], + "confidence": "medium", + "notes": "Greater Melbourne approx. 3000-3207. Victoria as a whole is 3000-3999." + } + }, + { + "city": "Sydney", + "state_province": "New South Wales", + "iso3166_2": "AU-NSW", + "country": "Australia", + "iso3166_1": "AU", + "latitude": -33.8688, + "longitude": 151.2093, + "local_languages": ["en-AU"], + "postal_code": { + "country_format_regex": "^\\d{4}$", + "city_regex": ["^(?:1[0-9]{3}|2[0-2][0-9]{2})$"], + "examples": ["2000", "2010", "1235"], + "confidence": "medium", + "notes": "Sydney metro approx. 2000-2249; 1000-1999 are Sydney PO Box / large-volume codes." + } + }, + { + "city": "Brisbane", + "state_province": "Queensland", + "iso3166_2": "AU-QLD", + "country": "Australia", + "iso3166_1": "AU", + "latitude": -27.4705, + "longitude": 153.026, + "local_languages": ["en-AU"], + "postal_code": { + "country_format_regex": "^\\d{4}$", + "city_regex": ["^4(?:0[0-9]{2}|1[0-9]{2}|20[0-7])$"], + "examples": ["4000", "4101"], + "confidence": "medium", + "notes": "Greater Brisbane approx. 4000-4207." + } + }, + { + "city": "Adelaide", + "state_province": "South Australia", + "iso3166_2": "AU-SA", + "country": "Australia", + "iso3166_1": "AU", + "latitude": -34.9285, + "longitude": 138.6007, + "local_languages": ["en-AU"], + "postal_code": { + "country_format_regex": "^\\d{4}$", + "city_regex": ["^5[01][0-9]{2}$"], + "examples": ["5000", "5067"], + "confidence": "medium", + "notes": "Adelaide metro approx. 5000-5199." + } + }, + { + "city": "Perth", + "state_province": "Western Australia", + "iso3166_2": "AU-WA", + "country": "Australia", + "iso3166_1": "AU", + "latitude": -31.9505, + "longitude": 115.8605, + "local_languages": ["en-AU"], + "postal_code": { + "country_format_regex": "^\\d{4}$", + "city_regex": ["^6[01][0-9]{2}$"], + "examples": ["6000", "6151"], + "confidence": "medium", + "notes": "Perth metro approx. 6000-6199." + } + }, + { + "city": "Hobart", + "state_province": "Tasmania", + "iso3166_2": "AU-TAS", + "country": "Australia", + "iso3166_1": "AU", + "latitude": -42.8821, + "longitude": 147.3272, + "local_languages": ["en-AU"], + "postal_code": { + "country_format_regex": "^\\d{4}$", + "city_regex": ["^70[0-9]{2}$"], + "examples": ["7000", "7050"], + "confidence": "medium", + "notes": "Hobart approx. 7000-7099." + } + }, + { + "city": "Wellington", + "state_province": "Wellington", + "iso3166_2": "NZ-WGN", + "country": "New Zealand", + "iso3166_1": "NZ", + "latitude": -41.2865, + "longitude": 174.7762, + "local_languages": ["en", "mi"], + "postal_code": { + "country_format_regex": "^\\d{4}$", + "city_regex": ["^60[0-9]{2}$"], + "examples": ["6011", "6021"], + "confidence": "medium", + "notes": "Wellington City = 6011-6037. Wider region extends to 5010-5999." + } + }, + { + "city": "Auckland", + "state_province": "Auckland", + "iso3166_2": "NZ-AUK", + "country": "New Zealand", + "iso3166_1": "NZ", + "latitude": -36.8485, + "longitude": 174.7633, + "local_languages": ["en", "mi"], + "postal_code": { + "country_format_regex": "^\\d{4}$", + "city_regex": ["^(?:0[6-9][0-9]{2}|10[0-9]{2}|2[0-6][0-9]{2})$"], + "examples": ["1010", "0620", "2013"], + "confidence": "low", + "notes": "Auckland is non-contiguous: 0600-0999 (north/west), 1010-1081 (central), 2010-2699 (south)." + } + }, + { + "city": "Christchurch", + "state_province": "Canterbury", + "iso3166_2": "NZ-CAN", + "country": "New Zealand", + "iso3166_1": "NZ", + "latitude": -43.532, + "longitude": 172.6306, + "local_languages": ["en", "mi"], + "postal_code": { + "country_format_regex": "^\\d{4}$", + "city_regex": ["^80[0-9]{2}$"], + "examples": ["8011", "8041"], + "confidence": "medium", + "notes": "Christchurch City = 8011-8062. Canterbury region extends 7600-8999." + } + }, + { + "city": "Nelson", + "state_province": "Nelson", + "iso3166_2": "NZ-NSN", + "country": "New Zealand", + "iso3166_1": "NZ", + "latitude": -41.2706, + "longitude": 173.284, + "local_languages": ["en", "mi"], + "postal_code": { + "country_format_regex": "^\\d{4}$", + "city_regex": ["^70[0-9]{2}$"], + "examples": ["7010", "7011"], + "confidence": "medium", + "notes": "Nelson City = 7010-7011 and 7040-7047." + } + }, + { + "city": "Munich", + "state_province": "Bavaria", + "iso3166_2": "DE-BY", + "country": "Germany", + "iso3166_1": "DE", + "latitude": 48.1351, + "longitude": 11.582, + "local_languages": ["de"], + "postal_code": { + "country_format_regex": "^\\d{5}$", + "city_regex": ["^8[01][0-9]{3}$"], + "examples": ["80331", "81929"], + "confidence": "high", + "notes": "Munich = 80331-81929." + } + }, + { + "city": "Berlin", + "state_province": "Berlin", + "iso3166_2": "DE-BE", + "country": "Germany", + "iso3166_1": "DE", + "latitude": 52.52, + "longitude": 13.405, + "local_languages": ["de"], + "postal_code": { + "country_format_regex": "^\\d{5}$", + "city_regex": ["^1[0-4][0-9]{3}$"], + "examples": ["10115", "14199"], + "confidence": "high", + "notes": "Berlin = 10115-14199." + } + }, + { + "city": "Cologne", + "state_province": "North Rhine-Westphalia", + "iso3166_2": "DE-NW", + "country": "Germany", + "iso3166_1": "DE", + "latitude": 50.9375, + "longitude": 6.9603, + "local_languages": ["de"], + "postal_code": { + "country_format_regex": "^\\d{5}$", + "city_regex": ["^(?:50[6-9][0-9]{2}|51[01][0-9]{2})$"], + "examples": ["50667", "51149"], + "confidence": "high", + "notes": "Cologne = 50667-51149." + } + }, + { + "city": "Bamberg", + "state_province": "Bavaria", + "iso3166_2": "DE-BY", + "country": "Germany", + "iso3166_1": "DE", + "latitude": 49.8916, + "longitude": 10.8916, + "local_languages": ["de"], + "postal_code": { + "country_format_regex": "^\\d{5}$", + "city_regex": ["^960(?:4[7-9]|5[0-2])$"], + "examples": ["96047", "96052"], + "confidence": "high", + "notes": "Bamberg = 96047-96052." + } + }, + { + "city": "Brussels", + "state_province": "Brussels-Capital", + "iso3166_2": "BE-BRU", + "country": "Belgium", + "iso3166_1": "BE", + "latitude": 50.8503, + "longitude": 4.3517, + "local_languages": ["fr", "nl"], + "postal_code": { + "country_format_regex": "^\\d{4}$", + "city_regex": ["^1(?:0[0-9]{2}|1[0-9]{2}|20[0-9]|210)$"], + "examples": ["1000", "1210"], + "confidence": "high", + "notes": "Brussels-Capital Region = 1000-1210." + } + }, + { + "city": "Antwerp", + "state_province": "Flanders", + "iso3166_2": "BE-VLG", + "country": "Belgium", + "iso3166_1": "BE", + "latitude": 51.2194, + "longitude": 4.4025, + "local_languages": ["nl"], + "postal_code": { + "country_format_regex": "^\\d{4}$", + "city_regex": ["^2[0-6][0-9]{2}$"], + "examples": ["2000", "2660"], + "confidence": "medium", + "notes": "City of Antwerp = 2000-2660; 2xxx also covers other Antwerp Province towns." + } + }, + { + "city": "Bruges", + "state_province": "Flanders", + "iso3166_2": "BE-VLG", + "country": "Belgium", + "iso3166_1": "BE", + "latitude": 51.2093, + "longitude": 3.2247, + "local_languages": ["nl"], + "postal_code": { + "country_format_regex": "^\\d{4}$", + "city_regex": ["^(?:8000|820[01]|831[01]|834[01]|838[01])$"], + "examples": ["8000", "8200", "8310", "8380"], + "confidence": "medium", + "notes": "Bruges uses discrete codes: 8000 (centre), 8200, 8310, 8340, 8380 (submunicipalities)." + } + }, + { + "city": "London", + "state_province": "England", + "iso3166_2": "GB-ENG", + "country": "United Kingdom", + "iso3166_1": "GB", + "latitude": 51.5074, + "longitude": -0.1278, + "local_languages": ["en-GB"], + "postal_code": { + "country_format_regex": "^[A-Z]{1,2}\\d[A-Z\\d]? ?\\d[A-Z]{2}$", + "city_regex": [ + "^(?:EC|WC|NW|SE|SW|E|N|W)[0-9][A-Z0-9]? ?[0-9][A-Z]{2}$", + "^(?:BR|CR|DA|EN|HA|IG|KT|RM|SM|TW|UB|WD)[0-9]{1,2} ?[0-9][A-Z]{2}$" + ], + "examples": ["EC1A 1BB", "SW1A 2AA", "N1 9GU"], + "confidence": "high", + "notes": "First pattern = inner London postcode areas; second = outer Greater London areas." + } + }, + { + "city": "Bristol", + "state_province": "England", + "iso3166_2": "GB-ENG", + "country": "United Kingdom", + "iso3166_1": "GB", + "latitude": 51.4545, + "longitude": -2.5879, + "local_languages": ["en-GB"], + "postal_code": { + "country_format_regex": "^[A-Z]{1,2}\\d[A-Z\\d]? ?\\d[A-Z]{2}$", + "city_regex": ["^BS[0-9]{1,2} ?[0-9][A-Z]{2}$"], + "examples": ["BS1 4DJ", "BS8 1TH"], + "confidence": "high", + "notes": "BS postcode area (also covers parts of North Somerset / South Gloucestershire)." + } + }, + { + "city": "Edinburgh", + "state_province": "Scotland", + "iso3166_2": "GB-SCT", + "country": "United Kingdom", + "iso3166_1": "GB", + "latitude": 55.9533, + "longitude": -3.1883, + "local_languages": ["en-GB", "gd"], + "postal_code": { + "country_format_regex": "^[A-Z]{1,2}\\d[A-Z\\d]? ?\\d[A-Z]{2}$", + "city_regex": ["^EH[0-9]{1,2} ?[0-9][A-Z]{2}$"], + "examples": ["EH1 1YZ", "EH8 9YL"], + "confidence": "high", + "notes": "EH postcode area." + } + }, + { + "city": "Glasgow", + "state_province": "Scotland", + "iso3166_2": "GB-SCT", + "country": "United Kingdom", + "iso3166_1": "GB", + "latitude": 55.8642, + "longitude": -4.2518, + "local_languages": ["en-GB", "gd"], + "postal_code": { + "country_format_regex": "^[A-Z]{1,2}\\d[A-Z\\d]? ?\\d[A-Z]{2}$", + "city_regex": ["^G[0-9]{1,2} ?[0-9][A-Z]{2}$"], + "examples": ["G1 1XW", "G12 8QQ"], + "confidence": "high", + "notes": "G postcode area." + } + }, + { + "city": "Prague", + "state_province": "Prague", + "iso3166_2": "CZ-10", + "country": "Czechia", + "iso3166_1": "CZ", + "latitude": 50.0755, + "longitude": 14.4378, + "local_languages": ["cs"], + "postal_code": { + "country_format_regex": "^\\d{3} ?\\d{2}$", + "city_regex": ["^1[0-9]{2} ?[0-9]{2}$"], + "examples": ["110 00", "190 00"], + "confidence": "high", + "notes": "Prague = 100 00 - 199 00." + } + }, + { + "city": "Pilsen", + "state_province": "Plzeň", + "iso3166_2": "CZ-32", + "country": "Czechia", + "iso3166_1": "CZ", + "latitude": 49.7384, + "longitude": 13.3736, + "local_languages": ["cs"], + "postal_code": { + "country_format_regex": "^\\d{3} ?\\d{2}$", + "city_regex": ["^3(?:0[0-9]|1[0-2]) ?[0-9]{2}$"], + "examples": ["301 00", "312 00"], + "confidence": "high", + "notes": "Plzen = 301 00 - 312 00." + } + }, + { + "city": "Amsterdam", + "state_province": "North Holland", + "iso3166_2": "NL-NH", + "country": "Netherlands", + "iso3166_1": "NL", + "latitude": 52.3676, + "longitude": 4.9041, + "local_languages": ["nl"], + "postal_code": { + "country_format_regex": "^\\d{4} ?[A-Z]{2}$", + "city_regex": ["^(?:10[0-9]{2}|110[0-9]) ?[A-Z]{2}$"], + "examples": ["1011 AB", "1098 XG"], + "confidence": "high", + "notes": "Amsterdam = 1000-1109 numeric part." + } + }, + { + "city": "Copenhagen", + "state_province": "Capital Region", + "iso3166_2": "DK-84", + "country": "Denmark", + "iso3166_1": "DK", + "latitude": 55.6761, + "longitude": 12.5683, + "local_languages": ["da"], + "postal_code": { + "country_format_regex": "^\\d{4}$", + "city_regex": ["^(?:1[0-4][0-9]{2}|2[0-5][0-9]{2})$"], + "examples": ["1050", "2100"], + "confidence": "medium", + "notes": "Copenhagen K = 1000-1499; Frederiksberg and surrounding districts = 1800-2500." + } + }, + { + "city": "Warsaw", + "state_province": "Masovian", + "iso3166_2": "PL-MZ", + "country": "Poland", + "iso3166_1": "PL", + "latitude": 52.2297, + "longitude": 21.0122, + "local_languages": ["pl"], + "postal_code": { + "country_format_regex": "^\\d{2}-\\d{3}$", + "city_regex": ["^0[0-4]-[0-9]{3}$"], + "examples": ["00-001", "04-999"], + "confidence": "high", + "notes": "Warsaw = 00-001 to 04-999." + } + }, + { + "city": "Krakow", + "state_province": "Lesser Poland", + "iso3166_2": "PL-MA", + "country": "Poland", + "iso3166_1": "PL", + "latitude": 50.0647, + "longitude": 19.945, + "local_languages": ["pl"], + "postal_code": { + "country_format_regex": "^\\d{2}-\\d{3}$", + "city_regex": ["^3[01]-[0-9]{3}$"], + "examples": ["30-001", "31-999"], + "confidence": "high", + "notes": "Krakow = 30-001 to 31-999." + } + }, + { + "city": "Rome", + "state_province": "Lazio", + "iso3166_2": "IT-62", + "country": "Italy", + "iso3166_1": "IT", + "latitude": 41.9028, + "longitude": 12.4964, + "local_languages": ["it"], + "postal_code": { + "country_format_regex": "^\\d{5}$", + "city_regex": ["^001[0-9]{2}$"], + "examples": ["00118", "00199"], + "confidence": "high", + "notes": "Rome = 00118-00199 (00120 = Vatican City, a separate state)." + } + }, + { + "city": "Milan", + "state_province": "Lombardy", + "iso3166_2": "IT-25", + "country": "Italy", + "iso3166_1": "IT", + "latitude": 45.4642, + "longitude": 9.19, + "local_languages": ["it"], + "postal_code": { + "country_format_regex": "^\\d{5}$", + "city_regex": ["^201[0-9]{2}$"], + "examples": ["20121", "20162"], + "confidence": "high", + "notes": "Milan city = 20121-20162." + } + }, + { + "city": "Barcelona", + "state_province": "Catalonia", + "iso3166_2": "ES-CT", + "country": "Spain", + "iso3166_1": "ES", + "latitude": 41.3851, + "longitude": 2.1734, + "local_languages": ["ca", "es"], + "postal_code": { + "country_format_regex": "^\\d{5}$", + "city_regex": ["^080[0-9]{2}$"], + "examples": ["08001", "08042"], + "confidence": "high", + "notes": "Barcelona city = 08001-08042. Province is 08xxx." + } + }, + { + "city": "Madrid", + "state_province": "Madrid", + "iso3166_2": "ES-MD", + "country": "Spain", + "iso3166_1": "ES", + "latitude": 40.4168, + "longitude": -3.7038, + "local_languages": ["es"], + "postal_code": { + "country_format_regex": "^\\d{5}$", + "city_regex": ["^280[0-9]{2}$"], + "examples": ["28001", "28080"], + "confidence": "high", + "notes": "Madrid city = 28001-28080. Community of Madrid is 28xxx." + } + }, + { + "city": "Paris", + "state_province": "Île-de-France", + "iso3166_2": "FR-IDF", + "country": "France", + "iso3166_1": "FR", + "latitude": 48.8566, + "longitude": 2.3522, + "local_languages": ["fr"], + "postal_code": { + "country_format_regex": "^\\d{5}$", + "city_regex": ["^75(?:0(?:0[1-9]|1[0-9]|20)|116)$"], + "examples": ["75001", "75020", "75116"], + "confidence": "high", + "notes": "Paris = 75001-75020, plus 75116 (a second code for the 16th arrondissement)." + } + }, + { + "city": "Lyon", + "state_province": "Auvergne-Rhône-Alpes", + "iso3166_2": "FR-ARA", + "country": "France", + "iso3166_1": "FR", + "latitude": 45.764, + "longitude": 4.8357, + "local_languages": ["fr"], + "postal_code": { + "country_format_regex": "^\\d{5}$", + "city_regex": ["^6900[1-9]$"], + "examples": ["69001", "69009"], + "confidence": "high", + "notes": "Lyon = 69001-69009 (nine arrondissements)." + } + }, + { + "city": "Stockholm", + "state_province": "Stockholm", + "iso3166_2": "SE-AB", + "country": "Sweden", + "iso3166_1": "SE", + "latitude": 59.3293, + "longitude": 18.0686, + "local_languages": ["sv"], + "postal_code": { + "country_format_regex": "^\\d{3} ?\\d{2}$", + "city_regex": ["^1[0-9]{2} ?[0-9]{2}$"], + "examples": ["111 22", "190 47"], + "confidence": "medium", + "notes": "Stockholm County = 1xx xx." + } + }, + { + "city": "Gothenburg", + "state_province": "Västra Götaland", + "iso3166_2": "SE-O", + "country": "Sweden", + "iso3166_1": "SE", + "latitude": 57.7089, + "longitude": 11.9746, + "local_languages": ["sv"], + "postal_code": { + "country_format_regex": "^\\d{3} ?\\d{2}$", + "city_regex": ["^4[0-2][0-9] ?[0-9]{2}$"], + "examples": ["411 05", "425 41"], + "confidence": "medium", + "notes": "Gothenburg = 400 00 - 425 99." + } + }, + { + "city": "Oslo", + "state_province": "Oslo", + "iso3166_2": "NO-03", + "country": "Norway", + "iso3166_1": "NO", + "latitude": 59.9139, + "longitude": 10.7522, + "local_languages": ["no"], + "postal_code": { + "country_format_regex": "^\\d{4}$", + "city_regex": ["^0[0-9]{3}$"], + "examples": ["0010", "0980"], + "confidence": "medium", + "notes": "Oslo = 0001-1295; the 0xxx block is the city proper." + } + }, + { + "city": "Dublin", + "state_province": "Leinster", + "iso3166_2": "IE-L", + "country": "Ireland", + "iso3166_1": "IE", + "latitude": 53.3498, + "longitude": -6.2603, + "local_languages": ["en", "ga"], + "postal_code": { + "country_format_regex": "^(?:D6W|[AC-FHKNPRTV-Y]\\d{2}) ?[0-9AC-FHKNPRTV-Y]{4}$", + "city_regex": [ + "^(?:D6W|D0[1-9]|D1[0-8]|D2[024]) ?[0-9AC-FHKNPRTV-Y]{4}$" + ], + "examples": ["D01 Y981", "D02 K303", "D6W XY00"], + "confidence": "high", + "notes": "Dublin routing keys: D01-D18, D20, D22, D24 and the exception D6W. Note D19, D21, D23 are not issued." + } + }, + { + "city": "Vienna", + "state_province": "Vienna", + "iso3166_2": "AT-9", + "country": "Austria", + "iso3166_1": "AT", + "latitude": 48.2082, + "longitude": 16.3738, + "local_languages": ["de-AT"], + "postal_code": { + "country_format_regex": "^\\d{4}$", + "city_regex": ["^1(?:0[1-9]0|1[0-9]0|2[0-3]0)$"], + "examples": ["1010", "1230"], + "confidence": "high", + "notes": "Vienna = 1010-1230, always ending in 0 (district number x10)." + } + }, + { + "city": "Zurich", + "state_province": "Zurich", + "iso3166_2": "CH-ZH", + "country": "Switzerland", + "iso3166_1": "CH", + "latitude": 47.3769, + "longitude": 8.5417, + "local_languages": ["de-CH"], + "postal_code": { + "country_format_regex": "^\\d{4}$", + "city_regex": ["^80[0-6][0-9]$"], + "examples": ["8001", "8064"], + "confidence": "high", + "notes": "City of Zurich = 8001-8064." + } + }, + { + "city": "Tallinn", + "state_province": "Harju", + "iso3166_2": "EE-37", + "country": "Estonia", + "iso3166_1": "EE", + "latitude": 59.437, + "longitude": 24.7536, + "local_languages": ["et"], + "postal_code": { + "country_format_regex": "^\\d{5}$", + "city_regex": ["^1[0-3][0-9]{3}$"], + "examples": ["10111", "13516"], + "confidence": "medium", + "notes": "Tallinn = 10xxx-13xxx." + } + }, + { + "city": "Denver", + "state_province": "Colorado", + "iso3166_2": "US-CO", + "country": "United States", + "iso3166_1": "US", + "latitude": 39.7392, + "longitude": -104.9903, + "local_languages": ["en-US"], + "postal_code": { + "country_format_regex": "^\\d{5}(?:-\\d{4})?$", + "city_regex": ["^80[2-3][0-9]{2}(?:-[0-9]{4})?$"], + "examples": ["80202", "80249"], + "confidence": "high", + "notes": "Denver = 80202-80249 plus 80290-80299." + } + }, + { + "city": "Portland", + "state_province": "Oregon", + "iso3166_2": "US-OR", + "country": "United States", + "iso3166_1": "US", + "latitude": 45.5152, + "longitude": -122.6784, + "local_languages": ["en-US"], + "postal_code": { + "country_format_regex": "^\\d{5}(?:-\\d{4})?$", + "city_regex": ["^972[0-9]{2}(?:-[0-9]{4})?$"], + "examples": ["97201", "97294"], + "confidence": "high", + "notes": "Portland, Oregon = 972xx." + } + }, + { + "city": "San Diego", + "state_province": "California", + "iso3166_2": "US-CA", + "country": "United States", + "iso3166_1": "US", + "latitude": 32.7157, + "longitude": -117.1611, + "local_languages": ["en-US", "es-US"], + "postal_code": { + "country_format_regex": "^\\d{5}(?:-\\d{4})?$", + "city_regex": ["^921[0-9]{2}(?:-[0-9]{4})?$"], + "examples": ["92101", "92199"], + "confidence": "high", + "notes": "San Diego = 921xx." + } + }, + { + "city": "Asheville", + "state_province": "North Carolina", + "iso3166_2": "US-NC", + "country": "United States", + "iso3166_1": "US", + "latitude": 35.5951, + "longitude": -82.5515, + "local_languages": ["en-US"], + "postal_code": { + "country_format_regex": "^\\d{5}(?:-\\d{4})?$", + "city_regex": ["^288[0-1][0-9](?:-[0-9]{4})?$"], + "examples": ["28801", "28816"], + "confidence": "high", + "notes": "Asheville = 28801-28816." + } + }, + { + "city": "Grand Rapids", + "state_province": "Michigan", + "iso3166_2": "US-MI", + "country": "United States", + "iso3166_1": "US", + "latitude": 42.9634, + "longitude": -85.6681, + "local_languages": ["en-US"], + "postal_code": { + "country_format_regex": "^\\d{5}(?:-\\d{4})?$", + "city_regex": ["^495[0-9]{2}(?:-[0-9]{4})?$"], + "examples": ["49503", "49546"], + "confidence": "high", + "notes": "Grand Rapids = 495xx." + } + }, + { + "city": "Chicago", + "state_province": "Illinois", + "iso3166_2": "US-IL", + "country": "United States", + "iso3166_1": "US", + "latitude": 41.8781, + "longitude": -87.6298, + "local_languages": ["en-US", "es-US"], + "postal_code": { + "country_format_regex": "^\\d{5}(?:-\\d{4})?$", + "city_regex": ["^60[6-8][0-9]{2}(?:-[0-9]{4})?$"], + "examples": ["60601", "60661"], + "confidence": "high", + "notes": "Chicago = 606xx-608xx." + } + }, + { + "city": "Seattle", + "state_province": "Washington", + "iso3166_2": "US-WA", + "country": "United States", + "iso3166_1": "US", + "latitude": 47.6062, + "longitude": -122.3321, + "local_languages": ["en-US"], + "postal_code": { + "country_format_regex": "^\\d{5}(?:-\\d{4})?$", + "city_regex": ["^981[0-9]{2}(?:-[0-9]{4})?$"], + "examples": ["98101", "98199"], + "confidence": "high", + "notes": "Seattle = 981xx." + } + }, + { + "city": "Austin", + "state_province": "Texas", + "iso3166_2": "US-TX", + "country": "United States", + "iso3166_1": "US", + "latitude": 30.2672, + "longitude": -97.7431, + "local_languages": ["en-US", "es-US"], + "postal_code": { + "country_format_regex": "^\\d{5}(?:-\\d{4})?$", + "city_regex": ["^787[0-9]{2}(?:-[0-9]{4})?$"], + "examples": ["78701", "78799"], + "confidence": "high", + "notes": "Austin = 787xx." + } + }, + { + "city": "Boston", + "state_province": "Massachusetts", + "iso3166_2": "US-MA", + "country": "United States", + "iso3166_1": "US", + "latitude": 42.3601, + "longitude": -71.0589, + "local_languages": ["en-US"], + "postal_code": { + "country_format_regex": "^\\d{5}(?:-\\d{4})?$", + "city_regex": ["^02[12][0-9]{2}(?:-[0-9]{4})?$"], + "examples": ["02108", "02215"], + "confidence": "high", + "notes": "Boston = 021xx and 022xx." + } + }, + { + "city": "Philadelphia", + "state_province": "Pennsylvania", + "iso3166_2": "US-PA", + "country": "United States", + "iso3166_1": "US", + "latitude": 39.9526, + "longitude": -75.1652, + "local_languages": ["en-US"], + "postal_code": { + "country_format_regex": "^\\d{5}(?:-\\d{4})?$", + "city_regex": ["^191[0-9]{2}(?:-[0-9]{4})?$"], + "examples": ["19102", "19147"], + "confidence": "high", + "notes": "Philadelphia = 191xx." + } + }, + { + "city": "Brooklyn", + "state_province": "New York", + "iso3166_2": "US-NY", + "country": "United States", + "iso3166_1": "US", + "latitude": 40.6782, + "longitude": -73.9442, + "local_languages": ["en-US", "es-US"], + "postal_code": { + "country_format_regex": "^\\d{5}(?:-\\d{4})?$", + "city_regex": ["^112[0-9]{2}(?:-[0-9]{4})?$"], + "examples": ["11201", "11256"], + "confidence": "high", + "notes": "Brooklyn = 112xx." + } + }, + { + "city": "Milwaukee", + "state_province": "Wisconsin", + "iso3166_2": "US-WI", + "country": "United States", + "iso3166_1": "US", + "latitude": 43.0389, + "longitude": -87.9065, + "local_languages": ["en-US"], + "postal_code": { + "country_format_regex": "^\\d{5}(?:-\\d{4})?$", + "city_regex": ["^532[0-9]{2}(?:-[0-9]{4})?$"], + "examples": ["53202", "53233"], + "confidence": "high", + "notes": "Milwaukee = 532xx." + } + }, + { + "city": "Richmond", + "state_province": "Virginia", + "iso3166_2": "US-VA", + "country": "United States", + "iso3166_1": "US", + "latitude": 37.5407, + "longitude": -77.436, + "local_languages": ["en-US"], + "postal_code": { + "country_format_regex": "^\\d{5}(?:-\\d{4})?$", + "city_regex": ["^232[0-9]{2}(?:-[0-9]{4})?$"], + "examples": ["23219", "23298"], + "confidence": "high", + "notes": "Richmond = 232xx." + } + }, + { + "city": "Cincinnati", + "state_province": "Ohio", + "iso3166_2": "US-OH", + "country": "United States", + "iso3166_1": "US", + "latitude": 39.1031, + "longitude": -84.512, + "local_languages": ["en-US"], + "postal_code": { + "country_format_regex": "^\\d{5}(?:-\\d{4})?$", + "city_regex": ["^452[0-9]{2}(?:-[0-9]{4})?$"], + "examples": ["45202", "45255"], + "confidence": "high", + "notes": "Cincinnati = 452xx." + } + }, + { + "city": "St. Louis", + "state_province": "Missouri", + "iso3166_2": "US-MO", + "country": "United States", + "iso3166_1": "US", + "latitude": 38.627, + "longitude": -90.1994, + "local_languages": ["en-US"], + "postal_code": { + "country_format_regex": "^\\d{5}(?:-\\d{4})?$", + "city_regex": ["^631[0-9]{2}(?:-[0-9]{4})?$"], + "examples": ["63101", "63139"], + "confidence": "high", + "notes": "St. Louis = 631xx." + } + }, + { + "city": "Tampa", + "state_province": "Florida", + "iso3166_2": "US-FL", + "country": "United States", + "iso3166_1": "US", + "latitude": 27.9506, + "longitude": -82.4572, + "local_languages": ["en-US", "es-US"], + "postal_code": { + "country_format_regex": "^\\d{5}(?:-\\d{4})?$", + "city_regex": ["^336[0-9]{2}(?:-[0-9]{4})?$"], + "examples": ["33602", "33647"], + "confidence": "high", + "notes": "Tampa = 336xx." + } + }, + { + "city": "Minneapolis", + "state_province": "Minnesota", + "iso3166_2": "US-MN", + "country": "United States", + "iso3166_1": "US", + "latitude": 44.9778, + "longitude": -93.265, + "local_languages": ["en-US"], + "postal_code": { + "country_format_regex": "^\\d{5}(?:-\\d{4})?$", + "city_regex": ["^554[0-9]{2}(?:-[0-9]{4})?$"], + "examples": ["55401", "55488"], + "confidence": "high", + "notes": "Minneapolis = 554xx." + } + }, + { + "city": "Burlington", + "state_province": "Vermont", + "iso3166_2": "US-VT", + "country": "United States", + "iso3166_1": "US", + "latitude": 44.4759, + "longitude": -73.2121, + "local_languages": ["en-US"], + "postal_code": { + "country_format_regex": "^\\d{5}(?:-\\d{4})?$", + "city_regex": ["^054[0-9]{2}(?:-[0-9]{4})?$"], + "examples": ["05401", "05408"], + "confidence": "high", + "notes": "Burlington = 054xx." + } + }, + { + "city": "Portland", + "state_province": "Maine", + "iso3166_2": "US-ME", + "country": "United States", + "iso3166_1": "US", + "latitude": 43.6591, + "longitude": -70.2568, + "local_languages": ["en-US"], + "postal_code": { + "country_format_regex": "^\\d{5}(?:-\\d{4})?$", + "city_regex": ["^041[0-9]{2}(?:-[0-9]{4})?$"], + "examples": ["04101", "04124"], + "confidence": "high", + "notes": "Portland, Maine = 041xx. Distinct from Portland, Oregon (972xx)." + } + }, + { + "city": "Atlanta", + "state_province": "Georgia", + "iso3166_2": "US-GA", + "country": "United States", + "iso3166_1": "US", + "latitude": 33.749, + "longitude": -84.388, + "local_languages": ["en-US"], + "postal_code": { + "country_format_regex": "^\\d{5}(?:-\\d{4})?$", + "city_regex": ["^30[3-9][0-9]{2}(?:-[0-9]{4})?$"], + "examples": ["30303", "30363"], + "confidence": "medium", + "notes": "Atlanta city = 303xx; the wider metro extends across 303xx-309xx." + } + }, + { + "city": "Toronto", + "state_province": "Ontario", + "iso3166_2": "CA-ON", + "country": "Canada", + "iso3166_1": "CA", + "latitude": 43.651, + "longitude": -79.347, + "local_languages": ["en-CA"], + "postal_code": { + "country_format_regex": "^[ABCEGHJ-NPRSTVXY]\\d[ABCEGHJ-NPRSTV-Z] ?\\d[ABCEGHJ-NPRSTV-Z]\\d$", + "city_regex": [ + "^M[0-9][ABCEGHJ-NPRSTV-Z] ?[0-9][ABCEGHJ-NPRSTV-Z][0-9]$" + ], + "examples": ["M5H 2N2", "M4B 1B3"], + "confidence": "high", + "notes": "Toronto = the entire 'M' forward sortation area." + } + }, + { + "city": "Vancouver", + "state_province": "British Columbia", + "iso3166_2": "CA-BC", + "country": "Canada", + "iso3166_1": "CA", + "latitude": 49.2827, + "longitude": -123.1207, + "local_languages": ["en-CA"], + "postal_code": { + "country_format_regex": "^[ABCEGHJ-NPRSTVXY]\\d[ABCEGHJ-NPRSTV-Z] ?\\d[ABCEGHJ-NPRSTV-Z]\\d$", + "city_regex": [ + "^V[5-7][ABCEGHJ-NPRSTV-Z] ?[0-9][ABCEGHJ-NPRSTV-Z][0-9]$" + ], + "examples": ["V6B 1A1", "V5K 0A1"], + "confidence": "medium", + "notes": "Vancouver proper = V5/V6; V7 covers North/West Vancouver and Richmond." + } + }, + { + "city": "Montreal", + "state_province": "Quebec", + "iso3166_2": "CA-QC", + "country": "Canada", + "iso3166_1": "CA", + "latitude": 45.5017, + "longitude": -73.5673, + "local_languages": ["fr-CA", "en-CA"], + "postal_code": { + "country_format_regex": "^[ABCEGHJ-NPRSTVXY]\\d[ABCEGHJ-NPRSTV-Z] ?\\d[ABCEGHJ-NPRSTV-Z]\\d$", + "city_regex": [ + "^H[0-9][ABCEGHJ-NPRSTV-Z] ?[0-9][ABCEGHJ-NPRSTV-Z][0-9]$" + ], + "examples": ["H3B 2Y7", "H2X 1Y4"], + "confidence": "high", + "notes": "Montreal = the entire 'H' forward sortation area." + } + }, + { + "city": "Calgary", + "state_province": "Alberta", + "iso3166_2": "CA-AB", + "country": "Canada", + "iso3166_1": "CA", + "latitude": 51.0447, + "longitude": -114.0719, + "local_languages": ["en-CA"], + "postal_code": { + "country_format_regex": "^[ABCEGHJ-NPRSTVXY]\\d[ABCEGHJ-NPRSTV-Z] ?\\d[ABCEGHJ-NPRSTV-Z]\\d$", + "city_regex": [ + "^T[1-3][ABCEGHJ-NPRSTV-Z] ?[0-9][ABCEGHJ-NPRSTV-Z][0-9]$" + ], + "examples": ["T2P 1J9", "T3A 0A1"], + "confidence": "medium", + "notes": "Calgary uses T1Y, T2A-T2Z and T3A-T3R; the T1/T3 blocks also include nearby communities." + } + }, + { + "city": "Halifax", + "state_province": "Nova Scotia", + "iso3166_2": "CA-NS", + "country": "Canada", + "iso3166_1": "CA", + "latitude": 44.6488, + "longitude": -63.5752, + "local_languages": ["en-CA"], + "postal_code": { + "country_format_regex": "^[ABCEGHJ-NPRSTVXY]\\d[ABCEGHJ-NPRSTV-Z] ?\\d[ABCEGHJ-NPRSTV-Z]\\d$", + "city_regex": ["^B3[ABCEGHJ-NPRSTV-Z] ?[0-9][ABCEGHJ-NPRSTV-Z][0-9]$"], + "examples": ["B3H 4R2", "B3J 1P3"], + "confidence": "medium", + "notes": "Halifax peninsula/urban = B3x. The wider municipality also uses B2x and B4x." + } + }, + { + "city": "Mexico City", + "state_province": "Mexico City", + "iso3166_2": "MX-CMX", + "country": "Mexico", + "iso3166_1": "MX", + "latitude": 19.4326, + "longitude": -99.1332, + "local_languages": ["es-MX"], + "postal_code": { + "country_format_regex": "^\\d{5}$", + "city_regex": ["^(?:0[1-9]|1[0-6])[0-9]{3}$"], + "examples": ["01000", "16900"], + "confidence": "high", + "notes": "Mexico City = 01000-16999." + } + }, + { + "city": "Tijuana", + "state_province": "Baja California", + "iso3166_2": "MX-BCN", + "country": "Mexico", + "iso3166_1": "MX", + "latitude": 32.5149, + "longitude": -117.0382, + "local_languages": ["es-MX"], + "postal_code": { + "country_format_regex": "^\\d{5}$", + "city_regex": ["^22[0-6][0-9]{2}$"], + "examples": ["22000", "22660"], + "confidence": "medium", + "notes": "Tijuana = 22000-22699." + } + }, + { + "city": "Monterrey", + "state_province": "Nuevo León", + "iso3166_2": "MX-NLE", + "country": "Mexico", + "iso3166_1": "MX", + "latitude": 25.6866, + "longitude": -100.3161, + "local_languages": ["es-MX"], + "postal_code": { + "country_format_regex": "^\\d{5}$", + "city_regex": ["^64[0-9]{3}$"], + "examples": ["64000", "64988"], + "confidence": "medium", + "notes": "Monterrey city = 64000-64999; the metro area extends into 65xxx-67xxx." + } + }, + { + "city": "Guadalajara", + "state_province": "Jalisco", + "iso3166_2": "MX-JAL", + "country": "Mexico", + "iso3166_1": "MX", + "latitude": 20.6597, + "longitude": -103.3496, + "local_languages": ["es-MX"], + "postal_code": { + "country_format_regex": "^\\d{5}$", + "city_regex": ["^44[0-9]{3}$"], + "examples": ["44100", "44990"], + "confidence": "medium", + "notes": "Guadalajara city = 44100-44999; Zapopan/Tlaquepaque use 45xxx." + } + }, + { + "city": "Ensenada", + "state_province": "Baja California", + "iso3166_2": "MX-BCN", + "country": "Mexico", + "iso3166_1": "MX", + "latitude": 31.8667, + "longitude": -116.5964, + "local_languages": ["es-MX"], + "postal_code": { + "country_format_regex": "^\\d{5}$", + "city_regex": ["^228[0-9]{2}$"], + "examples": ["22800", "22890"], + "confidence": "medium", + "notes": "Ensenada = 22800-22899." + } + } +] diff --git a/tooling/pipeline/includes/biergarten_pipeline.h b/tooling/pipeline/includes/biergarten_pipeline.h index f9061fd..5cd0437 100644 --- a/tooling/pipeline/includes/biergarten_pipeline.h +++ b/tooling/pipeline/includes/biergarten_pipeline.h @@ -64,6 +64,10 @@ #include "services/logging/log_producer.h" #include "services/logging/logger.h" +// --- services: postal_code --- +#include "services/postal_code/mock_postal_code_service.h" +#include "services/postal_code/postal_code_service.h" + // --- services: prompting --- #include "services/prompting/prompt_directory.h" diff --git a/tooling/pipeline/includes/biergarten_pipeline_orchestrator.h b/tooling/pipeline/includes/biergarten_pipeline_orchestrator.h index c65219a..4e5a827 100644 --- a/tooling/pipeline/includes/biergarten_pipeline_orchestrator.h +++ b/tooling/pipeline/includes/biergarten_pipeline_orchestrator.h @@ -16,7 +16,7 @@ #include "services/database/export_service.h" #include "services/enrichment/enrichment_service.h" #include "services/logging/logger.h" - +#include "services/postal_code/postal_code_service.h" /** * @brief Main orchestrator for the Biergarten data generation pipeline. * @@ -34,6 +34,7 @@ class BiergartenPipelineOrchestrator { * @param exporter Database backend for persisting generated records. * @param curated_data_service Loads curated location, persona, and name * data used to seed generation. + * @param postal_code_service * @param application_options CLI configuration and paths. */ BiergartenPipelineOrchestrator( @@ -42,6 +43,7 @@ class BiergartenPipelineOrchestrator { std::unique_ptr generator, std::unique_ptr exporter, std::unique_ptr curated_data_service, + std::unique_ptr postal_code_service, const ApplicationOptions& application_options); /** @@ -78,6 +80,7 @@ class BiergartenPipelineOrchestrator { */ std::unique_ptr exporter_; std::unique_ptr curated_data_service_; + std::unique_ptr postal_code_service_; ApplicationOptions application_options_; @@ -87,7 +90,7 @@ class BiergartenPipelineOrchestrator { * @return Vector of locations randomly sampled per * PipelineOptions::location_count. */ - std::vector QueryCitiesWithCountries(); + std::vector QueryCitiesWithCountries(); /** * @brief Generate breweries for enriched cities. diff --git a/tooling/pipeline/includes/data_generation/data_generator.h b/tooling/pipeline/includes/data_generation/data_generator.h index b3408e4..7e7a84e 100644 --- a/tooling/pipeline/includes/data_generation/data_generator.h +++ b/tooling/pipeline/includes/data_generation/data_generator.h @@ -24,7 +24,7 @@ class DataGenerator { * @param region_context Additional regional context text. * @return Brewery generation result. */ - virtual BreweryResult GenerateBrewery(const Location& location, + virtual BreweryResult GenerateBrewery(const City& location, const std::string& region_context) = 0; /** diff --git a/tooling/pipeline/includes/data_generation/llama_generator.h b/tooling/pipeline/includes/data_generation/llama_generator.h index 7d4dadb..5c7b9a7 100644 --- a/tooling/pipeline/includes/data_generation/llama_generator.h +++ b/tooling/pipeline/includes/data_generation/llama_generator.h @@ -55,7 +55,7 @@ class LlamaGenerator final : public DataGenerator { * @param region_context Additional regional context. * @return Generated brewery result. */ - BreweryResult GenerateBrewery(const Location& location, + BreweryResult GenerateBrewery(const City& location, const std::string& region_context) override; /** diff --git a/tooling/pipeline/includes/data_generation/mock_generator.h b/tooling/pipeline/includes/data_generation/mock_generator.h index 62467d8..f458557 100644 --- a/tooling/pipeline/includes/data_generation/mock_generator.h +++ b/tooling/pipeline/includes/data_generation/mock_generator.h @@ -24,7 +24,7 @@ class MockGenerator final : public DataGenerator { * @param region_context Unused for mock generation. * @return Generated brewery result. */ - BreweryResult GenerateBrewery(const Location& location, + BreweryResult GenerateBrewery(const City& location, const std::string& region_context) override; /** @@ -46,7 +46,7 @@ class MockGenerator final : public DataGenerator { * @param location City and country names. * @return Deterministic hash value. */ - static size_t DeterministicHash(const Location& location); + static size_t DeterministicHash(const City& location); /** * @brief Combines city, persona, and name into a stable hash value. @@ -56,7 +56,7 @@ class MockGenerator final : public DataGenerator { * @param name Sampled first/last name. * @return Deterministic hash value. */ - static size_t DeterministicHash(const Location& location, + static size_t DeterministicHash(const City& location, const UserPersona& persona, const Name& name); // Hash stride constants for deterministic distribution across fixed-size diff --git a/tooling/pipeline/includes/data_model/generated_models.h b/tooling/pipeline/includes/data_model/generated_models.h index 84a5fcf..7492d23 100644 --- a/tooling/pipeline/includes/data_model/generated_models.h +++ b/tooling/pipeline/includes/data_model/generated_models.h @@ -87,15 +87,27 @@ struct UserResult { * @brief Enriched city data with Wikipedia context. */ struct EnrichedCity { - Location location; + City location; std::string region_context{}; }; +/** + * @brief A brewery's street-level address, distinct from the shared `City` + * it belongs to. Mirrors the backend's `BreweryPostLocation` shape. + * + * Only `postal_code` is populated today -- street-address generation has no + * fixture data or service yet. + */ +struct Address { + std::string postal_code{}; +}; + /** * @brief Helper struct to store generated brewery data. */ struct BreweryRecord { - Location location; + City location; + Address address; BreweryResult brewery; }; @@ -107,7 +119,7 @@ struct BreweryRecord { * consumer can register real accounts from the pipeline's SQLite export. */ struct UserRecord { - Location location; + City location; UserResult user; std::string email{}; std::string date_of_birth{}; diff --git a/tooling/pipeline/includes/data_model/models.h b/tooling/pipeline/includes/data_model/models.h index 8ed7355..e07a778 100644 --- a/tooling/pipeline/includes/data_model/models.h +++ b/tooling/pipeline/includes/data_model/models.h @@ -22,13 +22,13 @@ class ILogger; namespace prog_opts = boost::program_options; // ============================================================================ -// Location Models +// City Models // ============================================================================ /** * @brief Canonical location record for city-level generation. */ -struct Location { +struct City { std::string city{}; std::string state_province{}; @@ -44,6 +44,18 @@ struct Location { */ std::string iso3166_1{}; + /** + * @brief A list of regular expressions used for valid postal codes in the + * region. + * + */ + std::vector postal_regex{}; + + /** + * @brief Example postal codes for the region. + */ + std::vector postal_code_examples{}; + /** * @brief Local language codes in priority order. */ diff --git a/tooling/pipeline/includes/services/curated_data/curated_data_service.h b/tooling/pipeline/includes/services/curated_data/curated_data_service.h index d6505ce..8a2815d 100644 --- a/tooling/pipeline/includes/services/curated_data/curated_data_service.h +++ b/tooling/pipeline/includes/services/curated_data/curated_data_service.h @@ -15,7 +15,7 @@ using ForenameList = std::vector; using SurnameList = std::vector; -using LocationsList = std::vector; +using CityList = std::vector; using PersonasList = std::vector; using ForenamesByCountryMap = std::unordered_map; using SurnamesByCountryMap = std::unordered_map; @@ -37,7 +37,7 @@ class ICuratedDataService { /** * @brief Loads all curated location records. */ - virtual const LocationsList& LoadLocations() = 0; + virtual const CityList& LoadCities() = 0; /** * @brief Loads all curated persona records. diff --git a/tooling/pipeline/includes/services/curated_data/curated_json_data_service.h b/tooling/pipeline/includes/services/curated_data/curated_json_data_service.h index 687b881..c8cd6ba 100644 --- a/tooling/pipeline/includes/services/curated_data/curated_json_data_service.h +++ b/tooling/pipeline/includes/services/curated_data/curated_json_data_service.h @@ -16,7 +16,7 @@ * CuratedJsonDataService. */ struct CuratedDataFilePaths { - std::filesystem::path locations_path; + std::filesystem::path cities_path; std::filesystem::path personas_path; std::filesystem::path forenames_path; std::filesystem::path surnames_path; @@ -27,7 +27,7 @@ struct CuratedDataFilePaths { */ class CuratedJsonDataService final : public ICuratedDataService { struct cache { - LocationsList locations; + CityList locations; PersonasList personas; ForenamesByCountryMap forenames_by_country; SurnamesByCountryMap surnames_by_country; @@ -43,9 +43,9 @@ class CuratedJsonDataService final : public ICuratedDataService { explicit CuratedJsonDataService(CuratedDataFilePaths filepaths); /** - * @brief Parses a JSON array file and returns all location records. + * @brief Parses a JSON array file and returns all city records. */ - const LocationsList& LoadLocations() override; + const CityList& LoadCities() override; /** * @brief Parses a JSON array file and returns all persona records. diff --git a/tooling/pipeline/includes/services/curated_data/mock_curated_data_service.h b/tooling/pipeline/includes/services/curated_data/mock_curated_data_service.h index 53aa5ef..1ab426d 100644 --- a/tooling/pipeline/includes/services/curated_data/mock_curated_data_service.h +++ b/tooling/pipeline/includes/services/curated_data/mock_curated_data_service.h @@ -20,7 +20,7 @@ class MockCuratedDataService final : public ICuratedDataService { public: MockCuratedDataService(); - const LocationsList& LoadLocations() override; + const CityList& LoadCities() override; const PersonasList& LoadPersonas() override; @@ -29,7 +29,7 @@ class MockCuratedDataService final : public ICuratedDataService { const SurnamesByCountryMap& LoadSurnamesByCountry() override; private: - LocationsList locations_; + CityList locations_; PersonasList personas_; ForenamesByCountryMap forenames_by_country_; SurnamesByCountryMap surnames_by_country_; diff --git a/tooling/pipeline/includes/services/database/sqlite_export_service.h b/tooling/pipeline/includes/services/database/sqlite_export_service.h index e5c4936..f595ebe 100644 --- a/tooling/pipeline/includes/services/database/sqlite_export_service.h +++ b/tooling/pipeline/includes/services/database/sqlite_export_service.h @@ -45,7 +45,7 @@ class SqliteExportService final : public IExportService { void RollbackAndCloseNoThrow() noexcept; [[nodiscard]] std::filesystem::path BuildDatabasePath() const; - [[nodiscard]] static std::string BuildLocationKey(const Location& location); + [[nodiscard]] static std::string BuildCityKey(const City& location); /** * @brief Returns the row id for @p location, inserting it first if it has @@ -54,18 +54,18 @@ class SqliteExportService final : public IExportService { * Shared by both ProcessRecord() overloads so breweries and users * referencing the same location resolve to the same row. */ - [[nodiscard]] sqlite3_int64 ResolveLocationId(const Location& location); + [[nodiscard]] sqlite3_int64 ResolveCityId(const City& location); std::unique_ptr date_time_provider_; std::filesystem::path output_path_; std::string run_timestamp_utc_; std::filesystem::path database_path_; SqliteDatabaseHandle db_handle_; - SqliteStatementHandle insert_location_stmt_; + SqliteStatementHandle insert_city_stmt_; SqliteStatementHandle insert_brewery_stmt_; SqliteStatementHandle insert_user_stmt_; bool transaction_open_ = false; - std::unordered_map location_cache_; + std::unordered_map city_cache_; }; #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_SQLITE_EXPORT_SERVICE_H_ diff --git a/tooling/pipeline/includes/services/database/sqlite_statement_helpers.h b/tooling/pipeline/includes/services/database/sqlite_statement_helpers.h index b2143c4..fb8491a 100644 --- a/tooling/pipeline/includes/services/database/sqlite_statement_helpers.h +++ b/tooling/pipeline/includes/services/database/sqlite_statement_helpers.h @@ -17,9 +17,9 @@ namespace sqlite_export_service_internal { -inline constexpr std::string_view kCreateLocationsTableSql = R"sql( +inline constexpr std::string_view kCreateCitiesTableSql = R"sql( -CREATE TABLE IF NOT EXISTS locations ( +CREATE TABLE IF NOT EXISTS cities ( id INTEGER PRIMARY KEY AUTOINCREMENT, city TEXT NOT NULL, state_province TEXT NOT NULL, @@ -38,15 +38,16 @@ inline constexpr std::string_view kCreateBreweriesTableSql = R"sql( CREATE TABLE IF NOT EXISTS breweries ( id INTEGER PRIMARY KEY AUTOINCREMENT, - location_id INTEGER NOT NULL, + city_id INTEGER NOT NULL, name_en TEXT NOT NULL, description_en TEXT NOT NULL, name_local TEXT NOT NULL, description_local TEXT NOT NULL, - FOREIGN KEY(location_id) REFERENCES locations(id) ON DELETE CASCADE + postal_code TEXT NOT NULL, + FOREIGN KEY(city_id) REFERENCES cities(id) ON DELETE CASCADE ); -CREATE INDEX IF NOT EXISTS idx_breweries_location_id ON breweries(location_id); +CREATE INDEX IF NOT EXISTS idx_breweries_city_id ON breweries(city_id); )sql"; @@ -54,7 +55,7 @@ inline constexpr std::string_view kCreateUsersTableSql = R"sql( CREATE TABLE IF NOT EXISTS users ( id INTEGER PRIMARY KEY AUTOINCREMENT, - location_id INTEGER NOT NULL, + city_id INTEGER NOT NULL, first_name TEXT NOT NULL, last_name TEXT NOT NULL, gender TEXT NOT NULL, @@ -63,15 +64,15 @@ CREATE TABLE IF NOT EXISTS users ( activity_weight REAL NOT NULL, email TEXT NOT NULL UNIQUE, date_of_birth TEXT NOT NULL, - FOREIGN KEY(location_id) REFERENCES locations(id) ON DELETE CASCADE + FOREIGN KEY(city_id) REFERENCES cities(id) ON DELETE CASCADE ); -CREATE INDEX IF NOT EXISTS idx_users_location_id ON users(location_id); +CREATE INDEX IF NOT EXISTS idx_users_city_id ON users(city_id); )sql"; -inline constexpr std::string_view kInsertLocationSql = R"sql( -INSERT INTO locations ( +inline constexpr std::string_view kInsertCitySql = R"sql( +INSERT INTO cities ( city, state_province, iso3166_2, @@ -85,17 +86,18 @@ INSERT INTO locations ( inline constexpr std::string_view kInsertBrewerySql = R"sql( INSERT INTO breweries ( - location_id, + city_id, name_en, description_en, name_local, - description_local -) VALUES (?, ?, ?, ?, ?); + description_local, + postal_code +) VALUES (?, ?, ?, ?, ?, ?); )sql"; inline constexpr std::string_view kInsertUserSql = R"sql( INSERT INTO users ( - location_id, + city_id, first_name, last_name, gender, @@ -109,27 +111,28 @@ INSERT INTO users ( // sqlite3_bind_*() parameter indices are 1-based, matching the "?" // placeholder order in the SQL above. -enum LocationBindIndex { - kLocationCityBindIndex = 1, - kLocationStateProvinceBindIndex, - kLocationIso31662BindIndex, - kLocationCountryBindIndex, - kLocationIso31661BindIndex, - kLocationLanguagesBindIndex, - kLocationLatitudeBindIndex, - kLocationLongitudeBindIndex, +enum CityBindIndex { + kCityNameBindIndex = 1, + kCityStateProvinceBindIndex, + kCityIso31662BindIndex, + kCityCountryBindIndex, + kCityIso31661BindIndex, + kCityLanguagesBindIndex, + kCityLatitudeBindIndex, + kCityLongitudeBindIndex, }; enum BreweryBindIndex { - kBreweryLocationIdBindIndex = 1, + kBreweryCityIdBindIndex = 1, kBreweryEnglishNameBindIndex, kBreweryEnglishDescriptionBindIndex, kBreweryLocalNameBindIndex, kBreweryLocalDescriptionBindIndex, + kBreweryPostalCodeBindIndex, }; enum UserBindIndex { - kUserLocationIdBindIndex = 1, + kUserCityIdBindIndex = 1, kUserFirstNameBindIndex, kUserLastNameBindIndex, kUserGenderBindIndex, diff --git a/tooling/pipeline/includes/services/enrichment/enrichment_service.h b/tooling/pipeline/includes/services/enrichment/enrichment_service.h index 9c79137..8070cdd 100644 --- a/tooling/pipeline/includes/services/enrichment/enrichment_service.h +++ b/tooling/pipeline/includes/services/enrichment/enrichment_service.h @@ -23,7 +23,7 @@ class IEnrichmentService { * @param loc Location to enrich. * @return Context text, or an empty string if unavailable. */ - virtual std::string GetLocationContext(const Location& loc) = 0; + virtual std::string GetLocationContext(const City& loc) = 0; }; #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_ENRICHMENT_SERVICE_H_ diff --git a/tooling/pipeline/includes/services/enrichment/mock_enrichment.h b/tooling/pipeline/includes/services/enrichment/mock_enrichment.h index 9a45b3f..88bc39a 100644 --- a/tooling/pipeline/includes/services/enrichment/mock_enrichment.h +++ b/tooling/pipeline/includes/services/enrichment/mock_enrichment.h @@ -15,7 +15,7 @@ */ class MockEnrichmentService final : public IEnrichmentService { public: - std::string GetLocationContext(const Location& /*loc*/) override { + std::string GetLocationContext(const City& /*loc*/) override { return {}; } }; diff --git a/tooling/pipeline/includes/services/enrichment/wikipedia_service.h b/tooling/pipeline/includes/services/enrichment/wikipedia_service.h index 1063aea..e33b94a 100644 --- a/tooling/pipeline/includes/services/enrichment/wikipedia_service.h +++ b/tooling/pipeline/includes/services/enrichment/wikipedia_service.h @@ -29,7 +29,7 @@ class WikipediaEnrichmentService final : public IEnrichmentService { /** * @brief Returns the Wikipedia-derived context for a location. */ - [[nodiscard]] std::string GetLocationContext(const Location& loc) override; + [[nodiscard]] std::string GetLocationContext(const City& loc) override; private: std::string FetchExtract(std::string_view query); diff --git a/tooling/pipeline/includes/services/postal_code/mock_postal_code_service.h b/tooling/pipeline/includes/services/postal_code/mock_postal_code_service.h new file mode 100644 index 0000000..5a0bbfe --- /dev/null +++ b/tooling/pipeline/includes/services/postal_code/mock_postal_code_service.h @@ -0,0 +1,32 @@ +#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_POSTAL_CODE_MOCK_POSTAL_CODE_SERVICE_H_ +#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_POSTAL_CODE_MOCK_POSTAL_CODE_SERVICE_H_ + +/** + * @file services/postal_code/mock_postal_code_service.h + * @brief Placeholder IPostalCodeService used until xeger-based generation is + * implemented. + */ + +#include +#include + +#include "data_model/models.h" +#include "services/postal_code/postal_code_service.h" + +/** + * @brief Postal code service that returns the location's first known + * example postal code rather than generating one from its regex. + */ +class MockPostalCodeService final : public IPostalCodeService { + public: + std::string GeneratePostalCode(const City& location) override { + if (location.postal_code_examples.empty()) { + throw std::runtime_error( + "MockPostalCodeService: location has no postal_code_examples: " + + location.city); + } + return location.postal_code_examples.front(); + } +}; + +#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_POSTAL_CODE_MOCK_POSTAL_CODE_SERVICE_H_ diff --git a/tooling/pipeline/includes/services/postal_code/postal_code_service.h b/tooling/pipeline/includes/services/postal_code/postal_code_service.h new file mode 100644 index 0000000..7cb5017 --- /dev/null +++ b/tooling/pipeline/includes/services/postal_code/postal_code_service.h @@ -0,0 +1,30 @@ +#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_POSTAL_CODE_POSTAL_CODE_SERVICE_H_ +#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_POSTAL_CODE_POSTAL_CODE_SERVICE_H_ + +/** + * @file services/postal_code/postal_code_service.h + * @brief Abstraction for generating a postal code for a location. + */ + +#include + +#include "data_model/models.h" + +/** + * @brief Interface for services that generate a postal code string matching + * a location's postal code format (`City::postal_regex`). + */ +class IPostalCodeService { + public: + virtual ~IPostalCodeService() = default; + + /** + * @brief Generates a postal code for the given location. + * + * @param location Location whose postal code format to generate for. + * @return A postal code string matching `location.postal_regex`. + */ + virtual std::string GeneratePostalCode(const City& location) = 0; +}; + +#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_POSTAL_CODE_POSTAL_CODE_SERVICE_H_ diff --git a/tooling/pipeline/locations.json b/tooling/pipeline/locations.json deleted file mode 100644 index 2b21673..0000000 --- a/tooling/pipeline/locations.json +++ /dev/null @@ -1,1002 +0,0 @@ -[ - { - "city": "Cape Town", - "state_province": "Western Cape", - "iso3166_2": "ZA-WC", - "country": "South Africa", - "iso3166_1": "ZA", - "latitude": -33.9249, - "longitude": 18.4241, - "local_languages": ["af", "en", "xh"] - }, - { - "city": "Johannesburg", - "state_province": "Gauteng", - "iso3166_2": "ZA-GT", - "country": "South Africa", - "iso3166_1": "ZA", - "latitude": -26.2041, - "longitude": 28.0473, - "local_languages": ["en", "zu", "st", "af"] - }, - { - "city": "Durban", - "state_province": "KwaZulu-Natal", - "iso3166_2": "ZA-NL", - "country": "South Africa", - "iso3166_1": "ZA", - "latitude": -29.8587, - "longitude": 31.0218, - "local_languages": ["zu", "en"] - }, - { - "city": "Franschhoek", - "state_province": "Western Cape", - "iso3166_2": "ZA-WC", - "country": "South Africa", - "iso3166_1": "ZA", - "latitude": -33.9146, - "longitude": 19.1198, - "local_languages": ["af", "en"] - }, - { - "city": "Nairobi", - "state_province": "Nairobi", - "iso3166_2": "KE-30", - "country": "Kenya", - "iso3166_1": "KE", - "latitude": -1.2921, - "longitude": 36.8219, - "local_languages": ["sw", "en"] - }, - { - "city": "Buenos Aires", - "state_province": "Buenos Aires City", - "iso3166_2": "AR-C", - "country": "Argentina", - "iso3166_1": "AR", - "latitude": -34.6037, - "longitude": -58.3816, - "local_languages": ["es-AR"] - }, - { - "city": "Bariloche", - "state_province": "Río Negro", - "iso3166_2": "AR-R", - "country": "Argentina", - "iso3166_1": "AR", - "latitude": -41.1335, - "longitude": -71.3103, - "local_languages": ["es-AR"] - }, - { - "city": "Bogotá", - "state_province": "Bogotá D.C.", - "iso3166_2": "CO-DC", - "country": "Colombia", - "iso3166_1": "CO", - "latitude": 4.711, - "longitude": -74.0721, - "local_languages": ["es-CO"] - }, - { - "city": "Medellín", - "state_province": "Antioquia", - "iso3166_2": "CO-ANT", - "country": "Colombia", - "iso3166_1": "CO", - "latitude": 6.2442, - "longitude": -75.5812, - "local_languages": ["es-CO"] - }, - { - "city": "São Paulo", - "state_province": "São Paulo", - "iso3166_2": "BR-SP", - "country": "Brazil", - "iso3166_1": "BR", - "latitude": -23.5505, - "longitude": -46.6333, - "local_languages": ["pt-BR"] - }, - { - "city": "Curitiba", - "state_province": "Paraná", - "iso3166_2": "BR-PR", - "country": "Brazil", - "iso3166_1": "BR", - "latitude": -25.4284, - "longitude": -49.2733, - "local_languages": ["pt-BR"] - }, - { - "city": "Rio de Janeiro", - "state_province": "Rio de Janeiro", - "iso3166_2": "BR-RJ", - "country": "Brazil", - "iso3166_1": "BR", - "latitude": -22.9068, - "longitude": -43.1729, - "local_languages": ["pt-BR"] - }, - { - "city": "Santiago", - "state_province": "Santiago Metropolitan", - "iso3166_2": "CL-RM", - "country": "Chile", - "iso3166_1": "CL", - "latitude": -33.4489, - "longitude": -70.6693, - "local_languages": ["es-CL"] - }, - { - "city": "Valdivia", - "state_province": "Los Ríos", - "iso3166_2": "CL-LR", - "country": "Chile", - "iso3166_1": "CL", - "latitude": -39.8142, - "longitude": -73.2459, - "local_languages": ["es-CL"] - }, - { - "city": "Lima", - "state_province": "Lima", - "iso3166_2": "PE-LMA", - "country": "Peru", - "iso3166_1": "PE", - "latitude": -12.0464, - "longitude": -77.0428, - "local_languages": ["es-PE"] - }, - { - "city": "Tokyo", - "state_province": "Tokyo", - "iso3166_2": "JP-13", - "country": "Japan", - "iso3166_1": "JP", - "latitude": 35.6762, - "longitude": 139.6503, - "local_languages": ["ja"] - }, - { - "city": "Osaka", - "state_province": "Osaka", - "iso3166_2": "JP-27", - "country": "Japan", - "iso3166_1": "JP", - "latitude": 34.6937, - "longitude": 135.5023, - "local_languages": ["ja"] - }, - { - "city": "Kyoto", - "state_province": "Kyoto", - "iso3166_2": "JP-26", - "country": "Japan", - "iso3166_1": "JP", - "latitude": 35.0116, - "longitude": 135.7681, - "local_languages": ["ja"] - }, - { - "city": "Sapporo", - "state_province": "Hokkaido", - "iso3166_2": "JP-01", - "country": "Japan", - "iso3166_1": "JP", - "latitude": 43.0618, - "longitude": 141.3545, - "local_languages": ["ja"] - }, - { - "city": "Seoul", - "state_province": "Seoul", - "iso3166_2": "KR-11", - "country": "South Korea", - "iso3166_1": "KR", - "latitude": 37.5665, - "longitude": 126.978, - "local_languages": ["ko"] - }, - { - "city": "Busan", - "state_province": "Busan", - "iso3166_2": "KR-26", - "country": "South Korea", - "iso3166_1": "KR", - "latitude": 35.1796, - "longitude": 129.0756, - "local_languages": ["ko"] - }, - { - "city": "Ho Chi Minh City", - "state_province": "Ho Chi Minh", - "iso3166_2": "VN-SG", - "country": "Vietnam", - "iso3166_1": "VN", - "latitude": 10.8231, - "longitude": 106.6297, - "local_languages": ["vi"] - }, - { - "city": "Hanoi", - "state_province": "Hanoi", - "iso3166_2": "VN-HN", - "country": "Vietnam", - "iso3166_1": "VN", - "latitude": 21.0285, - "longitude": 105.8542, - "local_languages": ["vi"] - }, - { - "city": "Da Nang", - "state_province": "Da Nang", - "iso3166_2": "VN-DN", - "country": "Vietnam", - "iso3166_1": "VN", - "latitude": 16.0544, - "longitude": 108.2022, - "local_languages": ["vi"] - }, - { - "city": "Bangkok", - "state_province": "Bangkok", - "iso3166_2": "TH-10", - "country": "Thailand", - "iso3166_1": "TH", - "latitude": 13.7563, - "longitude": 100.5018, - "local_languages": ["th"] - }, - { - "city": "Taipei", - "state_province": "Taipei", - "iso3166_2": "TW-TPE", - "country": "Taiwan", - "iso3166_1": "TW", - "latitude": 25.033, - "longitude": 121.5654, - "local_languages": ["zh-TW"] - }, - { - "city": "Beijing", - "state_province": "Beijing", - "iso3166_2": "CN-BJ", - "country": "China", - "iso3166_1": "CN", - "latitude": 39.9042, - "longitude": 116.4074, - "local_languages": ["zh-CN"] - }, - { - "city": "Shanghai", - "state_province": "Shanghai", - "iso3166_2": "CN-SH", - "country": "China", - "iso3166_1": "CN", - "latitude": 31.2304, - "longitude": 121.4737, - "local_languages": ["zh-CN"] - }, - { - "city": "Bengaluru", - "state_province": "Karnataka", - "iso3166_2": "IN-KA", - "country": "India", - "iso3166_1": "IN", - "latitude": 12.9716, - "longitude": 77.5946, - "local_languages": ["kn", "en"] - }, - { - "city": "Singapore", - "state_province": "Central Singapore", - "iso3166_2": "SG-01", - "country": "Singapore", - "iso3166_1": "SG", - "latitude": 1.3521, - "longitude": 103.8198, - "local_languages": ["en", "zh", "ms", "ta"] - }, - { - "city": "Melbourne", - "state_province": "Victoria", - "iso3166_2": "AU-VIC", - "country": "Australia", - "iso3166_1": "AU", - "latitude": -37.8136, - "longitude": 144.9631, - "local_languages": ["en-AU"] - }, - { - "city": "Sydney", - "state_province": "New South Wales", - "iso3166_2": "AU-NSW", - "country": "Australia", - "iso3166_1": "AU", - "latitude": -33.8688, - "longitude": 151.2093, - "local_languages": ["en-AU"] - }, - { - "city": "Brisbane", - "state_province": "Queensland", - "iso3166_2": "AU-QLD", - "country": "Australia", - "iso3166_1": "AU", - "latitude": -27.4705, - "longitude": 153.026, - "local_languages": ["en-AU"] - }, - { - "city": "Adelaide", - "state_province": "South Australia", - "iso3166_2": "AU-SA", - "country": "Australia", - "iso3166_1": "AU", - "latitude": -34.9285, - "longitude": 138.6007, - "local_languages": ["en-AU"] - }, - { - "city": "Perth", - "state_province": "Western Australia", - "iso3166_2": "AU-WA", - "country": "Australia", - "iso3166_1": "AU", - "latitude": -31.9505, - "longitude": 115.8605, - "local_languages": ["en-AU"] - }, - { - "city": "Hobart", - "state_province": "Tasmania", - "iso3166_2": "AU-TAS", - "country": "Australia", - "iso3166_1": "AU", - "latitude": -42.8821, - "longitude": 147.3272, - "local_languages": ["en-AU"] - }, - { - "city": "Wellington", - "state_province": "Wellington", - "iso3166_2": "NZ-WGN", - "country": "New Zealand", - "iso3166_1": "NZ", - "latitude": -41.2865, - "longitude": 174.7762, - "local_languages": ["en", "mi"] - }, - { - "city": "Auckland", - "state_province": "Auckland", - "iso3166_2": "NZ-AUK", - "country": "New Zealand", - "iso3166_1": "NZ", - "latitude": -36.8485, - "longitude": 174.7633, - "local_languages": ["en", "mi"] - }, - { - "city": "Christchurch", - "state_province": "Canterbury", - "iso3166_2": "NZ-CAN", - "country": "New Zealand", - "iso3166_1": "NZ", - "latitude": -43.532, - "longitude": 172.6306, - "local_languages": ["en", "mi"] - }, - { - "city": "Nelson", - "state_province": "Nelson", - "iso3166_2": "NZ-NSN", - "country": "New Zealand", - "iso3166_1": "NZ", - "latitude": -41.2706, - "longitude": 173.284, - "local_languages": ["en", "mi"] - }, - { - "city": "Munich", - "state_province": "Bavaria", - "iso3166_2": "DE-BY", - "country": "Germany", - "iso3166_1": "DE", - "latitude": 48.1351, - "longitude": 11.582, - "local_languages": ["de"] - }, - { - "city": "Berlin", - "state_province": "Berlin", - "iso3166_2": "DE-BE", - "country": "Germany", - "iso3166_1": "DE", - "latitude": 52.52, - "longitude": 13.405, - "local_languages": ["de"] - }, - { - "city": "Cologne", - "state_province": "North Rhine-Westphalia", - "iso3166_2": "DE-NW", - "country": "Germany", - "iso3166_1": "DE", - "latitude": 50.9375, - "longitude": 6.9603, - "local_languages": ["de"] - }, - { - "city": "Bamberg", - "state_province": "Bavaria", - "iso3166_2": "DE-BY", - "country": "Germany", - "iso3166_1": "DE", - "latitude": 49.8916, - "longitude": 10.8916, - "local_languages": ["de"] - }, - { - "city": "Brussels", - "state_province": "Brussels-Capital", - "iso3166_2": "BE-BRU", - "country": "Belgium", - "iso3166_1": "BE", - "latitude": 50.8503, - "longitude": 4.3517, - "local_languages": ["fr", "nl"] - }, - { - "city": "Antwerp", - "state_province": "Flanders", - "iso3166_2": "BE-VLG", - "country": "Belgium", - "iso3166_1": "BE", - "latitude": 51.2194, - "longitude": 4.4025, - "local_languages": ["nl"] - }, - { - "city": "Bruges", - "state_province": "Flanders", - "iso3166_2": "BE-VLG", - "country": "Belgium", - "iso3166_1": "BE", - "latitude": 51.2093, - "longitude": 3.2247, - "local_languages": ["nl"] - }, - { - "city": "London", - "state_province": "England", - "iso3166_2": "GB-ENG", - "country": "United Kingdom", - "iso3166_1": "GB", - "latitude": 51.5074, - "longitude": -0.1278, - "local_languages": ["en-GB"] - }, - { - "city": "Bristol", - "state_province": "England", - "iso3166_2": "GB-ENG", - "country": "United Kingdom", - "iso3166_1": "GB", - "latitude": 51.4545, - "longitude": -2.5879, - "local_languages": ["en-GB"] - }, - { - "city": "Edinburgh", - "state_province": "Scotland", - "iso3166_2": "GB-SCT", - "country": "United Kingdom", - "iso3166_1": "GB", - "latitude": 55.9533, - "longitude": -3.1883, - "local_languages": ["en-GB", "gd"] - }, - { - "city": "Glasgow", - "state_province": "Scotland", - "iso3166_2": "GB-SCT", - "country": "United Kingdom", - "iso3166_1": "GB", - "latitude": 55.8642, - "longitude": -4.2518, - "local_languages": ["en-GB", "gd"] - }, - { - "city": "Prague", - "state_province": "Prague", - "iso3166_2": "CZ-10", - "country": "Czechia", - "iso3166_1": "CZ", - "latitude": 50.0755, - "longitude": 14.4378, - "local_languages": ["cs"] - }, - { - "city": "Pilsen", - "state_province": "Plzeň", - "iso3166_2": "CZ-32", - "country": "Czechia", - "iso3166_1": "CZ", - "latitude": 49.7384, - "longitude": 13.3736, - "local_languages": ["cs"] - }, - { - "city": "Amsterdam", - "state_province": "North Holland", - "iso3166_2": "NL-NH", - "country": "Netherlands", - "iso3166_1": "NL", - "latitude": 52.3676, - "longitude": 4.9041, - "local_languages": ["nl"] - }, - { - "city": "Copenhagen", - "state_province": "Capital Region", - "iso3166_2": "DK-84", - "country": "Denmark", - "iso3166_1": "DK", - "latitude": 55.6761, - "longitude": 12.5683, - "local_languages": ["da"] - }, - { - "city": "Warsaw", - "state_province": "Masovian", - "iso3166_2": "PL-MZ", - "country": "Poland", - "iso3166_1": "PL", - "latitude": 52.2297, - "longitude": 21.0122, - "local_languages": ["pl"] - }, - { - "city": "Krakow", - "state_province": "Lesser Poland", - "iso3166_2": "PL-MA", - "country": "Poland", - "iso3166_1": "PL", - "latitude": 50.0647, - "longitude": 19.945, - "local_languages": ["pl"] - }, - { - "city": "Rome", - "state_province": "Lazio", - "iso3166_2": "IT-62", - "country": "Italy", - "iso3166_1": "IT", - "latitude": 41.9028, - "longitude": 12.4964, - "local_languages": ["it"] - }, - { - "city": "Milan", - "state_province": "Lombardy", - "iso3166_2": "IT-25", - "country": "Italy", - "iso3166_1": "IT", - "latitude": 45.4642, - "longitude": 9.19, - "local_languages": ["it"] - }, - { - "city": "Barcelona", - "state_province": "Catalonia", - "iso3166_2": "ES-CT", - "country": "Spain", - "iso3166_1": "ES", - "latitude": 41.3851, - "longitude": 2.1734, - "local_languages": ["ca", "es"] - }, - { - "city": "Madrid", - "state_province": "Madrid", - "iso3166_2": "ES-MD", - "country": "Spain", - "iso3166_1": "ES", - "latitude": 40.4168, - "longitude": -3.7038, - "local_languages": ["es"] - }, - { - "city": "Paris", - "state_province": "Île-de-France", - "iso3166_2": "FR-IDF", - "country": "France", - "iso3166_1": "FR", - "latitude": 48.8566, - "longitude": 2.3522, - "local_languages": ["fr"] - }, - { - "city": "Lyon", - "state_province": "Auvergne-Rhône-Alpes", - "iso3166_2": "FR-ARA", - "country": "France", - "iso3166_1": "FR", - "latitude": 45.764, - "longitude": 4.8357, - "local_languages": ["fr"] - }, - { - "city": "Stockholm", - "state_province": "Stockholm", - "iso3166_2": "SE-AB", - "country": "Sweden", - "iso3166_1": "SE", - "latitude": 59.3293, - "longitude": 18.0686, - "local_languages": ["sv"] - }, - { - "city": "Gothenburg", - "state_province": "Västra Götaland", - "iso3166_2": "SE-O", - "country": "Sweden", - "iso3166_1": "SE", - "latitude": 57.7089, - "longitude": 11.9746, - "local_languages": ["sv"] - }, - { - "city": "Oslo", - "state_province": "Oslo", - "iso3166_2": "NO-03", - "country": "Norway", - "iso3166_1": "NO", - "latitude": 59.9139, - "longitude": 10.7522, - "local_languages": ["no"] - }, - { - "city": "Dublin", - "state_province": "Leinster", - "iso3166_2": "IE-L", - "country": "Ireland", - "iso3166_1": "IE", - "latitude": 53.3498, - "longitude": -6.2603, - "local_languages": ["en", "ga"] - }, - { - "city": "Vienna", - "state_province": "Vienna", - "iso3166_2": "AT-9", - "country": "Austria", - "iso3166_1": "AT", - "latitude": 48.2082, - "longitude": 16.3738, - "local_languages": ["de-AT"] - }, - { - "city": "Zurich", - "state_province": "Zurich", - "iso3166_2": "CH-ZH", - "country": "Switzerland", - "iso3166_1": "CH", - "latitude": 47.3769, - "longitude": 8.5417, - "local_languages": ["de-CH"] - }, - { - "city": "Tallinn", - "state_province": "Harju", - "iso3166_2": "EE-37", - "country": "Estonia", - "iso3166_1": "EE", - "latitude": 59.437, - "longitude": 24.7536, - "local_languages": ["et"] - }, - { - "city": "Denver", - "state_province": "Colorado", - "iso3166_2": "US-CO", - "country": "United States", - "iso3166_1": "US", - "latitude": 39.7392, - "longitude": -104.9903, - "local_languages": ["en-US"] - }, - { - "city": "Portland", - "state_province": "Oregon", - "iso3166_2": "US-OR", - "country": "United States", - "iso3166_1": "US", - "latitude": 45.5152, - "longitude": -122.6784, - "local_languages": ["en-US"] - }, - { - "city": "San Diego", - "state_province": "California", - "iso3166_2": "US-CA", - "country": "United States", - "iso3166_1": "US", - "latitude": 32.7157, - "longitude": -117.1611, - "local_languages": ["en-US", "es-US"] - }, - { - "city": "Asheville", - "state_province": "North Carolina", - "iso3166_2": "US-NC", - "country": "United States", - "iso3166_1": "US", - "latitude": 35.5951, - "longitude": -82.5515, - "local_languages": ["en-US"] - }, - { - "city": "Grand Rapids", - "state_province": "Michigan", - "iso3166_2": "US-MI", - "country": "United States", - "iso3166_1": "US", - "latitude": 42.9634, - "longitude": -85.6681, - "local_languages": ["en-US"] - }, - { - "city": "Chicago", - "state_province": "Illinois", - "iso3166_2": "US-IL", - "country": "United States", - "iso3166_1": "US", - "latitude": 41.8781, - "longitude": -87.6298, - "local_languages": ["en-US", "es-US"] - }, - { - "city": "Seattle", - "state_province": "Washington", - "iso3166_2": "US-WA", - "country": "United States", - "iso3166_1": "US", - "latitude": 47.6062, - "longitude": -122.3321, - "local_languages": ["en-US"] - }, - { - "city": "Austin", - "state_province": "Texas", - "iso3166_2": "US-TX", - "country": "United States", - "iso3166_1": "US", - "latitude": 30.2672, - "longitude": -97.7431, - "local_languages": ["en-US", "es-US"] - }, - { - "city": "Boston", - "state_province": "Massachusetts", - "iso3166_2": "US-MA", - "country": "United States", - "iso3166_1": "US", - "latitude": 42.3601, - "longitude": -71.0589, - "local_languages": ["en-US"] - }, - { - "city": "Philadelphia", - "state_province": "Pennsylvania", - "iso3166_2": "US-PA", - "country": "United States", - "iso3166_1": "US", - "latitude": 39.9526, - "longitude": -75.1652, - "local_languages": ["en-US"] - }, - { - "city": "Brooklyn", - "state_province": "New York", - "iso3166_2": "US-NY", - "country": "United States", - "iso3166_1": "US", - "latitude": 40.6782, - "longitude": -73.9442, - "local_languages": ["en-US", "es-US"] - }, - { - "city": "Milwaukee", - "state_province": "Wisconsin", - "iso3166_2": "US-WI", - "country": "United States", - "iso3166_1": "US", - "latitude": 43.0389, - "longitude": -87.9065, - "local_languages": ["en-US"] - }, - { - "city": "Richmond", - "state_province": "Virginia", - "iso3166_2": "US-VA", - "country": "United States", - "iso3166_1": "US", - "latitude": 37.5407, - "longitude": -77.436, - "local_languages": ["en-US"] - }, - { - "city": "Cincinnati", - "state_province": "Ohio", - "iso3166_2": "US-OH", - "country": "United States", - "iso3166_1": "US", - "latitude": 39.1031, - "longitude": -84.512, - "local_languages": ["en-US"] - }, - { - "city": "St. Louis", - "state_province": "Missouri", - "iso3166_2": "US-MO", - "country": "United States", - "iso3166_1": "US", - "latitude": 38.627, - "longitude": -90.1994, - "local_languages": ["en-US"] - }, - { - "city": "Tampa", - "state_province": "Florida", - "iso3166_2": "US-FL", - "country": "United States", - "iso3166_1": "US", - "latitude": 27.9506, - "longitude": -82.4572, - "local_languages": ["en-US", "es-US"] - }, - { - "city": "Minneapolis", - "state_province": "Minnesota", - "iso3166_2": "US-MN", - "country": "United States", - "iso3166_1": "US", - "latitude": 44.9778, - "longitude": -93.265, - "local_languages": ["en-US"] - }, - { - "city": "Burlington", - "state_province": "Vermont", - "iso3166_2": "US-VT", - "country": "United States", - "iso3166_1": "US", - "latitude": 44.4759, - "longitude": -73.2121, - "local_languages": ["en-US"] - }, - { - "city": "Portland", - "state_province": "Maine", - "iso3166_2": "US-ME", - "country": "United States", - "iso3166_1": "US", - "latitude": 43.6591, - "longitude": -70.2568, - "local_languages": ["en-US"] - }, - { - "city": "Atlanta", - "state_province": "Georgia", - "iso3166_2": "US-GA", - "country": "United States", - "iso3166_1": "US", - "latitude": 33.749, - "longitude": -84.388, - "local_languages": ["en-US"] - }, - { - "city": "Toronto", - "state_province": "Ontario", - "iso3166_2": "CA-ON", - "country": "Canada", - "iso3166_1": "CA", - "latitude": 43.651, - "longitude": -79.347, - "local_languages": ["en-CA"] - }, - { - "city": "Vancouver", - "state_province": "British Columbia", - "iso3166_2": "CA-BC", - "country": "Canada", - "iso3166_1": "CA", - "latitude": 49.2827, - "longitude": -123.1207, - "local_languages": ["en-CA"] - }, - { - "city": "Montreal", - "state_province": "Quebec", - "iso3166_2": "CA-QC", - "country": "Canada", - "iso3166_1": "CA", - "latitude": 45.5017, - "longitude": -73.5673, - "local_languages": ["fr-CA", "en-CA"] - }, - { - "city": "Calgary", - "state_province": "Alberta", - "iso3166_2": "CA-AB", - "country": "Canada", - "iso3166_1": "CA", - "latitude": 51.0447, - "longitude": -114.0719, - "local_languages": ["en-CA"] - }, - { - "city": "Halifax", - "state_province": "Nova Scotia", - "iso3166_2": "CA-NS", - "country": "Canada", - "iso3166_1": "CA", - "latitude": 44.6488, - "longitude": -63.5752, - "local_languages": ["en-CA"] - }, - { - "city": "Mexico City", - "state_province": "Mexico City", - "iso3166_2": "MX-CMX", - "country": "Mexico", - "iso3166_1": "MX", - "latitude": 19.4326, - "longitude": -99.1332, - "local_languages": ["es-MX"] - }, - { - "city": "Tijuana", - "state_province": "Baja California", - "iso3166_2": "MX-BCN", - "country": "Mexico", - "iso3166_1": "MX", - "latitude": 32.5149, - "longitude": -117.0382, - "local_languages": ["es-MX"] - }, - { - "city": "Monterrey", - "state_province": "Nuevo León", - "iso3166_2": "MX-NLE", - "country": "Mexico", - "iso3166_1": "MX", - "latitude": 25.6866, - "longitude": -100.3161, - "local_languages": ["es-MX"] - }, - { - "city": "Guadalajara", - "state_province": "Jalisco", - "iso3166_2": "MX-JAL", - "country": "Mexico", - "iso3166_1": "MX", - "latitude": 20.6597, - "longitude": -103.3496, - "local_languages": ["es-MX"] - }, - { - "city": "Ensenada", - "state_province": "Baja California", - "iso3166_2": "MX-BCN", - "country": "Mexico", - "iso3166_1": "MX", - "latitude": 31.8667, - "longitude": -116.5964, - "local_languages": ["es-MX"] - } -] diff --git a/tooling/pipeline/runpod/Dockerfile b/tooling/pipeline/runpod/Dockerfile index 09fa455..b3d419f 100644 --- a/tooling/pipeline/runpod/Dockerfile +++ b/tooling/pipeline/runpod/Dockerfile @@ -52,7 +52,7 @@ WORKDIR /app/build COPY --from=builder /app/build/biergarten-pipeline ./ # Copy required config files -COPY locations.json /app/build/ +COPY cities.json /app/build/ COPY beer-styles.json /app/build/ # Copy prompt templates diff --git a/tooling/pipeline/src/biergarten_pipeline_orchestrator/biergarten_pipeline_orchestrator.cc b/tooling/pipeline/src/biergarten_pipeline_orchestrator/biergarten_pipeline_orchestrator.cc index eae518a..297eb2f 100644 --- a/tooling/pipeline/src/biergarten_pipeline_orchestrator/biergarten_pipeline_orchestrator.cc +++ b/tooling/pipeline/src/biergarten_pipeline_orchestrator/biergarten_pipeline_orchestrator.cc @@ -7,16 +7,20 @@ #include +#include "services/postal_code/postal_code_service.h" + BiergartenPipelineOrchestrator::BiergartenPipelineOrchestrator( std::shared_ptr logger, std::unique_ptr context_service, std::unique_ptr generator, std::unique_ptr exporter, std::unique_ptr curated_data_service, + std::unique_ptr postal_code_service, const ApplicationOptions& application_options) : logger_(std::move(logger)), context_service_(std::move(context_service)), generator_(std::move(generator)), exporter_(std::move(exporter)), curated_data_service_(std::move(curated_data_service)), + postal_code_service_(std::move(postal_code_service)), application_options_(application_options) {} \ No newline at end of file diff --git a/tooling/pipeline/src/biergarten_pipeline_orchestrator/generate_breweries.cc b/tooling/pipeline/src/biergarten_pipeline_orchestrator/generate_breweries.cc index 0ee023b..70d575f 100644 --- a/tooling/pipeline/src/biergarten_pipeline_orchestrator/generate_breweries.cc +++ b/tooling/pipeline/src/biergarten_pipeline_orchestrator/generate_breweries.cc @@ -22,12 +22,16 @@ void BiergartenPipelineOrchestrator::GenerateBreweries( const auto generate_record = [this, &skipped_count]( - const Location& location, + const City& location, const std::string& region_context) -> std::optional { try { const BreweryResult brewery = generator_->GenerateBrewery(location, region_context); - return BreweryRecord{.location = location, .brewery = brewery}; + const std::string postal_code = + postal_code_service_->GeneratePostalCode(location); + return BreweryRecord{.location = location, + .address = Address{.postal_code = postal_code}, + .brewery = brewery}; } catch (const std::exception& e) { ++skipped_count; diff --git a/tooling/pipeline/src/biergarten_pipeline_orchestrator/log_results.cc b/tooling/pipeline/src/biergarten_pipeline_orchestrator/log_results.cc index 9587ced..1ab7393 100644 --- a/tooling/pipeline/src/biergarten_pipeline_orchestrator/log_results.cc +++ b/tooling/pipeline/src/biergarten_pipeline_orchestrator/log_results.cc @@ -14,12 +14,13 @@ void BiergartenPipelineOrchestrator::LogResults() const { boost::json::array brewery_output; - for (const auto& [location, brewery] : generated_breweries_) { + for (const auto& [location, address, brewery] : generated_breweries_) { brewery_output.push_back(boost::json::object{ {"name_en", brewery.name_en}, {"description_en", brewery.description_en}, {"name_local", brewery.name_local}, {"description_local", brewery.description_local}, + {"postal_code", address.postal_code}, {"location", boost::json::object{ {"city", location.city}, {"country", location.country}, diff --git a/tooling/pipeline/src/biergarten_pipeline_orchestrator/query_cities_with_countries.cc b/tooling/pipeline/src/biergarten_pipeline_orchestrator/query_cities_with_countries.cc index f3549dd..675edd0 100644 --- a/tooling/pipeline/src/biergarten_pipeline_orchestrator/query_cities_with_countries.cc +++ b/tooling/pipeline/src/biergarten_pipeline_orchestrator/query_cities_with_countries.cc @@ -14,14 +14,14 @@ #include "services/curated_data/curated_json_data_service.h" #include "services/logging/logger.h" -std::vector +std::vector BiergartenPipelineOrchestrator::QueryCitiesWithCountries() { logger_->Log({.level = LogLevel::Info, .phase = PipelinePhase::Startup, .message = "=== GEOGRAPHIC DATA OVERVIEW ==="}); - const std::vector& all_locations = - curated_data_service_->LoadLocations(); + const std::vector& all_locations = + curated_data_service_->LoadCities(); const size_t sample_count = std::min( static_cast(application_options_.pipeline.location_count), @@ -31,7 +31,7 @@ BiergartenPipelineOrchestrator::QueryCitiesWithCountries() { static_cast>( sample_count); - std::vector sampled_locations; + std::vector sampled_locations; sampled_locations.reserve(sample_count); std::random_device random_generator; diff --git a/tooling/pipeline/src/biergarten_pipeline_orchestrator/run.cc b/tooling/pipeline/src/biergarten_pipeline_orchestrator/run.cc index 5ada0f6..6f20b41 100644 --- a/tooling/pipeline/src/biergarten_pipeline_orchestrator/run.cc +++ b/tooling/pipeline/src/biergarten_pipeline_orchestrator/run.cc @@ -14,7 +14,7 @@ bool BiergartenPipelineOrchestrator::Run() { try { exporter_->Initialize(); - std::vector cities = QueryCitiesWithCountries(); + std::vector cities = QueryCitiesWithCountries(); std::vector enriched; enriched.reserve(cities.size()); diff --git a/tooling/pipeline/src/data_generation/llama/generate_brewery.cc b/tooling/pipeline/src/data_generation/llama/generate_brewery.cc index 30c4ac6..75e064d 100644 --- a/tooling/pipeline/src/data_generation/llama/generate_brewery.cc +++ b/tooling/pipeline/src/data_generation/llama/generate_brewery.cc @@ -36,7 +36,7 @@ static std::string FormatLocalLanguageCodes( static constexpr int kBreweryInitialMaxTokens = 2800; BreweryResult LlamaGenerator::GenerateBrewery( - const Location& location, const std::string& region_context) { + const City& location, const std::string& region_context) { /** * Preprocess and truncate region context to manageable size */ diff --git a/tooling/pipeline/src/data_generation/mock/deterministic_hash.cc b/tooling/pipeline/src/data_generation/mock/deterministic_hash.cc index 06d46b7..26adc6a 100644 --- a/tooling/pipeline/src/data_generation/mock/deterministic_hash.cc +++ b/tooling/pipeline/src/data_generation/mock/deterministic_hash.cc @@ -8,14 +8,14 @@ #include "data_generation/mock_generator.h" -size_t MockGenerator::DeterministicHash(const Location& location) { +size_t MockGenerator::DeterministicHash(const City& location) { size_t seed = 0; boost::hash_combine(seed, location.city); boost::hash_combine(seed, location.country); return seed; } -size_t MockGenerator::DeterministicHash(const Location& location, +size_t MockGenerator::DeterministicHash(const City& location, const UserPersona& persona, const Name& name) { size_t seed = DeterministicHash(location); diff --git a/tooling/pipeline/src/data_generation/mock/generate_brewery.cc b/tooling/pipeline/src/data_generation/mock/generate_brewery.cc index 20a4ff6..0aabf2d 100644 --- a/tooling/pipeline/src/data_generation/mock/generate_brewery.cc +++ b/tooling/pipeline/src/data_generation/mock/generate_brewery.cc @@ -11,7 +11,7 @@ #include "data_generation/mock_generator.h" BreweryResult MockGenerator::GenerateBrewery( - const Location& location, const std::string& /*region_context*/) { + const City& location, const std::string& /*region_context*/) { const size_t hash = DeterministicHash(location); const std::string_view adjective = diff --git a/tooling/pipeline/src/main.cc b/tooling/pipeline/src/main.cc index 3086ff6..c26b085 100644 --- a/tooling/pipeline/src/main.cc +++ b/tooling/pipeline/src/main.cc @@ -102,12 +102,14 @@ int main(const int argc, char** argv) { return std::make_unique( CuratedDataFilePaths{ - .locations_path = "locations.json", + .cities_path = "cities.json", .personas_path = "personas.json", .forenames_path = "forenames-by-country.json", .surnames_path = "surnames-by-country.json", }); }), + di::bind().to(), + di::bind().to([options, log_producer] { if (options.generator.use_mocked) { { @@ -226,4 +228,4 @@ int main(const int argc, char** argv) { return shutdown(EXIT_FAILURE); } -} \ No newline at end of file +} diff --git a/tooling/pipeline/src/services/curated_data/curated_json_data_service.cc b/tooling/pipeline/src/services/curated_data/curated_json_data_service.cc index d9a14c3..4e08a1b 100644 --- a/tooling/pipeline/src/services/curated_data/curated_json_data_service.cc +++ b/tooling/pipeline/src/services/curated_data/curated_json_data_service.cc @@ -1,6 +1,6 @@ /** * @file json_handling/json_loader.cc - * @brief Parses curated location JSON input into strongly typed Location + * @brief Parses curated location JSON input into strongly typed City * records with strict field validation and descriptive error reporting. */ @@ -109,31 +109,38 @@ std::string ReadFirstOfStringArray(const boost::json::object& object, CuratedJsonDataService::CuratedJsonDataService(CuratedDataFilePaths filepaths) : filepaths_(std::move(filepaths)) {} -const LocationsList& CuratedJsonDataService::LoadLocations() { +const CityList& CuratedJsonDataService::LoadCities() { if (!cache_.locations.empty()) { return cache_.locations; } const boost::json::value root = - ParseJsonFile(filepaths_.locations_path, "locations"); + ParseJsonFile(filepaths_.cities_path, "cities"); if (!root.is_array()) { throw std::runtime_error( - "Invalid locations JSON: root element must be an array"); + "Invalid cities JSON: root element must be an array"); } - LocationsList locations; + CityList locations; const auto& items = root.as_array(); locations.reserve(items.size()); for (const auto& item : items) { if (!item.is_object()) { throw std::runtime_error( - "Invalid locations JSON: each entry must be an object"); + "Invalid cities JSON: each entry must be an object"); } const auto& object = item.as_object(); - locations.push_back(Location{ + + const boost::json::value* postal_code = object.if_contains("postal_code"); + if (postal_code == nullptr || !postal_code->is_object()) { + throw std::runtime_error( + "Missing or invalid object field: postal_code"); + } + + locations.push_back(City{ .city = ReadRequiredString(object, "city"), .state_province = ReadRequiredString(object, "state_province"), .iso3166_2 = ReadRequiredString(object, "iso3166_2"), @@ -142,8 +149,23 @@ const LocationsList& CuratedJsonDataService::LoadLocations() { .local_languages = ReadRequiredStringArray(object, "local_languages"), .latitude = ReadRequiredNumber(object, "latitude"), .longitude = ReadRequiredNumber(object, "longitude"), + .postal_regex = + ReadRequiredStringArray(postal_code->as_object(), "city_regex"), + .postal_code_examples = + ReadRequiredStringArray(postal_code->as_object(), "examples"), }); } + + for (auto location : locations) { + std::cout << "Location: " << location.city << ", " + << location.state_province << ", " << location.iso3166_2 << ", " + << location.country << ", " << location.iso3166_1 << ", " + << location.latitude << ", " << location.longitude << std::endl; + + for (const auto& regex : location.postal_regex) { + std::cout << " Postal regex: " << regex << std::endl; + } + } cache_.locations = std::move(locations); return cache_.locations; } diff --git a/tooling/pipeline/src/services/curated_data/mock_curated_data_service.cc b/tooling/pipeline/src/services/curated_data/mock_curated_data_service.cc index dda4184..31264a7 100644 --- a/tooling/pipeline/src/services/curated_data/mock_curated_data_service.cc +++ b/tooling/pipeline/src/services/curated_data/mock_curated_data_service.cc @@ -7,38 +7,46 @@ MockCuratedDataService::MockCuratedDataService() : locations_{ - Location{.city = "Portland", + City{.city = "Portland", .state_province = "Oregon", .iso3166_2 = "US-OR", .country = "United States", .iso3166_1 = "US", .local_languages = {"en"}, .latitude = 45.5152, - .longitude = -122.6784}, - Location{.city = "Munich", + .longitude = -122.6784, + .postal_regex = {"^972[0-9]{2}(?:-[0-9]{4})?$"}, + .postal_code_examples = {"97201", "97294"}}, + City{.city = "Munich", .state_province = "Bavaria", .iso3166_2 = "DE-BY", .country = "Germany", .iso3166_1 = "DE", .local_languages = {"de"}, .latitude = 48.1351, - .longitude = 11.5820}, - Location{.city = "Lyon", + .longitude = 11.5820, + .postal_regex = {"^8[01][0-9]{3}$"}, + .postal_code_examples = {"80331", "81929"}}, + City{.city = "Lyon", .state_province = "Auvergne-Rhone-Alpes", .iso3166_2 = "FR-ARA", .country = "France", .iso3166_1 = "FR", .local_languages = {"fr"}, .latitude = 45.7640, - .longitude = 4.8357}, - Location{.city = "Brussels", + .longitude = 4.8357, + .postal_regex = {"^6900[1-9]$"}, + .postal_code_examples = {"69001", "69009"}}, + City{.city = "Brussels", .state_province = "Brussels-Capital", .iso3166_2 = "BE-BRU", .country = "Belgium", .iso3166_1 = "BE", .local_languages = {"nl", "fr"}, .latitude = 50.8503, - .longitude = 4.3517}, + .longitude = 4.3517, + .postal_regex = {"^1(?:0[0-9]{2}|1[0-9]{2}|20[0-9]|210)$"}, + .postal_code_examples = {"1000", "1210"}}, }, personas_{ UserPersona{.name = "Hophead Explorer", @@ -83,7 +91,7 @@ MockCuratedDataService::MockCuratedDataService() {"BE", SurnameList{"Peeters", "Janssens"}}, } {} -const LocationsList& MockCuratedDataService::LoadLocations() { +const CityList& MockCuratedDataService::LoadCities() { return locations_; } diff --git a/tooling/pipeline/src/services/enrichment/wikipedia/get_summary.cc b/tooling/pipeline/src/services/enrichment/wikipedia/get_summary.cc index 342f84a..07cfde7 100644 --- a/tooling/pipeline/src/services/enrichment/wikipedia/get_summary.cc +++ b/tooling/pipeline/src/services/enrichment/wikipedia/get_summary.cc @@ -11,7 +11,7 @@ #include "services/enrichment/wikipedia_service.h" std::string WikipediaEnrichmentService::GetLocationContext( - const Location& loc) { + const City& loc) { using namespace std::literals::chrono_literals; if (!this->client_) { if (logger_) { diff --git a/tooling/pipeline/src/services/sqlite/finalize.cc b/tooling/pipeline/src/services/sqlite/finalize.cc index c4e4ff6..40fe643 100644 --- a/tooling/pipeline/src/services/sqlite/finalize.cc +++ b/tooling/pipeline/src/services/sqlite/finalize.cc @@ -16,7 +16,7 @@ void SqliteExportService::Finalize() { try { insert_user_stmt_.reset(); insert_brewery_stmt_.reset(); - insert_location_stmt_.reset(); + insert_city_stmt_.reset(); if (transaction_open_) { sqlite_export_service_internal::ExecSql( db_handle_, "COMMIT;", "Failed to commit SQLite transaction"); @@ -24,7 +24,7 @@ void SqliteExportService::Finalize() { } db_handle_.reset(); - location_cache_.clear(); + city_cache_.clear(); } catch (...) { RollbackAndCloseNoThrow(); throw; diff --git a/tooling/pipeline/src/services/sqlite/initialize.cc b/tooling/pipeline/src/services/sqlite/initialize.cc index 4c7def2..e23e194 100644 --- a/tooling/pipeline/src/services/sqlite/initialize.cc +++ b/tooling/pipeline/src/services/sqlite/initialize.cc @@ -28,8 +28,8 @@ std::filesystem::path SqliteExportService::BuildDatabasePath() const { void SqliteExportService::InitializeSchema() const { sqlite_export_service_internal::ExecSql( - db_handle_, sqlite_export_service_internal::kCreateLocationsTableSql, - "Failed to create SQLite locations table"); + db_handle_, sqlite_export_service_internal::kCreateCitiesTableSql, + "Failed to create SQLite cities table"); sqlite_export_service_internal::ExecSql( db_handle_, sqlite_export_service_internal::kCreateBreweriesTableSql, "Failed to create SQLite breweries table"); @@ -39,9 +39,9 @@ void SqliteExportService::InitializeSchema() const { } void SqliteExportService::PrepareStatements() { - insert_location_stmt_ = sqlite_export_service_internal::PrepareStatement( - db_handle_, sqlite_export_service_internal::kInsertLocationSql, - "Failed to prepare SQLite location insert statement"); + insert_city_stmt_ = sqlite_export_service_internal::PrepareStatement( + db_handle_, sqlite_export_service_internal::kInsertCitySql, + "Failed to prepare SQLite city insert statement"); insert_brewery_stmt_ = sqlite_export_service_internal::PrepareStatement( db_handle_, sqlite_export_service_internal::kInsertBrewerySql, "Failed to prepare SQLite brewery insert statement"); @@ -62,9 +62,9 @@ void SqliteExportService::RollbackAndCloseNoThrow() noexcept { insert_user_stmt_.reset(); insert_brewery_stmt_.reset(); - insert_location_stmt_.reset(); + insert_city_stmt_.reset(); db_handle_.reset(); - location_cache_.clear(); + city_cache_.clear(); } void SqliteExportService::Initialize() { diff --git a/tooling/pipeline/src/services/sqlite/process_record.cc b/tooling/pipeline/src/services/sqlite/process_record.cc index 9acc0fc..ac35ee3 100644 --- a/tooling/pipeline/src/services/sqlite/process_record.cc +++ b/tooling/pipeline/src/services/sqlite/process_record.cc @@ -1,7 +1,7 @@ /** * @file services/sqlite/process_record.cc * @brief SqliteExportService::ProcessRecord() implementation - * and the shared location-resolution helper. + * and the shared city-resolution helper. */ #include @@ -14,7 +14,7 @@ constexpr int kLocationPrecision = 17; -std::string SqliteExportService::BuildLocationKey(const Location& location) { +std::string SqliteExportService::BuildCityKey(const City& location) { std::ostringstream key_stream; key_stream << location.city << '\n' << location.state_province << '\n' @@ -30,76 +30,76 @@ std::string SqliteExportService::BuildLocationKey(const Location& location) { return key_stream.str(); } -sqlite3_int64 SqliteExportService::ResolveLocationId(const Location& location) { - const std::string location_key = BuildLocationKey(location); - const auto cached_location = location_cache_.find(location_key); - if (cached_location != location_cache_.end()) { - return cached_location->second; +sqlite3_int64 SqliteExportService::ResolveCityId(const City& location) { + const std::string city_key = BuildCityKey(location); + const auto cached_city = city_cache_.find(city_key); + if (cached_city != city_cache_.end()) { + return cached_city->second; } const std::string local_languages_json = sqlite_export_service_internal::SerializeVector(location.local_languages); sqlite_export_service_internal::Bind( - insert_location_stmt_, + insert_city_stmt_, sqlite_export_service_internal::BoundParam{ - .index = sqlite_export_service_internal::kLocationCityBindIndex, + .index = sqlite_export_service_internal::kCityNameBindIndex, .value = location.city, - .action = "Failed to bind SQLite location city"}); + .action = "Failed to bind SQLite city name"}); sqlite_export_service_internal::Bind( - insert_location_stmt_, + insert_city_stmt_, sqlite_export_service_internal::BoundParam{ .index = - sqlite_export_service_internal::kLocationStateProvinceBindIndex, + sqlite_export_service_internal::kCityStateProvinceBindIndex, .value = location.state_province, - .action = "Failed to bind SQLite location state/province"}); + .action = "Failed to bind SQLite city state/province"}); sqlite_export_service_internal::Bind( - insert_location_stmt_, + insert_city_stmt_, sqlite_export_service_internal::BoundParam{ - .index = sqlite_export_service_internal::kLocationIso31662BindIndex, + .index = sqlite_export_service_internal::kCityIso31662BindIndex, .value = location.iso3166_2, - .action = "Failed to bind SQLite location ISO 3166-2 code"}); + .action = "Failed to bind SQLite city ISO 3166-2 code"}); sqlite_export_service_internal::Bind( - insert_location_stmt_, + insert_city_stmt_, sqlite_export_service_internal::BoundParam{ - .index = sqlite_export_service_internal::kLocationCountryBindIndex, + .index = sqlite_export_service_internal::kCityCountryBindIndex, .value = location.country, - .action = "Failed to bind SQLite location country"}); + .action = "Failed to bind SQLite city country"}); sqlite_export_service_internal::Bind( - insert_location_stmt_, + insert_city_stmt_, sqlite_export_service_internal::BoundParam{ - .index = sqlite_export_service_internal::kLocationIso31661BindIndex, + .index = sqlite_export_service_internal::kCityIso31661BindIndex, .value = location.iso3166_1, - .action = "Failed to bind SQLite location ISO 3166-1 code"}); + .action = "Failed to bind SQLite city ISO 3166-1 code"}); sqlite_export_service_internal::Bind( - insert_location_stmt_, + insert_city_stmt_, sqlite_export_service_internal::BoundParam{ - .index = sqlite_export_service_internal::kLocationLanguagesBindIndex, + .index = sqlite_export_service_internal::kCityLanguagesBindIndex, .value = local_languages_json, - .action = "Failed to bind SQLite location languages"}); + .action = "Failed to bind SQLite city languages"}); sqlite_export_service_internal::Bind( - insert_location_stmt_, + insert_city_stmt_, sqlite_export_service_internal::BoundParam{ - .index = sqlite_export_service_internal::kLocationLatitudeBindIndex, + .index = sqlite_export_service_internal::kCityLatitudeBindIndex, .value = location.latitude, - .action = "Failed to bind SQLite location latitude"}); + .action = "Failed to bind SQLite city latitude"}); sqlite_export_service_internal::Bind( - insert_location_stmt_, + insert_city_stmt_, sqlite_export_service_internal::BoundParam{ - .index = sqlite_export_service_internal::kLocationLongitudeBindIndex, + .index = sqlite_export_service_internal::kCityLongitudeBindIndex, .value = location.longitude, - .action = "Failed to bind SQLite location longitude"}); + .action = "Failed to bind SQLite city longitude"}); sqlite_export_service_internal::StepStatement( - db_handle_, insert_location_stmt_, - "Failed to insert SQLite location row"); + db_handle_, insert_city_stmt_, + "Failed to insert SQLite city row"); - const sqlite3_int64 location_id = + const sqlite3_int64 city_id = sqlite_export_service_internal::LastInsertRowId(db_handle_); - location_cache_.emplace(location_key, location_id); - sqlite_export_service_internal::ResetStatement(insert_location_stmt_); + city_cache_.emplace(city_key, city_id); + sqlite_export_service_internal::ResetStatement(insert_city_stmt_); - return location_id; + return city_id; } uint64_t SqliteExportService::ProcessRecord(const BreweryRecord& brewery) { @@ -107,14 +107,14 @@ uint64_t SqliteExportService::ProcessRecord(const BreweryRecord& brewery) { throw std::runtime_error("SQLite export service is not initialized"); } - const sqlite3_int64 location_id = ResolveLocationId(brewery.location); + const sqlite3_int64 city_id = ResolveCityId(brewery.location); sqlite_export_service_internal::Bind( insert_brewery_stmt_, sqlite_export_service_internal::BoundParam{ - .index = sqlite_export_service_internal::kBreweryLocationIdBindIndex, - .value = location_id, - .action = "Failed to bind SQLite brewery location id"}); + .index = sqlite_export_service_internal::kBreweryCityIdBindIndex, + .value = city_id, + .action = "Failed to bind SQLite brewery city id"}); sqlite_export_service_internal::Bind( insert_brewery_stmt_, @@ -146,6 +146,13 @@ uint64_t SqliteExportService::ProcessRecord(const BreweryRecord& brewery) { .value = brewery.brewery.description_local, .action = "Failed to bind SQLite brewery local description"}); + sqlite_export_service_internal::Bind( + insert_brewery_stmt_, + sqlite_export_service_internal::BoundParam{ + .index = sqlite_export_service_internal::kBreweryPostalCodeBindIndex, + .value = brewery.address.postal_code, + .action = "Failed to bind SQLite brewery postal code"}); + sqlite_export_service_internal::StepStatement( db_handle_, insert_brewery_stmt_, "Failed to insert SQLite brewery row"); diff --git a/tooling/pipeline/src/services/sqlite/process_user_record.cc b/tooling/pipeline/src/services/sqlite/process_user_record.cc index 34bac07..9288386 100644 --- a/tooling/pipeline/src/services/sqlite/process_user_record.cc +++ b/tooling/pipeline/src/services/sqlite/process_user_record.cc @@ -13,14 +13,14 @@ uint64_t SqliteExportService::ProcessRecord(const UserRecord& user) { throw std::runtime_error("SQLite export service is not initialized"); } - const sqlite3_int64 location_id = ResolveLocationId(user.location); + const sqlite3_int64 city_id = ResolveCityId(user.location); sqlite_export_service_internal::Bind( insert_user_stmt_, sqlite_export_service_internal::BoundParam{ - .index = sqlite_export_service_internal::kUserLocationIdBindIndex, - .value = location_id, - .action = "Failed to bind SQLite user location id"}); + .index = sqlite_export_service_internal::kUserCityIdBindIndex, + .value = city_id, + .action = "Failed to bind SQLite user city id"}); sqlite_export_service_internal::Bind( insert_user_stmt_, sqlite_export_service_internal::BoundParam{