mirror of
https://github.com/aaronpo97/the-biergarten-app.git
synced 2026-10-14 17:54:42 +00:00
Refactor documentation and diagrams for web/pipeline. (#272)
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> for assistance with diagram creation and documentation prose updates.
This commit is contained in:
Aaron Po
@@ -1,4 +1,13 @@
|
||||
# Ethics, bias, and known issues
|
||||
---
|
||||
title: Pipeline ethics, bias, and known issues
|
||||
last-updated: 2026-08-31
|
||||
tags:
|
||||
- pipeline
|
||||
- ethics
|
||||
- bias
|
||||
- known-issues
|
||||
- ai-generated
|
||||
---
|
||||
|
||||
This document covers the ethical context of the Biergarten Pipeline's output,
|
||||
the model's biases, and known issues including hallucinated brewing science and
|
||||
@@ -47,7 +56,7 @@ as a source of truth for brewing techniques or cultural representation.**
|
||||
|
||||
**Natural language processing is a powerful tool for data analysis and
|
||||
generation, but its output should be treated with scrutiny. Human language is
|
||||
not simply data points to be analyzed — it carries cultural and human meaning
|
||||
not simply data points to be analyzed: it carries cultural and human meaning
|
||||
that artificial intelligence cannot capture.**
|
||||
|
||||
---
|
||||
@@ -102,7 +111,7 @@ vendored verbatim, unmodified, from
|
||||
(the `common-forenames-by-country.json` / `common-surnames-by-country.json`
|
||||
release assets), released under **CC0** (public domain). That dataset's own
|
||||
forename/surname lists are pulled from Wikipedia's "Lists of most common
|
||||
surnames" and "List of most popular given names" as of the week of 2023-07-08 —
|
||||
surnames" and "List of most popular given names" as of the week of 2023-07-08;
|
||||
see that project's README for full provenance. Names are not LLM-generated; this
|
||||
is curated fixture data per ROADMAP.md §2. Per-forename gender from the source
|
||||
data is preserved through to the sampled `Name` (rather than discarded during
|
||||
@@ -113,7 +122,7 @@ for surnames) rather than trimmed to `locations.json`'s current country list, so
|
||||
it doesn't need re-sourcing if more countries are added later. `SampleName()` (a
|
||||
free helper in `generate_users.cc`) returns no result for a country present in
|
||||
neither file; of the countries in `locations.json`, that's currently `KE`, `SE`,
|
||||
`SG`, `TH`, `VN`, and `ZA` — `GenerateUsers` skips cities in those countries the
|
||||
`SG`, `TH`, `VN`, and `ZA`; `GenerateUsers` skips cities in those countries the
|
||||
same way brewery generation skips cities whose enrichment lookup fails.
|
||||
|
||||
---
|
||||
@@ -290,11 +299,20 @@ Each brewery and user address carries a longitude/latitude pair, sampled
|
||||
uniformly at random within a 5 km disc centred on the city's curated
|
||||
`latitude`/`longitude` from `locations.json`.
|
||||
|
||||
The resulting point is **plausible by construction** — it always falls within
|
||||
the city's vicinity — but it is not looked up from any real address registry.
|
||||
It does not correspond to a real address, business, or resident, and should
|
||||
not be treated as accurate geocoding data. This is consistent with the rest of
|
||||
the dataset: fixture data for a proof-of-concept, not a source of truth.
|
||||
The coordinate pair itself is **plausible by construction**, always falling
|
||||
within the city's vicinity, but does not correspond to a real business or
|
||||
resident, and should not be treated as accurate geocoding data. This is
|
||||
consistent with the rest of the dataset: fixture data for a proof-of-concept,
|
||||
not a source of truth.
|
||||
|
||||
For brewery addresses specifically (not user addresses, which carry no street
|
||||
address at all), `address_line1`/`postal_code` _are_ looked up from a real
|
||||
address registry — `NominatimAddressService` reverse-geocodes the synthetic
|
||||
coordinate pair against the live OpenStreetMap/Nominatim database, so the street
|
||||
address is a genuine nearby address, just not the address of an actual brewery.
|
||||
This lookup only happens for non-`--mocked` runs; `--mocked` runs use
|
||||
`MockAddressService`'s fixed placeholder address instead, which is not looked up
|
||||
from any real registry.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -1,4 +1,13 @@
|
||||
# Biergarten Pipeline
|
||||
---
|
||||
title: Biergarten Pipeline — C++ CLI for generating seed data
|
||||
last-updated: 2026-09-02
|
||||
tags:
|
||||
- pipeline
|
||||
- cpp
|
||||
- llm
|
||||
- seed-data
|
||||
- docker
|
||||
---
|
||||
|
||||
A C++20 command-line pipeline that samples city records from local JSON,
|
||||
enriches each with Wikipedia context, and generates bilingual brewery names and
|
||||
@@ -7,12 +16,12 @@ deterministic mock.
|
||||
|
||||
> **This pipeline produces AI-generated data.** It is not a source of truth for
|
||||
> brewing techniques, cultural representation, or local-language accuracy. See
|
||||
> [ETHICS-AND-KNOWN-ISSUES.md](./ETHICS-AND-KNOWN-ISSUES.md) for a full
|
||||
> [ETHICS-AND-KNOWN-ISSUES.md](./ETHICS-AND-KNOWN-ISSUES.md) for full
|
||||
> documentation of limitations, hallucination patterns, and bias.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
## Table of contents
|
||||
|
||||
- [How it fits the main app](#how-it-fits-the-main-app)
|
||||
- [Quick start](#quick-start)
|
||||
@@ -101,25 +110,27 @@ Each run writes a fresh dated SQLite file such as
|
||||
|
||||
#### CLI flags
|
||||
|
||||
| Flag | Purpose |
|
||||
| ------------------ | ------------------------------------------------------------------------------------------------------------ |
|
||||
| `--mocked` | Deterministic mock generator, no model required. |
|
||||
| `--model, -m` | Path to a GGUF file. Required unless `--mocked` is set. |
|
||||
| `--prompt-dir` | Directory containing prompt files (for example, `BREWERY_GENERATION.md`). Required unless `--mocked` is set. |
|
||||
| `--output, -o` | Directory for generated SQLite artifacts. Default: `output`. |
|
||||
| `--log-path` | Path for application logs. Default: `pipeline.log`. |
|
||||
| `--location-count` | Number of cities to sample from `locations.json` per run. Default: `10`. |
|
||||
| `--temperature` | Sampling temperature. Default: `1.0`. |
|
||||
| `--top-p` | Nucleus sampling. Default: `0.95`. |
|
||||
| `--top-k` | Top-k sampling. Default: `64`. |
|
||||
| `--n-ctx` | Context window size. Default: `8192`. |
|
||||
| `--seed` | Random seed. Default: `-1` (random at runtime). |
|
||||
| `--n-gpu-layers` | Number of model layers to offload to GPU. Default: `0`. |
|
||||
| `--help, -h` | Print usage and exit. |
|
||||
| Flag | Purpose |
|
||||
| ------------------ | ---------------------------------------------------------------------------------------------------------------- |
|
||||
| `--mocked` | Deterministic mock generator, no model required. |
|
||||
| `--model, -m` | Path to a GGUF file. Selects the local Llama.cpp generator. |
|
||||
| `--openai` | Use the OpenAI API (Chat Completions) for brewery/user data. Requires the `OPENAI_API_KEY` environment variable. |
|
||||
| `--openai-model` | OpenAI model ID used when `--openai` is set. Default: `gpt-4o-mini`. |
|
||||
| `--prompt-dir` | Directory containing prompt files (for example, `BREWERY_GENERATION.md`). Required unless `--mocked` is set. |
|
||||
| `--output, -o` | Directory for generated SQLite artifacts. Default: `output`. |
|
||||
| `--log-path` | Path for application logs. Default: `pipeline.log`. |
|
||||
| `--location-count` | Number of cities to sample from `locations.json` per run. Default: `10`. |
|
||||
| `--temperature` | Sampling temperature. Default: `1.0`. |
|
||||
| `--top-p` | Nucleus sampling. Default: `0.95`. |
|
||||
| `--top-k` | Top-k sampling. Default: `64`. |
|
||||
| `--n-ctx` | Context window size. Default: `8192`. |
|
||||
| `--seed` | Random seed. Default: `-1` (random at runtime). |
|
||||
| `--n-gpu-layers` | Number of model layers to offload to GPU. Default: `0`. |
|
||||
| `--help, -h` | Print usage and exit. |
|
||||
|
||||
`--mocked` and `--model` are mutually exclusive. Omitting both exits with an
|
||||
error before the pipeline starts. Sampling flags are ignored when `--mocked` is
|
||||
set.
|
||||
`--mocked`, `--model`, and `--openai` are mutually exclusive. Omitting all three
|
||||
exits with an error before the pipeline starts. Sampling flags are ignored
|
||||
unless `--model` is set (Llama).
|
||||
|
||||
The post-build step copies `prompts/` into `build/prompts/`. Rebuild after
|
||||
editing any prompt file.
|
||||
@@ -176,7 +187,7 @@ output to confirm the fast path was taken.
|
||||
### Run the container
|
||||
|
||||
The container always runs the model-backed path; there is no `--mocked`
|
||||
container mode (use a native build for that — see [Quick start](#quick-start)).
|
||||
container mode (use a native build for that; see [Quick start](#quick-start)).
|
||||
The entrypoint, `runpod/start.sh`, downloads the GGUF model automatically if it
|
||||
is not already present at the configured path.
|
||||
|
||||
@@ -191,7 +202,7 @@ docker run --rm \
|
||||
|
||||
By default this downloads `google_gemma-4-E4B-it-Q6_K.gguf` to `./models/` on
|
||||
first run if it isn't already there. To use a pre-downloaded model, place it at
|
||||
that path first — see [Model](#model) above.
|
||||
that path first; see [Model](#model).
|
||||
|
||||
#### Environment variables
|
||||
|
||||
@@ -211,8 +222,9 @@ configurable via environment variable.
|
||||
|
||||
Use a GPU pod template. Mount persistent storage for `/workspace/models`,
|
||||
`/workspace/output`, and `/workspace/logs`. See
|
||||
`tooling/pipeline/runpod/pod-template.yaml` for a starter template — set the
|
||||
environment variables listed above to match your run.
|
||||
`tooling/pipeline/runpod/pod-template.yaml` for a starter template; set the
|
||||
environment variables listed in [Environment variables](#environment-variables)
|
||||
to match your run.
|
||||
|
||||
---
|
||||
|
||||
@@ -226,8 +238,8 @@ environment variables listed above to match your run.
|
||||
| Sample | `BiergartenPipelineOrchestrator::QueryCitiesWithCountries()` samples `--location-count` locations per run (default `10`). |
|
||||
| Enrich | `WikipediaEnrichmentService` fetches brewing and beer-related context. Keeps going when a lookup fails. `--mocked` runs use `MockEnrichmentService` instead and skip Wikipedia entirely. |
|
||||
| Generate Users | `GenerateUsers()` samples a persona and a forename/surname pair per enriched city (skipping countries with no name data), then `MockGenerator` or `LlamaGenerator` produces a username, bio, and activity weight around the sampled name. |
|
||||
| Generate Breweries | `MockGenerator` or `LlamaGenerator` produces brewery names and descriptions in English and the local language. |
|
||||
| Store | `SqliteExportService` writes each successful user and brewery into a fresh dated `.sqlite` database with normalized `locations`, `users`, and `breweries` tables. |
|
||||
| Generate Breweries | `MockGenerator`, `LlamaGenerator`, or `OpenAIGenerator` produces brewery names and descriptions in English and the local language, then `IAddressService::ReverseGeocode()` resolves a street address for the brewery's jittered coordinates. |
|
||||
| Store | `SqliteExportService` writes each successful user and brewery into a fresh dated `.sqlite` database with normalized `cities`, `breweries`, `brewery_addresses`, `users`, and `user_addresses` tables. |
|
||||
| Log | `spdlog` writes results and warnings to the console. |
|
||||
|
||||
If name sampling, enrichment, or generation fails for a city, that city is
|
||||
@@ -236,67 +248,36 @@ skipped and the pipeline continues. `GenerateUsers()` runs before
|
||||
|
||||
### Key components
|
||||
|
||||
- `src/main.cc` — argument parsing and Boost.DI composition root.
|
||||
- `CuratedJsonDataService` — implements `ICuratedDataService`; takes a
|
||||
`CuratedDataFilePaths` DTO (locations/personas/forenames/surnames paths) in
|
||||
its constructor, then parses and validates curated location, persona, and
|
||||
forename/surname JSON, memoizing each result after its first load on a given
|
||||
instance. `MockCuratedDataService` is the in-memory substitute (4 fixed
|
||||
locations, 3 personas, and name data for `US`/`DE`/`FR`/`BE`) used in
|
||||
`--mocked` runs.
|
||||
- `WikipediaEnrichmentService` — queries Wikipedia extracts, caches results,
|
||||
returns empty context on failure. `MockEnrichmentService` is the no-op
|
||||
substitute used in `--mocked` runs.
|
||||
- `LlamaGenerator` — formats prompts for Gemma 4, validates JSON output for both
|
||||
`GenerateBrewery` and `GenerateUser`, retries malformed responses up to three
|
||||
times with corrective feedback in the retry prompt. The token budget is fixed
|
||||
across attempts; it is not raised automatically on truncation.
|
||||
- `MockGenerator` — stable hash-based output so the same city/persona/name input
|
||||
always produces the same brewery or user.
|
||||
- `SqliteExportService` — creates a dated SQLite file per run and persists each
|
||||
successful user and brewery into normalized tables.
|
||||
- Brewery payloads include English and local-language name and description
|
||||
fields. User payloads carry a sampled first/last name and gender, an
|
||||
LLM-generated username/bio/activity weight, and a programmatically generated
|
||||
(not LLM-authored) unique email and date of birth.
|
||||
| Component | Description |
|
||||
| ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `src/main.cc` | Argument parsing and the Boost.DI composition root. |
|
||||
| `CuratedJsonDataService` | Implements `ICuratedDataService`; takes a `CuratedDataFilePaths` DTO (locations/personas/forenames/surnames paths) in its constructor, parses and validates curated location, persona, and forename/surname JSON, and memoizes each result after its first load on a given instance. `MockCuratedDataService` is the in-memory substitute (4 fixed locations, 3 personas, and name data for `US`/`DE`/`FR`/`BE`) used in `--mocked` runs. |
|
||||
| `WikipediaEnrichmentService` | Queries Wikipedia extracts, caches results, and returns empty context on failure. `MockEnrichmentService` is the no-op substitute used in `--mocked` runs. |
|
||||
| `LlamaGenerator` | Formats prompts for Gemma 4, validates JSON output for both `GenerateBrewery` and `GenerateUser`, and retries malformed responses up to three times with corrective feedback in the retry prompt. The token budget is fixed across attempts; it is not raised automatically on truncation. |
|
||||
| `MockGenerator` | Uses stable hash-based output so the same city/persona/name input always produces the same brewery or user. |
|
||||
| `NominatimAddressService` | Reverse-geocodes each brewery's jittered coordinates into a street address (`address_line1`, `postal_code`) via the public [Nominatim API](https://nominatim.openstreetmap.org), respecting its one-request-per-second usage policy. Falls back to a placeholder `address_line1` ("Address unavailable") and empty `postal_code` when the lookup fails or returns no usable street address. `MockAddressService` is the no-network substitute (`"123 Mock Street"`, `"00000"`) used in `--mocked` runs. Only breweries get a resolved street address; generated users carry a coordinate pair with no address fields. |
|
||||
| `SqliteExportService` | Creates a dated SQLite file per run and persists each successful user and brewery into normalized tables. |
|
||||
| Brewery/user payloads | Brewery payloads include English and local-language name and description fields. User payloads carry a sampled first/last name and gender, an LLM-generated username/bio/activity weight, and a programmatically generated (not LLM-authored) unique email and date of birth. |
|
||||
|
||||
### Runtime behaviour
|
||||
|
||||
`WikipediaEnrichmentService` fetches two Wikipedia extracts per city: a generic
|
||||
"brewing" extract and a "beer in `{country}`" extract. It does not currently
|
||||
query a city- or region-specific page. Each query string is cached after its
|
||||
first successful (or empty) lookup.
|
||||
|
||||
`GetLocationContext()` returns an empty string when the web client is
|
||||
unavailable or when lookup/parsing fails.
|
||||
|
||||
`LlamaGenerator` validates model output as structured JSON. On validation
|
||||
failure it retries up to three times, replaying the previous error message in
|
||||
the next prompt so the model can self-correct. All runs to date have produced
|
||||
valid output on the first pass; the retry path is kept for resilience.
|
||||
|
||||
`MockGenerator` uses stable hashes for repeatable output in demos and Storybook
|
||||
runs.
|
||||
|
||||
`CuratedJsonDataService` memoizes each of `LoadLocations()`, `LoadPersonas()`,
|
||||
`LoadForenamesByCountry()`, and `LoadSurnamesByCountry()` independently the
|
||||
first time each is called, since `BiergartenPipelineOrchestrator` owns a single
|
||||
`ICuratedDataService` instance for the whole run — later calls return the cached
|
||||
result instead of re-parsing.
|
||||
|
||||
`GenerateUsers()` samples a forename/surname pair per city via `SampleName()`,
|
||||
keyed by the city's ISO 3166-1 code. Countries present in `locations.json` but
|
||||
absent from either name fixture (currently `KE`, `SE`, `SG`, `TH`, `VN`, `ZA`)
|
||||
are skipped, the same way a failed enrichment or generation call skips a city —
|
||||
see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section.
|
||||
| Component / behavior | Details |
|
||||
| ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `WikipediaEnrichmentService` | Fetches two Wikipedia extracts per city: a generic "brewing" extract and a "beer in `{country}`" extract. It does not currently query a city- or region-specific page. Each query string is cached after its first successful (or empty) lookup. |
|
||||
| `GetLocationContext()` | Returns an empty string when the web client is unavailable or when lookup/parsing fails. |
|
||||
| `LlamaGenerator` | Validates model output as structured JSON. On validation failure it retries up to three times, replaying the previous error message in the next prompt so the model can self-correct. All runs to date have produced valid output on the first pass; the retry path is kept for resilience. |
|
||||
| `MockGenerator` | Uses stable hashes for repeatable output in demos and Storybook runs. |
|
||||
| `NominatimAddressService.ReverseGeocode()` | Sleeps one second after every request to respect Nominatim's usage policy, and returns `std::nullopt` (logged as a warning, not a skipped city) if the request fails, the response has no `address` object, or neither `house_number` nor `road` is present — the brewery is still generated and exported, just with a placeholder address. |
|
||||
| `CuratedJsonDataService` | Memoizes each of `LoadLocations()`, `LoadPersonas()`, `LoadForenamesByCountry()`, and `LoadSurnamesByCountry()` independently the first time each is called, since `BiergartenPipelineOrchestrator` owns a single `ICuratedDataService` instance for the whole run; later calls return the cached result instead of re-parsing. |
|
||||
| `GenerateUsers()` | Samples a forename/surname pair per city via `SampleName()`, keyed by the city's ISO 3166-1 code. Countries present in `locations.json` but absent from either name fixture (currently `KE`, `SE`, `SG`, `TH`, `VN`, `ZA`) are skipped, the same way a failed enrichment or generation call skips a city; see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section. |
|
||||
|
||||
### Process flow - activity diagram
|
||||
|
||||
|
||||
See [Activity diagram](./diagrams/current/activity.md).
|
||||
|
||||
### Architectural overview - class diagram
|
||||
|
||||
|
||||
See [Class diagram](./diagrams/current/class.md).
|
||||
|
||||
---
|
||||
|
||||
@@ -307,12 +288,14 @@ Each successful run stores a `BreweryRecord` pair with the source location and a
|
||||
`UserResult` payload. The same generated records are also written to a fresh
|
||||
SQLite export file named with the current UTC timestamp.
|
||||
|
||||
| Field | Meaning |
|
||||
| ------------------- | ------------------------------------------ |
|
||||
| `name_en` | Brewery name in English. |
|
||||
| `description_en` | Brewery description in English. |
|
||||
| `name_local` | Brewery name in the local language. |
|
||||
| `description_local` | Brewery description in the local language. |
|
||||
| Field | Meaning |
|
||||
| ------------------- | ------------------------------------------------------------------------------------- |
|
||||
| `name_en` | Brewery name in English. |
|
||||
| `description_en` | Brewery description in English. |
|
||||
| `name_local` | Brewery name in the local language. |
|
||||
| `description_local` | Brewery description in the local language. |
|
||||
| `address_line1` | Street address resolved via reverse geocoding, or a placeholder if resolution failed. |
|
||||
| `postal_code` | Postal code resolved via reverse geocoding, or empty if resolution failed. |
|
||||
|
||||
| Field | Meaning |
|
||||
| ----------------- | -------------------------------------------------------------------------------------- |
|
||||
@@ -357,15 +340,14 @@ The build fetches Boost.DI, spdlog, and SQLite via CMake. llama.cpp is fetched
|
||||
only when a system installation is not detected. Metal is enabled on Apple
|
||||
Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present.
|
||||
|
||||
> **Code Style:** Code style generally follows the Google Style Guide and
|
||||
> features modern C++20 throughout. Formatting follows the Google C++ Style
|
||||
> Guide, with the exception of indentation width (3 spaces vs. Google's 2).
|
||||
> **Code style:** The code follows the Google C++ Style Guide, except that
|
||||
> indentation is 3 spaces instead of Google's 2, and uses C++20 throughout.
|
||||
|
||||
---
|
||||
|
||||
## Tested hardware
|
||||
|
||||
### ARM macOS — M1 Pro
|
||||
### ARM macOS (M1 Pro)
|
||||
|
||||
| | |
|
||||
| --------- | --------------------------------- |
|
||||
@@ -376,7 +358,7 @@ Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present.
|
||||
| Model | Gemma 4 E4B |
|
||||
| Inference | llama.cpp with Metal |
|
||||
|
||||
### x86_64 Linux — NVIDIA RTX 2000
|
||||
### x86_64 Linux (NVIDIA RTX 2000)
|
||||
|
||||
| | |
|
||||
| --------- | ------------------------------ |
|
||||
@@ -387,7 +369,7 @@ Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present.
|
||||
| Model | Gemma 4 E4B |
|
||||
| Inference | llama.cpp with CUDA 12.x |
|
||||
|
||||
### x86_64 Linux — Docker / RunPod (NVIDIA CUDA)
|
||||
### x86_64 Linux (Docker / RunPod, NVIDIA CUDA)
|
||||
|
||||
| | |
|
||||
| --------- | ------------------------------------------- |
|
||||
@@ -407,7 +389,7 @@ Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present.
|
||||
- Keep `locations.json` structured enough to support discovery and future
|
||||
filtering.
|
||||
- `personas.json`, `forenames-by-country.json`, and `surnames-by-country.json`
|
||||
are curated/vendored fixture data, not LLM-generated — see
|
||||
are curated/vendored fixture data, not LLM-generated; see
|
||||
ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for provenance.
|
||||
- Treat SQLite output as seed material for the app's brewery and user domains,
|
||||
not production data.
|
||||
@@ -435,30 +417,32 @@ Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present.
|
||||
|
||||
Paths below are relative to `tooling/pipeline/`.
|
||||
|
||||
- `src/main.cc` — argument parsing and DI composition root.
|
||||
- `src/biergarten_pipeline_orchestrator/` — orchestration, sampling, logging,
|
||||
and export.
|
||||
- `src/services/curated_data/` — `CuratedJsonDataService`, the file-backed
|
||||
`ICuratedDataService`, and `MockCuratedDataService`, the in-memory
|
||||
`ICuratedDataService` used in `--mocked` runs.
|
||||
- `src/services/enrichment/wikipedia/` — enrichment service and cache.
|
||||
- `src/services/sqlite/` — SQLite export implementation.
|
||||
- `src/data_generation/llama/` — local inference, prompt loading, output
|
||||
validation.
|
||||
- `src/data_generation/mock/` — deterministic fallback.
|
||||
- `runpod/` — container build and runtime launcher.
|
||||
| Path | Purpose |
|
||||
| --------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `src/main.cc` | Argument parsing and DI composition root. |
|
||||
| `src/biergarten_pipeline_orchestrator/` | Orchestration, sampling, logging, and export. |
|
||||
| `src/services/curated_data/` | `CuratedJsonDataService`, the file-backed `ICuratedDataService`, and `MockCuratedDataService`, the in-memory `ICuratedDataService` used in `--mocked` runs. |
|
||||
| `src/services/enrichment/wikipedia/` | Enrichment service and cache. |
|
||||
| `src/services/sqlite/` | SQLite export implementation. |
|
||||
| `src/data_generation/llama/` | Local inference, prompt loading, and output validation. |
|
||||
| `src/data_generation/mock/` | Deterministic fallback. |
|
||||
| `runpod/` | Container build and runtime launcher. |
|
||||
|
||||
---
|
||||
|
||||
## Next steps
|
||||
|
||||
The pipeline currently produces city-aware brewery and user records and dated
|
||||
SQLite exports. The next passes add additional fixture types so the app can
|
||||
exercise the full brewery and social domains without live data. For the detailed
|
||||
engineering breakdown of what's needed to reach the architecture in
|
||||
[`diagrams/planned/`](./diagrams/planned/), see [ROADMAP.md](./ROADMAP.md).
|
||||
SQLite exports.
|
||||
|
||||
### Testing — very high priority
|
||||
The next passes add additional fixture types so the app can exercise the full
|
||||
brewery and social domains without live data.
|
||||
|
||||
For the detailed engineering breakdown of what's needed to reach the
|
||||
architecture in [`diagrams/planned/`](./diagrams/planned/), see
|
||||
[ROADMAP.md](./ROADMAP.md).
|
||||
|
||||
### Testing
|
||||
|
||||
- Unit test JSON validation and retry logic against malformed, truncated, and
|
||||
empty model outputs.
|
||||
@@ -481,7 +465,7 @@ enough to exercise search, sort, and category filters.
|
||||
### Check-in system
|
||||
|
||||
Produce timestamped check-in events between users and breweries. Use a J-curve
|
||||
activity profile — a small set of users accounts for most check-ins, the rest
|
||||
activity profile: a small set of users accounts for most check-ins, the rest
|
||||
appear occasionally. Add bursty behaviour around weekends and travel periods.
|
||||
|
||||
### Beer ratings
|
||||
|
||||
@@ -1,21 +1,24 @@
|
||||
# Pipeline Roadmap — Reaching the Planned Architecture
|
||||
---
|
||||
title: Pipeline roadmap — reaching the planned architecture
|
||||
last-updated: 2026-09-01
|
||||
tags:
|
||||
- pipeline
|
||||
- roadmap
|
||||
- cpp
|
||||
- concurrency
|
||||
---
|
||||
|
||||
This is the engineering breakdown for closing the gap between the **current**
|
||||
pipeline and the **planned** architecture documented in
|
||||
[`diagrams/planned/class.puml`](./diagrams/planned/class.puml) and
|
||||
[`diagrams/planned/activity.puml`](./diagrams/planned/activity.puml). Nothing in
|
||||
`diagrams/planned/` is implemented yet — this file tracks what it would take to
|
||||
get there. For the current implementation, see [README.md](./README.md).
|
||||
Engineering breakdown for closing the gap between the **current** pipeline and
|
||||
the **planned** architecture in
|
||||
[`diagrams/planned/class.md`](./diagrams/planned/class.md) /
|
||||
[`diagrams/planned/activity.md`](./diagrams/planned/activity.md). Items are
|
||||
grouped by layer, roughly by dependency order. For the current implementation,
|
||||
see [README.md](./README.md).
|
||||
|
||||
Items are grouped by layer and roughly ordered by dependency: later groups build
|
||||
on types and services introduced earlier.
|
||||
|
||||
The only concurrency in the current pipeline is the log dispatcher thread
|
||||
(`main.cc` spawns it to drain a `BoundedChannel<LogEntry>` into spdlog for the
|
||||
whole run). Sampling, enrichment, generation, and export all happen
|
||||
synchronously on the main thread inside `BiergartenPipelineOrchestrator::Run()`
|
||||
— §6 below is what introduces the additional worker threads the planned
|
||||
architecture needs.
|
||||
The only concurrency today is a log dispatcher thread draining a
|
||||
`BoundedChannel<LogEntry>` into spdlog; sampling, enrichment, generation, and
|
||||
export run synchronously on the main thread — §6 introduces the planned worker
|
||||
threads.
|
||||
|
||||
---
|
||||
|
||||
@@ -23,222 +26,183 @@ architecture needs.
|
||||
|
||||
`tooling/pipeline/includes/data_model/generated_models.h` and `models.h`.
|
||||
|
||||
- [ ] Add `Completeness` enum (`Full`, `Partial`, `Absent`) and a
|
||||
`LocationContext` struct (`text`, `completeness`, `char_count`). Replace
|
||||
`EnrichedCity::region_context` (currently a plain `std::string`) with
|
||||
`context : LocationContext`.
|
||||
- [ ] Add `Completeness` enum (`Full`/`Partial`/`Absent`) and a
|
||||
`LocationContext` struct (`text`, `completeness`, `char_count`); replace
|
||||
`EnrichedCity::region_context` (a plain `string` today) with it.
|
||||
- [ ] Add `BeerStyle` (`name`, `description`, `min_abv`, `max_abv`, `min_ibu`,
|
||||
`max_ibu`).
|
||||
- [ ] Add `BeerResult`, `CheckinResult`, `RatingResult` result payloads.
|
||||
- [ ] Add `BeerResult`, `CheckinResult`, `RatingResult` payloads.
|
||||
- [ ] Add `GenerationMetadata` (`generation_id`, `generated_time`,
|
||||
`context_provided`, `generated_with`).
|
||||
- [x] Add `first_name`, `last_name`, `gender`, and `activity_weight` to
|
||||
`UserResult` (currently just `username`, `bio`). `first_name`/
|
||||
`last_name`/`gender` are copied from the sampled `Name` (see below), not
|
||||
- [x] `UserResult` gained `first_name`/`last_name`/`gender`/`activity_weight`
|
||||
(was just `username`/`bio`), copied from the sampled `Name`, not
|
||||
LLM-invented.
|
||||
- [x] Add `Name` (`first_name`, `last_name`, `gender`) — the sampled result
|
||||
handed to `DataGenerator::GenerateUser`. Add `ForenameEntry` (`name`,
|
||||
`gender`). Landed as a flatter shape than originally planned here: no
|
||||
`NamesByCountry` wrapper class. `curated_data_service.h` instead declares
|
||||
`ForenameList = std::vector<ForenameEntry>` and
|
||||
`SurnameList = std::vector<std::string>`, and `ICuratedDataService`
|
||||
exposes two maps keyed by ISO 3166-1 code —
|
||||
`ForenamesByCountryMap = unordered_map<string, ForenameList>` and
|
||||
`SurnamesByCountryMap = unordered_map<string, SurnameList>` — directly
|
||||
(see §2). Sampling is a free function,
|
||||
`SampleName(forenames_by_country, surnames_by_country, iso3166_1, rng)`,
|
||||
in `generate_users.cc`'s anonymous namespace, not a method on a class.
|
||||
Pairing a forename with a surname happens at sample time so gender
|
||||
(present per-forename in the source data) is never discarded the way a
|
||||
pre-flattened `{first_name, last_name}` list would lose it; see §2.
|
||||
- [x] Added `Name` (`first_name`, `last_name`, `gender`) and `ForenameEntry`
|
||||
(`name`, `gender`). Flatter than planned: no `NamesByCountry` wrapper —
|
||||
`ICuratedDataService` exposes `ForenamesByCountryMap`/
|
||||
`SurnamesByCountryMap` (keyed by ISO 3166-1) directly, and sampling is a
|
||||
free function, `SampleName(forenames, surnames, iso3166_1, rng)`, not a
|
||||
class method.
|
||||
- [ ] Extend `GeneratedBrewery` with `brewery_id`, `context_completeness`,
|
||||
`metadata` (currently just `{location, brewery}`).
|
||||
- [ ] Add `GeneratedBeer`, `GeneratedCheckin`, `GeneratedRating`,
|
||||
`GeneratedFollow` aggregate structs.
|
||||
- [x] Add `UserRecord` (`location`, `user : UserResult`, `email`,
|
||||
`date_of_birth`) as the current stand-in for the planned `GeneratedUser` —
|
||||
`email` and `date_of_birth` are programmatically generated by the
|
||||
orchestrator, never LLM-authored, so a downstream auth-account seeding
|
||||
consumer can register real accounts from the pipeline's SQLite export. No
|
||||
`password`, `user_id`, or `metadata` field yet, matching `BreweryRecord`'s
|
||||
equally pre-`metadata` shape. Already wired into
|
||||
`IExportService`/`SqliteExportService`: a `users` table exists and
|
||||
`GenerateUsers()` exports each successful record via
|
||||
`ProcessRecord(const UserRecord&)` (see §5) in addition to logging and
|
||||
holding `generated_users_` in memory. Still missing vs. the planned
|
||||
`GeneratedUser`: `user_id`, `password`, and `metadata`.
|
||||
- [x] Add `UserPersona` (`name`, `description`, `style_affinities`).
|
||||
- [x] Added `UserRecord` (`location`, `user`, `email`, `date_of_birth`), the
|
||||
current stand-in for the planned `GeneratedUser`. `email`/ `date_of_birth`
|
||||
are programmatic (never LLM-authored), for downstream auth-account
|
||||
seeding. Missing `user_id`/`password`/`metadata`; already exported via
|
||||
`ProcessRecord(const UserRecord&)` (§5).
|
||||
- [x] Added `UserPersona` (`name`, `description`, `style_affinities`).
|
||||
|
||||
## 2. Data Preloading
|
||||
|
||||
`tooling/pipeline/includes/services/curated_data/`, fixture files.
|
||||
|
||||
- [x] Extract an interface; have `CuratedJsonDataService` implement it. Landed
|
||||
as `ICuratedDataService` (`services/curated_data/curated_data_service.h`),
|
||||
not `DataPreloader` — `LoadLocations()`, `LoadPersonas()`,
|
||||
`LoadForenamesByCountry()`, and `LoadSurnamesByCountry()` are all virtual
|
||||
methods returning `const&` and take no arguments. `CuratedJsonDataService`
|
||||
(`services/curated_data/curated_json_data_service.h`, originally landed as
|
||||
`JsonLoader` under `json_handling/` before being renamed and moved) takes
|
||||
a `CuratedDataFilePaths` DTO (`locations_path`, `personas_path`,
|
||||
`forenames_path`, `surnames_path`) in its constructor rather than a path
|
||||
per `Load*()` call, and memoizes each result in a private `cache` struct
|
||||
(`locations`, `personas`, `forenames_by_country`, `surnames_by_country`),
|
||||
so a second call on the same instance returns the cached result instead of
|
||||
re-parsing — safe because `CuratedJsonDataService` outlives every call
|
||||
site (owned by `BiergartenPipelineOrchestrator` via
|
||||
`unique_ptr<ICuratedDataService>` for the whole run). `main.cc`'s DI
|
||||
injector also gained a `MockCuratedDataService` (fixed in-memory data: 4
|
||||
locations across `US`/`DE`/`FR`/`BE`, 3 personas, and matching
|
||||
forename/surname sets for those four countries), bound in place of
|
||||
`CuratedJsonDataService` under `--mocked`, mirroring the existing
|
||||
`MockEnrichmentService`/`MockGenerator` pattern.
|
||||
- [ ] Add `LoadBeerStyles()`. `beer-styles.json` already exists in the repo and
|
||||
is already copied into the Docker image (`runpod/Dockerfile`), but no
|
||||
loader reads it yet, and the native CMake build doesn't copy it into
|
||||
`build/` at all — `CMakeLists.txt`'s "Runtime Assets" step only copies
|
||||
`locations.json` and `prompts/`.
|
||||
- [x] Add `LoadPersonas()` and author `personas.json` (doesn't exist yet).
|
||||
- [x] Add name-by-country loading, parsing
|
||||
`tooling/pipeline/forenames-by-country.json` and
|
||||
`surnames-by-country.json`. Landed as two separate methods —
|
||||
`LoadForenamesByCountry()` and `LoadSurnamesByCountry()` — each returning
|
||||
a flat `ForenamesByCountryMap` / `SurnamesByCountryMap` (aliases for
|
||||
`unordered_map<string, ForenameList>` /
|
||||
`unordered_map<string, SurnameList>`) keyed by ISO 3166-1 code, rather
|
||||
than one combined `LoadNamesByCountry() : NamesByCountry` call. Both files
|
||||
are vendored verbatim (unmodified, full multinational coverage) from
|
||||
`sigpwned/popular-names-by-country-dataset` (CC0) — see
|
||||
ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for
|
||||
provenance. Deliberately not pre-paired or filtered down to just the
|
||||
countries in `locations.json`: keeping the source shape (including
|
||||
per-forename gender) intact means the loader can support more countries
|
||||
later, or gender-aware persona/bio generation, without re-sourcing data.
|
||||
- [x] Extracted `ICuratedDataService` (not `DataPreloader`); `Load*()` methods
|
||||
take no arguments, return `const&`. `CuratedJsonDataService` (renamed from
|
||||
`JsonLoader`) takes a `CuratedDataFilePaths` DTO at construction and
|
||||
memoizes each `Load*()` result. `MockCuratedDataService` (fixed data for
|
||||
`US`/`DE`/`FR`/`BE`) binds in its place under `--mocked`.
|
||||
- [ ] Add `LoadBeerStyles()`. `beer-styles.json` exists and is copied into the
|
||||
Docker image, but no loader reads it and the native CMake build doesn't
|
||||
copy it into `build/` at all.
|
||||
- [x] Added `LoadPersonas()` and authored `personas.json`.
|
||||
- [x] Added `LoadForenamesByCountry()` / `LoadSurnamesByCountry()` (two methods,
|
||||
not one combined `LoadNamesByCountry()`), parsing vendored CC0 data from
|
||||
`sigpwned/popular-names-by-country-dataset` — see
|
||||
ETHICS-AND-KNOWN-ISSUES.md for provenance. Deliberately not pre-paired or
|
||||
filtered to `locations.json`'s countries, to keep per-forename gender and
|
||||
support more countries later without re-sourcing.
|
||||
|
||||
## 3. Policy / Strategy Layer
|
||||
|
||||
Entirely new — no `includes/policy/` (or equivalent) directory exists today.
|
||||
Entirely new — no `includes/policy/` (or equivalent) exists today.
|
||||
|
||||
- [ ] `ContextStrategy` interface + `BreweryContextStrategy` /
|
||||
`BeerContextStrategy`. Today
|
||||
`WikipediaEnrichmentService::GetLocationContext` hardcodes a generic
|
||||
`"brewing"` query and a `"beer in {country}"` query directly — no
|
||||
per-phase strategy selection.
|
||||
- [ ] `SamplingStrategy` interface + `UniformSamplingStrategy`, replacing the
|
||||
inline `std::ranges::sample(...)` call in
|
||||
`BiergartenPipelineOrchestrator::QueryCitiesWithCountries()`.
|
||||
- [ ] `BeerSelectionStrategy` interface + `RandomBeerSelectionStrategy`, to pick
|
||||
styles per brewery from the `BeerStyle` palette (depends on §1 and §2).
|
||||
- [ ] `CheckinDistributionStrategy` interface + `JCurveCheckinStrategy` /
|
||||
`RandomCheckinStrategy` — this is the "Check-In System" item already
|
||||
called out in README.md's Next Steps, made concrete.
|
||||
- [ ] `FollowGenerationStrategy` interface + `RandomFollowStrategy` /
|
||||
- [ ] `ContextStrategy` + `BreweryContextStrategy`/`BeerContextStrategy`. Today
|
||||
`WikipediaEnrichmentService::GetLocationContext` hardcodes a `"brewing"`
|
||||
query and a `"beer in {country}"` query directly.
|
||||
- [ ] `SamplingStrategy` + `UniformSamplingStrategy`, replacing the inline
|
||||
`std::ranges::sample(...)` in `QueryCitiesWithCountries()`.
|
||||
- [ ] `BeerSelectionStrategy` + `RandomBeerSelectionStrategy` (depends on §1,
|
||||
§2).
|
||||
- [ ] `CheckinDistributionStrategy` + `JCurveCheckinStrategy` /
|
||||
`RandomCheckinStrategy` — the "Check-In System" README next step, made
|
||||
concrete.
|
||||
- [ ] `FollowGenerationStrategy` + `RandomFollowStrategy` /
|
||||
`ActivityWeightedFollowStrategy`.
|
||||
|
||||
## 4. Data Generation
|
||||
|
||||
`tooling/pipeline/includes/data_generation/`, `src/data_generation/`.
|
||||
|
||||
- [ ] Extend the `DataGenerator` interface with `GenerateBeer`,
|
||||
`GenerateCheckin`, `GenerateRating` (today: only `GenerateBrewery` and
|
||||
`GenerateUser`).
|
||||
- [x] Implement `LlamaGenerator::GenerateUser` for real, with a retry loop
|
||||
mirroring `GenerateBrewery` (GBNF grammar, `ValidateUserJson`, up to 3
|
||||
attempts with corrective feedback) and a new `prompts/USER_GENERATION.md`.
|
||||
Changed the `DataGenerator::GenerateUser` signature to
|
||||
`(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult`,
|
||||
matching what the activity diagram actually passes.
|
||||
`MockGenerator::GenerateUser` updated to match the new signature too.
|
||||
- [ ] Implement `MockGenerator::GenerateBeer` / `GenerateCheckin` /
|
||||
`GenerateRating`.
|
||||
- [ ] Extend `DataGenerator` with `GenerateBeer`, `GenerateCheckin`,
|
||||
`GenerateRating` (today: only `GenerateBrewery`/`GenerateUser`).
|
||||
- [x] Added `OpenAIGenerator`, a third backend calling the OpenAI Chat
|
||||
Completions API with Structured Outputs (`json_schema`, `strict`) —
|
||||
schema-valid JSON with no `IPromptFormatter`/GBNF grammar needed. Selected
|
||||
via `--openai`/`GeneratorMode::kOpenAI`; landed ahead of §3/§4. Now shown
|
||||
in `diagrams/planned/class.md` alongside `MockGenerator`/`LlamaGenerator`,
|
||||
noting only `GenerateBrewery`/`GenerateUser` are implemented so far.
|
||||
- [x] Implemented `LlamaGenerator::GenerateUser` for real (GBNF grammar,
|
||||
`ValidateUserJson`, 3 retries with corrective feedback,
|
||||
`prompts/USER_GENERATION.md`). Signature is
|
||||
`(city, persona, name) : UserResult`, matching the activity diagram.
|
||||
- [ ] Implement
|
||||
`MockGenerator::GenerateBeer`/`GenerateCheckin`/`GenerateRating`.
|
||||
- [ ] Add `IPromptFormatter::ExpectedArchitecture()` and
|
||||
`LlamaGenerator::ValidateModelArchitecture()` so loading a GGUF that
|
||||
doesn't match the configured chat template fails fast instead of silently
|
||||
producing degraded output.
|
||||
- [ ] Add prompt template files for beer/checkin/rating generation next to the
|
||||
existing `prompts/BREWERY_GENERATION.md`.
|
||||
`LlamaGenerator::ValidateModelArchitecture()` so a mismatched GGUF chat
|
||||
template fails fast instead of degrading silently.
|
||||
- [ ] Add prompt templates for beer/checkin/rating, next to
|
||||
`prompts/BREWERY_GENERATION.md`.
|
||||
|
||||
## 5. Export Service
|
||||
|
||||
`tooling/pipeline/includes/services/database/`, `src/services/sqlite/`.
|
||||
|
||||
- [x] `IExportService::ProcessRecord(const UserRecord&)` and
|
||||
`SqliteExportService`'s `users` table (see `kCreateUsersTableSql` /
|
||||
`kInsertUserSql` in
|
||||
`includes/services/database/sqlite_statement_helpers.h`) are implemented —
|
||||
`GenerateUsers()` exports every successful user alongside its resolved
|
||||
- [x] `IExportService::ProcessRecord(const UserRecord&)` and the `users` table
|
||||
are implemented — every successful user is exported with its resolved
|
||||
`location_id`.
|
||||
- [ ] Extend `IExportService` with `ProcessBeer`, `ProcessCheckin`,
|
||||
`ProcessRating`, `ProcessFollow` (today
|
||||
`ProcessRecord(const BreweryRecord&)` and
|
||||
`ProcessRecord(const UserRecord&)` exist).
|
||||
`ProcessRating`, `ProcessFollow`.
|
||||
- [ ] Add `beers`, `checkins`, `ratings`, `follows` tables to
|
||||
`SqliteExportService::InitializeSchema()` (`locations`, `breweries`, and
|
||||
`users` already exist) — see `kCreateLocationsTableSql` /
|
||||
`kCreateBreweriesTableSql` / `kCreateUsersTableSql` in
|
||||
`includes/services/database/sqlite_statement_helpers.h` for the existing
|
||||
pattern to follow.
|
||||
- [ ] Add a `brewery_cache_` alongside the existing `location_cache_`, and move
|
||||
from one open transaction for the whole run to per-phase batched commits
|
||||
(`BEGIN` / `COMMIT & BEGIN` on a batch-size threshold), as shown in the
|
||||
planned activity diagram.
|
||||
`InitializeSchema()` (`cities`/`breweries`/`brewery_addresses`/`users`/
|
||||
`user_addresses` already exist — see `sqlite_statement_helpers.h` for the
|
||||
pattern).
|
||||
- [ ] Add a `brewery_cache_` alongside `city_cache_`, and move from one open
|
||||
transaction for the whole run to per-phase batched commits (`BEGIN` /
|
||||
`COMMIT & BEGIN` on a batch threshold), per the planned activity diagram.
|
||||
|
||||
## 6. Concurrency & Orchestration
|
||||
|
||||
`tooling/pipeline/includes/concurrency/`,
|
||||
`src/biergarten_pipeline_orchestrator/`.
|
||||
|
||||
- [ ] `BoundedChannel<T>` already exists and is production-tested — but it's
|
||||
only wired to the single log channel today. Stand up the per-phase
|
||||
producer/consumer channels (`loc_ch`, `exp_ch`, etc.) the planned activity
|
||||
diagram describes, each with a dedicated LLM worker thread and SQLite
|
||||
worker thread.
|
||||
- [ ] Rewrite `BiergartenPipelineOrchestrator::Run()` from one synchronous pass
|
||||
over `generated_breweries_` into phased methods — `RunUserPhase` →
|
||||
`RunBreweryAndBeerPhase` (with a `RunBeerPhase` sub-step once
|
||||
`brewery_pool_` is populated) → `RunCheckinPhase` / `RunFollowPhase`
|
||||
(these two can run in parallel) → `RunRatingPhase` — backed by
|
||||
`user_pool_`, `brewery_pool_`, `beer_pool_`, `checkin_pool_`,
|
||||
`follow_pool_`.
|
||||
- [ ] Honor the structural-concurrency requirement already called out in a
|
||||
comment on `BiergartenPipelineOrchestrator::Run()` in
|
||||
`biergarten_pipeline_orchestrator.h`: once real worker threads exist, they
|
||||
must be structurally joined (e.g. via `std::jthread`) before `Run()`
|
||||
returns, so no worker logs to a closed channel during teardown.
|
||||
- [ ] `BoundedChannel<T>` exists and is production-tested but only wired to the
|
||||
log channel. Stand up the per-phase producer/consumer channels (`loc_ch`,
|
||||
`exp_ch`, etc.) with a dedicated LLM worker thread and SQLite worker
|
||||
thread each; the brewery phase needs a third, a Geocode Worker between
|
||||
them on its own `geo_ch` (§9).
|
||||
- [ ] Rewrite `Run()` from one synchronous pass into phased methods —
|
||||
`RunUserPhase` → `RunBreweryAndBeerPhase` (`RunBeerPhase` sub-step once
|
||||
`brewery_pool_` fills) → `RunCheckinPhase`/`RunFollowPhase` (parallel) →
|
||||
`RunRatingPhase` — backed by `user_pool_`, `brewery_pool_`, `beer_pool_`,
|
||||
`checkin_pool_`, `follow_pool_`.
|
||||
- [ ] Structurally join worker threads (e.g. `std::jthread`) before `Run()`
|
||||
returns, per the existing comment in `biergarten_pipeline_orchestrator.h`,
|
||||
so nothing logs to a closed channel during teardown.
|
||||
|
||||
## 7. Enrichment
|
||||
|
||||
- [ ] Decide whether to restore the city/region-specific Wikipedia query that's
|
||||
currently commented out in
|
||||
`src/services/enrichment/wikipedia/get_summary.cc` (`GetLocationContext`).
|
||||
The `ContextStrategy` work in §3 is a natural place to reintroduce it via
|
||||
`BreweryContextStrategy::QueriesFor()`.
|
||||
- [ ] Decide whether to restore the city/region-specific Wikipedia query
|
||||
currently commented out in `get_summary.cc` — the §3 `ContextStrategy`
|
||||
work is a natural place to reintroduce it.
|
||||
- [ ] Pre-warm caches at startup (`PreWarmBeerStyleCache`,
|
||||
`PreWarmLocationCache` in the planned activity diagram) instead of
|
||||
fetching lazily per record, so the streaming phases never block on a cold
|
||||
cache mid-run.
|
||||
`PreWarmLocationCache`) instead of fetching lazily per record.
|
||||
|
||||
## 8. Fixtures / Build
|
||||
|
||||
- [x] Author `personas.json`, `forenames-by-country.json`, and
|
||||
`surnames-by-country.json` (see §2).
|
||||
- [x] Authored `personas.json`, `forenames-by-country.json`,
|
||||
`surnames-by-country.json` (§2).
|
||||
- [ ] Fix the `beer-styles.json` build-tree gap: add it to the "Runtime Assets"
|
||||
`configure_file` step in `CMakeLists.txt` so native builds and the Docker
|
||||
image agree on what's available at runtime.
|
||||
`configure_file` step in `CMakeLists.txt`.
|
||||
|
||||
## 9. Geocoding
|
||||
|
||||
`tooling/pipeline/includes/services/address/`, `src/services/address/` — added
|
||||
entirely after `diagrams/planned/` was authored; wasn't tracked here until now.
|
||||
|
||||
- [x] Added `IAddressService`
|
||||
(`ReverseGeocode(lon, lat) : optional<AddressLookupResult>`),
|
||||
`NominatimAddressService` (public Nominatim API, 1s sleep per request,
|
||||
`nullopt` on failure or no usable address), and `MockAddressService`
|
||||
(fixed placeholder for `--mocked`). Called once per brewery, after
|
||||
`GenerateBrewery` and before export, to resolve
|
||||
`BreweryAddress::address_line1`/`postal_code` — users aren't geocoded.
|
||||
`diagrams/planned/class.md` now models this as an
|
||||
`InfrastructureGeocoding` namespace owned by the orchestrator.
|
||||
- [ ] Give the brewery phase its own **Geocode Worker** thread, between the LLM
|
||||
worker and the SQLite worker (`loc_ch` → LLM Worker → `geo_ch` → Geocode
|
||||
Worker → `exp_ch` → SQLite Worker), rather than calling `ReverseGeocode`
|
||||
inline — Nominatim's rate limit is unrelated to inference speed, so
|
||||
coupling them stalls whichever side is slower and needlessly cools the
|
||||
LLM's KV cache. It never skips a record on lookup failure, only attaches a
|
||||
placeholder; shown as its own subgraph (`B_GEO`) in
|
||||
`diagrams/planned/activity.md`. One worker per phase is safe today since
|
||||
`RunUserPhase` joins before `RunBreweryPhase` starts; concurrent phases
|
||||
plus user-geocoding would need a shared rate-limited client instead, since
|
||||
the throttle is global, not per-thread.
|
||||
|
||||
---
|
||||
|
||||
## Suggested build order
|
||||
|
||||
The planned activity diagram's own phase notes ("brewery*pool* is now fully
|
||||
populated. Phase 1b may begin.", etc.) imply a dependency order. Roughly:
|
||||
The planned activity diagram's phase notes imply a dependency order:
|
||||
|
||||
1. Domain models (§1) and data preloading (§2) — nothing else compiles without
|
||||
these.
|
||||
2. Export schema (§5) — so every later generation phase has somewhere to land
|
||||
its output.
|
||||
2. Export schema (§5) — so every generation phase has somewhere to land output.
|
||||
3. Policy/strategy layer (§3) and generator interface (§4).
|
||||
4. Concurrency/orchestration rewrite (§6), which is the only piece that actually
|
||||
wires §1–§5 into the phased, parallel pipeline shown in the planned diagrams.
|
||||
5. Enrichment cache pre-warming and the city/region query decision (§7) — useful
|
||||
at any point, but most valuable once phases run concurrently.
|
||||
4. Concurrency/orchestration rewrite (§6) — wires §1–§5 into the phased,
|
||||
parallel pipeline the planned diagrams show.
|
||||
5. Enrichment cache pre-warming and the city/region query decision (§7) — most
|
||||
valuable once phases run concurrently.
|
||||
|
||||
@@ -1,34 +0,0 @@
|
||||
skinparam shadowing false
|
||||
skinparam backgroundColor #FCFCF7
|
||||
skinparam defaultFontName "DM Sans"
|
||||
skinparam defaultFontColor #14180C
|
||||
skinparam titleFontName "Volkhov"
|
||||
skinparam titleFontColor #14180C
|
||||
skinparam ArrowColor #656F33
|
||||
skinparam NoteBackgroundColor #DBEEDD
|
||||
skinparam NoteFontColor #14180C
|
||||
skinparam NoteBorderColor #4A5837
|
||||
skinparam SwimlaneBorderColor #4A5837
|
||||
skinparam SwimlaneBorderThickness 1
|
||||
skinparam activityStartColor #EBECE3
|
||||
skinparam activityEndColor #4A5837
|
||||
skinparam activityStopColor #4A5837
|
||||
skinparam ActivityBackgroundColor #EBECE3
|
||||
skinparam ActivityBorderColor #4A5837
|
||||
skinparam ActivityDiamondBackgroundColor #CBD2B5
|
||||
skinparam ActivityDiamondBorderColor #4A5837
|
||||
skinparam packageStyle rectangle
|
||||
skinparam packageBackgroundColor #F1F3EA
|
||||
skinparam packageBorderColor #4A5837
|
||||
skinparam packageFontColor #14180C
|
||||
skinparam classBackgroundColor #EBECE3
|
||||
skinparam classBorderColor #4A5837
|
||||
skinparam classFontColor #14180C
|
||||
skinparam classAttributeFontColor #3F4724
|
||||
skinparam classStereotypeFontColor #4A5837
|
||||
skinparam interfaceBackgroundColor #DBEEDD
|
||||
skinparam interfaceBorderColor #4A5837
|
||||
skinparam interfaceFontColor #14180C
|
||||
skinparam enumBackgroundColor #E4E6D8
|
||||
skinparam enumBorderColor #4A5837
|
||||
skinparam enumFontColor #14180C
|
||||
11
docs/pipeline/diagrams/current/activity.md
Normal file
11
docs/pipeline/diagrams/current/activity.md
Normal file
@@ -0,0 +1,11 @@
|
||||
# The Biergarten Data Pipeline (Current — Synchronous Data Path)
|
||||
|
||||
The only concurrency in the current pipeline is a dedicated log dispatcher
|
||||
thread that drains `log_channel` and forwards entries to spdlog for the entire
|
||||
run (joined during shutdown after `log_channel` closes).
|
||||
`BiergartenPipelineOrchestrator::Run()` itself is synchronous — sampling,
|
||||
enrichment, generation, and export all happen on the main thread.
|
||||
|
||||
```mermaid
|
||||
--8<-- "pipeline/diagrams/current/activity.mmd"
|
||||
```
|
||||
86
docs/pipeline/diagrams/current/activity.mmd
Normal file
86
docs/pipeline/diagrams/current/activity.mmd
Normal file
@@ -0,0 +1,86 @@
|
||||
flowchart TD
|
||||
Start([start]) --> ParseArgs["main.cc: ParseArguments(argc, argv)"]
|
||||
ParseArgs --> ArgsValid{Arguments valid?}
|
||||
ArgsValid -- no --> LogErr[Log error / usage info] --> Stop1([stop])
|
||||
ArgsValid -- yes --> Init["main.cc: Init OpenSSL global state and LlamaBackendState"]
|
||||
Init --> DI["main.cc: di::make_injector(...)<br/>binds ILogger, IEnrichmentService, DataGenerator, IExportService, IPromptFormatter, WebClient based on --mocked vs. model-backed mode"]
|
||||
DI --> LogThread["main.cc: create BoundedChannel log_channel, spawn log dispatcher thread (LogDispatcher::Run)"]
|
||||
LogThread --> CreateOrch["main.cc: injector.create BiergartenPipelineOrchestrator"]
|
||||
CreateOrch --> Run["BiergartenPipelineOrchestrator::Run()"]
|
||||
|
||||
Run --> SqliteInit["SqliteExportService.Initialize()<br/>builds biergarten_seed_<UTC datetime>.sqlite (numeric suffix if it exists), opens DB connection, executes schema DDL, begins transaction"]
|
||||
SqliteInit --> QueryLoc[Orchestrator: QueryLocations]
|
||||
QueryLoc --> LoadLoc["CuratedJsonDataService.LoadLocations()"]
|
||||
LoadLoc --> SampleLoc["std::ranges::sample(all_locations, --location-count) — defaults to 10"]
|
||||
|
||||
SampleLoc --> EnrichLoop{For each sampled Location}
|
||||
EnrichLoop --> GetContext["IEnrichmentService.GetLocationContext(loc)<br/>WikipediaEnrichmentService fetches a generic 'brewing' extract and a 'beer in {country}' extract (not city/region-specific); MockEnrichmentService (--mocked) returns an empty string"]
|
||||
GetContext --> LookupFailed{Lookup failed?}
|
||||
LookupFailed -- yes --> SkipCity1[Log warning, skip city] --> EnrichLoop
|
||||
LookupFailed -- no --> StoreEnriched["Store EnrichedCity{Location, region_context}"] --> EnrichLoop
|
||||
EnrichLoop -- done --> GenUsers[Orchestrator: GenerateUsers enriched_cities]
|
||||
SkipCity1 ~~~ GenUsers
|
||||
StoreEnriched ~~~ GenUsers
|
||||
|
||||
GenUsers --> LoadUserData["CuratedJsonDataService: LoadPersonas / LoadForenamesByCountry / LoadSurnamesByCountry<br/>each memoized after first parse; MockCuratedDataService (--mocked) returns a fixed in-memory dataset instead, skipping these JSON files"]
|
||||
|
||||
LoadUserData --> UserLoop{For each EnrichedCity}
|
||||
UserLoop --> SampleName["SampleName(forenames_by_country, surnames_by_country, location.iso3166_1)"]
|
||||
SampleName --> NamesAvail{Names available for country?}
|
||||
NamesAvail -- no --> SkipCity2[Log warning, skip city] --> UserLoop
|
||||
NamesAvail -- yes --> SamplePersona["Sample UserPersona (uniform random)"]
|
||||
SamplePersona --> UserGenMode{Generator mode}
|
||||
|
||||
UserGenMode -- MockGenerator --> UserMockGen[DeterministicHash and Format]
|
||||
UserGenMode -- LlamaGenerator --> UserLlamaLoad["prompt_directory_.Load(USER_GENERATION) → USER_GENERATION.md in --prompt-dir"]
|
||||
UserLlamaLoad --> UserLlamaRetry["Infer(...) with kUserJsonGrammar → ValidateUserJson<br/>retry up to 3 attempts, appending validation error to the retry prompt on failure"]
|
||||
UserGenMode -- OpenAIGenerator --> UserOAILoad["prompt_directory_.Load(USER_GENERATION) — same prompt file as LlamaGenerator, content is backend-agnostic"]
|
||||
UserOAILoad --> UserOAIRetry["CallChatCompletionsApi(...) with kUserJsonSchema → ValidateUserJson<br/>retry up to 2 attempts — Structured Outputs (json_schema, strict) already guarantees schema shape, so retries only cover transient failures / placeholder text"]
|
||||
|
||||
UserMockGen --> UserGenDone{Generation successful?}
|
||||
UserLlamaRetry --> UserGenDone
|
||||
UserOAIRetry --> UserGenDone
|
||||
|
||||
UserGenDone -- yes --> UserPostProc["GetRandomCoordsWithinRange (5km jitter), BuildEmail, GenerateDateOfBirth<br/>email and date_of_birth are generated programmatically, never LLM-authored; generation + coordinate jitter run inside one try/catch per record"]
|
||||
UserGenDone -- no --> UserGenFail["Log warning: generation failed, skipping"] --> UserLoop
|
||||
|
||||
UserPostProc --> UserExport["SqliteExportService.ProcessRecord(UserRecord)<br/>reuse cached city_id or insert City and cache its id; insert User + UserAddress (FK city_id)"]
|
||||
UserExport --> UserExportFail{Exception during insert?}
|
||||
UserExportFail -- yes --> UserExportWarn["Log warning: SQLite export failed"] --> UserLoop
|
||||
UserExportFail -- no --> UserLoop
|
||||
UserLoop -- done --> GenBrew[Orchestrator: GenerateBreweries enriched_cities]
|
||||
SkipCity2 ~~~ GenBrew
|
||||
UserGenFail ~~~ GenBrew
|
||||
UserExportWarn ~~~ GenBrew
|
||||
UserExportFail ~~~ GenBrew
|
||||
|
||||
GenBrew --> BrewLoop{For each EnrichedCity}
|
||||
BrewLoop --> BrewGenMode{Generator mode}
|
||||
|
||||
BrewGenMode -- MockGenerator --> BrewMockGen[DeterministicHash and Format]
|
||||
BrewGenMode -- LlamaGenerator --> BrewLlamaLoad["PrepareRegionContext, prompt_directory_.Load(BREWERY_GENERATION) → BREWERY_GENERATION.md"]
|
||||
BrewLlamaLoad --> BrewLlamaRetry["Infer(...) with kBreweryJsonGrammar → ValidateBreweryJson<br/>retry up to 3 attempts; max_tokens is fixed across retries, not raised on truncation"]
|
||||
BrewGenMode -- OpenAIGenerator --> BrewOAILoad["PrepareRegionContext, prompt_directory_.Load(BREWERY_GENERATION) — same prompt file as LlamaGenerator"]
|
||||
BrewOAILoad --> BrewOAIRetry["CallChatCompletionsApi(...) with kBreweryJsonSchema → ValidateBreweryJson<br/>retry up to 2 attempts — Structured Outputs already guarantees schema shape"]
|
||||
|
||||
BrewMockGen --> BrewGenDone{Generation successful?}
|
||||
BrewLlamaRetry --> BrewGenDone
|
||||
BrewOAIRetry --> BrewGenDone
|
||||
|
||||
BrewGenDone -- yes --> BrewPostProc["GetRandomCoordsWithinRange (5km jitter)<br/>generation + jitter run inside one try/catch per record"]
|
||||
BrewGenDone -- no --> BrewGenFail["Log warning: generation failed, skipping"] --> BrewLoop
|
||||
|
||||
BrewPostProc --> BrewGeocode["IAddressService.ReverseGeocode(longitude, latitude)<br/>NominatimAddressService hits the live Nominatim API (1 req/sec) and returns nullopt on failure; MockAddressService (--mocked) returns a fixed placeholder address instead"]
|
||||
BrewGeocode --> BrewGeocodeFailed{Geocode resolved?}
|
||||
BrewGeocodeFailed -- no --> BrewPlaceholder["Log warning: could not reverse geocode; use placeholder address_line1, empty postal_code"] --> BrewExport
|
||||
BrewGeocodeFailed -- yes --> BrewExport["SqliteExportService.ProcessRecord(BreweryRecord)<br/>reuse cached city_id or insert City and cache its id; insert Brewery + BreweryAddress (FK city_id)"]
|
||||
BrewExport --> BrewExportFail{Exception during insert?}
|
||||
BrewExportFail -- yes --> BrewExportWarn["Log warning: SQLite export failed (data loss prevented per-record; pipeline continues)"] --> BrewLoop
|
||||
BrewExportFail -- no --> BrewLoop
|
||||
BrewLoop -- done --> Finalize["SqliteExportService.Finalize()<br/>commits transaction, closes database connection"]
|
||||
BrewGenFail ~~~ Finalize
|
||||
BrewExportWarn ~~~ Finalize
|
||||
BrewExportFail ~~~ Finalize
|
||||
|
||||
Finalize --> CloseLog["main.cc: close log_channel, join log dispatcher thread"]
|
||||
CloseLog --> Stop2(["return 0 / stop"])
|
||||
@@ -1,293 +0,0 @@
|
||||
@startuml
|
||||
skinparam style strictuml
|
||||
skinparam defaultFontName "DM Sans"
|
||||
skinparam defaultFontSize 14
|
||||
skinparam titleFontName "Volkhov"
|
||||
skinparam titleFontSize 20
|
||||
skinparam backgroundColor #FAFCF9
|
||||
skinparam defaultFontColor #28342A
|
||||
skinparam titleFontColor #28342A
|
||||
skinparam ArrowColor #628A5B
|
||||
skinparam NoteBackgroundColor #EAF0E8
|
||||
skinparam NoteBorderColor #547461
|
||||
skinparam ActivityBackgroundColor #FAFCF9
|
||||
skinparam ActivityBorderColor #547461
|
||||
skinparam ActivityDiamondBackgroundColor #FAFCF9
|
||||
skinparam ActivityDiamondBorderColor #628A5B
|
||||
skinparam ActivityBarColor #628A5B
|
||||
skinparam SwimlaneBorderColor #547461
|
||||
skinparam SwimlaneBorderThickness 0.3
|
||||
|
||||
title The Biergarten Data Pipeline (Current — Synchronous Data Path)
|
||||
|
||||
|#F2F6F0|main.cc|
|
||||
start
|
||||
:Create BoundedChannel<LogEntry> log_channel;
|
||||
:Spawn log dispatcher thread\n(LogDispatcher::Run());
|
||||
note right
|
||||
The only concurrency in the current pipeline:
|
||||
a dedicated thread drains log_channel and
|
||||
forwards entries to spdlog for the entire run.
|
||||
Joined during shutdown after log_channel closes.
|
||||
end note
|
||||
:ParseArguments(argc, argv);
|
||||
if (Are arguments valid?) then (no)
|
||||
:Log error / usage info;
|
||||
stop
|
||||
else (yes)
|
||||
endif
|
||||
|
||||
:Init OpenSSL global state & LlamaBackendState;
|
||||
:di::make_injector(...);
|
||||
note right
|
||||
Binds ILogger, IEnrichmentService, DataGenerator,
|
||||
IExportService, IPromptFormatter, WebClient based
|
||||
on --mocked vs. model-backed mode.
|
||||
end note
|
||||
:injector.create<std::unique_ptr<BiergartenPipelineOrchestrator>>();
|
||||
:BiergartenPipelineOrchestrator::Run();
|
||||
note right
|
||||
Run() itself is synchronous — sampling, enrichment,
|
||||
generation, and export all happen on this thread.
|
||||
end note
|
||||
|
||||
|#EAF0E8|BiergartenPipelineOrchestrator|
|
||||
:Initialize SQLite export;
|
||||
|
||||
|#E0EAE0|SqliteExportService|
|
||||
:GetUtcTimestamp() from SystemDateTimeProvider;
|
||||
:Initialize();
|
||||
note right
|
||||
Builds a fresh biergarten_seed_<UTC datetime>.sqlite filename
|
||||
Appends a numeric suffix if the timestamp already exists
|
||||
Opens DB Connection
|
||||
Executes Schema DDL
|
||||
Begins Transaction
|
||||
end note
|
||||
|
||||
|#EAF0E8|BiergartenPipelineOrchestrator|
|
||||
:QueryLocations();
|
||||
|
||||
|#E2EBDC|JsonLoader|
|
||||
:JsonLoader::LoadLocations("locations.json");
|
||||
:std::ranges::sample(all_locations, --location-count);
|
||||
note right
|
||||
--location-count defaults to 10.
|
||||
end note
|
||||
|
||||
|#EAF0E8|BiergartenPipelineOrchestrator|
|
||||
while (For each sampled Location?) is (Remaining cities)
|
||||
|#DCE8D8|IEnrichmentService|
|
||||
:GetLocationContext(loc);
|
||||
note right
|
||||
WikipediaEnrichmentService fetches a generic
|
||||
"brewing" extract and a "beer in {country}" extract
|
||||
(not a city/region-specific page). MockEnrichmentService
|
||||
(--mocked) returns an empty string instead.
|
||||
end note
|
||||
|#EAF0E8|BiergartenPipelineOrchestrator|
|
||||
if (Lookup failed?) then (yes)
|
||||
:Log warning, skip city;
|
||||
else (no)
|
||||
:Store EnrichedCity{Location, region_context};
|
||||
endif
|
||||
endwhile (Done)
|
||||
|
||||
|#EAF0E8|BiergartenPipelineOrchestrator|
|
||||
:GenerateUsers(enriched_cities);
|
||||
|
||||
|#E2EBDC|JsonLoader|
|
||||
:LoadPersonas("personas.json");
|
||||
:LoadForenamesByCountry("forenames-by-country.json");
|
||||
:LoadSurnamesByCountry("surnames-by-country.json");
|
||||
note right
|
||||
Each call is cached after its first parse.
|
||||
MockCuratedDataService (--mocked) returns a
|
||||
fixed in-memory dataset instead and skips
|
||||
these three JSON files entirely.
|
||||
end note
|
||||
|
||||
|#EAF0E8|BiergartenPipelineOrchestrator|
|
||||
while (For each EnrichedCity?) is (Remaining cities)
|
||||
:SampleName(forenames_by_country,\n surnames_by_country, location.iso3166_1);
|
||||
if (Names available for country?) then (no)
|
||||
:Log warning, skip city;
|
||||
else (yes)
|
||||
:Sample UserPersona (uniform random);
|
||||
|
||||
|#E5EDE1|DataGenerator|
|
||||
if (Generator Mode) then (MockGenerator)
|
||||
:DeterministicHash & Format;
|
||||
elseif (Generator Mode) then (LlamaGenerator)
|
||||
:prompt_directory_->Load("USER_GENERATION");
|
||||
note right
|
||||
Resolves to USER_GENERATION.md inside --prompt-dir.
|
||||
end note
|
||||
repeat
|
||||
:Infer(system_prompt, user_prompt, max_tokens, kUserJsonGrammar);
|
||||
:ValidateUserJson(raw, user);
|
||||
if (Is JSON Valid?) then (yes)
|
||||
break
|
||||
else (no)
|
||||
:Attempt++;
|
||||
:Append validation error to retry prompt;
|
||||
endif
|
||||
repeat while (Attempt < 3?) is (yes)
|
||||
else (OpenAIGenerator)
|
||||
:prompt_directory_->Load("USER_GENERATION");
|
||||
note right
|
||||
Same prompt file as LlamaGenerator - content
|
||||
is backend-agnostic.
|
||||
end note
|
||||
repeat
|
||||
:CallChatCompletionsApi(system_prompt, user_prompt,\nkUserJsonSchema, max_tokens);
|
||||
:ValidateUserJson(raw, user);
|
||||
if (Is JSON Valid?) then (yes)
|
||||
break
|
||||
else (no)
|
||||
:Attempt++;
|
||||
:Append validation error to retry prompt;
|
||||
endif
|
||||
repeat while (Attempt < 2?) is (yes)
|
||||
note right
|
||||
Structured Outputs (response_format:
|
||||
json_schema, strict: true) already
|
||||
guarantees schema shape, so only 2
|
||||
attempts are needed - retries here cover
|
||||
transient failures / placeholder text,
|
||||
not malformed JSON.
|
||||
end note
|
||||
endif
|
||||
|
||||
|#EAF0E8|BiergartenPipelineOrchestrator|
|
||||
if (Generation successful?) then (yes)
|
||||
:GetRandomCoordsWithinRange(city_centre, 5km, rng);
|
||||
:BuildEmail(name, used_email_local_parts);
|
||||
:GenerateDateOfBirth(rng);
|
||||
note right
|
||||
email and date_of_birth are generated
|
||||
programmatically, never LLM-authored.
|
||||
Generation + coordinate jitter run inside
|
||||
one try/catch per record. UserRecord.address
|
||||
now carries {city, longitude, latitude}.
|
||||
end note
|
||||
|
||||
|#E0EAE0|SqliteExportService|
|
||||
:ProcessRecord(UserRecord);
|
||||
if (Location in cache?) then (yes)
|
||||
:Reuse location_id;
|
||||
else (no)
|
||||
:Insert Location & Cache ID;
|
||||
endif
|
||||
:Insert User (FK: location_id);
|
||||
|
||||
if (Exception caught during insert?) then (yes)
|
||||
|#EAF0E8|BiergartenPipelineOrchestrator|
|
||||
:Log warning "SQLite export failed";
|
||||
else (no)
|
||||
endif
|
||||
else (no)
|
||||
:Log warning "Generation failed, skipping...";
|
||||
endif
|
||||
endif
|
||||
endwhile (Done)
|
||||
|
||||
|#EAF0E8|BiergartenPipelineOrchestrator|
|
||||
:GenerateBreweries(enriched_cities);
|
||||
|
||||
|#E5EDE1|DataGenerator|
|
||||
while (For each EnrichedCity?) is (Remaining cities)
|
||||
if (Generator Mode) then (MockGenerator)
|
||||
:DeterministicHash & Format;
|
||||
elseif (Generator Mode) then (LlamaGenerator)
|
||||
:PrepareRegionContext;
|
||||
:prompt_directory_->Load("BREWERY_GENERATION");
|
||||
note right
|
||||
Resolves to BREWERY_GENERATION.md inside --prompt-dir.
|
||||
end note
|
||||
repeat
|
||||
:Infer(system_prompt, user_prompt, max_tokens, kBreweryJsonGrammar);
|
||||
:ValidateBreweryJson(raw, brewery);
|
||||
if (Is JSON Valid?) then (yes)
|
||||
break
|
||||
else (no)
|
||||
:Attempt++;
|
||||
:Append validation error to retry prompt;
|
||||
endif
|
||||
repeat while (Attempt < 3?) is (yes)
|
||||
note right
|
||||
max_tokens is fixed across retries —
|
||||
it is not raised on truncation.
|
||||
end note
|
||||
else (OpenAIGenerator)
|
||||
:PrepareRegionContext;
|
||||
:prompt_directory_->Load("BREWERY_GENERATION");
|
||||
note right
|
||||
Same prompt file as LlamaGenerator - content
|
||||
is backend-agnostic.
|
||||
end note
|
||||
repeat
|
||||
:CallChatCompletionsApi(system_prompt, user_prompt,\nkBreweryJsonSchema, max_tokens);
|
||||
:ValidateBreweryJson(raw, brewery);
|
||||
if (Is JSON Valid?) then (yes)
|
||||
break
|
||||
else (no)
|
||||
:Attempt++;
|
||||
:Append validation error to retry prompt;
|
||||
endif
|
||||
repeat while (Attempt < 2?) is (yes)
|
||||
note right
|
||||
Structured Outputs already guarantees schema
|
||||
shape, so only 2 attempts are needed - retries
|
||||
here cover transient failures / placeholder
|
||||
text, not malformed JSON.
|
||||
end note
|
||||
endif
|
||||
|
||||
|#EAF0E8|BiergartenPipelineOrchestrator|
|
||||
if (Generation successful?) then (yes)
|
||||
:GetRandomCoordsWithinRange(city_centre, 5km, rng);
|
||||
note right
|
||||
Generation + coordinate jitter run inside one
|
||||
try/catch per record. BreweryRecord.address
|
||||
now carries {city, longitude, latitude}.
|
||||
end note
|
||||
|
||||
|#E0EAE0|SqliteExportService|
|
||||
:ProcessRecord(GeneratedBrewery);
|
||||
if (Location in cache?) then (yes)
|
||||
:Reuse location_id;
|
||||
else (no)
|
||||
:Insert Location & Cache ID;
|
||||
endif
|
||||
:Insert Brewery (FK: location_id);
|
||||
|
||||
if (Exception caught during insert?) then (yes)
|
||||
|#EAF0E8|BiergartenPipelineOrchestrator|
|
||||
:Log warning "SQLite export failed";
|
||||
note right
|
||||
Data loss is prevented per-record.
|
||||
The pipeline continues running.
|
||||
end note
|
||||
else (no)
|
||||
endif
|
||||
else (no)
|
||||
:Log warning "Generation failed, skipping...";
|
||||
endif
|
||||
|#E5EDE1|DataGenerator|
|
||||
endwhile (Done)
|
||||
|
||||
|#E0EAE0|SqliteExportService|
|
||||
:Finalize();
|
||||
note right
|
||||
Commits Transaction
|
||||
Closes Database Connection
|
||||
end note
|
||||
|
||||
|#F2F6F0|main.cc|
|
||||
:Close log_channel;
|
||||
:Join log dispatcher thread;
|
||||
:Return 0;
|
||||
stop
|
||||
|
||||
@enduml
|
||||
5
docs/pipeline/diagrams/current/class.md
Normal file
5
docs/pipeline/diagrams/current/class.md
Normal file
@@ -0,0 +1,5 @@
|
||||
# The Biergarten Data Pipeline — Class Diagram (Current)
|
||||
|
||||
```mermaid
|
||||
--8<-- "pipeline/diagrams/current/class.mmd"
|
||||
```
|
||||
285
docs/pipeline/diagrams/current/class.mmd
Normal file
285
docs/pipeline/diagrams/current/class.mmd
Normal file
@@ -0,0 +1,285 @@
|
||||
classDiagram
|
||||
class BiergartenPipelineOrchestrator {
|
||||
-shared_ptr~ILogger~ logger_
|
||||
-unique_ptr~IEnrichmentService~ context_service_
|
||||
-unique_ptr~DataGenerator~ generator_
|
||||
-unique_ptr~IExportService~ exporter_
|
||||
-unique_ptr~ICuratedDataService~ curated_data_service_
|
||||
-unique_ptr~IAddressService~ address_service_
|
||||
-ApplicationOptions application_options_
|
||||
-vector~BreweryRecord~ generated_breweries_
|
||||
-vector~UserRecord~ generated_users_
|
||||
+Run() bool
|
||||
-QueryLocations() vector~City~
|
||||
-GenerateBreweries(cities) void
|
||||
-GenerateUsers(cities) void
|
||||
-LogResults() void
|
||||
}
|
||||
note for BiergartenPipelineOrchestrator "GenerateBreweries/GenerateUsers jitter each address via GetRandomCoordsWithinRange(), uniformly within 5km of the city centre. GenerateBreweries additionally reverse-geocodes the jittered coordinates via address_service_."
|
||||
|
||||
class LogLevel {
|
||||
<<enumeration>>
|
||||
Debug
|
||||
Info
|
||||
Warn
|
||||
Error
|
||||
}
|
||||
|
||||
class PipelinePhase {
|
||||
<<enumeration>>
|
||||
Startup
|
||||
Enrichment
|
||||
UserGeneration
|
||||
BreweryAndBeerGeneration
|
||||
CheckinGeneration
|
||||
RatingGeneration
|
||||
FollowGeneration
|
||||
Teardown
|
||||
}
|
||||
|
||||
class LogDTO {
|
||||
<<struct>>
|
||||
+LogLevel level
|
||||
+PipelinePhase phase
|
||||
+string message
|
||||
}
|
||||
|
||||
class LogEntry {
|
||||
<<struct>>
|
||||
+time_point timestamp
|
||||
+source_location origin
|
||||
+thread::id thread_id
|
||||
+LogLevel level
|
||||
+PipelinePhase phase
|
||||
+string message
|
||||
}
|
||||
|
||||
class ILogger {
|
||||
<<interface>>
|
||||
+Log(payload) void
|
||||
-DoLog(entry)* void
|
||||
}
|
||||
|
||||
class LogProducer {
|
||||
-BoundedChannel~LogEntry~ channel_
|
||||
-DoLog(entry) void
|
||||
}
|
||||
|
||||
class LogDispatcher {
|
||||
-BoundedChannel~LogEntry~ channel_
|
||||
+Run() void
|
||||
-ToSpdlogLevel(level) level_enum
|
||||
}
|
||||
|
||||
class IEnrichmentService {
|
||||
<<interface>>
|
||||
+GetLocationContext(loc) string
|
||||
}
|
||||
|
||||
class MockEnrichmentService {
|
||||
+GetLocationContext(loc) string
|
||||
}
|
||||
|
||||
class WikipediaEnrichmentService {
|
||||
-unique_ptr~WebClient~ client_
|
||||
-unordered_map~string, string~ extract_cache_
|
||||
+GetLocationContext(loc) string
|
||||
-FetchExtract(query) string
|
||||
}
|
||||
|
||||
class WebClient {
|
||||
<<interface>>
|
||||
+Get(url) string
|
||||
+UrlEncode(value) string
|
||||
}
|
||||
|
||||
class HttpWebClient {
|
||||
+Get(url) string
|
||||
+UrlEncode(value) string
|
||||
}
|
||||
|
||||
class DataGenerator {
|
||||
<<interface>>
|
||||
+GenerateBrewery(city) BreweryResult
|
||||
+GenerateUser(city, persona, name) UserResult
|
||||
}
|
||||
|
||||
class MockGenerator {
|
||||
+GenerateBrewery(city) BreweryResult
|
||||
+GenerateUser(city, persona, name) UserResult
|
||||
-DeterministicHash(location) size_t
|
||||
-DeterministicHash(location, persona, name) size_t
|
||||
}
|
||||
|
||||
class LlamaGenerator {
|
||||
-ModelHandle model_
|
||||
-ContextHandle context_
|
||||
-unique_ptr~IPromptFormatter~ prompt_formatter_
|
||||
-unique_ptr~IPromptDirectory~ prompt_directory_
|
||||
-mt19937 rng_
|
||||
+GenerateBrewery(...) BreweryResult
|
||||
+GenerateUser(...) UserResult
|
||||
-Load(model_path) void
|
||||
-Infer(...) string
|
||||
-InferFormatted(...) string
|
||||
}
|
||||
|
||||
class OpenAIGenerator {
|
||||
-string api_key_
|
||||
-string model_
|
||||
-shared_ptr~ILogger~ logger_
|
||||
-unique_ptr~IPromptDirectory~ prompt_directory_
|
||||
-unique_ptr~WebClient~ web_client_
|
||||
+GenerateBrewery(city) BreweryResult
|
||||
+GenerateUser(city, persona, name) UserResult
|
||||
-CallChatCompletionsApi(...) string
|
||||
}
|
||||
note for OpenAIGenerator "Chat Completions with Structured Outputs (json_schema, strict) guarantees schema-valid JSON, replacing llama.cpp's GBNF grammar — no IPromptFormatter needed"
|
||||
|
||||
class IPromptFormatter {
|
||||
<<interface>>
|
||||
+Format(system_prompt, user_prompt) string
|
||||
}
|
||||
|
||||
class Gemma4JinjaPromptFormatter {
|
||||
+Format(system_prompt, user_prompt) string
|
||||
}
|
||||
|
||||
class IPromptDirectory {
|
||||
<<interface>>
|
||||
+Load(key) string
|
||||
}
|
||||
|
||||
class PromptDirectory {
|
||||
-path prompt_dir_
|
||||
-unordered_map~string, string~ cache_
|
||||
+Load(key) string
|
||||
}
|
||||
|
||||
class ICuratedDataService {
|
||||
<<interface>>
|
||||
+LoadLocations() vector~Location~
|
||||
+LoadPersonas() vector~UserPersona~
|
||||
+LoadForenamesByCountry() unordered_map~string, forename_list~
|
||||
+LoadSurnamesByCountry() unordered_map~string, surname_list~
|
||||
}
|
||||
|
||||
class CuratedJsonDataService {
|
||||
-CuratedDataFilePaths filepaths_
|
||||
-cache cache_
|
||||
+LoadLocations() vector~Location~
|
||||
+LoadPersonas() vector~UserPersona~
|
||||
+LoadForenamesByCountry() unordered_map~string, forename_list~
|
||||
+LoadSurnamesByCountry() unordered_map~string, surname_list~
|
||||
}
|
||||
note for CuratedJsonDataService "Takes a CuratedDataFilePaths DTO (locations/personas/forenames/surnames paths) once at construction, not per Load* call. Each Load* method memoizes its result in cache_ after its first call."
|
||||
|
||||
class MockCuratedDataService {
|
||||
-vector~Location~ locations_
|
||||
-vector~UserPersona~ personas_
|
||||
-unordered_map~string, forename_list~ forenames_by_country_
|
||||
-unordered_map~string, surname_list~ surnames_by_country_
|
||||
+LoadLocations() vector~Location~
|
||||
+LoadPersonas() vector~UserPersona~
|
||||
+LoadForenamesByCountry() unordered_map~string, forename_list~
|
||||
+LoadSurnamesByCountry() unordered_map~string, surname_list~
|
||||
}
|
||||
note for MockCuratedDataService "Fixed 4-location/3-persona dataset (US, DE, FR, BE) for --mocked runs"
|
||||
|
||||
class IAddressService {
|
||||
<<interface>>
|
||||
+ReverseGeocode(longitude, latitude) optional~AddressLookupResult~
|
||||
}
|
||||
|
||||
class NominatimAddressService {
|
||||
-unique_ptr~WebClient~ client_
|
||||
-shared_ptr~ILogger~ logger_
|
||||
+ReverseGeocode(longitude, latitude) optional~AddressLookupResult~
|
||||
}
|
||||
note for NominatimAddressService "Queries the public Nominatim API; sleeps 1s per request per its usage policy; returns nullopt on failure or when no usable street address is found"
|
||||
|
||||
class MockAddressService {
|
||||
+ReverseGeocode(longitude, latitude) optional~AddressLookupResult~
|
||||
}
|
||||
note for MockAddressService "Returns a fixed placeholder address ('123 Mock Street', '00000') for --mocked runs, no network call"
|
||||
|
||||
class IExportService {
|
||||
<<interface>>
|
||||
+Initialize() void
|
||||
+ProcessRecord(brewery) uint64_t
|
||||
+ProcessRecord(user) uint64_t
|
||||
+Finalize() void
|
||||
}
|
||||
|
||||
class SqliteExportService {
|
||||
-unique_ptr~IDateTimeProvider~ date_time_provider_
|
||||
-string run_timestamp_utc_
|
||||
-path database_path_
|
||||
-SqliteDatabaseHandle db_handle_
|
||||
-SqliteStatementHandle insert_city_stmt_
|
||||
-SqliteStatementHandle insert_brewery_stmt_
|
||||
-SqliteStatementHandle insert_brewery_address_stmt_
|
||||
-SqliteStatementHandle insert_user_stmt_
|
||||
-SqliteStatementHandle insert_user_address_stmt_
|
||||
-bool transaction_open_
|
||||
-unordered_map~string, sqlite3_int64~ city_cache_
|
||||
+Initialize() void
|
||||
+ProcessRecord(brewery) uint64_t
|
||||
+ProcessRecord(user) uint64_t
|
||||
+Finalize() void
|
||||
-InitializeSchema() void
|
||||
-ResolveCityId(city) sqlite3_int64
|
||||
}
|
||||
|
||||
class IDateTimeProvider {
|
||||
<<interface>>
|
||||
+GetUtcTimestamp() string
|
||||
}
|
||||
|
||||
class SystemDateTimeProvider {
|
||||
+GetUtcTimestamp() string
|
||||
}
|
||||
|
||||
BiergartenPipelineOrchestrator *-- ILogger : owns
|
||||
BiergartenPipelineOrchestrator *-- IEnrichmentService : owns
|
||||
BiergartenPipelineOrchestrator *-- DataGenerator : owns
|
||||
BiergartenPipelineOrchestrator *-- IExportService : owns
|
||||
BiergartenPipelineOrchestrator *-- ICuratedDataService : owns
|
||||
BiergartenPipelineOrchestrator *-- IAddressService : owns
|
||||
|
||||
LogEntry *-- LogLevel
|
||||
LogEntry *-- PipelinePhase
|
||||
LogDTO *-- LogLevel
|
||||
LogDTO *-- PipelinePhase
|
||||
ILogger <|.. LogProducer : implements
|
||||
LogProducer ..> LogEntry : emits
|
||||
LogDispatcher ..> LogEntry : consumes
|
||||
|
||||
IEnrichmentService <|.. WikipediaEnrichmentService : implements
|
||||
IEnrichmentService <|.. MockEnrichmentService : implements
|
||||
WikipediaEnrichmentService *-- WebClient : owns
|
||||
|
||||
WebClient <|.. HttpWebClient : implements
|
||||
|
||||
DataGenerator <|.. MockGenerator : implements
|
||||
DataGenerator <|.. LlamaGenerator : implements
|
||||
DataGenerator <|.. OpenAIGenerator : implements
|
||||
|
||||
LlamaGenerator *-- IPromptFormatter : uses
|
||||
LlamaGenerator *-- IPromptDirectory : uses
|
||||
|
||||
OpenAIGenerator *-- IPromptDirectory : uses
|
||||
OpenAIGenerator *-- WebClient : uses
|
||||
|
||||
IPromptFormatter <|.. Gemma4JinjaPromptFormatter : implements
|
||||
IPromptDirectory <|.. PromptDirectory : implements
|
||||
|
||||
ICuratedDataService <|.. CuratedJsonDataService : implements
|
||||
ICuratedDataService <|.. MockCuratedDataService : implements
|
||||
|
||||
IAddressService <|.. NominatimAddressService : implements
|
||||
IAddressService <|.. MockAddressService : implements
|
||||
|
||||
IExportService <|.. SqliteExportService : implements
|
||||
SqliteExportService *-- IDateTimeProvider : owns
|
||||
IDateTimeProvider <|.. SystemDateTimeProvider : implements
|
||||
@@ -1,301 +0,0 @@
|
||||
@startuml
|
||||
skinparam style strictuml
|
||||
skinparam defaultFontName "DM Sans"
|
||||
skinparam defaultFontSize 14
|
||||
skinparam titleFontName "Volkhov"
|
||||
skinparam titleFontSize 20
|
||||
skinparam backgroundColor #FAFCF9
|
||||
skinparam defaultFontColor #28342A
|
||||
skinparam titleFontColor #28342A
|
||||
skinparam ArrowColor #628A5B
|
||||
|
||||
skinparam class {
|
||||
BackgroundColor #FAFCF9
|
||||
HeaderBackgroundColor #EAF0E8
|
||||
BorderColor #547461
|
||||
ArrowColor #628A5B
|
||||
FontColor #28342A
|
||||
}
|
||||
|
||||
skinparam note {
|
||||
BackgroundColor #EAF0E8
|
||||
BorderColor #547461
|
||||
FontColor #28342A
|
||||
}
|
||||
|
||||
title The Biergarten Data Pipeline - Class Diagram
|
||||
|
||||
class BiergartenPipelineOrchestrator {
|
||||
- logger_ : std::shared_ptr<ILogger>
|
||||
- context_service_ : std::unique_ptr<IEnrichmentService>
|
||||
- generator_ : std::unique_ptr<DataGenerator>
|
||||
- exporter_ : std::unique_ptr<IExportService>
|
||||
- curated_data_service_ : std::unique_ptr<ICuratedDataService>
|
||||
- application_options_ : ApplicationOptions
|
||||
- generated_breweries_ : std::vector<BreweryRecord>
|
||||
- generated_users_ : std::vector<UserRecord>
|
||||
+ Run() : bool
|
||||
- QueryLocations() : std::vector<City>
|
||||
- GenerateBreweries(cities : std::span<const EnrichedCity>) : void
|
||||
- GenerateUsers(cities : std::span<const EnrichedCity>) : void
|
||||
- LogResults() const : void
|
||||
}
|
||||
|
||||
note right of BiergartenPipelineOrchestrator
|
||||
GenerateBreweries and GenerateUsers both call the
|
||||
free function GetRandomCoordsWithinRange()
|
||||
(biergarten_pipeline_orchestrator/coords.h) to jitter
|
||||
a coordinate pair uniformly within 5km of the city
|
||||
centre for each generated record's address.
|
||||
end note
|
||||
|
||||
class LogLevel <<enumeration>> {
|
||||
Debug
|
||||
Info
|
||||
Warn
|
||||
Error
|
||||
}
|
||||
|
||||
class PipelinePhase <<enumeration>> {
|
||||
Startup
|
||||
Enrichment
|
||||
UserGeneration
|
||||
BreweryAndBeerGeneration
|
||||
CheckinGeneration
|
||||
RatingGeneration
|
||||
FollowGeneration
|
||||
Teardown
|
||||
}
|
||||
|
||||
struct LogDTO {
|
||||
+ level : LogLevel
|
||||
+ phase : PipelinePhase
|
||||
+ message : std::string
|
||||
}
|
||||
|
||||
struct LogEntry {
|
||||
+ timestamp : std::chrono::system_clock::time_point
|
||||
+ origin : std::source_location
|
||||
+ thread_id : std::thread::id
|
||||
+ level : LogLevel
|
||||
+ phase : PipelinePhase
|
||||
+ message : std::string
|
||||
}
|
||||
|
||||
interface ILogger <<interface>> {
|
||||
+ Log(payload : LogDTO) : void
|
||||
- {abstract} DoLog(entry : LogEntry) : void
|
||||
}
|
||||
|
||||
class LogProducer {
|
||||
- channel_ : BoundedChannel<LogEntry>&
|
||||
- DoLog(entry : LogEntry) : void
|
||||
}
|
||||
|
||||
class LogDispatcher {
|
||||
- channel_ : BoundedChannel<LogEntry>&
|
||||
+ Run() : void
|
||||
- ToSpdlogLevel(level) : spdlog::level::level_enum
|
||||
}
|
||||
|
||||
interface IEnrichmentService <<interface>> {
|
||||
+ GetLocationContext(loc : const Location&) : std::string
|
||||
}
|
||||
|
||||
class MockEnrichmentService {
|
||||
+ GetLocationContext(loc : const Location&) : std::string
|
||||
}
|
||||
|
||||
class WikipediaEnrichmentService {
|
||||
- client_ : std::unique_ptr<WebClient>
|
||||
- extract_cache_ : std::unordered_map<std::string, std::string>
|
||||
+ GetLocationContext(loc : const Location&) : std::string
|
||||
- FetchExtract(query : std::string_view) : std::string
|
||||
}
|
||||
|
||||
interface WebClient <<interface>> {
|
||||
+ Get(url : const std::string&) : std::string
|
||||
+ UrlEncode(value : const std::string&) : std::string
|
||||
}
|
||||
|
||||
class HttpWebClient {
|
||||
+ Get(url : const std::string&) : std::string
|
||||
+ UrlEncode(value : const std::string&) : std::string
|
||||
}
|
||||
|
||||
interface DataGenerator <<interface>> {
|
||||
+ GenerateBrewery(city : const EnrichedCity&) : BreweryResult
|
||||
+ GenerateUser(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult
|
||||
}
|
||||
|
||||
class MockGenerator {
|
||||
+ GenerateBrewery(city : const EnrichedCity&) : BreweryResult
|
||||
+ GenerateUser(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult
|
||||
- DeterministicHash(location : const Location&) : size_t
|
||||
- DeterministicHash(location : const Location&, persona : const UserPersona&, name : const Name&) : size_t
|
||||
}
|
||||
|
||||
class LlamaGenerator {
|
||||
- model_ : ModelHandle
|
||||
- context_ : ContextHandle
|
||||
- prompt_formatter_ : std::unique_ptr<IPromptFormatter>
|
||||
- prompt_directory_ : std::unique_ptr<IPromptDirectory>
|
||||
- rng_ : std::mt19937
|
||||
+ GenerateBrewery(...) : BreweryResult
|
||||
+ GenerateUser(...) : UserResult
|
||||
- Load(model_path : const std::string&) : void
|
||||
- Infer(...) : std::string
|
||||
- InferFormatted(...) : std::string
|
||||
}
|
||||
|
||||
class OpenAIGenerator {
|
||||
- api_key_ : std::string
|
||||
- model_ : std::string
|
||||
- logger_ : std::shared_ptr<ILogger>
|
||||
- prompt_directory_ : std::unique_ptr<IPromptDirectory>
|
||||
- web_client_ : std::unique_ptr<WebClient>
|
||||
+ GenerateBrewery(city : const EnrichedCity&) : BreweryResult
|
||||
+ GenerateUser(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult
|
||||
- CallChatCompletionsApi(...) : std::string
|
||||
}
|
||||
|
||||
note right of OpenAIGenerator
|
||||
Chat Completions with Structured Outputs
|
||||
(response_format: json_schema, strict: true)
|
||||
guarantees schema-valid JSON, replacing
|
||||
llama.cpp's GBNF grammar constraint -
|
||||
no IPromptFormatter needed.
|
||||
end note
|
||||
|
||||
interface IPromptFormatter <<interface>> {
|
||||
+ Format(system_prompt : std::string_view, user_prompt : std::string_view) : std::string
|
||||
}
|
||||
|
||||
class Gemma4JinjaPromptFormatter {
|
||||
+ Format(system_prompt : std::string_view, user_prompt : std::string_view) : std::string
|
||||
}
|
||||
|
||||
interface IPromptDirectory <<interface>> {
|
||||
+ Load(key : std::string_view) : std::string
|
||||
}
|
||||
|
||||
class PromptDirectory {
|
||||
- prompt_dir_ : std::filesystem::path
|
||||
- cache_ : std::unordered_map<std::string, std::string>
|
||||
+ Load(key : std::string_view) : std::string
|
||||
}
|
||||
|
||||
interface ICuratedDataService <<interface>> {
|
||||
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
|
||||
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
|
||||
+ LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
|
||||
+ LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
|
||||
}
|
||||
|
||||
class JsonLoader {
|
||||
- cache_ : cache
|
||||
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
|
||||
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
|
||||
+ LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
|
||||
+ LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
|
||||
}
|
||||
|
||||
note right of JsonLoader
|
||||
Each Load* method memoizes its result in
|
||||
cache_ on first call; later calls on the
|
||||
same instance return the cached value
|
||||
without re-parsing.
|
||||
end note
|
||||
|
||||
class MockCuratedDataService {
|
||||
- locations_ : std::vector<Location>
|
||||
- personas_ : std::vector<UserPersona>
|
||||
- forenames_by_country_ : std::unordered_map<std::string, forename_list>
|
||||
- surnames_by_country_ : std::unordered_map<std::string, surname_list>
|
||||
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
|
||||
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
|
||||
+ LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
|
||||
+ LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
|
||||
}
|
||||
|
||||
note right of MockCuratedDataService
|
||||
Fixed 4-location / 3-persona dataset
|
||||
(US, DE, FR, BE) used in --mocked runs;
|
||||
filepath arguments are ignored.
|
||||
end note
|
||||
|
||||
interface IExportService <<interface>> {
|
||||
+ Initialize() : void
|
||||
+ ProcessRecord(brewery : const BreweryRecord&) : uint64_t
|
||||
+ ProcessRecord(user : const UserRecord&) : uint64_t
|
||||
+ Finalize() : void
|
||||
}
|
||||
|
||||
class SqliteExportService {
|
||||
- date_time_provider_ : std::unique_ptr<IDateTimeProvider>
|
||||
- run_timestamp_utc_ : std::string
|
||||
- database_path_ : std::filesystem::path
|
||||
- db_handle_ : SqliteDatabaseHandle
|
||||
- insert_location_stmt_ : SqliteStatementHandle
|
||||
- insert_brewery_stmt_ : SqliteStatementHandle
|
||||
- insert_user_stmt_ : SqliteStatementHandle
|
||||
- transaction_open_ : bool
|
||||
- location_cache_ : std::unordered_map<std::string, sqlite3_int64>
|
||||
+ Initialize() : void
|
||||
+ ProcessRecord(brewery : const BreweryRecord&) : uint64_t
|
||||
+ ProcessRecord(user : const UserRecord&) : uint64_t
|
||||
+ Finalize() : void
|
||||
- InitializeSchema() : void
|
||||
- ResolveLocationId(location : const Location&) : sqlite3_int64
|
||||
}
|
||||
|
||||
interface IDateTimeProvider <<interface>> {
|
||||
+ GetUtcTimestamp() : std::string
|
||||
}
|
||||
|
||||
class SystemDateTimeProvider {
|
||||
+ GetUtcTimestamp() : std::string
|
||||
}
|
||||
|
||||
' Structural Relationships / Dependency Injection
|
||||
BiergartenPipelineOrchestrator *-- ILogger : owns
|
||||
BiergartenPipelineOrchestrator *-- IEnrichmentService : owns
|
||||
BiergartenPipelineOrchestrator *-- DataGenerator : owns
|
||||
BiergartenPipelineOrchestrator *-- IExportService : owns
|
||||
BiergartenPipelineOrchestrator *-- ICuratedDataService : owns
|
||||
|
||||
LogEntry *-- LogLevel
|
||||
LogEntry *-- PipelinePhase
|
||||
LogDTO *-- LogLevel
|
||||
LogDTO *-- PipelinePhase
|
||||
ILogger <|.. LogProducer : implements
|
||||
LogProducer ..> LogEntry : emits
|
||||
LogDispatcher ..> LogEntry : consumes
|
||||
|
||||
IEnrichmentService <|.. WikipediaEnrichmentService : implements
|
||||
IEnrichmentService <|.. MockEnrichmentService : implements
|
||||
WikipediaEnrichmentService *-- WebClient : owns
|
||||
|
||||
WebClient <|.. HttpWebClient : implements
|
||||
|
||||
DataGenerator <|.. MockGenerator : implements
|
||||
DataGenerator <|.. LlamaGenerator : implements
|
||||
DataGenerator <|.. OpenAIGenerator : implements
|
||||
|
||||
LlamaGenerator *-- IPromptFormatter : uses
|
||||
LlamaGenerator *-- IPromptDirectory : uses
|
||||
|
||||
OpenAIGenerator *-- IPromptDirectory : uses
|
||||
OpenAIGenerator *-- WebClient : uses
|
||||
|
||||
IPromptFormatter <|.. Gemma4JinjaPromptFormatter : implements
|
||||
IPromptDirectory <|.. PromptDirectory : implements
|
||||
|
||||
ICuratedDataService <|.. JsonLoader : implements
|
||||
ICuratedDataService <|.. MockCuratedDataService : implements
|
||||
|
||||
IExportService <|.. SqliteExportService : implements
|
||||
SqliteExportService *-- IDateTimeProvider : owns
|
||||
IDateTimeProvider <|.. SystemDateTimeProvider : implements
|
||||
|
||||
@enduml
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
16
docs/pipeline/diagrams/planned/activity.md
Normal file
16
docs/pipeline/diagrams/planned/activity.md
Normal file
@@ -0,0 +1,16 @@
|
||||
# The Biergarten Data Pipeline — Activity Diagram (Planned)
|
||||
|
||||
Each generation phase runs a producer/consumer chain concurrently: an
|
||||
**Orchestrator** producer feeds a bounded channel, an **LLM Worker** consumes it
|
||||
and generates records (closing its output channel when the input is drained and
|
||||
closed), and a **SQLite Worker** consumes that output channel and commits in
|
||||
batches. The brewery phase inserts a fourth stage, a **Geocode Worker**, between
|
||||
the LLM Worker and the SQLite Worker: reverse-geocoding is network-bound and
|
||||
rate-limited independently of LLM inference (Nominatim's 1-request-per-second
|
||||
policy), so keeping it off the LLM Worker's thread means neither stage stalls
|
||||
waiting on the other's bottleneck. The Checkin and Follow phases run in parallel
|
||||
with each other once the user and brewery pools are populated.
|
||||
|
||||
```mermaid
|
||||
--8<-- "pipeline/diagrams/planned/activity.mmd"
|
||||
```
|
||||
208
docs/pipeline/diagrams/planned/activity.mmd
Normal file
208
docs/pipeline/diagrams/planned/activity.mmd
Normal file
@@ -0,0 +1,208 @@
|
||||
flowchart TD
|
||||
Start([start]) --> ParseArgs["Main: ParseArguments(argc, argv)"]
|
||||
ParseArgs --> InvalidArgs{Invalid args?}
|
||||
InvalidArgs -- yes --> LogErr["spdlog::error"] --> Stop1([stop])
|
||||
InvalidArgs -- no --> InitOpenSSL["Main: Init OpenSSL global state and LlamaBackendState"]
|
||||
InitOpenSSL --> BuildDI["Main: Build DI injector"]
|
||||
BuildDI --> InitSqlite["Main: Initialize SqliteExportService<br/>opens SQLite connection; transactions now managed per-phase via batching"]
|
||||
InitSqlite --> LogSetup["Main: create BoundedChannel log_ch, spawn Log Worker thread<br/>drains log_ch for the pipeline lifetime; all workers emit LogEntry via PipelineLogger, never spdlog directly"]
|
||||
LogSetup --> OrchRun["BiergartenPipelineOrchestrator::Run()"]
|
||||
|
||||
OrchRun --> PL1["JsonLoader.LoadBeerStyles → EnrichmentService.PreWarmBeerStyleCache"]
|
||||
OrchRun --> PL2["JsonLoader.LoadLocations → EnrichmentService.PreWarmLocationCache"]
|
||||
OrchRun --> PL3["JsonLoader.LoadForenamesByCountry"]
|
||||
OrchRun --> PL4["JsonLoader.LoadSurnamesByCountry"]
|
||||
OrchRun --> PL5["JsonLoader.LoadPersonas"]
|
||||
PL1 & PL2 & PL3 & PL4 & PL5 --> UserPhase["Orchestrator: RunUserPhase(sampled_locations)<br/>each Load* call memoized after first parse; MockCuratedDataService (--mocked) returns a fixed in-memory dataset instead, skipping these JSON files"]
|
||||
|
||||
UserPhase --> UCreate["Create BoundedChannels (loc_ch, exp_ch)"]
|
||||
|
||||
subgraph USERGEN["Phase 0 — User Generation (concurrent)"]
|
||||
direction LR
|
||||
subgraph U_PROD["Orchestrator (producer)"]
|
||||
UP1["Loop: send Location → loc_ch"]
|
||||
UP1 --> UP2["Close loc_ch"]
|
||||
end
|
||||
subgraph U_LLM["LLM Worker"]
|
||||
UL1{"loc_ch has items?"}
|
||||
UL1 -- yes --> UL2["Receive Location"]
|
||||
UL2 --> UL3["GetLocationContextFromCache(location) — guaranteed cache hit from startup"]
|
||||
UL3 --> UL4["IPersonaSelectionStrategy.SelectPersona(personas_palette_) — guaranteed cache hit; returns Persona{style_affinities, abv_range, ibu_preference, checkin_weight}"]
|
||||
UL4 --> UL5["SampleName(forenames_by_country, surnames_by_country, iso3166_1, rng)<br/>deterministic, no LLM; free function; skips the city if either map has no entry for iso3166_1"]
|
||||
UL5 --> UL6["GenerateUser(enriched_city, persona, sampled_name) via DataGenerator<br/>LLM receives EnrichedCity context + persona description + sampled name"]
|
||||
UL6 --> UL7["PipelineLogger.Log(Info, UserGeneration, city, user_id, llm)"]
|
||||
UL7 --> UL8["Send GeneratedUser → exp_ch"]
|
||||
UL8 --> UL1
|
||||
UL1 -- no --> UL9["Close exp_ch"]
|
||||
end
|
||||
subgraph U_SQL["SQLite Worker"]
|
||||
US1["BEGIN TRANSACTION"] --> US2{"exp_ch has items?"}
|
||||
US2 -- yes --> US3["Receive GeneratedUser"]
|
||||
US3 --> US4["ProcessUser(user)"]
|
||||
US4 --> US5["PipelineLogger.Log(Info, UserGeneration, city, user_id, sqlite)"]
|
||||
US5 --> US6["Append → user_pool_"]
|
||||
US6 --> US7{"Batch size reached?"}
|
||||
US7 -- yes --> US8["COMMIT and BEGIN"] --> US2
|
||||
US7 -- no --> US2
|
||||
US2 -- no --> US9["COMMIT (Final)"]
|
||||
end
|
||||
end
|
||||
|
||||
UCreate --> UP1
|
||||
UP2 -.->|loc_ch| UL1
|
||||
UL9 -.->|exp_ch| US2
|
||||
UL9 & US9 --> UserJoin["Orchestrator: Join LLM Worker, SQLite Worker"]
|
||||
|
||||
UserJoin --> BreweryPhase["Orchestrator: RunBreweryPhase(sampled_locations)"]
|
||||
BreweryPhase --> BCreate["Create BoundedChannels (loc_ch, geo_ch, exp_ch)"]
|
||||
|
||||
subgraph BREWGEN["Phase 1a — Brewery Generation (concurrent)"]
|
||||
direction LR
|
||||
subgraph B_PROD["Orchestrator (producer)"]
|
||||
BP1["Loop: sample User from user_pool_, pair with Location"]
|
||||
BP1 --> BP2["Send BreweryTask(Location, User) → loc_ch"]
|
||||
BP2 --> BP3["Close loc_ch"]
|
||||
end
|
||||
subgraph B_LLM["LLM Worker"]
|
||||
BL1{"loc_ch has items?"}
|
||||
BL1 -- yes --> BL2["Receive BreweryTask(Location, User)"]
|
||||
BL2 --> BL3["GetLocationContextFromCache(task.location) — guaranteed cache hit"]
|
||||
BL3 --> BL4["GenerateBrewery(enriched_city, context, task.user) via DataGenerator<br/>KV cache stays warm; brewery linked to sampled owner_user_id"]
|
||||
BL4 --> BL5["PipelineLogger.Log(Info, BreweryGeneration, city, brewery_id, llm)"]
|
||||
BL5 --> BL6["Send GeneratedBrewery → geo_ch"]
|
||||
BL6 --> BL1
|
||||
BL1 -- no --> BL7["Close geo_ch"]
|
||||
end
|
||||
subgraph B_GEO["Geocode Worker"]
|
||||
BG1{"geo_ch has items?"}
|
||||
BG1 -- yes --> BG2["Receive GeneratedBrewery"]
|
||||
BG2 --> BG3["AddressService.ReverseGeocode(brewery.longitude, brewery.latitude)<br/>rate-limited independently of LLM throughput — NominatimAddressService sleeps 1s per request; MockAddressService (--mocked) returns a fixed placeholder, no network call"]
|
||||
BG3 --> BG4{"Address resolved?"}
|
||||
BG4 -- yes --> BG5["Attach resolved AddressLookupResult to brewery"]
|
||||
BG4 -- no --> BG6["Log warning: could not reverse geocode; attach placeholder address_line1, empty postal_code — brewery is never dropped"]
|
||||
BG5 --> BG7["PipelineLogger.Log(Info, BreweryGeneration, city, brewery_id, geocode)"]
|
||||
BG6 --> BG7
|
||||
BG7 --> BG8["Send GeneratedBrewery → exp_ch"]
|
||||
BG8 --> BG1
|
||||
BG1 -- no --> BG9["Close exp_ch"]
|
||||
end
|
||||
subgraph B_SQL["SQLite Worker"]
|
||||
BS1["BEGIN TRANSACTION"] --> BS2{"exp_ch has items?"}
|
||||
BS2 -- yes --> BS3["Receive GeneratedBrewery"]
|
||||
BS3 --> BS4["ProcessBrewery(brewery)"]
|
||||
BS4 --> BS5["PipelineLogger.Log(Info, BreweryGeneration, city, brewery_id, sqlite)"]
|
||||
BS5 --> BS6["Append → brewery_pool_"]
|
||||
BS6 --> BS7{"Batch size reached?"}
|
||||
BS7 -- yes --> BS8["COMMIT and BEGIN"] --> BS2
|
||||
BS7 -- no --> BS2
|
||||
BS2 -- no --> BS9["COMMIT (Final)"]
|
||||
end
|
||||
end
|
||||
|
||||
BCreate --> BP1
|
||||
BP3 -.->|loc_ch| BL1
|
||||
BL7 -.->|geo_ch| BG1
|
||||
BG9 -.->|exp_ch| BS2
|
||||
BL7 & BG9 & BS9 --> BrewJoin["Orchestrator: Join LLM Worker, Geocode Worker, SQLite Worker<br/>brewery_pool_ is now fully populated"]
|
||||
|
||||
BrewJoin --> BeerPhase["Orchestrator: RunBeerPhase()"]
|
||||
BeerPhase --> BeCreate["Create BoundedChannels (brew_ch, exp_ch)"]
|
||||
|
||||
subgraph BEERGEN["Phase 1b — Beer Generation (concurrent)"]
|
||||
direction LR
|
||||
subgraph BE_PROD["Orchestrator (producer)"]
|
||||
BeP1["Loop: send Breweries → brew_ch"]
|
||||
BeP1 --> BeP2["Close brew_ch"]
|
||||
end
|
||||
subgraph BE_LLM["LLM Worker"]
|
||||
BeL1{"brew_ch has items?"}
|
||||
BeL1 -- yes --> BeL2["Receive GeneratedBrewery"]
|
||||
BeL2 --> BeL3["IBeerSelectionStrategy.SelectStyles(brewery, beer_style_palette_)"]
|
||||
BeL3 --> BeL4{"For each selected BeerStyle"}
|
||||
BeL4 -- remaining --> BeL5["GetStyleContextFromCache(style) — guaranteed cache hit; KV cache stays warm across all beer generations in this phase"]
|
||||
BeL5 --> BeL6["GenerateBeer(brewery, style_context) via DataGenerator"]
|
||||
BeL6 --> BeL7["Attach GeneratedBeer to bundle"] --> BeL4
|
||||
BeL4 -- done --> BeL8["PipelineLogger.Log(Info, BeerGeneration, city, brewery_id, llm)"]
|
||||
BeL8 --> BeL9["Send BeersBundle → exp_ch"]
|
||||
BeL9 --> BeL1
|
||||
BeL1 -- no --> BeL10["Close exp_ch"]
|
||||
end
|
||||
subgraph BE_SQL["SQLite Worker"]
|
||||
BeS1["BEGIN TRANSACTION"] --> BeS2{"exp_ch has items?"}
|
||||
BeS2 -- yes --> BeS3["Receive BeersBundle"]
|
||||
BeS3 --> BeS4{"For each beer in bundle"}
|
||||
BeS4 -- remaining --> BeS5["Set beer.brewery_id from bundle"]
|
||||
BeS5 --> BeS6["ProcessBeer(beer)"]
|
||||
BeS6 --> BeS7["Append → beer_pool_"] --> BeS4
|
||||
BeS4 -- done --> BeS8["PipelineLogger.Log(Info, BeerGeneration, city, brewery_id, sqlite)"]
|
||||
BeS8 --> BeS9{"Batch size reached?"}
|
||||
BeS9 -- yes --> BeS10["COMMIT and BEGIN"] --> BeS2
|
||||
BeS9 -- no --> BeS2
|
||||
BeS2 -- no --> BeS11["COMMIT (Final)"]
|
||||
end
|
||||
end
|
||||
|
||||
BeCreate --> BeP1
|
||||
BeP2 -.->|brew_ch| BeL1
|
||||
BeL10 -.->|exp_ch| BeS2
|
||||
BeL10 & BeS11 --> BeerJoin["Orchestrator: Join LLM Worker, SQLite Worker<br/>brewery_pool_ and beer_pool_ are now completely populated — Checkin and Follow phases may run in parallel"]
|
||||
|
||||
BeerJoin --> CheckinPhase["Orchestrator: RunCheckinPhase()"]
|
||||
BeerJoin --> FollowPhase["Orchestrator: RunFollowPhase()"]
|
||||
|
||||
subgraph CHECKIN["Checkin Generation"]
|
||||
CK1["ICheckinDistributionStrategy.AssignActivityWeights(user_pool_)<br/>weights seeded from persona.checkin_weight; J-curve profile emerges from persona distribution"]
|
||||
CK1 --> CK2["BEGIN TRANSACTION"]
|
||||
CK2 --> CK3{"For each GeneratedUser in user_pool_"}
|
||||
CK3 -- remaining --> CK4["CheckinsForUser(user, brewery_pool_.size())"]
|
||||
CK4 --> CK5{"For each checkin index"}
|
||||
CK5 -- remaining --> CK6["TimestampFor(user, index)"]
|
||||
CK6 --> CK7["Select brewery from brewery_pool_"]
|
||||
CK7 --> CK8["GenerateCheckin(user, brewery, timestamp) via DataGenerator"]
|
||||
CK8 --> CK9["ProcessCheckin(checkin)"]
|
||||
CK9 --> CK10["PipelineLogger.Log(Info, CheckinGeneration, nullopt, checkin_id, sqlite)"]
|
||||
CK10 --> CK11["Append → checkin_pool_"]
|
||||
CK11 --> CK12{"Batch size reached?"}
|
||||
CK12 -- yes --> CK13["COMMIT and BEGIN"] --> CK5
|
||||
CK12 -- no --> CK5
|
||||
CK5 -- done --> CK3
|
||||
CK3 -- done --> CK14["COMMIT (Final)"]
|
||||
end
|
||||
|
||||
subgraph FOLLOW["Follow Generation"]
|
||||
FL1["IFollowGenerationStrategy.AssignFollowWeights(user_pool_)<br/>RandomFollowStrategy: uniform weights; ActivityWeightedFollowStrategy: derived from user.activity_weight so high-activity users attract more followers"]
|
||||
FL1 --> FL2["BEGIN TRANSACTION"]
|
||||
FL2 --> FL3["IFollowGenerationStrategy.GenerateFollows(user_pool_)<br/>self-follow constraint (follower_id != followed_id) enforced here and at the DB schema level"]
|
||||
FL3 --> FL4{"For each GeneratedFollow"}
|
||||
FL4 -- remaining --> FL5["ProcessFollow(follow)"]
|
||||
FL5 --> FL6["PipelineLogger.Log(Info, FollowGeneration, nullopt, follower_id, sqlite)"]
|
||||
FL6 --> FL7["Append → follow_pool_"]
|
||||
FL7 --> FL8{"Batch size reached?"}
|
||||
FL8 -- yes --> FL9["COMMIT and BEGIN"] --> FL4
|
||||
FL8 -- no --> FL4
|
||||
FL4 -- done --> FL10["COMMIT (Final)"]
|
||||
end
|
||||
|
||||
CheckinPhase --> CK1
|
||||
FollowPhase --> FL1
|
||||
CK14 & FL10 --> PhaseJoin["Orchestrator: Join CheckinPhase, FollowPhase<br/>checkin_pool_ and follow_pool_ are now fully populated"]
|
||||
|
||||
PhaseJoin --> RatingPhase["Orchestrator: RunRatingPhase()<br/>beer selection biased by user.persona.style_affinities and abv_range; rating skew modulated per persona"]
|
||||
RatingPhase --> RT1["BEGIN TRANSACTION"]
|
||||
RT1 --> RT2{"For each GeneratedCheckin in checkin_pool_"}
|
||||
RT2 -- remaining --> RT3["Match brewery_id, select beer from beer_pool_ (same brewery_id, biased by persona affinities)"]
|
||||
RT3 --> RT4{"Beer exists for brewery?"}
|
||||
RT4 -- yes --> RT5["GenerateRating(user, beer, checkin_id) via DataGenerator"]
|
||||
RT5 --> RT6["ProcessRating(rating)"]
|
||||
RT6 --> RT7["PipelineLogger.Log(Info, RatingGeneration, nullopt, rating_id, sqlite)"]
|
||||
RT7 --> RT8{"Batch size reached?"}
|
||||
RT8 -- yes --> RT9["COMMIT and BEGIN"] --> RT2
|
||||
RT8 -- no --> RT2
|
||||
RT4 -- no --> RT10["Log warning; skip — brewery has no beers"] --> RT2
|
||||
RT2 -- done --> RT11["COMMIT (Final)"]
|
||||
|
||||
RT11 --> Teardown["Orchestrator: Finalize SqliteExportService — safely closes the DB connection"]
|
||||
Teardown --> CloseLogCh["Orchestrator: close log_ch"]
|
||||
CloseLogCh --> MainDone["Main: spdlog::info 'Pipeline complete in X ms'"]
|
||||
MainDone --> JoinLog["Main: Join Log Worker — drain guarantees no LogEntry is dropped at shutdown"]
|
||||
JoinLog --> Stop2([stop])
|
||||
@@ -1,372 +0,0 @@
|
||||
@startuml biergarten_activity
|
||||
!include ../biergarten-weizen-theme.puml
|
||||
skinparam defaultFontSize 13
|
||||
skinparam titleFontSize 20
|
||||
|
||||
title The Biergarten Data Pipeline — Activity Diagram
|
||||
|
||||
|Main|
|
||||
start
|
||||
:ParseArguments(argc, argv);
|
||||
if (Invalid args?) then (yes)
|
||||
:spdlog::error;
|
||||
stop
|
||||
else (no)
|
||||
endif
|
||||
:Init OpenSSL global state & LlamaBackendState;
|
||||
:Build DI injector;
|
||||
|
||||
:Initialize SqliteExportService;
|
||||
note right
|
||||
Opens SQLite connection.
|
||||
(Transactions are now managed
|
||||
per-phase via batching).
|
||||
end note
|
||||
|
||||
:Create BoundedChannel<LogEntry> log_ch;
|
||||
:Spawn Log Worker thread;
|
||||
note right
|
||||
Log worker drains log_ch for the
|
||||
entire pipeline lifetime.
|
||||
All workers emit LogEntry structs
|
||||
via PipelineLogger -- never spdlog directly.
|
||||
end note
|
||||
|
||||
:BiergartenPipelineOrchestrator::Run();
|
||||
|BiergartenPipelineOrchestrator::Run()|
|
||||
|
||||
fork
|
||||
:JsonLoader::LoadBeerStyles("beer-styles.json");
|
||||
:EnrichmentService::PreWarmBeerStyleCache(beer_styles);
|
||||
fork again
|
||||
:JsonLoader::LoadLocations("locations.json");
|
||||
:EnrichmentService::PreWarmLocationCache(sampled_locations);
|
||||
end fork
|
||||
fork
|
||||
:JsonLoader::LoadForenamesByCountry(\n "forenames-by-country.json");
|
||||
fork again
|
||||
:JsonLoader::LoadSurnamesByCountry(\n "surnames-by-country.json");
|
||||
fork again
|
||||
:JsonLoader::LoadPersonas("personas.json");
|
||||
end fork
|
||||
note right
|
||||
Each call is memoized after its first parse
|
||||
(implemented today). MockCuratedDataService
|
||||
returns a fixed in-memory dataset instead
|
||||
under --mocked, skipping these JSON files.
|
||||
end note
|
||||
|
||||
' ═══════════════════════════════════════════
|
||||
' PHASE 0 — USER GENERATION
|
||||
' ═══════════════════════════════════════════
|
||||
|Orchestrator|
|
||||
:RunUserPhase(sampled_locations);
|
||||
:Create BoundedChannels\n(loc_ch, exp_ch);
|
||||
|
||||
fork
|
||||
|Orchestrator|
|
||||
:Loop: Send Locations -> loc_ch;
|
||||
:Close loc_ch;
|
||||
note right
|
||||
Producer closes loc_ch.
|
||||
LLM Worker while loop
|
||||
terminates on empty + closed.
|
||||
end note
|
||||
fork again
|
||||
|LLM Worker|
|
||||
while (loc_ch has items?) is (yes)
|
||||
:Receive Location;
|
||||
|
||||
:GetLocationContextFromCache(location);
|
||||
note right
|
||||
Guaranteed cache hit from startup.
|
||||
end note
|
||||
|
||||
:IPersonaSelectionStrategy::SelectPersona(\n personas_palette_);
|
||||
note right
|
||||
Guaranteed cache hit from startup.
|
||||
Returns a Persona struct carrying
|
||||
style_affinities, abv_range,
|
||||
ibu_preference, checkin_weight.
|
||||
end note
|
||||
|
||||
:SampleName(forenames_by_country,\n surnames_by_country, location.iso3166_1, rng);
|
||||
note right
|
||||
Deterministic lookup -- no LLM involved.
|
||||
Free function (implemented today in
|
||||
generate_users.cc), not a class method.
|
||||
Name selected from pre-keyed maps
|
||||
and passed into the generation prompt.
|
||||
Skips the city if either map has no
|
||||
entry for iso3166_1.
|
||||
end note
|
||||
|
||||
:GenerateUser(enriched_city, persona, sampled_name)\nvia DataGenerator;
|
||||
note right
|
||||
LLM receives: EnrichedCity context + persona
|
||||
description + sampled name. Generates
|
||||
bio and preference signals grounded
|
||||
in locale and persona.
|
||||
end note
|
||||
|
||||
:PipelineLogger::Log(Info, UserGeneration,\n city, user_id, "llm");
|
||||
:Send GeneratedUser -> exp_ch;
|
||||
endwhile (no)
|
||||
:Close exp_ch;
|
||||
note right
|
||||
Producer closes exp_ch.
|
||||
SQLite Worker while loop
|
||||
terminates on empty + closed.
|
||||
end note
|
||||
fork again
|
||||
|SQLite Worker|
|
||||
:BEGIN TRANSACTION;
|
||||
while (exp_ch has items?) is (yes)
|
||||
:Receive GeneratedUser;
|
||||
:ProcessUser(user);
|
||||
:PipelineLogger::Log(Info, UserGeneration,\n city, user_id, "sqlite");
|
||||
:Append -> user_pool_;
|
||||
if (Batch size reached?) then (yes)
|
||||
:COMMIT & BEGIN;
|
||||
else (no)
|
||||
endif
|
||||
endwhile (no)
|
||||
:COMMIT (Final);
|
||||
end fork
|
||||
|
||||
|Orchestrator|
|
||||
:Join LLM Worker, SQLite Worker;
|
||||
|
||||
' ═══════════════════════════════════════════
|
||||
' PHASE 1a — BREWERY GENERATION
|
||||
' ═══════════════════════════════════════════
|
||||
:RunBreweryPhase(sampled_locations);
|
||||
:Create BoundedChannels\n(loc_ch, exp_ch);
|
||||
|
||||
fork
|
||||
|Orchestrator|
|
||||
:Loop: Sample User from user_pool_
|
||||
and pair with Location;
|
||||
:Send BreweryTask(Location, User) -> loc_ch;
|
||||
:Close loc_ch;
|
||||
fork again
|
||||
|LLM Worker|
|
||||
while (loc_ch has items?) is (yes)
|
||||
:Receive BreweryTask(Location, User);
|
||||
|
||||
:GetLocationContextFromCache(task.location);
|
||||
note right
|
||||
Guaranteed cache hit from startup.
|
||||
end note
|
||||
|
||||
:GenerateBrewery(enriched_city, context, task.user)\nvia DataGenerator;
|
||||
note right
|
||||
KV cache stays warm.
|
||||
Brewery is linked to the sampled owner_user_id.
|
||||
end note
|
||||
:PipelineLogger::Log(Info,\n BreweryGeneration,\n city, brewery_id, "llm");
|
||||
:Send GeneratedBrewery -> exp_ch;
|
||||
endwhile (no)
|
||||
:Close exp_ch;
|
||||
fork again
|
||||
|SQLite Worker|
|
||||
:BEGIN TRANSACTION;
|
||||
while (exp_ch has items?) is (yes)
|
||||
:Receive GeneratedBrewery;
|
||||
:ProcessBrewery(brewery);
|
||||
:PipelineLogger::Log(Info,\n BreweryGeneration,\n city, brewery_id, "sqlite");
|
||||
:Append -> brewery_pool_;
|
||||
if (Batch size reached?) then (yes)
|
||||
:COMMIT & BEGIN;
|
||||
else (no)
|
||||
endif
|
||||
endwhile (no)
|
||||
:COMMIT (Final);
|
||||
end fork
|
||||
|
||||
|Orchestrator|
|
||||
:Join LLM Worker, SQLite Worker;
|
||||
note right
|
||||
brewery_pool_ is now fully populated.
|
||||
Phase 1b may begin.
|
||||
end note
|
||||
|
||||
' ═══════════════════════════════════════════
|
||||
' PHASE 1b — BEER GENERATION
|
||||
' ═══════════════════════════════════════════
|
||||
:RunBeerPhase();
|
||||
:Create BoundedChannels\n(brew_ch, exp_ch);
|
||||
|
||||
fork
|
||||
|Orchestrator|
|
||||
:Loop: Send Breweries -> brew_ch;
|
||||
:Close brew_ch;
|
||||
fork again
|
||||
|LLM Worker|
|
||||
while (brew_ch has items?) is (yes)
|
||||
:Receive GeneratedBrewery;
|
||||
:IBeerSelectionStrategy::SelectStyles(\n brewery, beer_style_palette_);
|
||||
|
||||
while (For each selected BeerStyle?) is (remaining)
|
||||
:GetStyleContextFromCache(style);
|
||||
note right
|
||||
Guaranteed cache hit from startup.
|
||||
KV cache stays warm across all
|
||||
beer generations -- system prompt
|
||||
does not change within this phase.
|
||||
end note
|
||||
:GenerateBeer(brewery, style_context)\nvia DataGenerator;
|
||||
:Attach GeneratedBeer to bundle;
|
||||
endwhile (done)
|
||||
|
||||
:PipelineLogger::Log(Info,\n BeerGeneration,\n city, brewery_id, "llm");
|
||||
:Send BeersBundle -> exp_ch;
|
||||
endwhile (no)
|
||||
:Close exp_ch;
|
||||
fork again
|
||||
|SQLite Worker|
|
||||
:BEGIN TRANSACTION;
|
||||
while (exp_ch has items?) is (yes)
|
||||
:Receive BeersBundle;
|
||||
while (For each beer in bundle?) is (remaining)
|
||||
:Set beer.brewery_id from bundle;
|
||||
:ProcessBeer(beer);
|
||||
:Append -> beer_pool_;
|
||||
endwhile (done)
|
||||
:PipelineLogger::Log(Info,\n BeerGeneration,\n city, brewery_id, "sqlite");
|
||||
if (Batch size reached?) then (yes)
|
||||
:COMMIT & BEGIN;
|
||||
else (no)
|
||||
endif
|
||||
endwhile (no)
|
||||
:COMMIT (Final);
|
||||
end fork
|
||||
|
||||
|Orchestrator|
|
||||
:Join LLM Worker, SQLite Worker;
|
||||
note right
|
||||
Both brewery_pool_ and beer_pool_
|
||||
are now completely populated.
|
||||
Checkin and Follow phases may
|
||||
now run in parallel.
|
||||
end note
|
||||
|
||||
' ═══════════════════════════════════════════
|
||||
' PHASE 2 — CHECKIN + FOLLOW GENERATION
|
||||
' (parallel — both depend only on user_pool_
|
||||
' and brewery_pool_ being fully populated)
|
||||
' ═══════════════════════════════════════════
|
||||
fork
|
||||
|Orchestrator|
|
||||
:RunCheckinPhase();
|
||||
:ICheckinDistributionStrategy::\nAssignActivityWeights(user_pool_);
|
||||
note right
|
||||
Weights seeded from each user's
|
||||
persona.checkin_weight. J-curve profile
|
||||
emerges from persona distribution.
|
||||
end note
|
||||
|
||||
:BEGIN TRANSACTION;
|
||||
while (For each GeneratedUser in user_pool_?) is (remaining)
|
||||
:CheckinsForUser(user, brewery_pool_.size());
|
||||
while (For each checkin index?) is (remaining)
|
||||
:TimestampFor(user, index);
|
||||
:Select brewery from brewery_pool_;
|
||||
:GenerateCheckin(user, brewery, timestamp)\nvia DataGenerator;
|
||||
:ProcessCheckin(checkin);
|
||||
:PipelineLogger::Log(Info, CheckinGeneration,\n nullopt, checkin_id, "sqlite");
|
||||
:Append -> checkin_pool_;
|
||||
if (Batch size reached?) then (yes)
|
||||
:COMMIT & BEGIN;
|
||||
else (no)
|
||||
endif
|
||||
endwhile (done)
|
||||
endwhile (done)
|
||||
:COMMIT (Final);
|
||||
|
||||
fork again
|
||||
|Orchestrator|
|
||||
:RunFollowPhase();
|
||||
:IFollowGenerationStrategy::\nAssignFollowWeights(user_pool_);
|
||||
note right
|
||||
For RandomFollowStrategy, weights
|
||||
are uniform. For ActivityWeightedFollowStrategy,
|
||||
weights derived from user.activity_weight
|
||||
so high-activity users attract more followers.
|
||||
end note
|
||||
|
||||
:BEGIN TRANSACTION;
|
||||
:IFollowGenerationStrategy::\nGenerateFollows(user_pool_);
|
||||
note right
|
||||
Self-follow constraint (follower_id != followed_id)
|
||||
enforced here and at the DB schema level.
|
||||
end note
|
||||
while (For each GeneratedFollow?) is (remaining)
|
||||
:ProcessFollow(follow);
|
||||
:PipelineLogger::Log(Info, FollowGeneration,\n nullopt, follower_id, "sqlite");
|
||||
:Append -> follow_pool_;
|
||||
if (Batch size reached?) then (yes)
|
||||
:COMMIT & BEGIN;
|
||||
else (no)
|
||||
endif
|
||||
endwhile (done)
|
||||
:COMMIT (Final);
|
||||
|
||||
end fork
|
||||
|
||||
|Orchestrator|
|
||||
:Join CheckinPhase, FollowPhase;
|
||||
note right
|
||||
checkin_pool_ and follow_pool_
|
||||
are now fully populated.
|
||||
Rating phase may begin.
|
||||
end note
|
||||
|
||||
' ═══════════════════════════════════════════
|
||||
' PHASE 3 — RATING GENERATION
|
||||
' ═══════════════════════════════════════════
|
||||
:RunRatingPhase();
|
||||
note right
|
||||
Beer selection biased by
|
||||
user.persona.style_affinities and abv_range.
|
||||
Rating skew modulated per persona.
|
||||
end note
|
||||
|
||||
:BEGIN TRANSACTION;
|
||||
while (For each GeneratedCheckin in checkin_pool_?) is (remaining)
|
||||
:Match brewery_id, select beer from beer_pool_\n(same brewery_id, biased by persona affinities);
|
||||
if (Beer exists for brewery?) then (yes)
|
||||
:GenerateRating(user, beer, checkin_id)\nvia DataGenerator;
|
||||
:ProcessRating(rating);
|
||||
:PipelineLogger::Log(Info, RatingGeneration,\n nullopt, rating_id, "sqlite");
|
||||
if (Batch size reached?) then (yes)
|
||||
:COMMIT & BEGIN;
|
||||
else (no)
|
||||
endif
|
||||
else (no)
|
||||
:PipelineLogger::Log(Warn, RatingGeneration,\n nullopt, brewery_id, "sqlite");
|
||||
:Skip -- brewery has no beers;
|
||||
endif
|
||||
endwhile (done)
|
||||
:COMMIT (Final);
|
||||
|
||||
' ═══════════════════════════════════════════
|
||||
' TEARDOWN
|
||||
' ═══════════════════════════════════════════
|
||||
|Orchestrator|
|
||||
:Finalize SqliteExportService;
|
||||
note right
|
||||
Safely closes the DB connection.
|
||||
end note
|
||||
:Close log_ch;
|
||||
|
||||
|Main|
|
||||
:spdlog::info "Pipeline complete in X ms";
|
||||
:Join Log Worker;
|
||||
note right
|
||||
Drain guarantees no LogEntry is
|
||||
dropped at shutdown.
|
||||
end note
|
||||
stop
|
||||
|
||||
@enduml
|
||||
17
docs/pipeline/diagrams/planned/class.md
Normal file
17
docs/pipeline/diagrams/planned/class.md
Normal file
@@ -0,0 +1,17 @@
|
||||
# Biergarten Data Pipeline — Class Diagram (Planned)
|
||||
|
||||
```mermaid
|
||||
--8<-- "pipeline/diagrams/planned/class.mmd"
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- `Logger`, `PipelineLogger`, and `LogWorker` are referenced only in
|
||||
relationships in the source PlantUML (not given their own class blocks there
|
||||
either) — they appear here as implicit nodes for the same reason.
|
||||
- Package grouping from the PlantUML source (`Domain: Models`,
|
||||
`Domain: Application Configuration`, `Domain: Policy`,
|
||||
`Infrastructure: Logging`, `Infrastructure: Pipeline Channel`,
|
||||
`Infrastructure: Data Preloading`, `Infrastructure: Enrichment`,
|
||||
`Infrastructure: Prompting`, `Infrastructure: Data Generation`,
|
||||
`Infrastructure: Data Export`) is preserved as Mermaid `namespace` blocks.
|
||||
643
docs/pipeline/diagrams/planned/class.mmd
Normal file
643
docs/pipeline/diagrams/planned/class.mmd
Normal file
@@ -0,0 +1,643 @@
|
||||
classDiagram
|
||||
namespace DomainModels {
|
||||
class Location {
|
||||
+string city
|
||||
+string state_province
|
||||
+string iso3166_2
|
||||
+string country
|
||||
+string iso3166_1
|
||||
+vector~string~ local_languages
|
||||
+double latitude
|
||||
+double longitude
|
||||
}
|
||||
|
||||
class LocationContext {
|
||||
+string text
|
||||
+Completeness completeness
|
||||
+size_t char_count
|
||||
}
|
||||
|
||||
class Completeness {
|
||||
<<enumeration>>
|
||||
Full
|
||||
Partial
|
||||
Absent
|
||||
}
|
||||
|
||||
class EnrichedCity {
|
||||
+Location location
|
||||
+LocationContext context
|
||||
}
|
||||
|
||||
class BeerStyle {
|
||||
+string name
|
||||
+string description
|
||||
+float min_abv
|
||||
+float max_abv
|
||||
+int min_ibu
|
||||
+int max_ibu
|
||||
}
|
||||
|
||||
class BreweryResult {
|
||||
+string name_en
|
||||
+string description_en
|
||||
+string name_local
|
||||
+string description_local
|
||||
}
|
||||
|
||||
class BeerResult {
|
||||
+string name_en
|
||||
+string description_en
|
||||
+string name_local
|
||||
+string description_local
|
||||
+string style
|
||||
+float abv
|
||||
+int ibu
|
||||
}
|
||||
|
||||
class UserResult {
|
||||
+string first_name
|
||||
+string last_name
|
||||
+string gender
|
||||
+string username
|
||||
+string bio
|
||||
+float activity_weight
|
||||
}
|
||||
|
||||
class Name {
|
||||
+string first_name
|
||||
+string last_name
|
||||
+string gender
|
||||
}
|
||||
|
||||
class ForenameEntry {
|
||||
+string name
|
||||
+string gender
|
||||
}
|
||||
|
||||
class CheckinResult {
|
||||
+string checked_in_at
|
||||
+string note
|
||||
}
|
||||
|
||||
class RatingResult {
|
||||
+float score
|
||||
+string note
|
||||
}
|
||||
|
||||
class GenerationMetadata {
|
||||
+uint64_t generation_id
|
||||
+string generated_time
|
||||
+bool context_provided
|
||||
+string generated_with
|
||||
}
|
||||
|
||||
class GeneratedBrewery {
|
||||
+uint64_t brewery_id
|
||||
+Location location
|
||||
+BreweryResult brewery
|
||||
+Completeness context_completeness
|
||||
+GenerationMetadata metadata
|
||||
}
|
||||
|
||||
class GeneratedBeer {
|
||||
+uint64_t beer_id
|
||||
+uint64_t brewery_id
|
||||
+Location location
|
||||
+BeerStyle style
|
||||
+BeerResult beer
|
||||
+GenerationMetadata metadata
|
||||
}
|
||||
|
||||
class GeneratedUser {
|
||||
+uint64_t user_id
|
||||
+Location location
|
||||
+UserResult user
|
||||
+string email
|
||||
+string date_of_birth
|
||||
+string password
|
||||
+GenerationMetadata metadata
|
||||
}
|
||||
|
||||
class GeneratedCheckin {
|
||||
+uint64_t checkin_id
|
||||
+uint64_t user_id
|
||||
+uint64_t brewery_id
|
||||
+CheckinResult checkin
|
||||
+GenerationMetadata metadata
|
||||
}
|
||||
|
||||
class GeneratedRating {
|
||||
+uint64_t user_id
|
||||
+uint64_t beer_id
|
||||
+uint64_t checkin_id
|
||||
+RatingResult rating
|
||||
+GenerationMetadata metadata
|
||||
}
|
||||
|
||||
class GeneratedFollow {
|
||||
+uint64_t follower_id
|
||||
+uint64_t followed_id
|
||||
+GenerationMetadata metadata
|
||||
}
|
||||
|
||||
class UserPersona {
|
||||
+string name
|
||||
+string description
|
||||
+vector~string~ style_affinities
|
||||
}
|
||||
}
|
||||
note for ForenameEntry "forename_list/surname_list are unordered_set aliases, keyed by ISO 3166-1 country code via ICuratedDataService. SampleName() is a free function, not a method, sourced from popular-names-by-country-dataset (CC0)."
|
||||
note for GeneratedUser "email, date_of_birth, and password are generated programmatically by the orchestrator (not LLM-authored) so a downstream auth-account seeding consumer can register real accounts from this export."
|
||||
ForenameEntry ..> Name : SampleName() produces
|
||||
LocationContext *-- Completeness
|
||||
|
||||
namespace DomainApplicationConfiguration {
|
||||
class SamplingOptions {
|
||||
+float temperature
|
||||
+float top_p
|
||||
+uint32_t top_k
|
||||
+uint32_t n_ctx
|
||||
+int seed
|
||||
}
|
||||
|
||||
class GeneratorMode {
|
||||
<<enumeration>>
|
||||
kMock
|
||||
kLlama
|
||||
kOpenAI
|
||||
}
|
||||
|
||||
class GeneratorOptions {
|
||||
+path model_path
|
||||
+GeneratorMode mode
|
||||
+optional~SamplingOptions~ sampling
|
||||
+string openai_api_key
|
||||
+string openai_model
|
||||
}
|
||||
|
||||
class PipelineOptions {
|
||||
+path output_path
|
||||
+path log_path
|
||||
}
|
||||
|
||||
class ApplicationOptions {
|
||||
+GeneratorOptions generator
|
||||
+PipelineOptions pipeline
|
||||
}
|
||||
}
|
||||
note for GeneratorOptions "Already implemented (models.h), not planned: the three-way GeneratorMode split (kMock/kLlama/kOpenAI) and the openai_api_key/openai_model fields ship today, ahead of the rest of this diagram."
|
||||
ApplicationOptions *-- GeneratorOptions
|
||||
ApplicationOptions *-- PipelineOptions
|
||||
GeneratorOptions *-- GeneratorMode
|
||||
GeneratorOptions o-- SamplingOptions
|
||||
|
||||
namespace DomainPolicy {
|
||||
class ContextStrategy {
|
||||
<<interface>>
|
||||
+QueriesFor(loc) vector~string~
|
||||
+MaxContextChars() size_t
|
||||
}
|
||||
|
||||
class BreweryContextStrategy {
|
||||
+QueriesFor(loc) vector~string~
|
||||
+MaxContextChars() size_t
|
||||
}
|
||||
|
||||
class BeerContextStrategy {
|
||||
+QueriesFor(loc) vector~string~
|
||||
+MaxContextChars() size_t
|
||||
}
|
||||
|
||||
class SamplingStrategy {
|
||||
<<interface>>
|
||||
+Sample(locations) vector~Location~
|
||||
}
|
||||
|
||||
class UniformSamplingStrategy {
|
||||
-size_t sample_size_
|
||||
+Sample(locations) vector~Location~
|
||||
}
|
||||
|
||||
class BeerSelectionStrategy {
|
||||
<<interface>>
|
||||
+SelectStyles(brewery, palette) vector~BeerStyle~
|
||||
}
|
||||
|
||||
class RandomBeerSelectionStrategy {
|
||||
-mt19937 rng_
|
||||
-size_t min_beers_
|
||||
-size_t max_beers_
|
||||
+SelectStyles(brewery, palette) vector~BeerStyle~
|
||||
}
|
||||
|
||||
class CheckinDistributionStrategy {
|
||||
<<interface>>
|
||||
+AssignActivityWeights(users) void
|
||||
+CheckinsForUser(user, brewery_count) size_t
|
||||
+TimestampFor(user, index) string
|
||||
}
|
||||
|
||||
class JCurveCheckinStrategy {
|
||||
-mt19937 rng_
|
||||
+AssignActivityWeights(users) void
|
||||
+CheckinsForUser(user, brewery_count) size_t
|
||||
+TimestampFor(user, index) string
|
||||
}
|
||||
|
||||
class RandomCheckinStrategy {
|
||||
-mt19937 rng_
|
||||
-size_t min_checkins_
|
||||
-size_t max_checkins_
|
||||
+AssignActivityWeights(users) void
|
||||
+CheckinsForUser(user, brewery_count) size_t
|
||||
+TimestampFor(user, index) string
|
||||
}
|
||||
|
||||
class FollowGenerationStrategy {
|
||||
<<interface>>
|
||||
+GenerateFollows(users) vector~GeneratedFollow~
|
||||
}
|
||||
|
||||
class RandomFollowStrategy {
|
||||
-mt19937 rng_
|
||||
-size_t min_follows_
|
||||
-size_t max_follows_
|
||||
+GenerateFollows(users) vector~GeneratedFollow~
|
||||
}
|
||||
|
||||
class ActivityWeightedFollowStrategy {
|
||||
-mt19937 rng_
|
||||
-size_t min_follows_
|
||||
-size_t max_follows_
|
||||
+GenerateFollows(users) vector~GeneratedFollow~
|
||||
}
|
||||
}
|
||||
|
||||
namespace InfrastructureLogging {
|
||||
class LogLevel {
|
||||
<<enumeration>>
|
||||
Debug
|
||||
Info
|
||||
Warn
|
||||
Error
|
||||
}
|
||||
|
||||
class PipelinePhase {
|
||||
<<enumeration>>
|
||||
Startup
|
||||
UserGeneration
|
||||
BreweryAndBeerGeneration
|
||||
CheckinGeneration
|
||||
RatingGeneration
|
||||
FollowGeneration
|
||||
Teardown
|
||||
}
|
||||
|
||||
class LogEntry {
|
||||
+time_point timestamp
|
||||
+LogLevel level
|
||||
+PipelinePhase phase
|
||||
+string message
|
||||
+optional~string~ worker
|
||||
}
|
||||
|
||||
class ILogger {
|
||||
<<interface>>
|
||||
+Log(entry) void
|
||||
}
|
||||
|
||||
class LogProducer {
|
||||
-BoundedChannel~LogEntry~ channel_
|
||||
+Log(entry) void
|
||||
}
|
||||
|
||||
class LogDispatcher {
|
||||
-BoundedChannel~LogEntry~ channel_
|
||||
+Run() void
|
||||
-ToSpdlogLevel(level) level_enum
|
||||
}
|
||||
}
|
||||
LogEntry *-- LogLevel
|
||||
LogEntry *-- PipelinePhase
|
||||
ILogger <|.. LogProducer
|
||||
LogProducer ..> LogEntry : emits
|
||||
LogDispatcher ..> LogEntry : consumes
|
||||
|
||||
namespace InfrastructurePipelineChannel {
|
||||
class BoundedChannel~T~ {
|
||||
-queue~T~ queue_
|
||||
-mutex mutex_
|
||||
-condition_variable not_full_
|
||||
-condition_variable not_empty_
|
||||
-size_t capacity_
|
||||
-bool closed_
|
||||
+Send(item) void
|
||||
+Receive() optional~T~
|
||||
+Close() void
|
||||
}
|
||||
}
|
||||
|
||||
namespace InfrastructureDataPreloading {
|
||||
class ICuratedDataService {
|
||||
<<interface>>
|
||||
+LoadLocations(filepath) vector~Location~
|
||||
+LoadBeerStyles(filepath) vector~BeerStyle~
|
||||
+LoadPersonas(filepath) vector~UserPersona~
|
||||
+LoadForenamesByCountry(filepath) unordered_map~string, forename_list~
|
||||
+LoadSurnamesByCountry(filepath) unordered_map~string, surname_list~
|
||||
}
|
||||
|
||||
class JsonLoader {
|
||||
+LoadLocations(filepath) vector~Location~
|
||||
+LoadBeerStyles(filepath) vector~BeerStyle~
|
||||
+LoadPersonas(filepath) vector~UserPersona~
|
||||
+LoadForenamesByCountry(filepath) unordered_map~string, forename_list~
|
||||
+LoadSurnamesByCountry(filepath) unordered_map~string, surname_list~
|
||||
}
|
||||
|
||||
class MockCuratedDataService {
|
||||
+LoadLocations(filepath) vector~Location~
|
||||
+LoadBeerStyles(filepath) vector~BeerStyle~
|
||||
+LoadPersonas(filepath) vector~UserPersona~
|
||||
+LoadForenamesByCountry(filepath) unordered_map~string, forename_list~
|
||||
+LoadSurnamesByCountry(filepath) unordered_map~string, surname_list~
|
||||
}
|
||||
}
|
||||
note for JsonLoader "Each Load* call is memoized after its first parse — implemented today, ahead of LoadBeerStyles which is still planned."
|
||||
note for MockCuratedDataService "Fixed in-memory dataset for --mocked mode, implemented today for Locations/Personas/Forenames/Surnames; would need a LoadBeerStyles fixture too once that method is implemented."
|
||||
|
||||
namespace InfrastructureEnrichment {
|
||||
class EnrichmentService {
|
||||
<<interface>>
|
||||
+GetLocationContext(loc, strategy) LocationContext
|
||||
}
|
||||
|
||||
class WikipediaService {
|
||||
-unique_ptr~WebClient~ client_
|
||||
-unordered_map~string, string~ extract_cache_
|
||||
+GetLocationContext(loc, strategy) LocationContext
|
||||
-FetchExtract(query) string
|
||||
}
|
||||
|
||||
class WebClient {
|
||||
<<interface>>
|
||||
+Get(url) string
|
||||
+UrlEncode(value) string
|
||||
}
|
||||
|
||||
class HttpWebClient {
|
||||
+Get(url) string
|
||||
+UrlEncode(value) string
|
||||
}
|
||||
}
|
||||
|
||||
namespace InfrastructureGeocoding {
|
||||
class AddressLookupResult {
|
||||
+string address_line1
|
||||
+string postal_code
|
||||
}
|
||||
|
||||
class AddressService {
|
||||
<<interface>>
|
||||
+ReverseGeocode(longitude, latitude) optional~AddressLookupResult~
|
||||
}
|
||||
|
||||
class NominatimAddressService {
|
||||
-unique_ptr~WebClient~ client_
|
||||
-Logger logger_
|
||||
+ReverseGeocode(longitude, latitude) optional~AddressLookupResult~
|
||||
}
|
||||
|
||||
class MockAddressService {
|
||||
+ReverseGeocode(longitude, latitude) optional~AddressLookupResult~
|
||||
}
|
||||
}
|
||||
note for NominatimAddressService "Already implemented (services/address/), not planned: queries the public Nominatim API, sleeping 1s per request per its usage policy; returns nullopt on failure or when no usable street address is found. Ships today, ahead of the rest of this diagram."
|
||||
note for MockAddressService "Already implemented, not planned: returns a fixed placeholder address ('123 Mock Street', '00000') for --mocked runs, no network call."
|
||||
AddressService <|.. NominatimAddressService
|
||||
AddressService <|.. MockAddressService
|
||||
AddressService ..> AddressLookupResult : returns
|
||||
NominatimAddressService *-- WebClient
|
||||
|
||||
namespace InfrastructurePrompting {
|
||||
class IPromptDirectory {
|
||||
<<interface>>
|
||||
+Load(key) string
|
||||
}
|
||||
|
||||
class PromptDirectory {
|
||||
-path prompt_dir_
|
||||
-unordered_map~string, string~ cache_
|
||||
+PromptDirectory(prompt_dir)
|
||||
+Load(key) string
|
||||
}
|
||||
}
|
||||
IPromptDirectory <|.. PromptDirectory
|
||||
|
||||
namespace InfrastructureDataGeneration {
|
||||
class DataGenerator {
|
||||
<<interface>>
|
||||
+GenerateBrewery(location, context) BreweryResult
|
||||
+GenerateBeer(brewery_id, location, context, style) BeerResult
|
||||
+GenerateUser(city, persona, name) UserResult
|
||||
+GenerateCheckin(user, brewery, timestamp) CheckinResult
|
||||
+GenerateRating(user, beer, checkin_id) RatingResult
|
||||
}
|
||||
|
||||
class MockGenerator {
|
||||
+GenerateBrewery(...) BreweryResult
|
||||
+GenerateBeer(...) BeerResult
|
||||
+GenerateUser(...) UserResult
|
||||
+GenerateCheckin(...) CheckinResult
|
||||
+GenerateRating(...) RatingResult
|
||||
-DeterministicHash(location) size_t
|
||||
}
|
||||
|
||||
class LlamaGenerator {
|
||||
-ModelHandle model_
|
||||
-ContextHandle context_
|
||||
-unique_ptr~PromptFormatter~ prompt_formatter_
|
||||
-unique_ptr~IPromptDirectory~ prompt_directory_
|
||||
-mt19937 rng_
|
||||
+GenerateBrewery(...) BreweryResult
|
||||
+GenerateBeer(...) BeerResult
|
||||
+GenerateUser(...) UserResult
|
||||
+GenerateCheckin(...) CheckinResult
|
||||
+GenerateRating(...) RatingResult
|
||||
-Load(opts) void
|
||||
-Infer(system_prompt, user_prompt, max_tokens, grammar) string
|
||||
-ValidateModelArchitecture() void
|
||||
}
|
||||
|
||||
class PromptFormatter {
|
||||
<<interface>>
|
||||
+Format(system_prompt, user_prompt) string
|
||||
+ExpectedArchitecture() string_view
|
||||
}
|
||||
|
||||
class Gemma4JinjaPromptFormatter {
|
||||
+Format(system_prompt, user_prompt) string
|
||||
+ExpectedArchitecture() string_view
|
||||
}
|
||||
|
||||
class OpenAIGenerator {
|
||||
-string api_key_
|
||||
-string model_
|
||||
-Logger logger_
|
||||
-unique_ptr~IPromptDirectory~ prompt_directory_
|
||||
-unique_ptr~WebClient~ web_client_
|
||||
+GenerateBrewery(location, context) BreweryResult
|
||||
+GenerateBeer(brewery_id, location, context, style) BeerResult
|
||||
+GenerateUser(city, persona, name) UserResult
|
||||
+GenerateCheckin(user, brewery, timestamp) CheckinResult
|
||||
+GenerateRating(user, beer, checkin_id) RatingResult
|
||||
-CallChatCompletionsApi(...) string
|
||||
}
|
||||
}
|
||||
note for OpenAIGenerator "Already implemented (data_generation/openai/), not planned: calls the OpenAI Chat Completions API with Structured Outputs (json_schema, strict) to guarantee schema-valid JSON, needing no IPromptFormatter or GBNF grammar. Ships today for GenerateBrewery/GenerateUser only, ahead of the rest of this diagram — GenerateBeer/GenerateCheckin/GenerateRating are shown here for interface completeness but not yet implemented on any backend."
|
||||
|
||||
namespace InfrastructureDataExport {
|
||||
class ExportService {
|
||||
<<interface>>
|
||||
+Initialize() void
|
||||
+ProcessBrewery(brewery) uint64_t
|
||||
+ProcessBeer(beer) uint64_t
|
||||
+ProcessUser(user) uint64_t
|
||||
+ProcessCheckin(checkin) uint64_t
|
||||
+ProcessRating(rating) void
|
||||
+ProcessFollow(follow) void
|
||||
+Finalize() void
|
||||
}
|
||||
|
||||
class SqliteExportService {
|
||||
-unique_ptr~DateTimeProvider~ date_time_provider_
|
||||
-SqliteDatabaseHandle db_handle_
|
||||
-SqliteStatementHandle insert_location_stmt_
|
||||
-SqliteStatementHandle insert_brewery_stmt_
|
||||
-SqliteStatementHandle insert_beer_stmt_
|
||||
-SqliteStatementHandle insert_user_stmt_
|
||||
-SqliteStatementHandle insert_checkin_stmt_
|
||||
-SqliteStatementHandle insert_rating_stmt_
|
||||
-SqliteStatementHandle insert_follow_stmt_
|
||||
-bool transaction_open_
|
||||
-unordered_map~string, uint64_t~ location_cache_
|
||||
-unordered_map~string, uint64_t~ brewery_cache_
|
||||
+Initialize() void
|
||||
+ProcessRecord(brewery) uint64_t
|
||||
+ProcessRecord(beer) uint64_t
|
||||
+ProcessRecord(user) uint64_t
|
||||
+ProcessRecord(checkin) uint64_t
|
||||
+ProcessRecord(rating) void
|
||||
+ProcessRecord(follow) void
|
||||
+Finalize() void
|
||||
-InitializeSchema() void
|
||||
-PrepareStatements() void
|
||||
-RollbackAndCloseNoThrow() void
|
||||
-FinalizeStatements() void
|
||||
}
|
||||
|
||||
class DateTimeProvider {
|
||||
<<interface>>
|
||||
+GetUtcTimestamp() string
|
||||
}
|
||||
|
||||
class SystemDateTimeProvider {
|
||||
+GetUtcTimestamp() string
|
||||
}
|
||||
}
|
||||
|
||||
class BiergartenPipelineOrchestrator {
|
||||
-unique_ptr~ICuratedDataService~ curated_data_service_
|
||||
-unique_ptr~EnrichmentService~ enrichment_service_
|
||||
-unique_ptr~DataGenerator~ generator_
|
||||
-unique_ptr~Logger~ logger_
|
||||
-unique_ptr~ExportService~ exporter_
|
||||
-unique_ptr~AddressService~ address_service_
|
||||
-unique_ptr~ContextStrategy~ brewery_context_strategy_
|
||||
-unique_ptr~SamplingStrategy~ sampling_strategy_
|
||||
-unique_ptr~BeerSelectionStrategy~ beer_selection_strategy_
|
||||
-unique_ptr~CheckinDistributionStrategy~ checkin_strategy_
|
||||
-unique_ptr~FollowGenerationStrategy~ follow_strategy_
|
||||
-vector~BeerStyle~ beer_style_palette_
|
||||
-ApplicationOptions options_
|
||||
-vector~GeneratedUser~ user_pool_
|
||||
-vector~GeneratedBrewery~ brewery_pool_
|
||||
-vector~GeneratedBeer~ beer_pool_
|
||||
-vector~GeneratedCheckin~ checkin_pool_
|
||||
-vector~GeneratedFollow~ follow_pool_
|
||||
+Run() bool
|
||||
-RunUserPhase(locations) void
|
||||
-RunBreweryAndBeerPhase(locations) void
|
||||
-RunCheckinPhase() void
|
||||
-RunRatingPhase() void
|
||||
-RunFollowPhase() void
|
||||
}
|
||||
|
||||
BiergartenPipelineOrchestrator *-- ICuratedDataService
|
||||
BiergartenPipelineOrchestrator *-- EnrichmentService
|
||||
BiergartenPipelineOrchestrator *-- DataGenerator
|
||||
BiergartenPipelineOrchestrator *-- ExportService
|
||||
BiergartenPipelineOrchestrator *-- AddressService
|
||||
BiergartenPipelineOrchestrator *-- CheckinDistributionStrategy
|
||||
BiergartenPipelineOrchestrator *-- FollowGenerationStrategy
|
||||
BiergartenPipelineOrchestrator *-- SamplingStrategy
|
||||
BiergartenPipelineOrchestrator *-- BeerSelectionStrategy
|
||||
BiergartenPipelineOrchestrator *-- ApplicationOptions
|
||||
BiergartenPipelineOrchestrator *-- Logger
|
||||
|
||||
BiergartenPipelineOrchestrator "1" *-- "0..*" GeneratedUser : user_pool_
|
||||
BiergartenPipelineOrchestrator "1" *-- "0..*" GeneratedBrewery : brewery_pool_
|
||||
BiergartenPipelineOrchestrator "1" *-- "0..*" GeneratedBeer : beer_pool_
|
||||
BiergartenPipelineOrchestrator "1" *-- "0..*" GeneratedCheckin : checkin_pool_
|
||||
BiergartenPipelineOrchestrator "1" *-- "0..*" GeneratedFollow : follow_pool_
|
||||
|
||||
ICuratedDataService <|.. JsonLoader
|
||||
ICuratedDataService <|.. MockCuratedDataService
|
||||
Logger <|.. PipelineLogger
|
||||
ContextStrategy <|.. BreweryContextStrategy
|
||||
ContextStrategy <|.. BeerContextStrategy
|
||||
SamplingStrategy <|.. UniformSamplingStrategy
|
||||
BeerSelectionStrategy <|.. RandomBeerSelectionStrategy
|
||||
CheckinDistributionStrategy <|.. JCurveCheckinStrategy
|
||||
CheckinDistributionStrategy <|.. RandomCheckinStrategy
|
||||
FollowGenerationStrategy <|.. RandomFollowStrategy
|
||||
FollowGenerationStrategy <|.. ActivityWeightedFollowStrategy
|
||||
EnrichmentService <|.. WikipediaService
|
||||
WebClient <|.. HttpWebClient
|
||||
DataGenerator <|.. MockGenerator
|
||||
DataGenerator <|.. LlamaGenerator
|
||||
DataGenerator <|.. OpenAIGenerator
|
||||
PromptFormatter <|.. Gemma4JinjaPromptFormatter
|
||||
ExportService <|.. SqliteExportService
|
||||
DateTimeProvider <|.. SystemDateTimeProvider
|
||||
|
||||
WikipediaService *-- WebClient
|
||||
WikipediaService ..> ContextStrategy
|
||||
LlamaGenerator *-- PromptFormatter
|
||||
LlamaGenerator *-- IPromptDirectory
|
||||
LlamaGenerator ..> GeneratorOptions
|
||||
OpenAIGenerator *-- IPromptDirectory
|
||||
OpenAIGenerator *-- WebClient
|
||||
SqliteExportService *-- DateTimeProvider
|
||||
|
||||
PipelineLogger o-- BoundedChannel : logs to
|
||||
LogWorker o-- BoundedChannel : drains from
|
||||
|
||||
EnrichedCity *-- Location
|
||||
EnrichedCity *-- LocationContext
|
||||
GeneratedBrewery *-- Location
|
||||
GeneratedBrewery *-- BreweryResult
|
||||
GeneratedBrewery *-- GenerationMetadata
|
||||
GeneratedBeer *-- Location
|
||||
GeneratedBeer *-- BeerStyle
|
||||
GeneratedBeer *-- BeerResult
|
||||
GeneratedBeer *-- GenerationMetadata
|
||||
GeneratedUser *-- Location
|
||||
GeneratedUser *-- UserResult
|
||||
GeneratedUser *-- GenerationMetadata
|
||||
GeneratedCheckin *-- CheckinResult
|
||||
GeneratedCheckin *-- GenerationMetadata
|
||||
GeneratedRating *-- RatingResult
|
||||
GeneratedRating *-- GenerationMetadata
|
||||
GeneratedFollow *-- GenerationMetadata
|
||||
@@ -1,646 +0,0 @@
|
||||
@startuml class_diagram
|
||||
|
||||
' ==========================================
|
||||
' CONFIGURATION & STYLING
|
||||
' ==========================================
|
||||
!include ../biergarten-weizen-theme.puml
|
||||
skinparam classAttributeFontSize 9
|
||||
skinparam defaultFontSize 25
|
||||
skinparam titleFontSize 30
|
||||
|
||||
title Biergarten Data Pipeline — Class Diagram
|
||||
|
||||
package "Domain: Models" {
|
||||
|
||||
class Location {
|
||||
+ city : std::string
|
||||
+ state_province : std::string
|
||||
+ iso3166_2 : std::string
|
||||
+ country : std::string
|
||||
+ iso3166_1 : std::string
|
||||
+ local_languages : std::vector<std::string>
|
||||
+ latitude : double
|
||||
+ longitude : double
|
||||
}
|
||||
|
||||
class LocationContext {
|
||||
+ text : std::string
|
||||
+ completeness : Completeness
|
||||
+ char_count : size_t
|
||||
}
|
||||
|
||||
enum Completeness {
|
||||
Full
|
||||
Partial
|
||||
Absent
|
||||
}
|
||||
|
||||
class EnrichedCity {
|
||||
+ location : Location
|
||||
+ context : LocationContext
|
||||
}
|
||||
|
||||
class BeerStyle {
|
||||
+ name : std::string
|
||||
+ description : std::string
|
||||
+ min_abv : float
|
||||
+ max_abv : float
|
||||
+ min_ibu : int
|
||||
+ max_ibu : int
|
||||
}
|
||||
|
||||
class BreweryResult {
|
||||
+ name_en : std::string
|
||||
+ description_en : std::string
|
||||
+ name_local : std::string
|
||||
+ description_local : std::string
|
||||
}
|
||||
|
||||
class BeerResult {
|
||||
+ name_en : std::string
|
||||
+ description_en : std::string
|
||||
+ name_local : std::string
|
||||
+ description_local : std::string
|
||||
+ style : std::string
|
||||
+ abv : float
|
||||
+ ibu : int
|
||||
}
|
||||
|
||||
class UserResult {
|
||||
+ first_name : std::string
|
||||
+ last_name : std::string
|
||||
+ gender : std::string
|
||||
+ username : std::string
|
||||
+ bio : std::string
|
||||
+ activity_weight : float
|
||||
}
|
||||
|
||||
class Name {
|
||||
+ first_name : std::string
|
||||
+ last_name : std::string
|
||||
+ gender : std::string
|
||||
}
|
||||
|
||||
class ForenameEntry {
|
||||
+ name : std::string
|
||||
+ gender : std::string
|
||||
}
|
||||
|
||||
note right of ForenameEntry
|
||||
forename_list = std::unordered_set<ForenameEntry>
|
||||
surname_list = std::unordered_set<std::string>
|
||||
(aliases declared in curated_data_service.h).
|
||||
ICuratedDataService::LoadForenamesByCountry() /
|
||||
LoadSurnamesByCountry() each return a flat
|
||||
std::unordered_map<std::string, forename_list>
|
||||
or <std::string, surname_list> keyed by ISO
|
||||
3166-1 country code -- no NamesByCountry
|
||||
wrapper class. SampleName(forenames_by_country,
|
||||
surnames_by_country, iso3166_1, rng) is a free
|
||||
function, not a method, so it can be called by
|
||||
any per-city worker without owning the maps.
|
||||
Sourced from popular-names-by-country-dataset
|
||||
(CC0) -- see ETHICS-AND-KNOWN-ISSUES.md. Gender
|
||||
is preserved per forename so it can be threaded
|
||||
through to persona/bio generation later; surnames
|
||||
carry no gender in the source data. Pairing
|
||||
happens at sample time, not at fixture-authoring
|
||||
time, so no information from the source dataset
|
||||
is discarded up front.
|
||||
end note
|
||||
|
||||
ForenameEntry ..> Name : SampleName() produces
|
||||
|
||||
class CheckinResult {
|
||||
+ checked_in_at : std::string
|
||||
+ note : std::string
|
||||
}
|
||||
|
||||
class RatingResult {
|
||||
+ score : float
|
||||
+ note : std::string
|
||||
}
|
||||
|
||||
class GenerationMetadata {
|
||||
+ generation_id : uint64_t
|
||||
+ generated_time : std::string
|
||||
+ context_provided : bool
|
||||
+ generated_with : std::string
|
||||
}
|
||||
|
||||
class GeneratedBrewery {
|
||||
+ brewery_id : uint64_t
|
||||
+ location : Location
|
||||
+ brewery : BreweryResult
|
||||
+ context_completeness : LocationContext::Completeness
|
||||
+ metadata : GenerationMetadata
|
||||
}
|
||||
|
||||
class GeneratedBeer {
|
||||
+ beer_id : uint64_t
|
||||
+ brewery_id : uint64_t
|
||||
+ location : Location
|
||||
+ style : BeerStyle
|
||||
+ beer : BeerResult
|
||||
+ metadata : GenerationMetadata
|
||||
}
|
||||
|
||||
class GeneratedUser {
|
||||
+ user_id : uint64_t
|
||||
+ location : Location
|
||||
+ user : UserResult
|
||||
+ email : std::string
|
||||
+ date_of_birth : std::string
|
||||
+ password : std::string
|
||||
+ metadata : GenerationMetadata
|
||||
}
|
||||
|
||||
note right of GeneratedUser
|
||||
email, date_of_birth, and password are
|
||||
programmatically generated by the orchestrator
|
||||
(not LLM-authored) so a downstream auth-account
|
||||
seeding consumer can register real accounts from
|
||||
this export. See ROADMAP.md §1.
|
||||
end note
|
||||
|
||||
class GeneratedCheckin {
|
||||
+ checkin_id : uint64_t
|
||||
+ user_id : uint64_t
|
||||
+ brewery_id : uint64_t
|
||||
+ checkin : CheckinResult
|
||||
+ metadata : GenerationMetadata
|
||||
}
|
||||
|
||||
class GeneratedRating {
|
||||
+ user_id : uint64_t
|
||||
+ beer_id : uint64_t
|
||||
+ checkin_id : uint64_t
|
||||
+ rating : RatingResult
|
||||
+ metadata : GenerationMetadata
|
||||
}
|
||||
|
||||
class GeneratedFollow {
|
||||
+ follower_id : uint64_t
|
||||
+ followed_id : uint64_t
|
||||
+ metadata : GenerationMetadata
|
||||
}
|
||||
|
||||
class UserPersona {
|
||||
+ name: std::string
|
||||
+ description: std::string
|
||||
+ style_affinities: std::vector<std::string>
|
||||
}
|
||||
|
||||
LocationContext *-- Completeness
|
||||
}
|
||||
|
||||
package "Domain: Application Configuration" {
|
||||
class SamplingOptions {
|
||||
+ temperature: float = 1.0F
|
||||
+ top_p: float = 0.95F
|
||||
+ top_k: uint32_t = 64
|
||||
+ n_ctx: uint32_t = 8192
|
||||
+ seed: int = -1
|
||||
}
|
||||
|
||||
class GeneratorOptions {
|
||||
+ model_path: std::filesystem::path
|
||||
+ use_mocked: bool = false
|
||||
+ sampling: std::optional<SamplingOptions>
|
||||
}
|
||||
|
||||
class PipelineOptions {
|
||||
+ output_path: std::filesystem::path
|
||||
+ log_path: std::filesystem::path
|
||||
}
|
||||
|
||||
class ApplicationOptions {
|
||||
+ generator: GeneratorOptions
|
||||
+ pipeline: PipelineOptions
|
||||
}
|
||||
|
||||
ApplicationOptions *-- GeneratorOptions
|
||||
ApplicationOptions *-- PipelineOptions
|
||||
GeneratorOptions o-- SamplingOptions
|
||||
}
|
||||
|
||||
package "Domain: Policy" {
|
||||
|
||||
interface ContextStrategy <<interface>> {
|
||||
+ QueriesFor(loc : const Location&) : std::vector<std::string>
|
||||
+ MaxContextChars() : size_t
|
||||
}
|
||||
|
||||
class BreweryContextStrategy {
|
||||
+ QueriesFor(loc : const Location&) : std::vector<std::string>
|
||||
+ MaxContextChars() : size_t
|
||||
}
|
||||
|
||||
class BeerContextStrategy {
|
||||
+ QueriesFor(loc : const Location&) : std::vector<std::string>
|
||||
+ MaxContextChars() : size_t
|
||||
}
|
||||
|
||||
interface SamplingStrategy <<interface>> {
|
||||
+ Sample(locations : const std::vector<Location>&) : std::vector<Location>
|
||||
}
|
||||
|
||||
class UniformSamplingStrategy {
|
||||
- sample_size_ : size_t
|
||||
+ Sample(locations : const std::vector<Location>&) : std::vector<Location>
|
||||
}
|
||||
|
||||
interface BeerSelectionStrategy <<interface>> {
|
||||
+ SelectStyles(brewery : const GeneratedBrewery&,\n palette : std::span<const BeerStyle>) : std::vector<BeerStyle>
|
||||
}
|
||||
|
||||
class RandomBeerSelectionStrategy {
|
||||
- rng_ : std::mt19937
|
||||
- min_beers_ : size_t
|
||||
- max_beers_ : size_t
|
||||
+ SelectStyles(brewery : const GeneratedBrewery&,\n palette : std::span<const BeerStyle>) : std::vector<BeerStyle>
|
||||
}
|
||||
|
||||
interface CheckinDistributionStrategy <<interface>> {
|
||||
+ AssignActivityWeights(users : std::vector<GeneratedUser>&) : void
|
||||
+ CheckinsForUser(user : const GeneratedUser&,\n brewery_count : size_t) : size_t
|
||||
+ TimestampFor(user : const GeneratedUser&,\n index : size_t) : std::string
|
||||
}
|
||||
|
||||
class JCurveCheckinStrategy {
|
||||
- rng_ : std::mt19937
|
||||
+ AssignActivityWeights(users : std::vector<GeneratedUser>&) : void
|
||||
+ CheckinsForUser(user : const GeneratedUser&,\n brewery_count : size_t) : size_t
|
||||
+ TimestampFor(user : const GeneratedUser&,\n index : size_t) : std::string
|
||||
}
|
||||
|
||||
class RandomCheckinStrategy {
|
||||
- rng_ : std::mt19937
|
||||
- min_checkins_ : size_t
|
||||
- max_checkins_ : size_t
|
||||
+ AssignActivityWeights(users : std::vector<GeneratedUser>&) : void
|
||||
+ CheckinsForUser(user : const GeneratedUser&,\n brewery_count : size_t) : size_t
|
||||
+ TimestampFor(user : const GeneratedUser&,\n index : size_t) : std::string
|
||||
}
|
||||
|
||||
interface FollowGenerationStrategy <<interface>> {
|
||||
+ GenerateFollows(users : const std::vector<GeneratedUser>&) : std::vector<GeneratedFollow>
|
||||
}
|
||||
|
||||
class RandomFollowStrategy {
|
||||
- rng_ : std::mt19937
|
||||
- min_follows_ : size_t
|
||||
- max_follows_ : size_t
|
||||
+ GenerateFollows(users : const std::vector<GeneratedUser>&) : std::vector<GeneratedFollow>
|
||||
}
|
||||
|
||||
class ActivityWeightedFollowStrategy {
|
||||
- rng_ : std::mt19937
|
||||
- min_follows_ : size_t
|
||||
- max_follows_ : size_t
|
||||
+ GenerateFollows(users : const std::vector<GeneratedUser>&) : std::vector<GeneratedFollow>
|
||||
}
|
||||
}
|
||||
|
||||
package "Infrastructure: Logging" {
|
||||
enum LogLevel {
|
||||
Debug
|
||||
Info
|
||||
Warn
|
||||
Error
|
||||
}
|
||||
|
||||
enum PipelinePhase {
|
||||
Startup
|
||||
UserGeneration
|
||||
BreweryAndBeerGeneration
|
||||
CheckinGeneration
|
||||
RatingGeneration
|
||||
FollowGeneration
|
||||
Teardown
|
||||
}
|
||||
|
||||
class LogEntry {
|
||||
+ timestamp : std::chrono::system_clock::time_point
|
||||
+ level : LogLevel
|
||||
+ phase : PipelinePhase
|
||||
+ message : std::string
|
||||
+ worker : std::optional<std::string>
|
||||
}
|
||||
|
||||
interface ILogger <<interface>> {
|
||||
+ Log(entry : const LogEntry&) : void
|
||||
}
|
||||
|
||||
class LogProducer {
|
||||
- channel_ : BoundedChannel<LogEntry>&
|
||||
+ Log(entry : const LogEntry&) : void
|
||||
}
|
||||
|
||||
class LogDispatcher {
|
||||
- channel_ : BoundedChannel<LogEntry>&
|
||||
+ Run() : void
|
||||
- ToSpdlogLevel(level) : spdlog::level::level_enum
|
||||
}
|
||||
|
||||
LogEntry *-- LogLevel
|
||||
LogEntry *-- PipelinePhase
|
||||
ILogger <|.. LogProducer
|
||||
LogProducer ..> LogEntry : emits
|
||||
LogDispatcher ..> LogEntry : consumes
|
||||
}
|
||||
|
||||
package "Infrastructure: Pipeline Channel" {
|
||||
|
||||
class "BoundedChannel<T>" as BoundedChannel {
|
||||
- queue_ : std::queue<T>
|
||||
- mutex_ : std::mutex
|
||||
- not_full_ : std::condition_variable
|
||||
- not_empty_ : std::condition_variable
|
||||
- capacity_ : size_t
|
||||
- closed_ : bool
|
||||
+ Send(item : T) : void
|
||||
+ Receive() : std::optional<T>
|
||||
+ Close() : void
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
package "Infrastructure: Data Preloading" {
|
||||
|
||||
interface ICuratedDataService <<interface>> {
|
||||
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
|
||||
+ LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&
|
||||
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
|
||||
+ LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
|
||||
+ LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
|
||||
}
|
||||
|
||||
class JsonLoader {
|
||||
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
|
||||
+ LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&
|
||||
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
|
||||
+ LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
|
||||
+ LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
|
||||
}
|
||||
|
||||
note right of JsonLoader
|
||||
Each Load* call is memoized after its first
|
||||
parse -- implemented today, ahead of the rest
|
||||
of this package (LoadBeerStyles is still planned).
|
||||
end note
|
||||
|
||||
class MockCuratedDataService {
|
||||
+ LoadLocations(filepath : const std::filesystem::path&) : const std::vector<Location>&
|
||||
+ LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&
|
||||
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
|
||||
+ LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
|
||||
+ LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
|
||||
}
|
||||
|
||||
note right of MockCuratedDataService
|
||||
Fixed in-memory dataset used in --mocked mode,
|
||||
implemented today for Locations/Personas/
|
||||
Forenames/Surnames; would need a LoadBeerStyles
|
||||
fixture too once that method is implemented.
|
||||
end note
|
||||
|
||||
}
|
||||
|
||||
package "Infrastructure: Enrichment" {
|
||||
|
||||
interface EnrichmentService <<interface>> {
|
||||
+ GetLocationContext(loc : const Location&,\n strategy : const ContextStrategy&) : LocationContext
|
||||
}
|
||||
|
||||
class WikipediaService {
|
||||
- client_ : std::unique_ptr<WebClient>
|
||||
- extract_cache_ : std::unordered_map<std::string, std::string>
|
||||
+ GetLocationContext(loc : const Location&,\n strategy : const ContextStrategy&) : LocationContext
|
||||
- FetchExtract(query : std::string_view) : std::string
|
||||
}
|
||||
|
||||
interface WebClient <<interface>> {
|
||||
+ Get(url : const std::string&) : std::string
|
||||
+ UrlEncode(value : const std::string&) : std::string
|
||||
}
|
||||
|
||||
class HttpWebClient {
|
||||
+ Get(url : const std::string&) : std::string
|
||||
+ UrlEncode(value : const std::string&) : std::string
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
package "Infrastructure: Prompting" {
|
||||
|
||||
interface IPromptDirectory <<interface>> {
|
||||
+ Load(key : std::string_view) : std::string
|
||||
}
|
||||
|
||||
class PromptDirectory {
|
||||
- prompt_dir_ : std::filesystem::path
|
||||
- cache_ : std::unordered_map<std::string, std::string>
|
||||
+ PromptDirectory(prompt_dir : const std::filesystem::path&)
|
||||
+ Load(key : std::string_view) : std::string
|
||||
}
|
||||
|
||||
IPromptDirectory <|.. PromptDirectory
|
||||
}
|
||||
|
||||
package "Infrastructure: Data Generation" {
|
||||
|
||||
interface DataGenerator <<interface>> {
|
||||
+ GenerateBrewery(location : const Location&,\n context : const LocationContext&) : BreweryResult
|
||||
+ GenerateBeer(brewery_id : uint64_t,\n location : const Location&,\n context : const LocationContext&,\n style : const BeerStyle&) : BeerResult
|
||||
+ GenerateUser(city : const EnrichedCity&,\n persona : const UserPersona&,\n name : const Name&) : UserResult
|
||||
+ GenerateCheckin(user : const GeneratedUser&,\n brewery : const GeneratedBrewery&,\n timestamp : const std::string&) : CheckinResult
|
||||
+ GenerateRating(user : const GeneratedUser&,\n beer : const GeneratedBeer&,\n checkin_id : uint64_t) : RatingResult
|
||||
}
|
||||
|
||||
class MockGenerator {
|
||||
+ GenerateBrewery(...) : BreweryResult
|
||||
+ GenerateBeer(...) : BeerResult
|
||||
+ GenerateUser(...) : UserResult
|
||||
+ GenerateCheckin(...) : CheckinResult
|
||||
+ GenerateRating(...) : RatingResult
|
||||
- DeterministicHash(location : const Location&) : size_t
|
||||
}
|
||||
|
||||
class LlamaGenerator {
|
||||
- model_ : ModelHandle
|
||||
- context_ : ContextHandle
|
||||
- prompt_formatter_ : std::unique_ptr<PromptFormatter>
|
||||
- prompt_directory_ : std::unique_ptr<IPromptDirectory>
|
||||
- rng_ : std::mt19937
|
||||
+ GenerateBrewery(...) : BreweryResult
|
||||
+ GenerateBeer(...) : BeerResult
|
||||
+ GenerateUser(...) : UserResult
|
||||
+ GenerateCheckin(...) : CheckinResult
|
||||
+ GenerateRating(...) : RatingResult
|
||||
- Load(opts : const GeneratorOptions&) : void
|
||||
- Infer(system_prompt, user_prompt,\n max_tokens, grammar) : std::string
|
||||
- ValidateModelArchitecture() : void
|
||||
}
|
||||
|
||||
interface PromptFormatter <<interface>> {
|
||||
+ Format(system_prompt : std::string_view,\n user_prompt : std::string_view) : std::string
|
||||
+ ExpectedArchitecture() : std::string_view
|
||||
}
|
||||
|
||||
class Gemma4JinjaPromptFormatter {
|
||||
+ Format(...) : std::string
|
||||
+ ExpectedArchitecture() : std::string_view
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
package "Infrastructure: Data Export" {
|
||||
|
||||
interface ExportService <<interface>> {
|
||||
+ Initialize() : void
|
||||
+ ProcessBrewery(brewery : const GeneratedBrewery&) : uint64_t
|
||||
+ ProcessBeer(beer : const GeneratedBeer&) : uint64_t
|
||||
+ ProcessUser(user : const GeneratedUser&) : uint64_t
|
||||
+ ProcessCheckin(checkin : const GeneratedCheckin&) : uint64_t
|
||||
+ ProcessRating(rating : const GeneratedRating&) : void
|
||||
+ ProcessFollow(follow : const GeneratedFollow&) : void
|
||||
+ Finalize() : void
|
||||
}
|
||||
|
||||
class SqliteExportService {
|
||||
- date_time_provider_ : std::unique_ptr<DateTimeProvider>
|
||||
- db_handle_ : SqliteDatabaseHandle
|
||||
- insert_location_stmt_ : SqliteStatementHandle
|
||||
- insert_brewery_stmt_ : SqliteStatementHandle
|
||||
- insert_beer_stmt_ : SqliteStatementHandle
|
||||
- insert_user_stmt_ : SqliteStatementHandle
|
||||
- insert_checkin_stmt_ : SqliteStatementHandle
|
||||
- insert_rating_stmt_ : SqliteStatementHandle
|
||||
- insert_follow_stmt_ : SqliteStatementHandle
|
||||
- transaction_open_ : bool
|
||||
- location_cache_ : std::unordered_map<std::string, uint64_t>
|
||||
- brewery_cache_ : std::unordered_map<std::string, uint64_t>
|
||||
+ Initialize() : void
|
||||
+ ProcessRecord(brewery : const GeneratedBrewery&) : uint64_t
|
||||
+ ProcessRecord(beer : const GeneratedBeer&) : uint64_t
|
||||
+ ProcessRecord(user : const GeneratedUser&) : uint64_t
|
||||
+ ProcessRecord(checkin : const GeneratedCheckin&) : uint64_t
|
||||
+ ProcessRecord(rating : const GeneratedRating&) : void
|
||||
+ ProcessRecord(follow : const GeneratedFollow&) : void
|
||||
+ Finalize() : void
|
||||
- InitializeSchema() : void
|
||||
- PrepareStatements() : void
|
||||
- RollbackAndCloseNoThrow() : void
|
||||
- FinalizeStatements() : void
|
||||
}
|
||||
|
||||
interface DateTimeProvider <<interface>> {
|
||||
+ GetUtcTimestamp() : std::string
|
||||
}
|
||||
|
||||
class SystemDateTimeProvider {
|
||||
+ GetUtcTimestamp() : std::string
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
class BiergartenPipelineOrchestrator {
|
||||
- curated_data_service_ : std::unique_ptr<ICuratedDataService>
|
||||
- enrichment_service_ : std::unique_ptr<EnrichmentService>
|
||||
- generator_ : std::unique_ptr<DataGenerator>
|
||||
- logger_ : std::unique_ptr<Logger>
|
||||
- exporter_ : std::unique_ptr<ExportService>
|
||||
- brewery_context_strategy_ : std::unique_ptr<ContextStrategy>
|
||||
- sampling_strategy_ : std::unique_ptr<SamplingStrategy>
|
||||
- beer_selection_strategy_ : std::unique_ptr<BeerSelectionStrategy>
|
||||
- checkin_strategy_ : std::unique_ptr<CheckinDistributionStrategy>
|
||||
- follow_strategy_ : std::unique_ptr<FollowGenerationStrategy>
|
||||
- beer_style_palette_ : std::vector<BeerStyle>
|
||||
- options_ : ApplicationOptions
|
||||
--
|
||||
- user_pool_ : std::vector<GeneratedUser>
|
||||
- brewery_pool_ : std::vector<GeneratedBrewery>
|
||||
- beer_pool_ : std::vector<GeneratedBeer>
|
||||
- checkin_pool_ : std::vector<GeneratedCheckin>
|
||||
- follow_pool_ : std::vector<GeneratedFollow>
|
||||
--
|
||||
+ Run() : bool
|
||||
- RunUserPhase(locations : const std::vector<Location>&) : void
|
||||
- RunBreweryAndBeerPhase(locations : const std::vector<Location>&) : void
|
||||
- RunCheckinPhase() : void
|
||||
- RunRatingPhase() : void
|
||||
- RunFollowPhase() : void
|
||||
}
|
||||
|
||||
' --- Orchestration Aggregations (Services & Strategies) ---
|
||||
BiergartenPipelineOrchestrator *-- ICuratedDataService
|
||||
BiergartenPipelineOrchestrator *-- EnrichmentService
|
||||
BiergartenPipelineOrchestrator *-- DataGenerator
|
||||
BiergartenPipelineOrchestrator *-- ExportService
|
||||
BiergartenPipelineOrchestrator *-- CheckinDistributionStrategy
|
||||
BiergartenPipelineOrchestrator *-- FollowGenerationStrategy
|
||||
BiergartenPipelineOrchestrator *-- SamplingStrategy
|
||||
BiergartenPipelineOrchestrator *-- BeerSelectionStrategy
|
||||
BiergartenPipelineOrchestrator *-- ApplicationOptions
|
||||
BiergartenPipelineOrchestrator *-- Logger
|
||||
|
||||
' --- Orchestration Aggregations (Data Pools) ---
|
||||
BiergartenPipelineOrchestrator *-- "0..*" GeneratedUser : user_pool_
|
||||
BiergartenPipelineOrchestrator *-- "0..*" GeneratedBrewery : brewery_pool_
|
||||
BiergartenPipelineOrchestrator *-- "0..*" GeneratedBeer : beer_pool_
|
||||
BiergartenPipelineOrchestrator *-- "0..*" GeneratedCheckin : checkin_pool_
|
||||
BiergartenPipelineOrchestrator *-- "0..*" GeneratedFollow : follow_pool_
|
||||
|
||||
' --- Interfaces & Implementations ---
|
||||
ICuratedDataService <|.. JsonLoader
|
||||
ICuratedDataService <|.. MockCuratedDataService
|
||||
Logger <|.. PipelineLogger
|
||||
ContextStrategy <|.. BreweryContextStrategy
|
||||
ContextStrategy <|.. BeerContextStrategy
|
||||
SamplingStrategy <|.. UniformSamplingStrategy
|
||||
BeerSelectionStrategy <|.. RandomBeerSelectionStrategy
|
||||
CheckinDistributionStrategy <|.. JCurveCheckinStrategy
|
||||
CheckinDistributionStrategy <|.. RandomCheckinStrategy
|
||||
FollowGenerationStrategy <|.. RandomFollowStrategy
|
||||
FollowGenerationStrategy <|.. ActivityWeightedFollowStrategy
|
||||
EnrichmentService <|.. WikipediaService
|
||||
WebClient <|.. HttpWebClient
|
||||
DataGenerator <|.. MockGenerator
|
||||
DataGenerator <|.. LlamaGenerator
|
||||
PromptFormatter <|.. Gemma4JinjaPromptFormatter
|
||||
ExportService <|.. SqliteExportService
|
||||
DateTimeProvider <|.. SystemDateTimeProvider
|
||||
|
||||
' --- Service Compositions & Dependencies ---
|
||||
WikipediaService *-- WebClient
|
||||
WikipediaService ..> ContextStrategy
|
||||
LlamaGenerator *-- PromptFormatter
|
||||
LlamaGenerator *-- IPromptDirectory
|
||||
LlamaGenerator ..> GeneratorOptions
|
||||
SqliteExportService *-- DateTimeProvider
|
||||
|
||||
' --- Cross-Component Aggregations (Held References) ---
|
||||
PipelineLogger o-- BoundedChannel : logs to
|
||||
LogWorker o-- BoundedChannel : drains from
|
||||
|
||||
' --- Domain Containment ---
|
||||
EnrichedCity *-- Location
|
||||
EnrichedCity *-- LocationContext
|
||||
GeneratedBrewery *-- Location
|
||||
GeneratedBrewery *-- BreweryResult
|
||||
GeneratedBrewery *-- GenerationMetadata
|
||||
GeneratedBeer *-- Location
|
||||
GeneratedBeer *-- BeerStyle
|
||||
GeneratedBeer *-- BeerResult
|
||||
GeneratedBeer *-- GenerationMetadata
|
||||
GeneratedUser *-- Location
|
||||
GeneratedUser *-- UserResult
|
||||
GeneratedUser *-- GenerationMetadata
|
||||
GeneratedCheckin *-- CheckinResult
|
||||
GeneratedCheckin *-- GenerationMetadata
|
||||
GeneratedRating *-- RatingResult
|
||||
GeneratedRating *-- GenerationMetadata
|
||||
GeneratedFollow *-- GenerationMetadata
|
||||
|
||||
@enduml
|
||||
File diff suppressed because one or more lines are too long
|
Before Width: | Height: | Size: 142 KiB |
File diff suppressed because one or more lines are too long
|
Before Width: | Height: | Size: 240 KiB |
Reference in New Issue
Block a user