Compare commits

42 Commits

Author SHA1 Message Date
Aaron Po
98f97b8e2c add symlink to web docs 2026-06-27 23:34:30 -04:00
Aaron Po
2adc1eae38 update pipeline results viewer 2026-06-27 23:32:08 -04:00
Aaron Po
9d2ce67a78 Run pipeling, update sqllite file 2026-06-27 22:40:22 -04:00
Aaron Po
d1ad1eb397 rename json loader 2026-06-27 15:16:34 -04:00
Aaron Po
9137f4bddb Do full Gemma-4 run of pipeline, transfer sqllite file to public dir of preview 2026-06-23 22:02:17 -04:00
Aaron Po
6883b05824 remove password field 2026-06-23 01:39:57 -04:00
Aaron Po
b631c0f5db add results viewer 2026-06-23 01:27:37 -04:00
Aaron Po
9a9ffd59e6 Formatted src/biergarten_pipeline_orchestrator 2026-06-23 01:04:52 -04:00
Aaron Po
35a5c0785c documentation updates, remove superfluous comments 2026-06-23 01:04:46 -04:00
Aaron Po
e590cf2543 edit .clang-format 2026-06-23 00:50:27 -04:00
Aaron Po
3b32766a83 edits 2026-06-23 00:48:20 -04:00
Aaron Po
8d6b83673b Update doxygen comments 2026-06-23 00:45:32 -04:00
Aaron Po
22b3f2b3f9 cleanup gbnf grammars 2026-06-23 00:38:53 -04:00
Aaron Po
ac701d661a Remove logger from JSON export 2026-06-23 00:19:28 -04:00
Aaron Po
3f978f4de4 Cleanup user and brewery generation exception logic 2026-06-21 23:09:29 -04:00
Aaron Po
ad97b0ea6c format 2026-06-21 22:14:42 -04:00
Aaron Po
4de0ea6638 Persist generated users to SQLite and code cleanup 2026-06-21 22:14:42 -04:00
Aaron Po
51b40a39c9 Add user generation feature 2026-06-21 22:14:42 -04:00
Aaron Po
880f73e004 update pipeline docs 2026-06-21 21:58:03 -04:00
Aaron Po
3711591db1 commit csharpier config update 2026-06-20 15:54:53 -04:00
Aaron Po
194509f9c9 update mailkit 2026-06-20 15:54:53 -04:00
Aaron Po
67d6350425 Format ./web/backend/Shared/Shared.Contracts 2026-06-20 15:54:53 -04:00
Aaron Po
cee2917556 Format ./web/backend/Shared/Shared.Application 2026-06-20 15:54:53 -04:00
Aaron Po
6f8e5dc722 Format ./web/backend/API/API.Specs 2026-06-20 15:54:53 -04:00
Aaron Po
db3ed3581d Format ./web/backend/API/API.Core 2026-06-20 15:54:53 -04:00
Aaron Po
b54f70ccbe Format ./web/backend/Domain/Domain.Exceptions 2026-06-20 15:54:53 -04:00
Aaron Po
d4734ae9e7 Format ./web/backend/Domain/Domain.Entities 2026-06-20 15:54:53 -04:00
Aaron Po
27f28d5207 Format ./web/backend/Database/Database.Seed 2026-06-20 15:54:53 -04:00
Aaron Po
bf0d29fa54 Format ./web/backend/Database/Database.Migrations 2026-06-20 15:54:53 -04:00
Aaron Po
4772f59a68 Format ./web/backend/Infrastructure/Infrastructure.Sql 2026-06-20 15:54:53 -04:00
Aaron Po
2e5d4923c0 Format ./web/backend/Infrastructure/Infrastructure.PasswordHashing 2026-06-20 15:54:53 -04:00
Aaron Po
62766b7638 Format ./web/backend/Infrastructure/Infrastructure.Jwt 2026-06-20 15:54:53 -04:00
Aaron Po
744d4c7f8c Format ./web/backend/Infrastructure/Infrastructure.Email.Templates 2026-06-20 15:54:53 -04:00
Aaron Po
2f28ac9350 Format ./web/backend/Infrastructure/Infrastructure.Email 2026-06-20 15:54:53 -04:00
Aaron Po
c8bddae817 Format ./web/backend/Features/Features.UserManagement.Tests 2026-06-20 15:54:53 -04:00
Aaron Po
b298bd02d3 Format ./web/backend/Features/Features.UserManagement 2026-06-20 15:54:53 -04:00
Aaron Po
e2e5eb89e3 Format ./web/backend/Features/Features.Emails.Tests 2026-06-20 15:54:53 -04:00
Aaron Po
3dd44fca5a Format ./web/backend/Features/Features.Emails 2026-06-20 15:54:53 -04:00
Aaron Po
0c55d6c83f Format ./web/backend/Features/Features.Breweries.Tests 2026-06-20 15:54:53 -04:00
Aaron Po
58cbc5b5ba Format ./web/backend/Features/Features.Breweries 2026-06-20 15:54:53 -04:00
Aaron Po
5bef24696b Format ./web/backend/Features/Features.Auth.Tests 2026-06-20 15:54:53 -04:00
Aaron Po
79ae18b3e2 Format ./web/backend/Features/Features.Auth 2026-06-20 15:54:53 -04:00
210 changed files with 58216 additions and 2098 deletions

View File

@@ -13,6 +13,7 @@ low-resource language failures.
- [Model Bias and Language Quality](#model-bias-and-language-quality) - [Model Bias and Language Quality](#model-bias-and-language-quality)
- [Western and Eurocentric Lens](#western-and-eurocentric-lens) - [Western and Eurocentric Lens](#western-and-eurocentric-lens)
- [Wikipedia Enrichment](#wikipedia-enrichment) - [Wikipedia Enrichment](#wikipedia-enrichment)
- [Names-by-Country Dataset](#names-by-country-dataset)
- [The "Avoid AI Phrases" Prompt Instruction](#the-avoid-ai-phrases-prompt-instruction) - [The "Avoid AI Phrases" Prompt Instruction](#the-avoid-ai-phrases-prompt-instruction)
- [Known Issues](#known-issues) - [Known Issues](#known-issues)
- [Hallucinated Brewing Techniques](#hallucinated-brewing-techniques) - [Hallucinated Brewing Techniques](#hallucinated-brewing-techniques)
@@ -91,6 +92,33 @@ generated descriptions.
--- ---
## Names-by-Country Dataset
`tooling/pipeline/forenames-by-country.json` and `surnames-by-country.json`
(used to sample a `Name` per ISO 3166-1 country code for user generation) are
vendored verbatim, unmodified, from
[sigpwned/popular-names-by-country-dataset](https://github.com/sigpwned/popular-names-by-country-dataset)
(the `common-forenames-by-country.json` / `common-surnames-by-country.json`
release assets), released under **CC0** (public domain). That dataset's own
forename/surname lists are pulled from Wikipedia's "Lists of most common
surnames" and "List of most popular given names" as of the week of
2023-07-08 — see that project's README for full provenance. Names are not
LLM-generated; this is curated fixture data per ROADMAP.md §2. Per-forename
gender from the source data is preserved through to the sampled `Name`
(rather than discarded during loading) so it's available for gender-aware
persona/bio generation later.
The full multinational dataset is kept as-is (106 countries for forenames,
75 for surnames) rather than trimmed to `locations.json`'s current country
list, so it doesn't need re-sourcing if more countries are added later.
`NamesByCountry::SampleName()` returns no result for a country present in
neither file; of the countries in `locations.json`, that's currently `KE`,
`SE`, `SG`, `TH`, `VN`, and `ZA` — `GenerateUsers` skips cities in those
countries the same way brewery generation skips cities whose enrichment
lookup fails.
---
## The "Avoid AI Phrases" Prompt Instruction ## The "Avoid AI Phrases" Prompt Instruction
The system prompt instructs the model to avoid common AI-generated phrasing The system prompt instructs the model to avoid common AI-generated phrasing
@@ -245,7 +273,7 @@ but is semantically incoherent. This comes from limited training-data coverage
rather than prompt engineering. rather than prompt engineering.
Output sample: Output sample:
[./out-sample/french-cities.example](out-sample/french-cities.example) [./french-cities.example](french-cities.example)
#### Proposed Mitigations #### Proposed Mitigations

View File

@@ -94,24 +94,27 @@ Each run writes a fresh dated SQLite file such as
./biergarten-pipeline \ ./biergarten-pipeline \
--model ../models/google_gemma-4-E4B-it-Q6_K.gguf \ --model ../models/google_gemma-4-E4B-it-Q6_K.gguf \
--prompt-dir prompts \ --prompt-dir prompts \
--location-count 25 \
--temperature 1.0 --top-p 0.95 --top-k 64 --n-ctx 8192 --seed -1 --temperature 1.0 --top-p 0.95 --top-k 64 --n-ctx 8192 --seed -1
``` ```
#### CLI Flags #### CLI Flags
| Flag | Purpose | | Flag | Purpose |
| --------------- | ---------------------------------------------------------------------------------------------------- | | ------------------ | ---------------------------------------------------------------------------------------------------- |
| `--mocked` | Deterministic mock generator, no model required. | | `--mocked` | Deterministic mock generator, no model required. |
| `--model, -m` | Path to a GGUF file. Required unless `--mocked` is set. | | `--model, -m` | Path to a GGUF file. Required unless `--mocked` is set. |
| `--prompt-dir` | Directory containing prompt files (e.g. `BREWERY_GENERATION.md`). Required unless `--mocked` is set. | | `--prompt-dir` | Directory containing prompt files (e.g. `BREWERY_GENERATION.md`). Required unless `--mocked` is set. |
| `--output, -o` | Directory for generated SQLite artifacts. Default: `output`. | | `--output, -o` | Directory for generated SQLite artifacts. Default: `output`. |
| `--log-path` | Path for application logs. Default: `pipeline.log`. | | `--log-path` | Path for application logs. Default: `pipeline.log`. |
| `--temperature` | Sampling temperature. Default: `1.0`. | | `--location-count` | Number of cities to sample from `locations.json` per run. Default: `10`. |
| `--top-p` | Nucleus sampling. Default: `0.95`. | | `--temperature` | Sampling temperature. Default: `1.0`. |
| `--top-k` | Top-k sampling. Default: `64`. | | `--top-p` | Nucleus sampling. Default: `0.95`. |
| `--n-ctx` | Context window size. Default: `8192`. | | `--top-k` | Top-k sampling. Default: `64`. |
| `--seed` | Random seed. Default: `-1` (random at runtime). | | `--n-ctx` | Context window size. Default: `8192`. |
| `--help, -h` | Print usage and exit. | | `--seed` | Random seed. Default: `-1` (random at runtime). |
| `--n-gpu-layers` | Number of model layers to offload to GPU. Default: `0`. |
| `--help, -h` | Print usage and exit. |
`--mocked` and `--model` are mutually exclusive. Omitting both exits with an `--mocked` and `--model` are mutually exclusive. Omitting both exits with an
error before the pipeline starts. Sampling flags are ignored when `--mocked` is error before the pipeline starts. Sampling flags are ignored when `--mocked` is
@@ -130,17 +133,20 @@ NVIDIA GPU.
### How it works ### How it works
The container uses a two-stage build. The first stage pulls prebuilt The container uses a two-stage build. The builder stage installs CMake/Ninja,
`libllama`, `libggml`, and backend plugin libraries (including `libggml-cuda.so` clones the matching llama.cpp release tag for its headers only (installed into
and the CPU variant plugins) from `ghcr.io/ggml-org/llama.cpp:full-cuda`. The `/usr/local/include`), and copies prebuilt shared libraries (`libllama`,
second stage copies those libraries into `/usr/local/lib` and runs `ldconfig` so `libggml`, and CUDA/CPU backend plugins) from `ghcr.io/ggml-org/llama.cpp:full-cuda`
the dynamic linker and `dlopen` calls from `ggml_backend_load_all()` can resolve into `/usr/local/lib`. With both headers and libraries present, CMake's
the CUDA backend plugin at runtime. llama.cpp headers are cloned at the matching system-library detection (see [Build](#build) above) finds them and skips the
tag and installed into `/usr/local/include`. CMake auto-detects both and skips FetchContent source build, keeping image build times short.
the FetchContent source build entirely, keeping image build times short.
`GGML_BACKEND_PATH` is set to `/usr/local/lib` so llama.cpp knows where to scan The runtime stage copies the compiled binary, the same prebuilt shared
for backend plugins. libraries, and config/prompt assets into a slim CUDA runtime image. It sets
`LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH` so the dynamic linker
resolves `libllama`/`libggml` at startup, and also co-locates
`libggml-cuda.so` and the CPU backend plugins next to the binary for
`ggml_backend_load_all()`'s `dlopen` scan.
### Build the image ### Build the image
@@ -165,44 +171,46 @@ docker build \
Look for `[biergarten] Found system llama.cpp — skipping FetchContent` in the Look for `[biergarten] Found system llama.cpp — skipping FetchContent` in the
output to confirm the fast path was taken. output to confirm the fast path was taken.
### Run in mocked mode ### Run the container
No model or GPU required. Useful for validating the pipeline logic and SQLite The container always runs the model-backed path; there is no `--mocked`
export path. container mode (use a native build for that — see [Quick Start](#quick-start)).
The entrypoint, `runpod/start.sh`, downloads the GGUF model automatically if
```bash it is not already present at the configured path.
docker run --rm \
-e BIERGARTEN_MODE=mocked \
-v "$PWD/output:/workspace/output" \
-v "$PWD/logs:/workspace/logs" \
biergarten-pipeline:latest
```
### Run in live mode
Mount your GGUF model before starting. The container validates the model path
before launching the binary.
```bash ```bash
docker run --rm \ docker run --rm \
--runtime=nvidia \ --runtime=nvidia \
-e BIERGARTEN_MODE=live \
-e GGML_BACKEND_PATH="/usr/local/lib/libggml-cuda.so" \
-v "$PWD/models:/workspace/models" \ -v "$PWD/models:/workspace/models" \
-v "$PWD/output:/workspace/output" \ -v "$PWD/output:/workspace/output" \
-v "$PWD/logs:/workspace/logs" \ -v "$PWD/logs:/workspace/logs" \
biergarten-pipeline:latest biergarten-pipeline:latest
``` ```
The model must be present at `./models/google_gemma-4-E4B-it-Q6_K.gguf` on the By default this downloads `google_gemma-4-E4B-it-Q6_K.gguf` to
host. See [Model](#model) above for the download command. `./models/` on first run if it isn't already there. To use a pre-downloaded
model, place it at that path first — see [Model](#model) above.
#### Environment variables
| Variable | Purpose |
| ------------------------- | ----------------------------------------------------------------- |
| `BIERGARTEN_MODEL_PATH` | GGUF model path. Default: `/workspace/models/google_gemma-4-E4B-it-Q6_K.gguf`. |
| `BIERGARTEN_OUTPUT_DIR` | SQLite output directory. Default: `/workspace/output`. |
| `BIERGARTEN_LOG_PATH` | Log file path. Default: `/workspace/logs/pipeline.log`. |
| `BIERGARTEN_GL_LAYERS` | GPU layers to offload (`--n-gpu-layers`). Default: `40`. |
| `BIERGARTEN_TEMPERATURE`, `BIERGARTEN_TOP_P`, `BIERGARTEN_TOP_K`, `BIERGARTEN_N_CTX`, `BIERGARTEN_SEED` | Optional sampling overrides, unset by default (binary defaults apply). |
| `BIERGARTEN_EXTRA_ARGS` | Additional raw CLI args appended verbatim. |
`--prompt-dir` is hardcoded to `/app/prompts` inside the container and is not
configurable via environment variable.
### RunPod deployment ### RunPod deployment
Use a GPU pod template. Mount persistent storage for `/workspace/models`, Use a GPU pod template. Mount persistent storage for `/workspace/models`,
`/workspace/output`, and `/workspace/logs`. Set `BIERGARTEN_MODE=live` in the `/workspace/output`, and `/workspace/logs`. See
template environment. See `tooling/pipeline/runpod/pod-template.yaml` for a `tooling/pipeline/runpod/pod-template.yaml` for a starter template — set the
starter template. environment variables listed above to match your run.
--- ---
@@ -213,8 +221,8 @@ starter template.
| Stage | Implementation | | Stage | Implementation |
| -------- | --------------------------------------------------------------------------------------------------------------------------------------- | | -------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| Load | `JsonLoader::LoadLocations()` reads `locations.json` into typed `Location` records. | | Load | `JsonLoader::LoadLocations()` reads `locations.json` into typed `Location` records. |
| Sample | `BiergartenDataGenerator::QueryCitiesWithCountries()` samples up to 50 locations per run. | | Sample | `BiergartenPipelineOrchestrator::QueryCitiesWithCountries()` samples `--location-count` locations per run (default `10`). |
| Enrich | `WikipediaService` fetches city and beer context. Keeps going when a lookup fails. | | Enrich | `WikipediaEnrichmentService` fetches brewing and beer-related context. Keeps going when a lookup fails. `--mocked` runs use `MockEnrichmentService` instead and skip Wikipedia entirely. |
| Generate | `MockGenerator` or `LlamaGenerator` produces brewery names and descriptions in English and the local language. | | Generate | `MockGenerator` or `LlamaGenerator` produces brewery names and descriptions in English and the local language. |
| Store | `SqliteExportService` writes each successful brewery into a fresh dated `.sqlite` database with normalized location and brewery tables. | | Store | `SqliteExportService` writes each successful brewery into a fresh dated `.sqlite` database with normalized location and brewery tables. |
| Log | `spdlog` writes results and warnings to the console. | | Log | `spdlog` writes results and warnings to the console. |
@@ -226,11 +234,13 @@ pipeline continues.
- `src/main.cc` — argument parsing and Boost.DI composition root. - `src/main.cc` — argument parsing and Boost.DI composition root.
- `JsonLoader` — validates curated location input. - `JsonLoader` — validates curated location input.
- `WikipediaService` — queries Wikipedia extracts, caches results, returns empty - `WikipediaEnrichmentService` — queries Wikipedia extracts, caches results,
context on failure. returns empty context on failure. `MockEnrichmentService` is the no-op
substitute used in `--mocked` runs.
- `LlamaGenerator` — formats prompts for Gemma 4, validates JSON output, retries - `LlamaGenerator` — formats prompts for Gemma 4, validates JSON output, retries
malformed responses up to three times. If output looks truncated, the retry malformed responses up to three times with corrective feedback in the
raises the token budget before trying again. retry prompt. The token budget is fixed across attempts; it is not raised
automatically on truncation.
- `MockGenerator` — stable hash-based output so the same city input always - `MockGenerator` — stable hash-based output so the same city input always
produces the same brewery. produces the same brewery.
- `SqliteExportService` — creates a dated SQLite file per run and persists each - `SqliteExportService` — creates a dated SQLite file per run and persists each
@@ -240,18 +250,18 @@ pipeline continues.
### Runtime Behaviour ### Runtime Behaviour
`WikipediaService` queries city, country, and beer-related Wikipedia extracts `WikipediaEnrichmentService` fetches two Wikipedia extracts per city: a
using its configured lookup, then caches the first successful response per query generic "brewing" extract and a "beer in `{country}`" extract. It does not
string. The fetched extract text is included in the prompt as context for currently query a city- or region-specific page. Each query string is cached
generation. after its first successful (or empty) lookup.
`GetLocationContext()` returns an empty string when the web client is `GetLocationContext()` returns an empty string when the web client is
unavailable or when lookup/parsing fails. unavailable or when lookup/parsing fails.
`LlamaGenerator` validates model output as structured JSON. The retry path `LlamaGenerator` validates model output as structured JSON. On validation
exists as a safety hatch for cases where the reasoning block consumes available failure it retries up to three times, replaying the previous error message in
token budget and compresses the JSON output space. All runs to date have the next prompt so the model can self-correct. All runs to date have produced
produced valid output on the first pass; the path is kept for resilience. valid output on the first pass; the retry path is kept for resilience.
`MockGenerator` uses stable hashes for repeatable output in demos and Storybook `MockGenerator` uses stable hashes for repeatable output in demos and Storybook
runs. runs.
@@ -292,7 +302,6 @@ code, latitude, and longitude for each entry.
| `local_languages` | Locale-aware copy selection | | `local_languages` | Locale-aware copy selection |
| `name_en`, `description_en` | Default English display content | | `name_en`, `description_en` | Default English display content |
| `name_local`, `description_local` | Local-language display content | | `name_local`, `description_local` | Local-language display content |
| `region_context` | Richer copy for cards and detail pages |
--- ---
@@ -368,29 +377,31 @@ Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present.
## Repo Layout ## Repo Layout
| Path | Purpose | | Path | Purpose |
| ---------------------------- | -------------------------------------------------- | | -------------------------------------- | -------------------------------------------------- |
| `includes/` | Public headers and shared models. | | `tooling/pipeline/includes/` | Public headers and shared models. |
| `src/` | Implementation files. | | `tooling/pipeline/src/` | Implementation files. |
| `locations.json` | Curated city input copied into the build tree. | | `tooling/pipeline/locations.json` | Curated city input copied into the build tree. |
| `prompts/` | System prompts used by the model-backed path. | | `tooling/pipeline/prompts/` | System prompts used by the model-backed path. |
| `diagrams/` | Architecture and pipeline diagrams. | | `tooling/pipeline/runpod/` | Dockerfile, launcher, and RunPod pod template. |
| `tooling/pipeline/runpod/` | Dockerfile, launcher, and RunPod pod template. | | `docs/pipeline/diagrams/` | Architecture and pipeline diagrams. |
| `ETHICS-AND-KNOWN-ISSUES.md` | Ethics, bias, hallucination analysis, mitigations. | | `docs/pipeline/ETHICS-AND-KNOWN-ISSUES.md` | Ethics, bias, hallucination analysis, mitigations. |
--- ---
## Code Tour ## Code Tour
Paths below are relative to `tooling/pipeline/`.
- `src/main.cc` — argument parsing and DI composition root. - `src/main.cc` — argument parsing and DI composition root.
- `src/biergarten_data_generator/` — orchestration, sampling, logging, and - `src/biergarten_pipeline_orchestrator/` — orchestration, sampling, logging,
export. and export.
- `src/services/wikipedia/` — enrichment service and cache. - `src/services/enrichment/wikipedia/` — enrichment service and cache.
- `src/services/sqlite/` — SQLite export implementation. - `src/services/sqlite/` — SQLite export implementation.
- `src/data_generation/llama/` — local inference, prompt loading, output - `src/data_generation/llama/` — local inference, prompt loading, output
validation. validation.
- `src/data_generation/mock/` — deterministic fallback. - `src/data_generation/mock/` — deterministic fallback.
- `tooling/pipeline/runpod/` — container build and runtime launcher. - `runpod/` — container build and runtime launcher.
--- ---
@@ -398,7 +409,9 @@ Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present.
The pipeline currently produces city-aware brewery records and dated SQLite The pipeline currently produces city-aware brewery records and dated SQLite
exports. The next passes add additional fixture types so the app can exercise exports. The next passes add additional fixture types so the app can exercise
the full brewery domain without live data. the full brewery domain without live data. For the detailed engineering
breakdown of what's needed to reach the architecture in
[`diagrams/planned/`](./diagrams/planned/), see [ROADMAP.md](./ROADMAP.md).
### Testing — Very High Priority ### Testing — Very High Priority

211
docs/pipeline/ROADMAP.md Normal file
View File

@@ -0,0 +1,211 @@
# Pipeline Roadmap — Reaching the Planned Architecture
This is the engineering breakdown for closing the gap between the **current**
pipeline and the **planned** architecture documented in
[`diagrams/planned/class.puml`](./diagrams/planned/class.puml) and
[`diagrams/planned/activity.puml`](./diagrams/planned/activity.puml). Nothing
in `diagrams/planned/` is implemented yet — this file tracks what it would
take to get there. For the current implementation, see
[README.md](./README.md).
Items are grouped by layer and roughly ordered by dependency: later groups
build on types and services introduced earlier.
The only concurrency in the current pipeline is the log dispatcher thread
(`main.cc` spawns it to drain a `BoundedChannel<LogEntry>` into spdlog for the
whole run). Sampling, enrichment, generation, and export all happen
synchronously on the main thread inside `BiergartenPipelineOrchestrator::Run()`
— §6 below is what introduces the additional worker threads the planned
architecture needs.
---
## 1. Domain Models
`tooling/pipeline/includes/data_model/generated_models.h` and `models.h`.
- [ ] Add `Completeness` enum (`Full`, `Partial`, `Absent`) and a
`LocationContext` struct (`text`, `completeness`, `char_count`).
Replace `EnrichedCity::region_context` (currently a plain
`std::string`) with `context : LocationContext`.
- [ ] Add `BeerStyle` (`name`, `description`, `min_abv`, `max_abv`,
`min_ibu`, `max_ibu`).
- [ ] Add `BeerResult`, `CheckinResult`, `RatingResult` result payloads.
- [ ] Add `GenerationMetadata` (`generation_id`, `generated_time`,
`context_provided`, `generated_with`).
- [x] Add `first_name`, `last_name`, `gender`, and `activity_weight` to
`UserResult` (currently just `username`, `bio`). `first_name`/
`last_name`/`gender` are copied from the sampled `Name` (see below),
not LLM-invented.
- [x] Add `Name` (`first_name`, `last_name`, `gender`) — the sampled result
handed to `DataGenerator::GenerateUser`. Add `ForenameEntry` (`name`,
`gender`) and a `NamesByCountry` class (not a bare typedef) that owns
two maps — `std::unordered_map<std::string, std::vector<ForenameEntry>>`
and `std::unordered_map<std::string, std::vector<std::string>>`, both
keyed by ISO 3166-1 code — and exposes `SampleName(iso3166_1, rng)`.
Pairing a forename with a surname happens at sample time so gender
(present per-forename in the source data) is never discarded the way a
pre-flattened `{first_name, last_name}` list would lose it; see
`LoadNamesByCountry()` below.
- [ ] Extend `GeneratedBrewery` with `brewery_id`, `context_completeness`,
`metadata` (currently just `{location, brewery}`).
- [ ] Add `GeneratedBeer`, `GeneratedCheckin`, `GeneratedRating`,
`GeneratedFollow` aggregate structs.
- [x] Add `GeneratedUser` aggregate struct. Carries `email`, `date_of_birth`,
and `password` — programmatically generated by the orchestrator, never
LLM-authored — so a downstream auth-account seeding consumer (outside
this repo tree's docs) can register real accounts from the pipeline's
SQLite export. Skips `user_id`/`metadata` for now, matching
`GeneratedBrewery`'s current (pre-`brewery_id`/`metadata`) shape. Not
yet wired into `IExportService`/`SqliteExportService` — `GenerateUsers`
currently only logs and holds `generated_users_` in memory; no `users`
table exists in the SQLite export yet (see §5).
- [x] Add `UserPersona` (`name`, `description`, `style_affinities`).
## 2. Data Preloading
`tooling/pipeline/includes/json_handling/`, fixture files.
- [ ] Extract a `DataPreloader` interface; have `JsonLoader` implement it.
`JsonLoader` now also exposes static `LoadPersonas()` and
`LoadNamesByCountry()` (see below), but as additional static methods,
not yet behind a formal interface.
- [ ] Add `LoadBeerStyles()`. `beer-styles.json` already exists in the repo
and is already copied into the Docker image
(`runpod/Dockerfile`), but no loader reads it yet, and the native
CMake build doesn't copy it into `build/` at all — `CMakeLists.txt`'s
"Runtime Assets" step only copies `locations.json` and `prompts/`.
- [x] Add `LoadPersonas()` and author `personas.json` (doesn't exist yet).
- [x] Add `LoadNamesByCountry(forenames_filepath, surnames_filepath)`,
parsing `tooling/pipeline/forenames-by-country.json` and
`surnames-by-country.json` into a `NamesByCountry`. Both files are
vendored verbatim (unmodified, full multinational coverage) from
`sigpwned/popular-names-by-country-dataset` (CC0) — see
ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for
provenance. Deliberately not pre-paired or filtered down to just the
countries in `locations.json`: keeping the source shape (including
per-forename gender) intact means the loader can support more
countries later, or gender-aware persona/bio generation, without
re-sourcing data.
## 3. Policy / Strategy Layer
Entirely new — no `includes/policy/` (or equivalent) directory exists today.
- [ ] `ContextStrategy` interface + `BreweryContextStrategy` /
`BeerContextStrategy`. Today `WikipediaEnrichmentService::GetLocationContext`
hardcodes a generic `"brewing"` query and a `"beer in {country}"`
query directly — no per-phase strategy selection.
- [ ] `SamplingStrategy` interface + `UniformSamplingStrategy`, replacing
the inline `std::ranges::sample(...)` call in
`BiergartenPipelineOrchestrator::QueryCitiesWithCountries()`.
- [ ] `BeerSelectionStrategy` interface + `RandomBeerSelectionStrategy`, to
pick styles per brewery from the `BeerStyle` palette (depends on §1
and §2).
- [ ] `CheckinDistributionStrategy` interface + `JCurveCheckinStrategy` /
`RandomCheckinStrategy` — this is the "Check-In System" item already
called out in README.md's Next Steps, made concrete.
- [ ] `FollowGenerationStrategy` interface + `RandomFollowStrategy` /
`ActivityWeightedFollowStrategy`.
## 4. Data Generation
`tooling/pipeline/includes/data_generation/`, `src/data_generation/`.
- [ ] Extend the `DataGenerator` interface with `GenerateBeer`,
`GenerateCheckin`, `GenerateRating` (today: only `GenerateBrewery` and
`GenerateUser`).
- [x] Implement `LlamaGenerator::GenerateUser` for real, with a retry loop
mirroring `GenerateBrewery` (GBNF grammar, `ValidateUserJson`, up to 3
attempts with corrective feedback) and a new
`prompts/USER_GENERATION.md`. Changed the `DataGenerator::GenerateUser`
signature to
`(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult`,
matching what the activity diagram actually passes. `MockGenerator::GenerateUser`
updated to match the new signature too.
- [ ] Implement `MockGenerator::GenerateBeer` / `GenerateCheckin` /
`GenerateRating`.
- [ ] Add `IPromptFormatter::ExpectedArchitecture()` and
`LlamaGenerator::ValidateModelArchitecture()` so loading a GGUF that
doesn't match the configured chat template fails fast instead of
silently producing degraded output.
- [ ] Add prompt template files for beer/checkin/rating generation next to
the existing `prompts/BREWERY_GENERATION.md`.
## 5. Export Service
`tooling/pipeline/includes/services/database/`, `src/services/sqlite/`.
- [ ] Extend `IExportService` with `ProcessBeer`, `ProcessUser`,
`ProcessCheckin`, `ProcessRating`, `ProcessFollow` (today only
`ProcessRecord(const GeneratedBrewery&)` exists).
- [ ] Add `beers`, `users`, `checkins`, `ratings`, `follows` tables to
`SqliteExportService::InitializeSchema()` — see
`kCreateLocationsTableSql` / `kCreateBreweriesTableSql` in
`includes/services/database/sqlite_statement_helpers.h` for the
existing pattern to follow.
- [ ] Add a `brewery_cache_` alongside the existing `location_cache_`, and
move from one open transaction for the whole run to per-phase batched
commits (`BEGIN` / `COMMIT & BEGIN` on a batch-size threshold), as
shown in the planned activity diagram.
## 6. Concurrency & Orchestration
`tooling/pipeline/includes/concurrency/`,
`src/biergarten_pipeline_orchestrator/`.
- [ ] `BoundedChannel<T>` already exists and is production-tested — but
it's only wired to the single log channel today. Stand up the
per-phase producer/consumer channels (`loc_ch`, `exp_ch`, etc.) the
planned activity diagram describes, each with a dedicated LLM worker
thread and SQLite worker thread.
- [ ] Rewrite `BiergartenPipelineOrchestrator::Run()` from one synchronous
pass over `generated_breweries_` into phased methods —
`RunUserPhase` → `RunBreweryAndBeerPhase` (with a `RunBeerPhase`
sub-step once `brewery_pool_` is populated) → `RunCheckinPhase` /
`RunFollowPhase` (these two can run in parallel) → `RunRatingPhase` —
backed by `user_pool_`, `brewery_pool_`, `beer_pool_`,
`checkin_pool_`, `follow_pool_`.
- [ ] Honor the structural-concurrency requirement already called out in a
comment on `BiergartenPipelineOrchestrator::Run()` in
`biergarten_pipeline_orchestrator.h`: once real worker threads exist,
they must be structurally joined (e.g. via `std::jthread`) before
`Run()` returns, so no worker logs to a closed channel during
teardown.
## 7. Enrichment
- [ ] Decide whether to restore the city/region-specific Wikipedia query
that's currently commented out in
`src/services/enrichment/wikipedia/get_summary.cc`
(`GetLocationContext`). The `ContextStrategy` work in §3 is a natural
place to reintroduce it via `BreweryContextStrategy::QueriesFor()`.
- [ ] Pre-warm caches at startup (`PreWarmBeerStyleCache`,
`PreWarmLocationCache` in the planned activity diagram) instead of
fetching lazily per record, so the streaming phases never block on a
cold cache mid-run.
## 8. Fixtures / Build
- [ ] Author `personas.json` and `names-by-country.json` (see §2).
- [ ] Fix the `beer-styles.json` build-tree gap: add it to the "Runtime
Assets" `configure_file` step in `CMakeLists.txt` so native builds and
the Docker image agree on what's available at runtime.
---
## Suggested build order
The planned activity diagram's own phase notes ("brewery*pool* is now fully
populated. Phase 1b may begin.", etc.) imply a dependency order. Roughly:
1. Domain models (§1) and data preloading (§2) — nothing else compiles
without these.
2. Export schema (§5) — so every later generation phase has somewhere to
land its output.
3. Policy/strategy layer (§3) and generator interface (§4).
4. Concurrency/orchestration rewrite (§6), which is the only piece that
actually wires §1–§5 into the phased, parallel pipeline shown in the
planned diagrams.
5. Enrichment cache pre-warming and the city/region query decision (§7) —
useful at any point, but most valuable once phases run concurrently.

View File

@@ -18,23 +18,40 @@ skinparam ActivityBarColor #628A5B
skinparam SwimlaneBorderColor #547461 skinparam SwimlaneBorderColor #547461
skinparam SwimlaneBorderThickness 0.3 skinparam SwimlaneBorderThickness 0.3
title The Biergarten Data Pipeline (Streaming Architecture) title The Biergarten Data Pipeline (Current — Synchronous Data Path)
|#F2F6F0|main.cc| |#F2F6F0|main.cc|
start start
:Create BoundedChannel<LogEntry> log_channel;
:Spawn log dispatcher thread\n(LogDispatcher::Run());
note right
The only concurrency in the current pipeline:
a dedicated thread drains log_channel and
forwards entries to spdlog for the entire run.
Joined during shutdown after log_channel closes.
end note
:ParseArguments(argc, argv); :ParseArguments(argc, argv);
if (Are arguments valid?) then (no) if (Are arguments valid?) then (no)
:spdlog::error usage info; :Log error / usage info;
stop stop
else (yes) else (yes)
endif endif
:Init OpenSSL global state & LlamaBackendState; :Init OpenSSL global state & LlamaBackendState;
:di::make_injector(...); :di::make_injector(...);
:injector.create<std::unique_ptr<BiergartenDataGenerator>>(); note right
:BiergartenDataGenerator::Run(); Binds ILogger, IEnrichmentService, DataGenerator,
IExportService, IPromptFormatter, WebClient based
on --mocked vs. model-backed mode.
end note
:injector.create<std::unique_ptr<BiergartenPipelineOrchestrator>>();
:BiergartenPipelineOrchestrator::Run();
note right
Run() itself is synchronous — sampling, enrichment,
generation, and export all happen on this thread.
end note
|#EAF0E8|BiergartenDataGenerator| |#EAF0E8|BiergartenPipelineOrchestrator|
:Initialize SQLite export; :Initialize SQLite export;
|#E0EAE0|SqliteExportService| |#E0EAE0|SqliteExportService|
@@ -48,23 +65,35 @@ note right
Begins Transaction Begins Transaction
end note end note
|#EAF0E8|BiergartenDataGenerator| |#EAF0E8|BiergartenPipelineOrchestrator|
:QueryCitiesWithCountries(); :QueryCitiesWithCountries();
|#E2EBDC|JsonLoader| |#E2EBDC|JsonLoader|
:JsonLoader::LoadLocations("locations.json"); :JsonLoader::LoadLocations("locations.json");
:std::ranges::sample(all_locations, 50); :std::ranges::sample(all_locations, --location-count);
note right
--location-count defaults to 10.
end note
|#EAF0E8|BiergartenDataGenerator| |#EAF0E8|BiergartenPipelineOrchestrator|
while (For each sampled Location?) is (Remaining cities) while (For each sampled Location?) is (Remaining cities)
|#DCE8D8|WikipediaService| |#DCE8D8|IEnrichmentService|
:GetLocationContext(loc); :GetLocationContext(loc);
:FetchExtracts(City, Country, Beer); note right
|#EAF0E8|BiergartenDataGenerator| WikipediaEnrichmentService fetches a generic
:Store EnrichedCity{Location, region_context}; "brewing" extract and a "beer in {country}" extract
(not a city/region-specific page). MockEnrichmentService
(--mocked) returns an empty string instead.
end note
|#EAF0E8|BiergartenPipelineOrchestrator|
if (Lookup failed?) then (yes)
:Log warning, skip city;
else (no)
:Store EnrichedCity{Location, region_context};
endif
endwhile (Done) endwhile (Done)
|#EAF0E8|BiergartenDataGenerator| |#EAF0E8|BiergartenPipelineOrchestrator|
:GenerateBreweries(enriched_cities); :GenerateBreweries(enriched_cities);
|#E5EDE1|DataGenerator| |#E5EDE1|DataGenerator|
@@ -73,7 +102,10 @@ while (For each EnrichedCity?) is (Remaining cities)
:DeterministicHash & Format; :DeterministicHash & Format;
else (LlamaGenerator) else (LlamaGenerator)
:PrepareRegionContext; :PrepareRegionContext;
:LoadBrewerySystemPrompt("prompts/system.md"); :prompt_directory_->Load("BREWERY_GENERATION");
note right
Resolves to BREWERY_GENERATION.md inside --prompt-dir.
end note
repeat repeat
:Infer(system_prompt, user_prompt, max_tokens, kBreweryJsonGrammar); :Infer(system_prompt, user_prompt, max_tokens, kBreweryJsonGrammar);
:ValidateBreweryJson(raw, brewery); :ValidateBreweryJson(raw, brewery);
@@ -81,11 +113,16 @@ while (For each EnrichedCity?) is (Remaining cities)
break break
else (no) else (no)
:Attempt++; :Attempt++;
:Append validation error to retry prompt;
endif endif
repeat while (Attempt < 3?) is (yes) repeat while (Attempt < 3?) is (yes)
note right
max_tokens is fixed across retries —
it is not raised on truncation.
end note
endif endif
|#EAF0E8|BiergartenDataGenerator| |#EAF0E8|BiergartenPipelineOrchestrator|
if (Generation successful?) then (yes) if (Generation successful?) then (yes)
|#E0EAE0|SqliteExportService| |#E0EAE0|SqliteExportService|
:ProcessRecord(GeneratedBrewery); :ProcessRecord(GeneratedBrewery);
@@ -97,8 +134,8 @@ while (For each EnrichedCity?) is (Remaining cities)
:Insert Brewery (FK: location_id); :Insert Brewery (FK: location_id);
if (Exception caught during insert?) then (yes) if (Exception caught during insert?) then (yes)
|#EAF0E8|BiergartenDataGenerator| |#EAF0E8|BiergartenPipelineOrchestrator|
:spdlog::warn "Failed to stream record to SQLite export"; :Log warning "SQLite export failed";
note right note right
Data loss is prevented per-record. Data loss is prevented per-record.
The pipeline continues running. The pipeline continues running.
@@ -106,7 +143,7 @@ while (For each EnrichedCity?) is (Remaining cities)
else (no) else (no)
endif endif
else (no) else (no)
:spdlog::warn "Generation failed, skipping..."; :Log warning "Generation failed, skipping...";
endif endif
|#E5EDE1|DataGenerator| |#E5EDE1|DataGenerator|
endwhile (Done) endwhile (Done)
@@ -119,6 +156,8 @@ note right
end note end note
|#F2F6F0|main.cc| |#F2F6F0|main.cc|
:Close log_channel;
:Join log dispatcher thread;
:Return 0; :Return 0;
stop stop

View File

@@ -25,11 +25,12 @@ skinparam note {
title The Biergarten Data Pipeline - Class Diagram title The Biergarten Data Pipeline - Class Diagram
class BiergartenDataGenerator { class BiergartenPipelineOrchestrator {
- logger_ : std::shared_ptr<ILogger> - logger_ : std::shared_ptr<ILogger>
- context_service_ : std::unique_ptr<IEnrichmentService> - context_service_ : std::unique_ptr<IEnrichmentService>
- generator_ : std::unique_ptr<DataGenerator> - generator_ : std::unique_ptr<DataGenerator>
- exporter_ : std::unique_ptr<IExportService> - exporter_ : std::unique_ptr<IExportService>
- application_options_ : ApplicationOptions
- generated_breweries_ : std::vector<GeneratedBrewery> - generated_breweries_ : std::vector<GeneratedBrewery>
+ Run() : bool + Run() : bool
- QueryCitiesWithCountries() : std::vector<Location> - QueryCitiesWithCountries() : std::vector<Location>
@@ -54,21 +55,29 @@ class PipelinePhase <<enumeration>> {
Teardown Teardown
} }
struct LogEntry { struct LogDTO {
+ timestamp : std::chrono::system_clock::time_point + level : LogLevel
+ phase : PipelinePhase
+ message : std::string
}
struct LogEntry {
+ timestamp : std::chrono::system_clock::time_point
+ origin : std::source_location
+ thread_id : std::thread::id
+ level : LogLevel + level : LogLevel
+ phase : PipelinePhase + phase : PipelinePhase
+ message : std::string + message : std::string
+ worker : std::optional<std::string>
} }
interface ILogger <<interface>> { interface ILogger <<interface>> {
+ Log(entry : const LogEntry&) : void + Log(payload : LogDTO) : void
- {abstract} DoLog(entry : LogEntry) : void
} }
class LogProducer { class LogProducer {
- channel_ : BoundedChannel<LogEntry>& - channel_ : BoundedChannel<LogEntry>&
+ Log(entry : const LogEntry&) : void - DoLog(entry : LogEntry) : void
} }
class LogDispatcher { class LogDispatcher {
@@ -81,7 +90,11 @@ interface IEnrichmentService <<interface>> {
+ GetLocationContext(loc : const Location&) : std::string + GetLocationContext(loc : const Location&) : std::string
} }
class WikipediaService { class MockEnrichmentService {
+ GetLocationContext(loc : const Location&) : std::string
}
class WikipediaEnrichmentService {
- client_ : std::unique_ptr<WebClient> - client_ : std::unique_ptr<WebClient>
- extract_cache_ : std::unordered_map<std::string, std::string> - extract_cache_ : std::unordered_map<std::string, std::string>
+ GetLocationContext(loc : const Location&) : std::string + GetLocationContext(loc : const Location&) : std::string
@@ -113,13 +126,13 @@ class LlamaGenerator {
- model_ : ModelHandle - model_ : ModelHandle
- context_ : ContextHandle - context_ : ContextHandle
- prompt_formatter_ : std::unique_ptr<IPromptFormatter> - prompt_formatter_ : std::unique_ptr<IPromptFormatter>
- prompt_directory_ : std::unique_ptr<IPromptDirectory>
- rng_ : std::mt19937 - rng_ : std::mt19937
+ GenerateBrewery(...) : BreweryResult + GenerateBrewery(...) : BreweryResult
+ GenerateUser(...) : UserResult + GenerateUser(...) : UserResult
- Load(model_path : const std::string&) : void - Load(model_path : const std::string&) : void
- Infer(...) : std::string - Infer(...) : std::string
- InferFormatted(...) : std::string - InferFormatted(...) : std::string
- LoadBrewerySystemPrompt(...) : std::string
} }
interface IPromptFormatter <<interface>> { interface IPromptFormatter <<interface>> {
@@ -130,6 +143,16 @@ class Gemma4JinjaPromptFormatter {
+ Format(system_prompt : std::string_view, user_prompt : std::string_view) : std::string + Format(system_prompt : std::string_view, user_prompt : std::string_view) : std::string
} }
interface IPromptDirectory <<interface>> {
+ Load(key : std::string_view) : std::string
}
class PromptDirectory {
- prompt_dir_ : std::filesystem::path
- cache_ : std::unordered_map<std::string, std::string>
+ Load(key : std::string_view) : std::string
}
class JsonLoader { class JsonLoader {
+ {static} LoadLocations(filepath : const std::filesystem::path&) : std::vector<Location> + {static} LoadLocations(filepath : const std::filesystem::path&) : std::vector<Location>
} }
@@ -164,19 +187,22 @@ class SystemDateTimeProvider {
} }
' Structural Relationships / Dependency Injection ' Structural Relationships / Dependency Injection
BiergartenDataGenerator *-- ILogger : owns BiergartenPipelineOrchestrator *-- ILogger : owns
BiergartenDataGenerator *-- IEnrichmentService : owns BiergartenPipelineOrchestrator *-- IEnrichmentService : owns
BiergartenDataGenerator *-- DataGenerator : owns BiergartenPipelineOrchestrator *-- DataGenerator : owns
BiergartenDataGenerator *-- IExportService : owns BiergartenPipelineOrchestrator *-- IExportService : owns
LogEntry *-- LogLevel LogEntry *-- LogLevel
LogEntry *-- PipelinePhase LogEntry *-- PipelinePhase
LogDTO *-- LogLevel
LogDTO *-- PipelinePhase
ILogger <|.. LogProducer : implements ILogger <|.. LogProducer : implements
LogProducer ..> LogEntry : emits LogProducer ..> LogEntry : emits
LogDispatcher ..> LogEntry : consumes LogDispatcher ..> LogEntry : consumes
IEnrichmentService <|.. WikipediaService : implements IEnrichmentService <|.. WikipediaEnrichmentService : implements
WikipediaService *-- WebClient : owns IEnrichmentService <|.. MockEnrichmentService : implements
WikipediaEnrichmentService *-- WebClient : owns
WebClient <|.. HttpWebClient : implements WebClient <|.. HttpWebClient : implements
@@ -184,10 +210,12 @@ DataGenerator <|.. MockGenerator : implements
DataGenerator <|.. LlamaGenerator : implements DataGenerator <|.. LlamaGenerator : implements
LlamaGenerator *-- IPromptFormatter : uses LlamaGenerator *-- IPromptFormatter : uses
LlamaGenerator *-- IPromptDirectory : uses
IPromptFormatter <|.. Gemma4JinjaPromptFormatter : implements IPromptFormatter <|.. Gemma4JinjaPromptFormatter : implements
IPromptDirectory <|.. PromptDirectory : implements
BiergartenDataGenerator ..> JsonLoader : uses BiergartenPipelineOrchestrator ..> JsonLoader : uses
IExportService <|.. SqliteExportService : implements IExportService <|.. SqliteExportService : implements
SqliteExportService *-- IDateTimeProvider : owns SqliteExportService *-- IDateTimeProvider : owns

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 44 KiB

After

Width:  |  Height:  |  Size: 55 KiB

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 44 KiB

After

Width:  |  Height:  |  Size: 71 KiB

View File

@@ -43,7 +43,7 @@ fork again
:EnrichmentService::PreWarmLocationCache(sampled_locations); :EnrichmentService::PreWarmLocationCache(sampled_locations);
end fork end fork
fork fork
:JsonLoader::LoadNamesByCountry("names-by-country.json"); :JsonLoader::LoadNamesByCountry(\n "forenames-by-country.json",\n "surnames-by-country.json");
fork again fork again
:JsonLoader::LoadPersonas("personas.json"); :JsonLoader::LoadPersonas("personas.json");
end fork end fork

View File

@@ -67,11 +67,47 @@ package "Domain: Models" {
} }
class UserResult { class UserResult {
+ first_name : std::string
+ last_name : std::string
+ gender : std::string
+ username : std::string + username : std::string
+ bio : std::string + bio : std::string
+ activity_weight : float + activity_weight : float
} }
class Name {
+ first_name : std::string
+ last_name : std::string
+ gender : std::string
}
class ForenameEntry {
+ name : std::string
+ gender : std::string
}
class NamesByCountry {
- forenames_by_country_ : std::unordered_map<std::string, std::vector<ForenameEntry>>
- surnames_by_country_ : std::unordered_map<std::string, std::vector<std::string>>
+ SampleName(iso3166_1 : const std::string&,\n rng : std::mt19937&) : std::optional<Name>
}
note right of NamesByCountry
Backed by two source-shaped maps,
keyed by ISO 3166-1 country code,
sourced from popular-names-by-country-dataset
(CC0) -- see ETHICS-AND-KNOWN-ISSUES.md.
Gender is preserved per forename so it can
be threaded through to persona/bio generation
later; surnames carry no gender in the source
data. Pairing happens at sample time, not at
fixture-authoring time, so no information from
the source dataset is discarded up front.
end note
NamesByCountry *-- "0..*" ForenameEntry
NamesByCountry ..> Name : produces
class CheckinResult { class CheckinResult {
+ checked_in_at : std::string + checked_in_at : std::string
+ note : std::string + note : std::string
@@ -110,9 +146,20 @@ package "Domain: Models" {
+ user_id : uint64_t + user_id : uint64_t
+ location : Location + location : Location
+ user : UserResult + user : UserResult
+ email : std::string
+ date_of_birth : std::string
+ password : std::string
+ metadata : GenerationMetadata + metadata : GenerationMetadata
} }
note right of GeneratedUser
email, date_of_birth, and password are
programmatically generated by the orchestrator
(not LLM-authored) so a downstream auth-account
seeding consumer can register real accounts from
this export. See ROADMAP.md §1.
end note
class GeneratedCheckin { class GeneratedCheckin {
+ checkin_id : uint64_t + checkin_id : uint64_t
+ user_id : uint64_t + user_id : uint64_t
@@ -321,15 +368,15 @@ package "Infrastructure: Data Preloading" {
interface DataPreloader <<interface>> { interface DataPreloader <<interface>> {
+ LoadLocations(filepath : const std::filesystem::path&) : std::vector<Location> + LoadLocations(filepath : const std::filesystem::path&) : std::vector<Location>
+ LoadBeerStyles(filepath : const std::filesystem::path&) : std::vector<BeerStyle> + LoadBeerStyles(filepath : const std::filesystem::path&) : std::vector<BeerStyle>
+ LoadPersonas(filepath : const std::filesystem::path&) : std::vector<Persona> + LoadPersonas(filepath : const std::filesystem::path&) : std::vector<UserPersona>
+ LoadNamesByCountry(filepath : const std::filesystem::path&) : NamesByCountry + LoadNamesByCountry(forenames_filepath : const std::filesystem::path&,\n surnames_filepath : const std::filesystem::path&) : NamesByCountry
} }
class JsonLoader { class JsonLoader {
+ LoadLocations(filepath : const std::filesystem::path&) : std::vector<Location> + LoadLocations(filepath : const std::filesystem::path&) : std::vector<Location>
+ LoadBeerStyles(filepath : const std::filesystem::path&) : std::vector<BeerStyle> + LoadBeerStyles(filepath : const std::filesystem::path&) : std::vector<BeerStyle>
+ LoadPersonas(filepath : const std::filesystem::path&) : std::vector<Persona> + LoadPersonas(filepath : const std::filesystem::path&) : std::vector<UserPersona>
+ LoadNamesByCountry(filepath : const std::filesystem::path&) : NamesByCountry + LoadNamesByCountry(forenames_filepath : const std::filesystem::path&,\n surnames_filepath : const std::filesystem::path&) : NamesByCountry
} }
} }
@@ -380,7 +427,7 @@ package "Infrastructure: Data Generation" {
interface DataGenerator <<interface>> { interface DataGenerator <<interface>> {
+ GenerateBrewery(location : const Location&,\n context : const LocationContext&) : BreweryResult + GenerateBrewery(location : const Location&,\n context : const LocationContext&) : BreweryResult
+ GenerateBeer(brewery_id : uint64_t,\n location : const Location&,\n context : const LocationContext&,\n style : const BeerStyle&) : BeerResult + GenerateBeer(brewery_id : uint64_t,\n location : const Location&,\n context : const LocationContext&,\n style : const BeerStyle&) : BeerResult
+ GenerateUser(location : const Location&) : UserResult + GenerateUser(city : const EnrichedCity&,\n persona : const UserPersona&,\n name : const Name&) : UserResult
+ GenerateCheckin(user : const GeneratedUser&,\n brewery : const GeneratedBrewery&,\n timestamp : const std::string&) : CheckinResult + GenerateCheckin(user : const GeneratedUser&,\n brewery : const GeneratedBrewery&,\n timestamp : const std::string&) : CheckinResult
+ GenerateRating(user : const GeneratedUser&,\n beer : const GeneratedBeer&,\n checkin_id : uint64_t) : RatingResult + GenerateRating(user : const GeneratedUser&,\n beer : const GeneratedBeer&,\n checkin_id : uint64_t) : RatingResult
} }

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 106 KiB

After

Width:  |  Height:  |  Size: 139 KiB

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 184 KiB

After

Width:  |  Height:  |  Size: 234 KiB

View File

@@ -0,0 +1,24 @@
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
lerna-debug.log*
node_modules
dist
dist-ssr
*.local
# Editor directories and files
.vscode/*
!.vscode/extensions.json
.idea
.DS_Store
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?

View File

@@ -0,0 +1,74 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<link rel="icon" type="image/svg+xml" href="/favicon.svg" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Pipeline Results Viewer</title>
</head>
<body>
<div class="dialog" id="window">
<div class="dialog-header">
<p>Pipeline Results Viewer</p>
</div>
<div class="tab-strip" role="tablist">
<button
type="button"
class="tab is-active"
id="tab-tables"
role="tab"
aria-selected="true"
aria-controls="panel-tables"
>
Tables
</button>
<button
type="button"
class="tab"
id="tab-query"
role="tab"
aria-selected="false"
aria-controls="panel-query"
>
Query
</button>
</div>
<div class="dialog-body">
<div
class="tab-panel"
id="panel-tables"
role="tabpanel"
aria-labelledby="tab-tables"
>
<div class="field-group">
<label for="table-select">Table</label>
<select id="table-select"></select>
</div>
<p id="status"></p>
<div id="table-container"></div>
</div>
<div
class="tab-panel"
id="panel-query"
role="tabpanel"
aria-labelledby="tab-query"
hidden
>
<div class="field-group field-group--stacked">
<label for="query-input">SQL Query</label>
<textarea
id="query-input"
rows="4"
spellcheck="false"
placeholder="SELECT * FROM ..."
></textarea>
</div>
<button type="button" id="query-run" class="btn">Run Query</button>
<p id="query-status"></p>
<div id="query-container"></div>
</div>
</div>
</div>
<script type="module" src="/src/main.ts"></script>
</body>
</html>

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,20 @@
{
"name": "pipeline-results-viewer",
"version": "0.0.0",
"private": true,
"type": "module",
"scripts": {
"dev": "vite",
"build": "tsc && vite build",
"preview": "vite preview"
},
"devDependencies": {
"@types/sql.js": "^1.4.11",
"sass": "^1.101.0",
"typescript": "^6.0.3",
"vite": "^8.0.12"
},
"dependencies": {
"sql.js": "^1.14.1"
}
}

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 9.3 KiB

View File

@@ -0,0 +1,147 @@
import initSqlJs, { type Database, type SqlValue } from "sql.js";
import sqlWasmUrl from "sql.js/dist/sql-wasm.wasm?url";
import "./style.scss";
const DB_URL = "/biergarten.sqlite";
const tabs = document.querySelectorAll<HTMLButtonElement>("[role='tab']");
const panels = document.querySelectorAll<HTMLDivElement>("[role='tabpanel']");
const tableSelect = document.querySelector<HTMLSelectElement>("#table-select")!;
const tableContainer =
document.querySelector<HTMLDivElement>("#table-container")!;
const status = document.querySelector<HTMLParagraphElement>("#status")!;
const queryInput = document.querySelector<HTMLTextAreaElement>("#query-input")!;
const queryRun = document.querySelector<HTMLButtonElement>("#query-run")!;
const queryStatus = document.querySelector<HTMLParagraphElement>("#query-status")!;
const queryContainer =
document.querySelector<HTMLDivElement>("#query-container")!;
for (const tab of tabs) {
tab.addEventListener("click", () => {
for (const otherTab of tabs) {
const isActive = otherTab === tab;
otherTab.classList.toggle("is-active", isActive);
otherTab.setAttribute("aria-selected", String(isActive));
}
for (const panel of panels) {
panel.hidden = panel.id !== tab.getAttribute("aria-controls");
}
});
}
const renderResultTable = (
container: HTMLDivElement,
columns: string[],
values: SqlValue[][],
) => {
const table = document.createElement("table");
const head = table.createTHead().insertRow();
for (const column of columns) {
const th = document.createElement("th");
th.textContent = column;
head.append(th);
}
const body = table.createTBody();
for (const row of values) {
const tr = body.insertRow();
for (const cell of row) {
const td = document.createElement("td");
td.textContent = String(cell ?? "");
tr.append(td);
}
}
container.append(table);
};
const renderTable = (db: Database, name: string) => {
const result = db.exec(`SELECT * FROM "${name}"`);
tableContainer.replaceChildren();
if (result.length === 0) {
tableContainer.append(
Object.assign(document.createElement("p"), { textContent: "No rows." }),
);
return;
}
const { columns, values } = result[0];
renderResultTable(tableContainer, columns, values);
};
const runQuery = (db: Database) => {
const sql = queryInput.value.trim();
queryContainer.replaceChildren();
if (!sql) {
queryStatus.textContent = "Enter a query to run.";
return;
}
try {
const results = db.exec(sql);
if (results.length === 0) {
queryStatus.textContent = "Query executed. No rows returned.";
return;
}
const { columns, values } = results[results.length - 1];
renderResultTable(queryContainer, columns, values);
queryStatus.textContent = `${values.length} row${values.length === 1 ? "" : "s"} returned.`;
} catch (error) {
queryStatus.textContent = `Error: ${error instanceof Error ? error.message : String(error)}`;
}
};
async function main(): Promise<void> {
status.textContent = "Loading database...";
const sqlJs = await initSqlJs({ locateFile: () => sqlWasmUrl });
const response = await fetch(DB_URL);
const buffer = await response.arrayBuffer();
const db = new sqlJs.Database(new Uint8Array(buffer));
const tables =
db
.exec(
`SELECT name
FROM sqlite_master
WHERE
type = 'table'
AND
name NOT LIKE 'sqlite_%'
ORDER BY name`,
)[0]
?.values.map((row) => String(row[0])) ?? [];
for (const name of tables) {
tableSelect.append(new Option(name, name));
}
tableSelect.addEventListener("change", () =>
renderTable(db, tableSelect.value),
);
if (tables.length > 0) {
renderTable(db, tables[0]);
}
status.textContent = "";
queryRun.addEventListener("click", () => runQuery(db));
queryInput.addEventListener("keydown", (event) => {
if ((event.metaKey || event.ctrlKey) && event.key === "Enter") {
event.preventDefault();
runQuery(db);
}
});
}
main().catch((error: unknown) => {
status.textContent = `Failed to load database: ${String(error)}`;
});

View File

@@ -0,0 +1,267 @@
$gray-100: #dedede;
$gray-300: #c3c3c3;
$gray-500: #808080;
$white: #ffffff;
$black: #000000;
$navy: #000080;
$blue: #1084d0;
$teal: #008080;
$color-surface: $gray-300;
$color-surface-alt: $gray-100;
$color-surface-sunken: $white;
$color-border-highlight: $gray-100;
$color-border-lowlight: $gray-500;
$color-border-light: $white;
$color-border-dark: $black;
$color-text: $black;
$color-text-on-accent: $white;
$color-text-status: $navy;
$color-accent-start: $navy;
$color-accent-end: $blue;
$color-desktop-bg: $teal;
$color-row-stripe: $gray-100;
$color-shadow: rgba($black, 0.4);
$color-focus-ring: $black;
$bevel-size: 2px;
$font-stack: Tahoma, "MS Sans Serif", Verdana, sans-serif;
@mixin bevel-raised($size: $bevel-size) {
border-style: solid;
border-width: $size;
border-top-color: $color-border-highlight;
border-left-color: $color-border-highlight;
border-right-color: $color-border-dark;
border-bottom-color: $color-border-dark;
box-shadow:
inset $size $size 0 0 $color-border-light,
inset (-$size) (-$size) 0 0 $color-border-lowlight;
}
@mixin bevel-sunken($size: $bevel-size) {
border-style: solid;
border-width: $size;
border-top-color: $color-border-lowlight;
border-left-color: $color-border-lowlight;
border-right-color: $color-border-light;
border-bottom-color: $color-border-light;
box-shadow:
inset $size $size 0 0 $color-border-dark,
inset (-$size) (-$size) 0 0 $color-surface-alt;
}
* {
box-sizing: border-box;
}
body {
margin: 0;
padding: 24px;
background: $color-desktop-bg;
font-family: $font-stack;
color: $color-text;
min-height: 100vh;
}
#window {
width: 100%;
max-width: 1600px;
margin: 0 auto;
background: $color-surface;
padding: 6px;
display: flex;
flex-direction: column;
gap: 6px;
@include bevel-raised(3px);
box-shadow: 8px 8px 0 0 $color-shadow;
}
.dialog-header {
background: linear-gradient(90deg, $color-accent-start, $color-accent-end);
color: $color-text-on-accent;
padding: 6px 8px;
display: flex;
align-items: center;
p {
margin: 0;
font-size: 14px;
font-weight: bold;
letter-spacing: 0.5px;
text-transform: uppercase;
}
}
.tab-strip {
display: flex;
gap: 4px;
padding: 8px 8px 0;
}
.tab {
position: relative;
top: 3px;
margin-bottom: -2px;
font-family: $font-stack;
font-size: 12px;
font-weight: bold;
text-transform: uppercase;
padding: 6px 14px;
background: $color-surface;
color: $color-text;
border-style: solid;
border-width: 2px;
border-top-color: $color-border-light;
border-left-color: $color-border-light;
border-right-color: $color-border-dark;
border-bottom-color: $color-border-dark;
border-radius: 0;
cursor: pointer;
&.is-active {
top: 0;
z-index: 1;
border-bottom-color: $color-surface;
}
&:focus {
outline: 1px dotted $color-focus-ring;
outline-offset: -4px;
}
}
.dialog-body {
border-top: 2px solid $color-border-dark;
padding: 12px;
}
.tab-panel {
display: flex;
flex-direction: column;
gap: 12px;
&[hidden] {
display: none;
}
}
.field-group {
display: flex;
align-items: center;
gap: 8px;
label {
font-size: 12px;
font-weight: bold;
text-transform: uppercase;
}
&.field-group--stacked {
flex-direction: column;
align-items: stretch;
}
}
select#table-select {
font-family: $font-stack;
font-size: 13px;
padding: 3px 6px;
background: $color-surface-sunken;
color: $color-text;
border-radius: 0;
@include bevel-sunken();
&:focus {
outline: 1px dotted $color-focus-ring;
outline-offset: 2px;
}
}
textarea#query-input {
font-family: $font-stack;
font-size: 13px;
padding: 6px;
background: $color-surface-sunken;
color: $color-text;
border-radius: 0;
resize: vertical;
@include bevel-sunken();
&:focus {
outline: 1px dotted $color-focus-ring;
outline-offset: 2px;
}
}
.btn {
align-self: flex-start;
font-family: $font-stack;
font-size: 12px;
font-weight: bold;
text-transform: uppercase;
padding: 6px 16px;
background: $color-surface;
color: $color-text;
cursor: pointer;
@include bevel-raised();
&:active {
@include bevel-sunken();
}
&:focus {
outline: 1px dotted $color-focus-ring;
outline-offset: -4px;
}
}
#status,
#query-status {
margin: 0;
font-size: 12px;
font-style: italic;
color: $color-text-status;
}
#table-container,
#query-container {
background: $color-surface-sunken;
padding: 4px;
overflow-x: auto;
@include bevel-sunken();
}
table {
border-collapse: collapse;
margin: 0 auto;
font-family: $font-stack;
}
th,
td {
border: 1px solid $color-border-lowlight;
padding: 4px 8px;
text-align: left;
font-size: 13px;
}
th {
background: $color-surface;
white-space: nowrap;
font-weight: bold;
text-transform: uppercase;
@include bevel-raised(1px);
}
td {
min-width: 120px;
max-width: 800px;
white-space: normal;
word-break: break-word;
background: $color-surface-sunken;
}
tbody tr:nth-child(even) td {
background: $color-row-stripe;
}

View File

@@ -0,0 +1,24 @@
{
"compilerOptions": {
"target": "es2023",
"module": "esnext",
"lib": ["ES2023", "DOM"],
"types": ["vite/client"],
"skipLibCheck": true,
/* Bundler mode */
"moduleResolution": "bundler",
"esModuleInterop": true,
"allowImportingTsExtensions": true,
"verbatimModuleSyntax": true,
"moduleDetection": "force",
"noEmit": true,
/* Linting */
"noUnusedLocals": true,
"noUnusedParameters": true,
"erasableSyntaxOnly": true,
"noFallthroughCasesInSwitch": true
},
"include": ["src"]
}

View File

@@ -151,6 +151,11 @@ target_sources(${PROJECT_NAME} PRIVATE
src/json_handling/json_loader.cc src/json_handling/json_loader.cc
) )
# --- data_model ---
target_sources(${PROJECT_NAME} PRIVATE
src/data_model/names_by_country.cc
)
# --- application_options --- # --- application_options ---
target_sources(${PROJECT_NAME} PRIVATE target_sources(${PROJECT_NAME} PRIVATE
src/application_options/parse_arguments.cc src/application_options/parse_arguments.cc
@@ -161,6 +166,7 @@ target_sources(${PROJECT_NAME} PRIVATE
src/biergarten_pipeline_orchestrator/log_results.cc src/biergarten_pipeline_orchestrator/log_results.cc
src/biergarten_pipeline_orchestrator/biergarten_pipeline_orchestrator.cc src/biergarten_pipeline_orchestrator/biergarten_pipeline_orchestrator.cc
src/biergarten_pipeline_orchestrator/generate_breweries.cc src/biergarten_pipeline_orchestrator/generate_breweries.cc
src/biergarten_pipeline_orchestrator/generate_users.cc
src/biergarten_pipeline_orchestrator/run.cc src/biergarten_pipeline_orchestrator/run.cc
src/biergarten_pipeline_orchestrator/query_cities_with_countries.cc src/biergarten_pipeline_orchestrator/query_cities_with_countries.cc
) )
@@ -204,6 +210,7 @@ target_sources(${PROJECT_NAME} PRIVATE
# --- services: sqlite --- # --- services: sqlite ---
target_sources(${PROJECT_NAME} PRIVATE target_sources(${PROJECT_NAME} PRIVATE
src/services/sqlite/process_record.cc src/services/sqlite/process_record.cc
src/services/sqlite/process_user_record.cc
src/services/sqlite/sqlite_export_service.cc src/services/sqlite/sqlite_export_service.cc
src/services/sqlite/finalize.cc src/services/sqlite/finalize.cc
src/services/sqlite/initialize.cc src/services/sqlite/initialize.cc
@@ -260,6 +267,21 @@ configure_file(
${CMAKE_BINARY_DIR}/locations.json ${CMAKE_BINARY_DIR}/locations.json
COPYONLY COPYONLY
) )
configure_file(
${CMAKE_SOURCE_DIR}/personas.json
${CMAKE_BINARY_DIR}/personas.json
COPYONLY
)
configure_file(
${CMAKE_SOURCE_DIR}/forenames-by-country.json
${CMAKE_BINARY_DIR}/forenames-by-country.json
COPYONLY
)
configure_file(
${CMAKE_SOURCE_DIR}/surnames-by-country.json
${CMAKE_BINARY_DIR}/surnames-by-country.json
COPYONLY
)
add_custom_command(TARGET ${PROJECT_NAME} POST_BUILD add_custom_command(TARGET ${PROJECT_NAME} POST_BUILD
COMMAND ${CMAKE_COMMAND} -E copy_directory COMMAND ${CMAKE_COMMAND} -E copy_directory
${CMAKE_SOURCE_DIR}/prompts ${CMAKE_SOURCE_DIR}/prompts

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,72 @@
#!/bin/bash
#
# Walks every directory under src/ and includes/ (including nested
# subdirectories), runs clang-format on the C/C++ files in that directory
# only (non-recursive), and commits the result with its own commit.
#
# Usage: ./format-per-directory.sh [-y]
# -y skip the confirmation prompt
set -euo pipefail
SKIP_CONFIRM=false
if [[ "${1:-}" == "-y" ]]; then
SKIP_CONFIRM=true
fi
cd "$(dirname "$0")"
if [ ! -f .clang-format ]; then
echo "ERROR: .clang-format file not found."
exit 1
fi
if ! command -v clang-format &>/dev/null; then
echo "ERROR: clang-format not found."
exit 1
fi
if ! command -v git &>/dev/null; then
echo "ERROR: git not found."
exit 1
fi
echo "WARNING: This script will format .cpp, .h, .cxx, .cc, .c, .hpp files"
echo "directory-by-directory under src/ and includes/, committing after each"
echo "directory is formatted."
if [[ "$SKIP_CONFIRM" == false ]]; then
read -p "Do you want to continue? (y/n) " -n 1 -r
echo
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
echo "Aborted."
exit 1
fi
fi
if [[ -n "$(git status --porcelain)" ]]; then
echo "ERROR: working tree is not clean. Commit or stash your changes first."
exit 1
fi
dirs=$(find src includes -type d | sort)
for dir in $dirs; do
files=$(find "$dir" -maxdepth 1 -type f \( -name "*.cpp" -o -name "*.hpp" -o -name "*.h" -o -name "*.c" -o -name "*.cc" -o -name "*.cxx" \))
if [[ -z "$files" ]]; then
continue
fi
echo "Formatting $dir..."
echo "$files" | xargs clang-format -i
if [[ -n "$(git status --porcelain -- "$dir")" ]]; then
git add -- "$dir"
git commit -m "Formatted $dir"
else
echo "No changes in $dir, skipping commit."
fi
done
echo "Done."

View File

@@ -2,11 +2,11 @@
#define BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_DATA_GENERATOR_H_ #define BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_DATA_GENERATOR_H_
/** /**
* @file biergarten_data_generator.h * @file biergarten_pipeline_orchestrator.h
* @brief Orchestration for end-to-end brewery data generation pipeline. * @brief Orchestration for end-to-end brewery and user data generation.
* *
* Intent: Coordinates location loading, enrichment, and generation phases * Coordinates location loading, enrichment, and generation phases to produce
* to produce a complete dataset. Coordinates dependencies via composition root. * a complete dataset, wiring dependencies selected at the composition root.
*/ */
#include <memory> #include <memory>
@@ -15,32 +15,36 @@
#include "data_generation/data_generator.h" #include "data_generation/data_generator.h"
#include "data_model/generated_models.h" #include "data_model/generated_models.h"
#include "services/curated_data/curated_data_service.h"
#include "services/database/export_service.h" #include "services/database/export_service.h"
#include "services/enrichment/enrichment_service.h" #include "services/enrichment/enrichment_service.h"
#include "services/logging/logger.h" #include "services/logging/logger.h"
/** /**
* @brief Main data generator class for the Biergarten pipeline. * @brief Main orchestrator for the Biergarten data generation pipeline.
* *
* This class encapsulates the core logic for generating brewery data. * Handles location loading, city enrichment, and brewery/user generation.
* It handles location loading, city enrichment, and brewery generation.
*/ */
class BiergartenPipelineOrchestrator { class BiergartenPipelineOrchestrator {
public: public:
/** /**
* @brief Constructs the orchestrator with injected pipeline dependencies. * @brief Constructs the orchestrator with injected pipeline dependencies.
* *
* @param context_service Provides regional context for locations. * @param logger Sink for pipeline diagnostics.
* @param generator Implementation (Llama or Mock) for brewery/user generation. * @param context_service Provides regional context for locations.
* @param exporter Database backend for persisting generated records. * @param generator Implementation (Llama or Mock) for brewery/user
* @param application_options CLI configuration and paths. * generation.
*/ * @param exporter Database backend for persisting generated records.
* @param curated_data_service Loads curated location, persona, and name
* data used to seed generation.
* @param application_options CLI configuration and paths.
*/
BiergartenPipelineOrchestrator( BiergartenPipelineOrchestrator(
std::shared_ptr<ILogger> logger, std::shared_ptr<ILogger> logger,
std::unique_ptr<IEnrichmentService> context_service, std::unique_ptr<IEnrichmentService> context_service,
std::unique_ptr<DataGenerator> generator, std::unique_ptr<DataGenerator> generator,
std::unique_ptr<IExportService> exporter, std::unique_ptr<IExportService> exporter,
std::unique_ptr<ICuratedDataService> curated_data_service,
const ApplicationOptions& application_options); const ApplicationOptions& application_options);
/** /**
@@ -49,38 +53,42 @@ class BiergartenPipelineOrchestrator {
* Performs the following steps: * Performs the following steps:
* 1. Load curated locations from JSON * 1. Load curated locations from JSON
* 2. Resolve context for each city using the injected context service * 2. Resolve context for each city using the injected context service
* 3. Generate brewery data for sampled cities * 3. Generate brewery and user data for sampled cities
* *
* @note STRUCTURAL CONCURRENCY REQUIREMENT: * @note STRUCTURAL CONCURRENCY REQUIREMENT:
* When transitioned to a multithreaded design, this method MUST structurally * When transitioned to a multithreaded design, this method MUST
* enforce that all deployed worker threads are joined before returning (e.g. * structurally enforce that all deployed worker threads are joined before
* by using std::jthread or a structured concurrency primitive). This ensures * returning (e.g. by using std::jthread or a structured concurrency
* workers do not attempt to log to a closed channel during application teardown. * primitive). This ensures workers do not attempt to log to a closed
* channel during application teardown.
* *
* @return true if successful, false if not * @return true if successful, false if not
*/ */
bool Run(); bool Run();
private: private:
/// @brief Logger instance for emitting pipeline messages.
std::shared_ptr<ILogger> logger_; std::shared_ptr<ILogger> logger_;
/// @brief Owning context provider dependency.
std::unique_ptr<IEnrichmentService> context_service_; std::unique_ptr<IEnrichmentService> context_service_;
/// @brief Generator dependency selected in the composition root. /**
* @brief Generator implementation selected at the composition root (Llama
* or Mock).
*/
std::unique_ptr<DataGenerator> generator_; std::unique_ptr<DataGenerator> generator_;
/// @brief Storage backend for generated brewery records. /**
std::unique_ptr<IExportService> exporter_; * @brief Storage backend for generated brewery and user records.
*/
std::unique_ptr<IExportService> exporter_;
std::unique_ptr<ICuratedDataService> curated_data_service_;
/// @brief CLI configuration: paths, model settings, generation parameters. ApplicationOptions application_options_;
ApplicationOptions application_options_;
/** /**
* @brief Load locations from JSON and sample cities. * @brief Load locations from JSON and sample cities.
* *
* @return Vector of sampled locations capped at 50 entries. * @return Vector of locations randomly sampled per
* PipelineOptions::location_count.
*/ */
std::vector<Location> QueryCitiesWithCountries(); std::vector<Location> QueryCitiesWithCountries();
@@ -92,11 +100,24 @@ class BiergartenPipelineOrchestrator {
void GenerateBreweries(std::span<const EnrichedCity> cities); void GenerateBreweries(std::span<const EnrichedCity> cities);
/** /**
* @brief Log the generated brewery results. * @brief Generate users grounded in sampled names and personas for
* enriched cities.
*
* Loads personas.json / forenames-by-country.json /
* surnames-by-country.json itself, mirroring how QueryCitiesWithCountries()
* owns its own locations.json load.
*
* @param cities Span of enriched city data.
*/
void GenerateUsers(std::span<const EnrichedCity> cities);
/**
* @brief Log the generated brewery and user results.
*/ */
void LogResults() const; void LogResults() const;
/// @brief Stores generated brewery data. std::vector<BreweryRecord> generated_breweries_;
std::vector<GeneratedBrewery> generated_breweries_; std::vector<UserRecord> generated_users_;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_DATA_GENERATOR_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_DATA_GENERATOR_H_

View File

@@ -9,10 +9,12 @@
/** /**
* @file bounded_channel.h * @file bounded_channel.h
* @brief Thread-safe, bounded multi-producer/multi-consumer synchronous channel. * @brief Thread-safe, bounded multi-producer/multi-consumer synchronous
* channel.
* *
* Intent: Enables asynchronous inter-thread communication with backpressure. * Intent: Enables asynchronous inter-thread communication with backpressure.
* Models a synchronous channel where producers/consumers block on capacity limits. * Models a synchronous channel where producers/consumers block on capacity
* limits.
*/ */
/** /**

View File

@@ -28,12 +28,16 @@ class DataGenerator {
const std::string& region_context) = 0; const std::string& region_context) = 0;
/** /**
* @brief Generates a user profile for a locale. * @brief Generates a user profile grounded in a sampled name and persona.
* *
* @param locale Locale hint used by generator. * @param city Enriched city the user is associated with.
* @param persona Persona archetype grounding the generated bio.
* @param name Sampled first/last name (and gender)
* @return User generation result. * @return User generation result.
*/ */
virtual UserResult GenerateUser(const std::string& locale) = 0; virtual UserResult GenerateUser(const EnrichedCity& city,
const UserPersona& persona,
const Name& name) = 0;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_DATA_GENERATOR_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_DATA_GENERATOR_H_

View File

@@ -0,0 +1,52 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_JSON_GRAMMARS_H_
#define BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_JSON_GRAMMARS_H_
/**
* @file data_generation/json_grammars.h
* @brief GBNF grammars constraining structured JSON output from
* LlamaGenerator inference calls.
*/
#include <string_view>
// GBNF grammar for structured user JSON output.
// thought-block permits the model to emit free-form reasoning before the
// JSON object (the prompts explicitly invite this); only the "{...}" tail is
// constrained to the expected shape.
inline constexpr std::string_view kUserJsonGrammar = R"json_user(
root ::= thought-block (
"{" ws
"\"username\"" ws ":" ws string ws "," ws
"\"bio\"" ws ":" ws string ws "," ws
"\"activity_weight\"" ws ":" ws number ws
"}" ws
)
thought-block ::= [^{]*
ws ::= [ \t\n\r]*
string ::= "\"" char+ "\""
char ::= [^"\\\x7F\x00-\x1F] | [\\] escape
escape ::= ["\\/bfnrt] | "u" hex hex hex hex
hex ::= [0-9a-fA-F]
number ::= "-"? ("0" | [1-9] [0-9]*) ("." [0-9]+)?
)json_user";
// GBNF grammar for structured brewery JSON output (see thought-block note
// above).
inline constexpr std::string_view kBreweryJsonGrammar = R"json_brewery(
root ::= thought-block (
"{" ws
"\"name_en\"" ws ":" ws string ws "," ws
"\"description_en\"" ws ":" ws string ws "," ws
"\"name_local\"" ws ":" ws string ws "," ws
"\"description_local\"" ws ":" ws string ws
"}" ws
)
thought-block ::= [^{]*
ws ::= [ \t\n\r]*
string ::= "\"" char+ "\""
char ::= [^"\\\x7F\x00-\x1F] | [\\] escape
escape ::= ["\\/bfnrt] | "u" hex hex hex hex
hex ::= [0-9a-fA-F]
)json_brewery";
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_JSON_GRAMMARS_H_

View File

@@ -44,14 +44,9 @@ class LlamaGenerator final : public DataGenerator {
~LlamaGenerator() override; ~LlamaGenerator() override;
// disable copy constructor
LlamaGenerator(const LlamaGenerator&) = delete; LlamaGenerator(const LlamaGenerator&) = delete;
// disable copy assignment operator
LlamaGenerator& operator=(const LlamaGenerator&) = delete; LlamaGenerator& operator=(const LlamaGenerator&) = delete;
// disable move constructor
LlamaGenerator(LlamaGenerator&&) = delete; LlamaGenerator(LlamaGenerator&&) = delete;
// disable move assignment operator
LlamaGenerator& operator=(LlamaGenerator&&) = delete; LlamaGenerator& operator=(LlamaGenerator&&) = delete;
/** /**
@@ -65,12 +60,15 @@ class LlamaGenerator final : public DataGenerator {
const std::string& region_context) override; const std::string& region_context) override;
/** /**
* @brief Generates a user profile for the provided locale. * @brief Generates a user profile grounded in a sampled name and persona.
* *
* @param locale Locale hint. * @param city Enriched city the user is associated with.
* @param persona Persona archetype grounding the generated bio.
* @param name Sampled first/last name -- not LLM-invented.
* @return Generated user profile. * @return Generated user profile.
*/ */
UserResult GenerateUser(const std::string& locale) override; UserResult GenerateUser(const EnrichedCity& city, const UserPersona& persona,
const Name& name) override;
private: private:
static constexpr int32_t kDefaultMaxTokens = 10000; static constexpr int32_t kDefaultMaxTokens = 10000;

View File

@@ -47,4 +47,18 @@ void AppendTokenPiece(const llama_vocab* vocab, llama_token token,
std::optional<std::string> ValidateBreweryJson(const std::string& raw, std::optional<std::string> ValidateBreweryJson(const std::string& raw,
BreweryResult& brewery_out); BreweryResult& brewery_out);
/**
* @brief Validates and parses user JSON output.
*
* Only populates `username`, `bio`, and `activity_weight` -- `first_name`
* and `last_name` are not LLM-authored and are set separately from the
* sampled Name.
*
* @param raw Raw model output.
* @param user_out Parsed user payload.
* @return Validation error message if invalid, or std::nullopt on success.
*/
std::optional<std::string> ValidateUserJson(const std::string& raw,
UserResult& user_out);
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_LLAMA_GENERATOR_HELPERS_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_LLAMA_GENERATOR_HELPERS_H_

View File

@@ -28,12 +28,16 @@ class MockGenerator final : public DataGenerator {
const std::string& region_context) override; const std::string& region_context) override;
/** /**
* @brief Generates deterministic user data for a locale. * @brief Generates deterministic user data grounded in a sampled name and
* persona.
* *
* @param locale Locale hint. * @param city Enriched city the user is associated with.
* @param persona Persona archetype.
* @param name Sampled first/last name, copied directly into the result.
* @return Generated user result. * @return Generated user result.
*/ */
UserResult GenerateUser(const std::string& locale) override; UserResult GenerateUser(const EnrichedCity& city, const UserPersona& persona,
const Name& name) override;
private: private:
/** /**
@@ -44,12 +48,25 @@ class MockGenerator final : public DataGenerator {
*/ */
static size_t DeterministicHash(const Location& location); static size_t DeterministicHash(const Location& location);
/**
* @brief Combines city, persona, and name into a stable hash value.
*
* @param location City and country names.
* @param persona Persona archetype.
* @param name Sampled first/last name.
* @return Deterministic hash value.
*/
static size_t DeterministicHash(const Location& location,
const UserPersona& persona, const Name& name);
// Hash stride constants for deterministic distribution across fixed-size // Hash stride constants for deterministic distribution across fixed-size
// arrays. These coprime strides spread hash values uniformly without // arrays. These coprime strides spread hash values uniformly without
// clustering, ensuring diverse output across different hash inputs. // clustering, ensuring diverse output across different hash inputs.
static constexpr size_t kNounHashStride = 7; static constexpr size_t kNounHashStride = 7;
static constexpr size_t kDescriptionHashStride = 13; static constexpr size_t kDescriptionHashStride = 13;
static constexpr size_t kBioHashStride = 11; static constexpr size_t kBioHashStride = 11;
static constexpr size_t kActivityWeightHashStride = 17;
static constexpr size_t kActivityWeightHashRange = 1000;
static constexpr std::array<std::string_view, 18> kBreweryAdjectives = { static constexpr std::array<std::string_view, 18> kBreweryAdjectives = {
"Craft", "Heritage", "Local", "Artisan", "Pioneer", "Golden", "Craft", "Heritage", "Local", "Artisan", "Pioneer", "Golden",
@@ -124,7 +141,8 @@ class MockGenerator final : public DataGenerator {
"Visits breweries for the stories, stays for the flagship pours.", "Visits breweries for the stories, stays for the flagship pours.",
"Craft beer fan mapping tasting notes and favorite brew routes.", "Craft beer fan mapping tasting notes and favorite brew routes.",
"Always ready to trade recommendations for underrated local breweries.", "Always ready to trade recommendations for underrated local breweries.",
"Keeping a running list of must-try collab releases and tap takeovers."}; "Keeping a running list of must-try collab releases and tap "
"takeovers."};
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_MOCK_GENERATOR_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_MOCK_GENERATOR_H_

View File

@@ -3,8 +3,8 @@
/** /**
* @file data_model/generated_models.h * @file data_model/generated_models.h
* @brief Generated output models from the pipeline: brewery/user results, enriched data, * @brief Generated output models from the pipeline: brewery/user results,
* and complete generation results. * enriched data, and complete generation results.
*/ */
#include <string> #include <string>
@@ -19,16 +19,24 @@
* @brief Generated brewery payload. * @brief Generated brewery payload.
*/ */
struct BreweryResult { struct BreweryResult {
/// @brief Brewery display name in English. /**
* @brief Brewery display name in English.
*/
std::string name_en; std::string name_en;
/// @brief Brewery description text in English. /**
* @brief Brewery description text in English.
*/
std::string description_en; std::string description_en;
/// @brief Brewery display name in the local language. /**
* @brief Brewery display name in the local language.
*/
std::string name_local; std::string name_local;
/// @brief Brewery description text in the local language. /**
* @brief Brewery description text in the local language.
*/
std::string description_local; std::string description_local;
}; };
@@ -36,11 +44,39 @@ struct BreweryResult {
* @brief Generated user profile payload. * @brief Generated user profile payload.
*/ */
struct UserResult { struct UserResult {
/// @brief Username handle. /**
* @brief First (given) name, copied from the sampled Name -- not
* LLM-invented.
*/
std::string first_name{};
/**
* @brief Last (family) name, copied from the sampled Name -- not
* LLM-invented.
*/
std::string last_name{};
/**
* @brief Gender associated with the sampled first name, copied from the
* sampled Name -- not LLM-invented.
*/
std::string gender{};
/**
* @brief Username handle.
*/
std::string username{}; std::string username{};
/// @brief Short user biography. /**
* @brief Short user biography.
*/
std::string bio{}; std::string bio{};
/**
* @brief Relative check-in/activity weight for this user, used to drive
* a J-curve activity profile in later pipeline phases.
*/
float activity_weight{};
}; };
// ============================================================================ // ============================================================================
@@ -58,9 +94,23 @@ struct EnrichedCity {
/** /**
* @brief Helper struct to store generated brewery data. * @brief Helper struct to store generated brewery data.
*/ */
struct GeneratedBrewery { struct BreweryRecord {
Location location; Location location;
BreweryResult brewery; BreweryResult brewery;
}; };
/**
* @brief Helper struct to store generated user data.
*
* `email` and `date_of_birth` are programmatically generated by the
* orchestrator (never LLM-authored) so a downstream auth-account seeding
* consumer can register real accounts from the pipeline's SQLite export.
*/
struct UserRecord {
Location location;
UserResult user;
std::string email{};
std::string date_of_birth{};
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_GENERATED_MODELS_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_GENERATED_MODELS_H_

View File

@@ -28,40 +28,91 @@ namespace prog_opts = boost::program_options;
* @brief Canonical location record for city-level generation. * @brief Canonical location record for city-level generation.
*/ */
struct Location { struct Location {
/// @brief City name.
std::string city{}; std::string city{};
/// @brief State or province name.
std::string state_province{}; std::string state_province{};
/// @brief ISO 3166-2 subdivision code. /**
* @brief ISO 3166-2 subdivision code.
*/
std::string iso3166_2{}; std::string iso3166_2{};
/// @brief Country name.
std::string country{}; std::string country{};
/// @brief ISO 3166-1 country code. /**
* @brief ISO 3166-1 country code.
*/
std::string iso3166_1{}; std::string iso3166_1{};
/// @brief Local language codes in priority order. /**
* @brief Local language codes in priority order.
*/
std::vector<std::string> local_languages{}; std::vector<std::string> local_languages{};
/// @brief Latitude in decimal degrees. /**
* @brief Latitude in decimal degrees.
*/
double latitude{}; double latitude{};
/// @brief Longitude in decimal degrees. /**
* @brief Longitude in decimal degrees.
*/
double longitude{}; double longitude{};
}; };
/** // ============================================================================
* @brief Non-owning brewery location input. // Name / Persona Models
*/ // ============================================================================
struct BreweryLocation {
/// @brief City name.
std::string_view city_name;
/// @brief Country name. /**
std::string_view country_name; * @brief A sampled first/last name pair, with the source forename's gender.
*
* Produced by NamesByCountry::SampleName();
*/
struct Name {
std::string first_name{};
std::string last_name{};
/**
* @brief Gender associated with the sampled forename (e.g. "M", "F"), as
* reported by the source dataset.
*/
std::string gender{};
};
/**
* @brief A single forename entry from the names-by-country fixture data.
*/
struct ForenameEntry {
/**
* @brief Romanized forename.
*/
std::string name{};
/**
* @brief Gender associated with this forename, as reported by the source
* dataset (e.g. "M", "F").
*/
std::string gender{};
};
/**
* @brief A persona archetype used to ground LLM-generated user bios.
*/
struct UserPersona {
/**
* @brief Persona display name (e.g. "Hophead Explorer").
*/
std::string name{};
/**
* @brief Short description of the persona's interests and voice.
*/
std::string description{};
/**
* @brief Beer styles this persona gravitates toward.
*/
std::vector<std::string> style_affinities{};
}; };
// ============================================================================ // ============================================================================
@@ -72,22 +123,34 @@ struct BreweryLocation {
* @brief LLM sampling parameters. * @brief LLM sampling parameters.
*/ */
struct SamplingOptions { struct SamplingOptions {
/// @brief LLM sampling temperature (0.0 to 1.0, higher = more random). /**
* @brief LLM sampling temperature (higher = more random).
*/
float temperature = 1.0F; float temperature = 1.0F;
/// @brief LLM nucleus sampling top-p parameter. /**
* @brief LLM nucleus sampling top-p parameter.
*/
float top_p = 0.95F; float top_p = 0.95F;
/// @brief LLM top-k sampling parameter. /**
* @brief LLM top-k sampling parameter.
*/
uint32_t top_k = 64; uint32_t top_k = 64;
/// @brief Context window size (tokens). /**
* @brief Context window size (tokens).
*/
uint32_t n_ctx = 8192; uint32_t n_ctx = 8192;
/// @brief Random seed (-1 for random, otherwise non-negative). /**
* @brief Random seed (-1 for random, otherwise non-negative).
*/
int seed = -1; int seed = -1;
/// @brief Number of layers to offload to GPU. /**
* @brief Number of layers to offload to GPU.
*/
int n_gpu_layers = 0; int n_gpu_layers = 0;
}; };
@@ -95,16 +158,20 @@ struct SamplingOptions {
* @brief Configuration for the LLM generator component. * @brief Configuration for the LLM generator component.
*/ */
struct GeneratorOptions { struct GeneratorOptions {
/// @brief Path to the LLM model file (gguf format). /**
* @brief Path to the LLM model file (gguf format).
*/
std::filesystem::path model_path; std::filesystem::path model_path;
/// @brief Use mocked generator instead of actual LLM inference. /**
* @brief Use mocked generator instead of actual LLM inference.
*/
bool use_mocked = false; bool use_mocked = false;
/**
* @brief Specific sampling parameters for this generator.
/// @brief Specific sampling parameters for this generator. * If nullopt, the application should use global defaults.
/// If nullopt, the application should use global defaults. */
std::optional<SamplingOptions> sampling; std::optional<SamplingOptions> sampling;
}; };
@@ -112,18 +179,23 @@ struct GeneratorOptions {
* @brief Configuration for the pipeline execution and output. * @brief Configuration for the pipeline execution and output.
*/ */
struct PipelineOptions { struct PipelineOptions {
/// @brief Directory for generated artifacts. /**
* @brief Directory for generated artifacts.
*/
std::filesystem::path output_path; std::filesystem::path output_path;
/// @brief Directory that contains named prompt files (e.g. /**
/// BREWERY_GENERATION.md). * @brief Directory that contains named prompt files (e.g.
* BREWERY_GENERATION.md).
*/
std::filesystem::path prompt_dir; std::filesystem::path prompt_dir;
/// @brief Path for application logs.
std::filesystem::path log_path; std::filesystem::path log_path;
/// @brief Number of locations to sample from the dataset /**
/// More locations -> more users/more breweries * @brief Number of locations to sample from the dataset
* More locations -> more users/more breweries
*/
uint32_t location_count; uint32_t location_count;
}; };
@@ -139,7 +211,7 @@ struct ApplicationOptions {
// Function Declarations // Function Declarations
// ============================================================================ // ============================================================================
std::optional<ApplicationOptions> ParseArguments(const int argc, char** argv, std::optional<ApplicationOptions> ParseArguments(
std::shared_ptr<ILogger> logger = nullptr); const int argc, char** argv, std::shared_ptr<ILogger> logger = nullptr);
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_MODELS_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_MODELS_H_

View File

@@ -0,0 +1,51 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_NAMES_BY_COUNTRY_H_
#define BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_NAMES_BY_COUNTRY_H_
/**
* @file data_model/names_by_country.h
* @brief Sampling over the names-by-country fixture data.
*/
#include <optional>
#include <random>
#include <string>
#include <unordered_map>
#include <vector>
#include "data_model/models.h"
/**
* @brief Samples locale-appropriate names from curated forename/surname
* fixture data, keyed by ISO 3166-1 country code.
*
* Forenames and surnames are kept in separate maps (mirroring the source
* dataset's own shape) so per-forename gender is preserved; pairing happens
* at sample time rather than being precomputed, so no information from the
* source dataset is discarded up front.
*/
class NamesByCountry {
public:
NamesByCountry(std::unordered_map<std::string, std::vector<ForenameEntry>>
forenames_by_country,
std::unordered_map<std::string, std::vector<std::string>>
surnames_by_country);
/**
* @brief Samples a first/last name pair for the given country.
*
* @param iso3166_1 ISO 3166-1 country code to sample for.
* @param rng Random source used for sampling.
* @return A sampled Name, or std::nullopt if the country has no forename
* or no surname entries.
*/
[[nodiscard]] std::optional<Name> SampleName(const std::string& iso3166_1,
std::mt19937& rng) const;
private:
std::unordered_map<std::string, std::vector<ForenameEntry>>
forenames_by_country_;
std::unordered_map<std::string, std::vector<std::string>>
surnames_by_country_;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_NAMES_BY_COUNTRY_H_

View File

@@ -3,23 +3,45 @@
/** /**
* @file json_handling/json_loader.h * @file json_handling/json_loader.h
* @brief Loader API for curated location data. * @brief JSON-backed implementation of ICuratedDataService.
*/ */
#include <filesystem> #include <filesystem>
#include <memory>
#include <vector> #include <vector>
#include "data_model/models.h" #include "data_model/models.h"
#include "services/logging/logger.h" #include "data_model/names_by_country.h"
#include "services/curated_data/curated_data_service.h"
/// @brief Loads curated world locations from a JSON file into memory. /**
class JsonLoader { * @brief Loads curated location, persona, and name data from JSON files.
*/
class JsonLoader final : public ICuratedDataService {
public: public:
/// @brief Parses a JSON array file and returns all location records. JsonLoader() = default;
static std::vector<Location> LoadLocations(
const std::filesystem::path& filepath, /**
std::shared_ptr<ILogger> logger = nullptr); * @brief Parses a JSON array file and returns all location records.
*/
std::vector<Location> LoadLocations(
const std::filesystem::path& filepath) override;
/**
* @brief Parses a JSON array file and returns all persona records.
*/
std::vector<UserPersona> LoadPersonas(
const std::filesystem::path& filepath) override;
/**
* @brief Parses the names-by-country fixture pair into a sampling-capable
* NamesByCountry.
*
* @param forenames_filepath Path to forenames-by-country.json.
* @param surnames_filepath Path to surnames-by-country.json.
*/
NamesByCountry LoadNamesByCountry(
const std::filesystem::path& forenames_filepath,
const std::filesystem::path& surnames_filepath) override;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_

View File

@@ -16,16 +16,10 @@
*/ */
class LlamaBackendState { class LlamaBackendState {
public: public:
/// @brief Initializes global llama backend state.
LlamaBackendState() { llama_backend_init(); } LlamaBackendState() { llama_backend_init(); }
/// @brief Cleans up global llama backend state.
~LlamaBackendState() { llama_backend_free(); } ~LlamaBackendState() { llama_backend_free(); }
/// @brief Non-copyable type.
LlamaBackendState(const LlamaBackendState&) = delete; LlamaBackendState(const LlamaBackendState&) = delete;
/// @brief Non-copyable type.
LlamaBackendState& operator=(const LlamaBackendState&) = delete; LlamaBackendState& operator=(const LlamaBackendState&) = delete;
}; };

View File

@@ -0,0 +1,53 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_CURATED_DATA_SERVICE_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_CURATED_DATA_SERVICE_H_
/**
* @file services/curated_data/curated_data_service.h
* @brief Abstraction for loading curated location, persona, and name data.
*/
#include <filesystem>
#include <vector>
#include "data_model/models.h"
#include "data_model/names_by_country.h"
/**
* @brief Interface for services that load curated data used to seed
* brewery/user generation.
*/
class ICuratedDataService {
public:
ICuratedDataService() = default;
virtual ~ICuratedDataService() = default;
ICuratedDataService(const ICuratedDataService&) = delete;
ICuratedDataService& operator=(const ICuratedDataService&) = delete;
ICuratedDataService(ICuratedDataService&&) = delete;
ICuratedDataService& operator=(ICuratedDataService&&) = delete;
/**
* @brief Loads all curated location records.
*/
virtual std::vector<Location> LoadLocations(
const std::filesystem::path& filepath) = 0;
/**
* @brief Loads all curated persona records.
*/
virtual std::vector<UserPersona> LoadPersonas(
const std::filesystem::path& filepath) = 0;
/**
* @brief Loads the names-by-country fixture pair into a sampling-capable
* NamesByCountry.
*
* @param forenames_filepath Path to forenames-by-country.json.
* @param surnames_filepath Path to surnames-by-country.json.
*/
virtual NamesByCountry LoadNamesByCountry(
const std::filesystem::path& forenames_filepath,
const std::filesystem::path& surnames_filepath) = 0;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_CURATED_DATA_SERVICE_H_

View File

@@ -16,8 +16,6 @@
class IExportService { class IExportService {
public: public:
IExportService() = default; IExportService() = default;
/// @brief Virtual destructor for polymorphic cleanup.
virtual ~IExportService() = default; virtual ~IExportService() = default;
IExportService(const IExportService&) = delete; IExportService(const IExportService&) = delete;
@@ -25,7 +23,9 @@ class IExportService {
IExportService(IExportService&&) = delete; IExportService(IExportService&&) = delete;
IExportService& operator=(IExportService&&) = delete; IExportService& operator=(IExportService&&) = delete;
/// @brief Prepares the export destination for a new run. /**
* @brief Prepares the export destination for a new run.
*/
virtual void Initialize() = 0; virtual void Initialize() = 0;
/** /**
@@ -33,9 +33,18 @@ class IExportService {
* *
* @param brewery Generated brewery payload to store. * @param brewery Generated brewery payload to store.
*/ */
virtual uint64_t ProcessRecord(const GeneratedBrewery& brewery) = 0; virtual uint64_t ProcessRecord(const BreweryRecord& brewery) = 0;
/// @brief Finalizes the export destination. /**
* @brief Persists one generated user record.
*
* @param user Generated user payload to store.
*/
virtual uint64_t ProcessRecord(const UserRecord& user) = 0;
/**
* @brief Finalizes the export destination.
*/
virtual void Finalize() = 0; virtual void Finalize() = 0;
}; };

View File

@@ -11,8 +11,8 @@
#include <string> #include <string>
#include <unordered_map> #include <unordered_map>
#include "data_model/models.h"
#include "../datetime/date_time_provider.h" #include "../datetime/date_time_provider.h"
#include "data_model/models.h"
#include "export_service.h" #include "export_service.h"
#include "sqlite_export_service_helpers.h" #include "sqlite_export_service_helpers.h"
@@ -30,7 +30,8 @@ class SqliteExportService final : public IExportService {
SqliteExportService& operator=(SqliteExportService&&) = delete; SqliteExportService& operator=(SqliteExportService&&) = delete;
void Initialize() override; void Initialize() override;
uint64_t ProcessRecord(const GeneratedBrewery& brewery) override; uint64_t ProcessRecord(const BreweryRecord& brewery) override;
uint64_t ProcessRecord(const UserRecord& user) override;
void Finalize() override; void Finalize() override;
private: private:
@@ -46,6 +47,15 @@ class SqliteExportService final : public IExportService {
[[nodiscard]] std::filesystem::path BuildDatabasePath() const; [[nodiscard]] std::filesystem::path BuildDatabasePath() const;
[[nodiscard]] static std::string BuildLocationKey(const Location& location); [[nodiscard]] static std::string BuildLocationKey(const Location& location);
/**
* @brief Returns the row id for @p location, inserting it first if it has
* not already been seen during this run.
*
* Shared by both ProcessRecord() overloads so breweries and users
* referencing the same location resolve to the same row.
*/
[[nodiscard]] sqlite3_int64 ResolveLocationId(const Location& location);
std::unique_ptr<IDateTimeProvider> date_time_provider_; std::unique_ptr<IDateTimeProvider> date_time_provider_;
std::filesystem::path output_path_; std::filesystem::path output_path_;
std::string run_timestamp_utc_; std::string run_timestamp_utc_;
@@ -53,6 +63,7 @@ class SqliteExportService final : public IExportService {
SqliteDatabaseHandle db_handle_; SqliteDatabaseHandle db_handle_;
SqliteStatementHandle insert_location_stmt_; SqliteStatementHandle insert_location_stmt_;
SqliteStatementHandle insert_brewery_stmt_; SqliteStatementHandle insert_brewery_stmt_;
SqliteStatementHandle insert_user_stmt_;
bool transaction_open_ = false; bool transaction_open_ = false;
std::unordered_map<std::string, sqlite3_int64> location_cache_; std::unordered_map<std::string, sqlite3_int64> location_cache_;
}; };

View File

@@ -50,6 +50,26 @@ CREATE INDEX IF NOT EXISTS idx_breweries_location_id ON breweries(location_id);
)sql"; )sql";
inline constexpr std::string_view kCreateUsersTableSql = R"sql(
CREATE TABLE IF NOT EXISTS users (
id INTEGER PRIMARY KEY AUTOINCREMENT,
location_id INTEGER NOT NULL,
first_name TEXT NOT NULL,
last_name TEXT NOT NULL,
gender TEXT NOT NULL,
username TEXT NOT NULL,
bio TEXT NOT NULL,
activity_weight REAL NOT NULL,
email TEXT NOT NULL UNIQUE,
date_of_birth TEXT NOT NULL,
FOREIGN KEY(location_id) REFERENCES locations(id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_users_location_id ON users(location_id);
)sql";
inline constexpr std::string_view kInsertLocationSql = R"sql( inline constexpr std::string_view kInsertLocationSql = R"sql(
INSERT INTO locations ( INSERT INTO locations (
city, city,
@@ -73,6 +93,22 @@ INSERT INTO breweries (
) VALUES (?, ?, ?, ?, ?); ) VALUES (?, ?, ?, ?, ?);
)sql"; )sql";
inline constexpr std::string_view kInsertUserSql = R"sql(
INSERT INTO users (
location_id,
first_name,
last_name,
gender,
username,
bio,
activity_weight,
email,
date_of_birth
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?);
)sql";
// sqlite3_bind_*() parameter indices are 1-based, matching the "?"
// placeholder order in the SQL above.
inline constexpr int kLocationCityBindIndex = 1; inline constexpr int kLocationCityBindIndex = 1;
inline constexpr int kLocationStateProvinceBindIndex = 2; inline constexpr int kLocationStateProvinceBindIndex = 2;
inline constexpr int kLocationIso31662BindIndex = 3; inline constexpr int kLocationIso31662BindIndex = 3;
@@ -88,6 +124,16 @@ inline constexpr int kBreweryEnglishDescriptionBindIndex = 3;
inline constexpr int kBreweryLocalNameBindIndex = 4; inline constexpr int kBreweryLocalNameBindIndex = 4;
inline constexpr int kBreweryLocalDescriptionBindIndex = 5; inline constexpr int kBreweryLocalDescriptionBindIndex = 5;
inline constexpr int kUserLocationIdBindIndex = 1;
inline constexpr int kUserFirstNameBindIndex = 2;
inline constexpr int kUserLastNameBindIndex = 3;
inline constexpr int kUserGenderBindIndex = 4;
inline constexpr int kUserUsernameBindIndex = 5;
inline constexpr int kUserBioBindIndex = 6;
inline constexpr int kUserActivityWeightBindIndex = 7;
inline constexpr int kUserEmailBindIndex = 8;
inline constexpr int kUserDateOfBirthBindIndex = 9;
SqliteStatementHandle PrepareStatement(const SqliteDatabaseHandle& db_handle, SqliteStatementHandle PrepareStatement(const SqliteDatabaseHandle& db_handle,
std::string_view sql, std::string_view sql,
const char* action); const char* action);

View File

@@ -18,7 +18,6 @@
*/ */
class IDateTimeProvider { class IDateTimeProvider {
public: public:
/// @brief Virtual destructor for polymorphic cleanup.
virtual ~IDateTimeProvider() = default; virtual ~IDateTimeProvider() = default;
IDateTimeProvider() = default; IDateTimeProvider() = default;

View File

@@ -15,7 +15,6 @@
*/ */
class IEnrichmentService { class IEnrichmentService {
public: public:
/// @brief Virtual destructor for polymorphic cleanup.
virtual ~IEnrichmentService() = default; virtual ~IEnrichmentService() = default;
/** /**

View File

@@ -1,13 +1,18 @@
//
// Created by aaronpo on 13/05/2026.
//
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_MOCK_ENRICHMENT_H_ #ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_MOCK_ENRICHMENT_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_MOCK_ENRICHMENT_H_ #define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_MOCK_ENRICHMENT_H_
/**
* @file services/enrichment/mock_enrichment.h
* @brief No-op IEnrichmentService used when network enrichment is disabled.
*/
#include <string> #include <string>
#include "enrichment_service.h" #include "enrichment_service.h"
/**
* @brief Enrichment service that returns no context for any location.
*/
class MockEnrichmentService final : public IEnrichmentService { class MockEnrichmentService final : public IEnrichmentService {
public: public:
std::string GetLocationContext(const Location& /*loc*/) override { std::string GetLocationContext(const Location& /*loc*/) override {

View File

@@ -15,21 +15,33 @@
#include "services/logging/logger.h" #include "services/logging/logger.h"
#include "web_client/web_client.h" #include "web_client/web_client.h"
/// @brief Provides Wikipedia summary lookups backed by cached raw extracts. /**
* @brief Provides Wikipedia summary lookups backed by cached raw extracts.
*/
class WikipediaEnrichmentService final : public IEnrichmentService { class WikipediaEnrichmentService final : public IEnrichmentService {
public: public:
/// @brief Creates a new Wikipedia service with the provided web client. /**
* @brief Creates a new Wikipedia service with the provided web client.
*/
explicit WikipediaEnrichmentService(std::unique_ptr<WebClient> client, explicit WikipediaEnrichmentService(std::unique_ptr<WebClient> client,
std::shared_ptr<ILogger> logger); std::shared_ptr<ILogger> logger);
/// @brief Returns the Wikipedia-derived context for a location. /**
* @brief Returns the Wikipedia-derived context for a location.
*/
[[nodiscard]] std::string GetLocationContext(const Location& loc) override; [[nodiscard]] std::string GetLocationContext(const Location& loc) override;
private: private:
std::string FetchExtract(std::string_view query); std::string FetchExtract(std::string_view query);
std::unique_ptr<WebClient> client_; std::unique_ptr<WebClient> client_;
std::shared_ptr<ILogger> logger_; std::shared_ptr<ILogger> logger_;
/// @brief Canonical cache for raw Wikipedia query extracts. /**
* @brief Cache for raw Wikipedia query extracts, keyed by query string.
*
* GetLocationContext() always queries "brewing" and reuses "beer in
* {country}" for every city in that country, so caching avoids refetching
* the same extract across locations in a run.
*/
std::unordered_map<std::string, std::string> extract_cache_; std::unordered_map<std::string, std::string> extract_cache_;
}; };

View File

@@ -20,10 +20,22 @@
* @brief Severity levels supported by the logging infra. * @brief Severity levels supported by the logging infra.
*/ */
enum class LogLevel { enum class LogLevel {
Debug, ///< Development/debugging information. /**
Info, ///< General informational messages. * @brief Development/debugging information.
Warn, ///< Warning conditions. */
Error, ///< Error conditions. Debug,
/**
* @brief General informational messages.
*/
Info,
/**
* @brief Warning conditions.
*/
Warn,
/**
* @brief Error conditions.
*/
Error,
}; };
/** /**
@@ -34,18 +46,44 @@ enum class LogLevel {
* pipeline that emitted it. * pipeline that emitted it.
*/ */
enum class PipelinePhase { enum class PipelinePhase {
Startup, ///< Initialization and validation. /**
UserGeneration, ///< User profile generation. * @brief Initialization and validation.
BreweryAndBeerGeneration, ///< Brewery and beer data generation. */
CheckinGeneration, ///< Checkin (visit) record generation. Startup,
RatingGeneration, ///< Rating and review generation. /**
FollowGeneration, ///< Follow relationship generation. * @brief Location/context enrichment (e.g. Wikipedia lookups).
Teardown, ///< Finalization and cleanup. */
Enrichment,
/**
* @brief User profile generation.
*/
UserGeneration,
/**
* @brief Brewery and beer data generation.
*/
BreweryAndBeerGeneration,
/**
* @brief Checkin (visit) record generation.
*/
CheckinGeneration,
/**
* @brief Rating and review generation.
*/
RatingGeneration,
/**
* @brief Follow relationship generation.
*/
FollowGeneration,
/**
* @brief Finalization and cleanup.
*/
Teardown,
}; };
/** /**
* @struct LogDTO * @struct LogDTO
* @brief User-provided subset of log fields. Used to capture call-site info transparently. * @brief User-provided subset of log fields. Used to capture call-site info
* transparently.
*/ */
struct LogDTO { struct LogDTO {
LogLevel level; LogLevel level;
@@ -64,25 +102,35 @@ struct LogDTO {
* before the entry is dispatched. * before the entry is dispatched.
*/ */
struct LogEntry { struct LogEntry {
/// @brief Timestamp when the entry was created. /**
* @brief Timestamp when the entry was created.
*/
std::chrono::system_clock::time_point timestamp{}; std::chrono::system_clock::time_point timestamp{};
/// @brief Source location where the log call was made. /**
* @brief Source location where the log call was made.
*/
std::source_location origin{}; std::source_location origin{};
/// @brief Thread responsible for emitting the log. /**
* @brief Thread responsible for emitting the log.
*/
std::thread::id thread_id{}; std::thread::id thread_id{};
/**
/// @brief Severity level of this entry. * @brief Severity level of this entry.
*/
LogLevel level; LogLevel level;
/// @brief Pipeline phase associated with the entry. /**
* @brief Pipeline phase associated with the entry.
*/
PipelinePhase phase; PipelinePhase phase;
/// @brief Log message text. /**
* @brief Log message text.
*/
std::string message; std::string message;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_ENTRY_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_ENTRY_H_

View File

@@ -39,7 +39,8 @@ class ILogger {
*/ */
void Log(LogDTO payload, void Log(LogDTO payload,
std::source_location origin = std::source_location::current(), std::source_location origin = std::source_location::current(),
std::chrono::system_clock::time_point timestamp = std::chrono::system_clock::now(), std::chrono::system_clock::time_point timestamp =
std::chrono::system_clock::now(),
std::thread::id thread_id = std::this_thread::get_id()) { std::thread::id thread_id = std::this_thread::get_id()) {
LogEntry entry; LogEntry entry;
entry.timestamp = timestamp; entry.timestamp = timestamp;

View File

@@ -1,19 +1,18 @@
/** /**
* @file web_client/http_web_client.h * @file web_client/http_web_client.h
* @brief cpp-httplib implementation of the WebClient interface. * @brief cpp-httplib implementation of the WebClient interface.
*/ */
#ifndef BIERGARTEN_PIPELINE_INCLUDES_WEB_CLIENT_HTTP_WEB_CLIENT_H_ #ifndef BIERGARTEN_PIPELINE_INCLUDES_WEB_CLIENT_HTTP_WEB_CLIENT_H_
#define BIERGARTEN_PIPELINE_INCLUDES_WEB_CLIENT_HTTP_WEB_CLIENT_H_ #define BIERGARTEN_PIPELINE_INCLUDES_WEB_CLIENT_HTTP_WEB_CLIENT_H_
#include "web_client/web_client.h"
#include "services/logging/logger.h"
#include <memory> #include <memory>
#include <string> #include <string>
#include <utility> #include <utility>
#include "services/logging/logger.h"
#include "web_client/web_client.h"
/** /**
* @brief WebClient implementation backed by cpp-httplib. * @brief WebClient implementation backed by cpp-httplib.
* *
@@ -26,7 +25,7 @@
* bound to a single origin at construction time. * bound to a single origin at construction time.
*/ */
class HttpWebClient final : public WebClient { class HttpWebClient final : public WebClient {
public: public:
explicit HttpWebClient(std::shared_ptr<ILogger> logger) explicit HttpWebClient(std::shared_ptr<ILogger> logger)
: logger_(std::move(logger)) {} : logger_(std::move(logger)) {}
~HttpWebClient() override = default; ~HttpWebClient() override = default;
@@ -34,7 +33,8 @@ public:
/** /**
* @brief Executes a blocking HTTP/HTTPS GET request against a full URL. * @brief Executes a blocking HTTP/HTTPS GET request against a full URL.
* *
* @param url Fully-qualified URL, e.g. "https://en.wikipedia.org/api/rest_v1/page/summary/Berlin" * @param url Fully-qualified URL, e.g.
* "https://en.wikipedia.org/api/rest_v1/page/summary/Berlin"
* @return Response body on HTTP 2xx; throws std::runtime_error otherwise. * @return Response body on HTTP 2xx; throws std::runtime_error otherwise.
*/ */
std::string Get(const std::string& url) override; std::string Get(const std::string& url) override;
@@ -52,5 +52,4 @@ public:
std::shared_ptr<ILogger> logger_; std::shared_ptr<ILogger> logger_;
}; };
#endif #endif

View File

@@ -13,7 +13,6 @@
*/ */
class WebClient { class WebClient {
public: public:
/// @brief Virtual destructor for polymorphic cleanup.
virtual ~WebClient() = default; virtual ~WebClient() = default;
/** /**

View File

@@ -0,0 +1,42 @@
[
{
"name": "Hophead Explorer",
"description": "Chases the newest hop varieties and double-digit IBUs, tracking hazy releases and limited tap takeovers across town.",
"style_affinities": ["American IPA", "Double IPA", "New England IPA", "Black IPA"]
},
{
"name": "Malt & Tradition Loyalist",
"description": "Prefers clean, balanced lagers brewed the way their region has brewed them for generations, and is quick to defend a well-made Helles.",
"style_affinities": ["Munich Helles", "Vienna Lager", "Märzen", "Festbier"]
},
{
"name": "Dark & Roasty Devotee",
"description": "Orders the darkest beer on the board year-round, drawn to roasted malt, coffee, and chocolate notes over carbonated lightness.",
"style_affinities": ["Imperial Stout", "Robust Porter", "Baltic Porter", "Oatmeal Stout"]
},
{
"name": "Sour & Wild Forager",
"description": "Seeks out barrel-aged, spontaneously fermented, and funky beers, and will travel for a fresh bottling of something wild.",
"style_affinities": ["Lambic", "Gueuze", "Flanders Red Ale", "Wild Ale"]
},
{
"name": "Session Sipper",
"description": "Values an easy-drinking pint that holds up over a long afternoon at the taproom more than raw intensity or novelty.",
"style_affinities": ["Session IPA", "Ordinary Bitter", "Cream Ale", "Blonde Ale"]
},
{
"name": "Belgian Abbey Pilgrim",
"description": "Collects abbey and trappist releases, drawn to the dried-fruit esters and warming strength of classic Belgian ales.",
"style_affinities": ["Belgian Tripel", "Belgian Dubbel", "Belgian Quadrupel", "Trappist Single"]
},
{
"name": "Wheat & Citrus Casual",
"description": "Reaches for something light, cloudy, and refreshing, usually with a citrus or coriander note, especially on a patio in summer.",
"style_affinities": ["Hefeweizen", "Witbier", "American Wheat Beer", "Berliner Weisse"]
},
{
"name": "Big & Boozy Connoisseur",
"description": "Sips slowly from a snifter, seeking out high-ABV releases meant for aging and savoring rather than quick refreshment.",
"style_affinities": ["English Barleywine", "Wee Heavy", "Doppelbock", "Old Ale"]
}
]

View File

@@ -0,0 +1,122 @@
# FULL SYSTEM PROMPT
You are a craft beer community profile writer. You write short, first-person
biographies for fictional users of a brewery-discovery app, grounded in the
exact name, locale, and persona provided. You do not invent the user's name --
it is given to you and must not be altered, translated, or abbreviated beyond
forming a username.
You will receive the inputs like this:
## NAME:
[First name] [Last name]
## GENDER:
[Gender associated with the given first name]
## CITY:
[City Name]
## COUNTRY:
[Country Name]
## PERSONA:
[Persona archetype name]
## PERSONA DESCRIPTION:
[What this persona cares about and how they talk about beer]
## STYLE AFFINITIES:
[Comma-separated beer styles this persona favors]
## CRITICAL OUTPUT FORMAT (READ CAREFULLY):
ABSOLUTELY NO MARKDOWN FORMATTING. Do NOT wrap your response in json or ```
blocks.
Do not add markdown, code fences, or postscript around the final JSON object.
Do not say "Here is the JSON" or "Enjoy!".
The JSON must contain exactly three keys ("username", "bio",
"activity_weight") in that order. Do not rename or add any other keys.
ESCAPE ALL QUOTES inside the bio field using \", or use single quotes (' ')
instead.
DO NOT use actual line breaks (\n) inside any string. Keep the bio as one
continuous string.
The bio must be between 25 and 60 words, written in first person, and must
read as something this specific person would write about themselves -- not a
generic profile blurb.
`activity_weight` is a number between 0 and 1 (inclusive) representing how
often this persona checks in at breweries relative to other users. Persona
archetypes implying frequent, habitual brewery visits should skew higher;
casual or occasional-visitor personas should skew lower. Do not default to
0.5 -- vary it meaningfully based on the persona description.
Expected JSON format:
```json
{
"username": "a handle derived from the given name",
"bio": "The first-person bio goes here.",
"activity_weight": 0.62
}
```
## CONTENT RULES AND CONSTRAINTS:
### THE USERNAME:
Derive the username from the given first and/or last name (e.g. lowercase,
abbreviated, or combined with a short word or number tied to the persona's
style affinities). Do not invent an unrelated handle. Do not include spaces.
### THE BIO:
Ground the bio in the supplied persona description and style affinities --
mention at least one specific beer style from the style affinities list, in a
way that sounds like a personal preference rather than a list. Reference the
city or country naturally if it fits, but do not force it.
### VOICE & PERSPECTIVE:
Write in the first person ("I"/"my"). The tone should match the persona:
an enthusiastic explorer sounds different from a relaxed, easy-drinking
regular. Do not write in second or third person.
### THE BLOCKLIST (FORBIDDEN CONCEPTS):
You absolutely cannot use the following words and phrases. Make sure your
final output doesn't have any of these:
- "hidden gem"
- "passion"
- "authentic"
- "craft beer enthusiast"
- "beer connoisseur"
- "journey"
#### FORBIDDEN WRITING PATTERNS
The following patterns are common AI writing pitfalls and must not appear in
the bio:
- Negative parallelism constructions: "It's not X, it's Y"
- Inflated significance phrases: "stands as a testament," "plays a vital
role," "deeply rooted"
- Superficial trailing analyses: sentences ending in -ing words that add
opinion without content
- Promotional travel-copy tone: "breathtaking," "must-visit," "vibrant"
- Overused conjunctive transitions used as sentence openers: "Moreover,"
"Furthermore," "In addition"
- Rule of three: do not consistently organise ideas or examples in triplets

8
tooling/pipeline/run.sh Normal file
View File

@@ -0,0 +1,8 @@
./biergarten-pipeline \
--model ../models/google_gemma-4-E4B-it-Q6_K.gguf \
--temperature 1.0 \
--top-p 0.95 \
--top-k 64 \
--n-ctx 8192 \
--location-count 15 \
--prompt-dir ../prompts

View File

@@ -10,11 +10,10 @@ dockerStartCmd: []
isPublic: false isPublic: false
isServerless: false isServerless: false
env: env:
BIERGARTEN_MODE: live
BIERGARTEN_MODEL_PATH: /workspace/models/google_gemma-4-E4B-it-Q6_K.gguf BIERGARTEN_MODEL_PATH: /workspace/models/google_gemma-4-E4B-it-Q6_K.gguf
BIERGARTEN_PROMPT_DIR: /workspace/app/build/prompts
BIERGARTEN_OUTPUT_DIR: /workspace/output BIERGARTEN_OUTPUT_DIR: /workspace/output
BIERGARTEN_LOG_PATH: /workspace/logs/pipeline.log BIERGARTEN_LOG_PATH: /workspace/logs/pipeline.log
BIERGARTEN_GL_LAYERS: "40"
BIERGARTEN_TEMPERATURE: "1.0" BIERGARTEN_TEMPERATURE: "1.0"
BIERGARTEN_TOP_P: "0.95" BIERGARTEN_TOP_P: "0.95"
BIERGARTEN_TOP_K: "64" BIERGARTEN_TOP_K: "64"

View File

@@ -15,8 +15,6 @@ std::optional<ApplicationOptions> ParseArguments(
opt("help,h", "Produce help message"); opt("help,h", "Produce help message");
// Defaults sourced from SamplingOptions{} so the CLI and LlamaGenerator
// share a single source of truth — changing the struct updates both.
auto add_sampling_options = [&]() -> void { auto add_sampling_options = [&]() -> void {
const SamplingOptions sampling_defaults{}; const SamplingOptions sampling_defaults{};
opt("temperature", opt("temperature",
@@ -53,7 +51,8 @@ std::optional<ApplicationOptions> ParseArguments(
prog_opts::value<std::string>()->default_value("pipeline.log"), prog_opts::value<std::string>()->default_value("pipeline.log"),
"Path for application logs"); "Path for application logs");
opt("prompt-dir", prog_opts::value<std::string>()->default_value(""), opt("prompt-dir", prog_opts::value<std::string>()->default_value(""),
"Directory containing named prompt files (e.g. BREWERY_GENERATION.md)." "Directory containing named prompt files (e.g. "
"BREWERY_GENERATION.md)."
" Required when not using --mocked."); " Required when not using --mocked.");
opt("location-count", prog_opts::value<uint32_t>()->default_value(10)); opt("location-count", prog_opts::value<uint32_t>()->default_value(10));
}; };
@@ -72,11 +71,11 @@ std::optional<ApplicationOptions> ParseArguments(
})(); })();
if (logger) { if (logger) {
logger->Log(LogDTO{.level = LogLevel::Info, logger->Log(LogDTO{.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = title}); .message = title});
logger->Log(LogDTO{.level = LogLevel::Info, logger->Log(LogDTO{.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = usage}); .message = usage});
} }
return std::nullopt; return std::nullopt;
} }
@@ -91,8 +90,8 @@ std::optional<ApplicationOptions> ParseArguments(
help_stream << "\n" << desc; help_stream << "\n" << desc;
if (logger) { if (logger) {
logger->Log(LogDTO{.level = LogLevel::Info, logger->Log(LogDTO{.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = help_stream.str()}); .message = help_stream.str()});
} }
return std::nullopt; return std::nullopt;
} }
@@ -108,14 +107,16 @@ std::optional<ApplicationOptions> ParseArguments(
const std::string model_path = var_map["model"].as<std::string>(); const std::string model_path = var_map["model"].as<std::string>();
const int n_gpu_layers = var_map["n-gpu-layers"].as<int>(); const int n_gpu_layers = var_map["n-gpu-layers"].as<int>();
// Enforce mutual exclusivity before any further configuration is applied. // Enforce mutual exclusivity before any further configuration is
// applied.
if (use_mocked && !model_path.empty()) { if (use_mocked && !model_path.empty()) {
const std::string msg = const std::string msg =
"Invalid arguments: --mocked and --model are mutually exclusive"; "Invalid arguments: --mocked and --model are mutually "
"exclusive";
if (logger) { if (logger) {
logger->Log(LogDTO{.level = LogLevel::Error, logger->Log(LogDTO{.level = LogLevel::Error,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = msg}); .message = msg});
} else { } else {
std::cerr << msg << std::endl; std::cerr << msg << std::endl;
} }
@@ -124,11 +125,12 @@ std::optional<ApplicationOptions> ParseArguments(
if (!use_mocked && model_path.empty()) { if (!use_mocked && model_path.empty()) {
const std::string msg = const std::string msg =
"Invalid arguments: either --mocked or --model must be specified"; "Invalid arguments: either --mocked or --model must be "
"specified";
if (logger) { if (logger) {
logger->Log(LogDTO{.level = LogLevel::Error, logger->Log(LogDTO{.level = LogLevel::Error,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = msg}); .message = msg});
} else { } else {
std::cerr << msg << std::endl; std::cerr << msg << std::endl;
} }
@@ -139,7 +141,8 @@ std::optional<ApplicationOptions> ParseArguments(
// generator has no use for it and should not require it to be present. // generator has no use for it and should not require it to be present.
if (!use_mocked && options.pipeline.prompt_dir.empty()) { if (!use_mocked && options.pipeline.prompt_dir.empty()) {
const std::string msg = const std::string msg =
"Invalid arguments: --prompt-dir is required when not using --mocked"; "Invalid arguments: --prompt-dir is required when not using "
"--mocked";
if (logger) { if (logger) {
logger->Log({.level = LogLevel::Error, logger->Log({.level = LogLevel::Error,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
@@ -152,7 +155,6 @@ std::optional<ApplicationOptions> ParseArguments(
options.generator.use_mocked = use_mocked; options.generator.use_mocked = use_mocked;
options.generator.model_path = model_path; options.generator.model_path = model_path;
// options.generator.n_gpu_layers = n_gpu_layers;
// Only populate sampling config when the user explicitly overrides at // Only populate sampling config when the user explicitly overrides at
// least one value. Leaving it as std::nullopt lets LlamaGenerator fall // least one value. Leaving it as std::nullopt lets LlamaGenerator fall
@@ -171,8 +173,8 @@ std::optional<ApplicationOptions> ParseArguments(
"Sampling parameters are ignored when using --mocked"; "Sampling parameters are ignored when using --mocked";
if (logger) { if (logger) {
logger->Log(LogDTO{.level = LogLevel::Warn, logger->Log(LogDTO{.level = LogLevel::Warn,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = msg}); .message = msg});
} else { } else {
std::cerr << msg << std::endl; std::cerr << msg << std::endl;
} }
@@ -197,8 +199,8 @@ std::optional<ApplicationOptions> ParseArguments(
exception.what(); exception.what();
if (logger) { if (logger) {
logger->Log(LogDTO{.level = LogLevel::Error, logger->Log(LogDTO{.level = LogLevel::Error,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = msg}); .message = msg});
} }
return std::nullopt; return std::nullopt;
} catch (...) { } catch (...) {
@@ -206,8 +208,8 @@ std::optional<ApplicationOptions> ParseArguments(
"Failed to parse command-line arguments: unknown error"; "Failed to parse command-line arguments: unknown error";
if (logger) { if (logger) {
logger->Log(LogDTO{.level = LogLevel::Error, logger->Log(LogDTO{.level = LogLevel::Error,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = msg}); .message = msg});
} }
return std::nullopt; return std::nullopt;
} }

View File

@@ -1,6 +1,6 @@
/** /**
* @file biergarten_pipeline_orchestrator/biergarten_pipeline_orchestrator.cc * @file biergarten_pipeline_orchestrator/biergarten_pipeline_orchestrator.cc
* @brief BiergartenDataGenerator constructor implementation. * @brief BiergartenPipelineOrchestrator constructor implementation.
*/ */
#include "biergarten_pipeline_orchestrator.h" #include "biergarten_pipeline_orchestrator.h"
@@ -12,9 +12,12 @@ BiergartenPipelineOrchestrator::BiergartenPipelineOrchestrator(
std::unique_ptr<IEnrichmentService> context_service, std::unique_ptr<IEnrichmentService> context_service,
std::unique_ptr<DataGenerator> generator, std::unique_ptr<DataGenerator> generator,
std::unique_ptr<IExportService> exporter, std::unique_ptr<IExportService> exporter,
const ApplicationOptions &app_options) std::unique_ptr<ICuratedDataService> curated_data_service,
: logger_(std::move(logger)), const ApplicationOptions& application_options)
context_service_(std::move(context_service)), : logger_(std::move(logger)),
generator_(std::move(generator)), context_service_(std::move(context_service)),
exporter_(std::move(exporter)), generator_(std::move(generator)),
application_options_(app_options) {} exporter_(std::move(exporter)),
curated_data_service_(std::move(curated_data_service)),
application_options_(application_options) {
}

View File

@@ -1,10 +1,11 @@
/** /**
* @file biergarten_pipeline_orchestrator/generate_breweries.cc * @file biergarten_pipeline_orchestrator/generate_breweries.cc
* @brief BiergartenDataGenerator::GenerateBreweries() implementation. * @brief BiergartenPipelineOrchestrator::GenerateBreweries() implementation.
*/ */
#include <chrono> #include <chrono>
#include <format> #include <format>
#include <optional>
#include "biergarten_pipeline_orchestrator.h" #include "biergarten_pipeline_orchestrator.h"
#include "services/logging/logger.h" #include "services/logging/logger.h"
@@ -19,50 +20,69 @@ void BiergartenPipelineOrchestrator::GenerateBreweries(
size_t skipped_count = 0; size_t skipped_count = 0;
size_t export_failed_count = 0; size_t export_failed_count = 0;
for (const auto& [location, region_context] : cities) { const auto generate_record =
[this, &skipped_count](
const Location& location,
const std::string& region_context) -> std::optional<BreweryRecord> {
try { try {
const BreweryResult brewery = const BreweryResult brewery =
generator_->GenerateBrewery(location, region_context); generator_->GenerateBrewery(location, region_context);
return BreweryRecord{.location = location, .brewery = brewery};
const GeneratedBrewery gen{.location = location, .brewery = brewery};
generated_breweries_.push_back(gen);
try {
exporter_->ProcessRecord(gen);
} catch (const std::exception& export_exception) {
++export_failed_count;
logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::BreweryAndBeerGeneration,
.message =
std::format("[Pipeline] Generated brewery for '{}' ({}) but SQLite export failed: {}",
location.city, location.country, export_exception.what())});
}
} catch (const std::exception& e) { } catch (const std::exception& e) {
++skipped_count; ++skipped_count;
logger_->Log({.level = LogLevel::Warn, logger_->Log(
.phase = PipelinePhase::BreweryAndBeerGeneration, {.level = LogLevel::Warn,
.message = std::format("[Pipeline] Skipping city '{}' ({}): brewery generation failed: {}", .phase = PipelinePhase::BreweryAndBeerGeneration,
location.city, location.country, e.what())}); .message = std::format("[Pipeline] Skipping city '{}' ({}): brewery "
"generation failed: {}",
location.city, location.country, e.what())});
return std::nullopt;
} }
};
const auto export_record = [this, &export_failed_count](
const BreweryRecord& record) {
try {
exporter_->ProcessRecord(record);
} catch (const std::exception& export_exception) {
++export_failed_count;
logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::BreweryAndBeerGeneration,
.message = std::format("[Pipeline] Generated brewery for '{}' ({}) "
"but SQLite export failed: {}",
record.location.city, record.location.country,
export_exception.what())});
}
};
for (const auto& [location, region_context] : cities) {
const std::optional<BreweryRecord> record =
generate_record(location, region_context);
if (!record.has_value()) {
continue;
}
generated_breweries_.push_back(*record);
export_record(*record);
} }
if (skipped_count > 0) { if (skipped_count > 0) {
logger_->Log({.level = LogLevel::Warn, logger_->Log(
.phase = PipelinePhase::BreweryAndBeerGeneration, {.level = LogLevel::Warn,
.message = std::format( .phase = PipelinePhase::BreweryAndBeerGeneration,
"[Pipeline] Skipped {} city/cities due to generation errors", .message = std::format(
skipped_count)}); "[Pipeline] Skipped {} city/cities due to generation errors",
skipped_count)});
} }
if (export_failed_count > 0) { if (export_failed_count > 0) {
logger_->Log({.level = LogLevel::Warn, logger_->Log(
.phase = PipelinePhase::Teardown, {.level = LogLevel::Warn,
.message = std::format( .phase = PipelinePhase::Teardown,
"[Pipeline] Failed to export {} generated brewery/breweries to SQLite", .message = std::format("[Pipeline] Failed to export {} generated "
export_failed_count)}); "brewery/breweries to SQLite",
export_failed_count)});
} }
} }

View File

@@ -0,0 +1,188 @@
/**
* @file biergarten_pipeline_orchestrator/generate_users.cc
* @brief BiergartenPipelineOrchestrator::GenerateUsers() implementation.
*/
#include <cctype>
#include <chrono>
#include <format>
#include <optional>
#include <random>
#include <stdexcept>
#include <string>
#include <string_view>
#include <unordered_set>
#include "biergarten_pipeline_orchestrator.h"
#include "json_handling/json_loader.h"
#include "services/logging/logger.h"
namespace {
std::string Sanitize(std::string_view value) {
std::string out;
out.reserve(value.size());
for (const char character : value) {
if (std::isalnum(static_cast<unsigned char>(character)) != 0) {
out.push_back(static_cast<char>(
std::tolower(static_cast<unsigned char>(character))));
}
}
return out;
}
std::string BuildEmail(const Name& name,
std::unordered_set<std::string>& used_local_parts) {
const std::string base =
std::format("{}.{}", Sanitize(name.first_name), Sanitize(name.last_name));
std::string local_part = base;
uint32_t suffix = 1;
while (used_local_parts.contains(local_part)) {
local_part = std::format("{}{}", base, suffix);
++suffix;
}
used_local_parts.insert(local_part);
return std::format("{}@thebiergarten.app", local_part);
}
std::string GenerateDateOfBirth(std::mt19937& rng) {
using namespace std::chrono;
constexpr int kMinAge = 19;
constexpr int kMaxAge = 48;
constexpr int kMaxDayOffset = 364;
std::uniform_int_distribution<int> age_dist(kMinAge, kMaxAge);
std::uniform_int_distribution<int> day_offset_dist(0, kMaxDayOffset);
const year_month_day today{floor<days>(system_clock::now())};
const year_month_day birth_year_anchor{today.year() - years{age_dist(rng)},
today.month(), today.day()};
const sys_days birth_date =
sys_days{birth_year_anchor} - days{day_offset_dist(rng)};
const year_month_day birth_ymd{birth_date};
return std::format("{:04}-{:02}-{:02}", static_cast<int>(birth_ymd.year()),
static_cast<unsigned>(birth_ymd.month()),
static_cast<unsigned>(birth_ymd.day()));
}
} // namespace
void BiergartenPipelineOrchestrator::GenerateUsers(
std::span<const EnrichedCity> cities) {
logger_->Log({.level = LogLevel::Info,
.phase = PipelinePhase::UserGeneration,
.message = "=== SAMPLE USER GENERATION ==="});
const std::vector<UserPersona> personas =
curated_data_service_->LoadPersonas("personas.json");
if (personas.empty()) {
throw std::runtime_error(
"No personas available in personas.json for user generation");
}
const NamesByCountry names_by_country =
curated_data_service_->LoadNamesByCountry(
"forenames-by-country.json", "surnames-by-country.json");
std::mt19937 rng(std::random_device{}());
std::uniform_int_distribution<size_t> persona_dist(0, personas.size() - 1);
generated_users_.clear();
size_t skipped_count = 0;
size_t export_failed_count = 0;
const auto generate_record =
[this, &rng, &skipped_count](
const EnrichedCity& city, const UserPersona& persona,
const Name& sampled_name) -> std::optional<UserRecord> {
try {
std::unordered_set<std::string> used_email_local_parts;
const UserResult user =
generator_->GenerateUser(city, persona, sampled_name);
return UserRecord{
.location = city.location,
.user = user,
.email = BuildEmail(sampled_name, used_email_local_parts),
.date_of_birth = GenerateDateOfBirth(rng),
};
} catch (const std::exception& e) {
++skipped_count;
logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration,
.message = std::format(
"[Pipeline] Skipping city '{}' ({}): "
"user generation failed: {}",
city.location.city, city.location.country, e.what())});
return std::nullopt;
}
};
const auto export_record = [this,
&export_failed_count](const UserRecord& record) {
try {
exporter_->ProcessRecord(record);
} catch (const std::exception& export_exception) {
++export_failed_count;
logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration,
.message = std::format("[Pipeline] Generated user for '{}' ({}) but "
"SQLite export failed: {}",
record.location.city, record.location.country,
export_exception.what())});
}
};
for (const auto& city : cities) {
const std::optional<Name> sampled_name =
names_by_country.SampleName(city.location.iso3166_1, rng);
if (!sampled_name.has_value()) {
++skipped_count;
logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration,
.message = std::format(
"[Pipeline] Skipping city '{}' ({}): no names "
"available for country '{}'",
city.location.city, city.location.country,
city.location.iso3166_1)});
continue;
}
const UserPersona& persona = personas[persona_dist(rng)];
const std::optional<UserRecord> record =
generate_record(city, persona, *sampled_name);
if (!record.has_value()) {
continue;
}
generated_users_.push_back(*record);
export_record(*record);
}
if (skipped_count > 0) {
logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration,
.message = std::format(
"[Pipeline] Skipped {} city/cities during user generation",
skipped_count)});
}
if (export_failed_count > 0) {
logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::Teardown,
.message = std::format("[Pipeline] Failed to export {} "
"generated user/users to SQLite",
export_failed_count)});
}
}

View File

@@ -1,6 +1,6 @@
/** /**
* @file biergarten_pipeline_orchestrator/log_results.cc * @file biergarten_pipeline_orchestrator/log_results.cc
* @brief BiergartenDataGenerator::LogResults() implementation. * @brief BiergartenPipelineOrchestrator::LogResults() implementation.
*/ */
#include <boost/json/array.hpp> #include <boost/json/array.hpp>
@@ -10,11 +10,12 @@
#include "../../includes/json_handling/pretty_print.h" #include "../../includes/json_handling/pretty_print.h"
#include "biergarten_pipeline_orchestrator.h" #include "biergarten_pipeline_orchestrator.h"
#include "services/logging/logger.h" #include "services/logging/logger.h"
void BiergartenPipelineOrchestrator::LogResults() const { void BiergartenPipelineOrchestrator::LogResults() const {
boost::json::array output; boost::json::array brewery_output;
for (const auto& [location, brewery] : generated_breweries_) { for (const auto& [location, brewery] : generated_breweries_) {
output.push_back(boost::json::object{ brewery_output.push_back(boost::json::object{
{"name_en", brewery.name_en}, {"name_en", brewery.name_en},
{"description_en", brewery.description_en}, {"description_en", brewery.description_en},
{"name_local", brewery.name_local}, {"name_local", brewery.name_local},
@@ -29,9 +30,30 @@ void BiergartenPipelineOrchestrator::LogResults() const {
}}}); }}});
} }
std::ostringstream oss; std::ostringstream brewery_oss;
PrettyPrint(oss, output); PrettyPrint(brewery_oss, brewery_output);
logger_->Log({.level = LogLevel::Info, logger_->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Teardown, .phase = PipelinePhase::Teardown,
.message = oss.str()}); .message = brewery_oss.str()});
boost::json::array user_output;
for (const auto& generated_user : generated_users_) {
user_output.push_back(boost::json::object{
{"first_name", generated_user.user.first_name},
{"last_name", generated_user.user.last_name},
{"gender", generated_user.user.gender},
{"username", generated_user.user.username},
{"bio", generated_user.user.bio},
{"activity_weight", generated_user.user.activity_weight},
{"email", generated_user.email},
{"date_of_birth", generated_user.date_of_birth},
});
}
std::ostringstream user_oss;
PrettyPrint(user_oss, user_output);
logger_->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Teardown,
.message = user_oss.str()});
} }

View File

@@ -1,6 +1,7 @@
/** /**
* @file biergarten_pipeline_orchestrator/query_cities_with_countries.cc * @file biergarten_pipeline_orchestrator/query_cities_with_countries.cc
* @brief BiergartenDataGenerator::QueryCitiesWithCountries() implementation. * @brief BiergartenPipelineOrchestrator::QueryCitiesWithCountries()
* implementation.
*/ */
#include <algorithm> #include <algorithm>
@@ -22,7 +23,7 @@ BiergartenPipelineOrchestrator::QueryCitiesWithCountries() {
const std::filesystem::path locations_path = "locations.json"; const std::filesystem::path locations_path = "locations.json";
auto all_locations = JsonLoader::LoadLocations(locations_path, logger_); auto all_locations = curated_data_service_->LoadLocations(locations_path);
const size_t sample_count = std::min( const size_t sample_count = std::min(
static_cast<size_t>(application_options_.pipeline.location_count), static_cast<size_t>(application_options_.pipeline.location_count),

View File

@@ -1,6 +1,6 @@
/** /**
* @file biergarten_pipeline_orchestrator/run.cc * @file biergarten_pipeline_orchestrator/run.cc
* @brief BiergartenDataGenerator::Run() implementation. * @brief BiergartenPipelineOrchestrator::Run() implementation.
*/ */
#include <chrono> #include <chrono>
@@ -22,42 +22,40 @@ bool BiergartenPipelineOrchestrator::Run() {
for (auto& city : cities) { for (auto& city : cities) {
try { try {
std::string region_context = context_service_->GetLocationContext(city); std::string region_context = context_service_->GetLocationContext(city);
// logger_->Log(LogLevel::Debug, PipelinePhase::UserGeneration,
// "[Pipeline] Context for '" + city.city + "' (" +
// city.iso3166_2 + ") gathered:\n" + region_context);
enriched.push_back( enriched.push_back(
EnrichedCity{.location = std::move(city), EnrichedCity{.location = std::move(city),
.region_context = std::move(region_context)}); .region_context = std::move(region_context)});
} catch (const std::exception& exception) { } catch (const std::exception& exception) {
++skipped_count; ++skipped_count;
logger_->Log( logger_->Log({.level = LogLevel::Warn,
{.level = LogLevel::Warn, .phase = PipelinePhase::Enrichment,
.phase = PipelinePhase::UserGeneration, .message = std::format(
.message = std::format( "[Pipeline] Skipping city '{}' ({}): context "
"[Pipeline] Skipping city '{}' ({}): context lookup failed: {}", "lookup failed: {}",
city.city, city.country, exception.what())}); city.city, city.country, exception.what())});
} }
} }
if (skipped_count > 0) { if (skipped_count > 0) {
logger_->Log({.level = LogLevel::Warn, logger_->Log(
.phase = PipelinePhase::UserGeneration, {.level = LogLevel::Warn,
.message = std::format( .phase = PipelinePhase::Enrichment,
"[Pipeline] Skipped {} city/cities due to context lookup errors", .message = std::format("[Pipeline] Skipped {} city/cities due "
skipped_count)}); "to context lookup errors",
skipped_count)});
} }
this->GenerateUsers(enriched);
this->GenerateBreweries(enriched); this->GenerateBreweries(enriched);
exporter_->Finalize(); exporter_->Finalize();
this->LogResults(); this->LogResults();
return true; return true;
} catch (const std::exception& e) { } catch (const std::exception& e) {
logger_->Log( logger_->Log({.level = LogLevel::Error,
{.level = LogLevel::Error, .phase = PipelinePhase::Teardown,
.phase = PipelinePhase::Teardown, .message = std::format(
.message = "Pipeline execution failed with error: {}", e.what())});
std::format("Pipeline execution failed with error: {}", e.what())});
return false; return false;
} }
} }

View File

@@ -12,6 +12,7 @@
#include <string_view> #include <string_view>
#include <vector> #include <vector>
#include "data_generation/json_grammars.h"
#include "data_generation/llama_generator.h" #include "data_generation/llama_generator.h"
#include "data_generation/llama_generator_helpers.h" #include "data_generation/llama_generator_helpers.h"
@@ -32,19 +33,6 @@ static std::string FormatLocalLanguageCodes(
return formatted; return formatted;
} }
// GBNF grammar for structured brewery JSON output.
// @TODO move to a separate gbnf file if it grows in complexity or is shared
// across modules.
static constexpr std::string_view kBreweryJsonGrammar = R"json_brewery(
root ::= thought-block "{" ws "\"name_en\"" ws ":" ws string ws "," ws "\"description_en\"" ws ":" ws string ws "," ws "\"name_local\"" ws ":" ws string ws "," ws "\"description_local\"" ws ":" ws string ws "}" ws
thought-block ::= [^{]*
ws ::= [ \t\n\r]*
string ::= "\"" char+ "\""
char ::= [^"\\\x7F\x00-\x1F] | [\\] escape
escape ::= ["\\/bfnrt] | "u" hex hex hex hex
hex ::= [0-9a-fA-F]
)json_brewery";
static constexpr int kBreweryInitialMaxTokens = 2800; static constexpr int kBreweryInitialMaxTokens = 2800;
BreweryResult LlamaGenerator::GenerateBrewery( BreweryResult LlamaGenerator::GenerateBrewery(
@@ -83,15 +71,14 @@ BreweryResult LlamaGenerator::GenerateBrewery(
/** /**
* RETRY LOOP with validation and error correction * RETRY LOOP with validation and error correction
* Attempts to generate valid brewery data up to 3 times, with feedback-based * Attempts to generate valid brewery data up to 3 times, with
* refinement * feedback-based refinement
*/ */
constexpr int max_attempts = 3; constexpr int max_attempts = 3;
std::string raw; std::string raw;
std::string last_error; std::string last_error;
// Token budget: too small risks truncating valid JSON mid-string. // Token budget: too small risks truncating valid JSON mid-string.
// Start conservatively but allow adaptive increases on truncation.
int max_tokens = kBreweryInitialMaxTokens; int max_tokens = kBreweryInitialMaxTokens;
// Limit output length to keep it concise and focused // Limit output length to keep it concise and focused
@@ -104,7 +91,7 @@ BreweryResult LlamaGenerator::GenerateBrewery(
{.level = LogLevel::Debug, {.level = LogLevel::Debug,
.phase = PipelinePhase::BreweryAndBeerGeneration, .phase = PipelinePhase::BreweryAndBeerGeneration,
.message = std::format("LlamaGenerator: raw output (attempt {}): {}", .message = std::format("LlamaGenerator: raw output (attempt {}): {}",
attempt + 1, raw)}); attempt + 1, raw)});
} }
// Validate output: parse JSON and check required fields // Validate output: parse JSON and check required fields
@@ -120,8 +107,9 @@ BreweryResult LlamaGenerator::GenerateBrewery(
logger_->Log( logger_->Log(
{.level = LogLevel::Info, {.level = LogLevel::Info,
.phase = PipelinePhase::BreweryAndBeerGeneration, .phase = PipelinePhase::BreweryAndBeerGeneration,
.message = std::format("LlamaGenerator: successfully generated brewery data on attempt {}", .message = std::format("LlamaGenerator: successfully generated "
attempt + 1)}); "brewery data on attempt {}",
attempt + 1)});
} }
return brewery; return brewery;
@@ -134,33 +122,38 @@ BreweryResult LlamaGenerator::GenerateBrewery(
logger_->Log( logger_->Log(
{.level = LogLevel::Warn, {.level = LogLevel::Warn,
.phase = PipelinePhase::BreweryAndBeerGeneration, .phase = PipelinePhase::BreweryAndBeerGeneration,
.message = .message = std::format(
std::format("LlamaGenerator: malformed brewery JSON (attempt {}): {}", "LlamaGenerator: malformed brewery JSON (attempt {}): {}",
attempt + 1, *validation_error)}); attempt + 1, *validation_error)});
} }
// Update prompt with error details to guide LLM toward correct output. // Update prompt with error details to guide LLM toward correct output.
user_prompt = std::format( user_prompt = std::format(
"Your previous response was invalid. Error: {}\nReturn the thought " "Your previous response was invalid. Error: {}\nReturn the thought "
"process before the JSON if needed, then return ONLY valid JSON with " "process before the JSON if needed, then return ONLY valid JSON "
"exactly these keys, in this exact order: {{\"name_en\": \"<English " "with "
"exactly these keys, in this exact order: {{\"name_en\": "
"\"<English "
"brewery name>\", \"description_en\": \"<English single-paragraph " "brewery name>\", \"description_en\": \"<English single-paragraph "
"description>\", \"name_local\": \"<local-language brewery name>\", " "description>\", \"name_local\": \"<local-language brewery "
"name>\", "
"\"description_local\": \"<local-language single-paragraph " "\"description_local\": \"<local-language single-paragraph "
"description>\"}}.\nDo not include markdown, comments, extra keys, or " "description>\"}}.\nDo not include markdown, comments, extra keys, "
"literal placeholder values.\n\nKeep the JSON strings concise enough " "or "
"literal placeholder values.\n\nKeep the JSON strings concise "
"enough "
"to fit within the token budget.\n\n{}", "to fit within the token budget.\n\n{}",
*validation_error, retry_location); *validation_error, retry_location);
} }
// All retry attempts exhausted: log failure and throw exception // All retry attempts exhausted: log failure and throw exception
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Error,
{.level = LogLevel::Error, .phase = PipelinePhase::BreweryAndBeerGeneration,
.phase = PipelinePhase::BreweryAndBeerGeneration, .message = std::format(
.message = std::format( "LlamaGenerator: malformed brewery "
"LlamaGenerator: malformed brewery response after {} attempts: {}", "response after {} attempts: {}",
max_attempts, last_error.empty() ? raw : last_error)}); max_attempts, last_error.empty() ? raw : last_error)});
} }
throw std::runtime_error("LlamaGenerator: malformed brewery response"); throw std::runtime_error("LlamaGenerator: malformed brewery response");
} }

View File

@@ -1,24 +1,112 @@
/** /**
* @file data_generation/llama/generate_user.cc * @file data_generation/llama/generate_user.cc
* @brief Generates locale-aware user profiles with strict two-line formatting, * @brief Builds persona/name-grounded user prompts, performs retry-based
* retry handling, and output sanitization for downstream parsing. * inference, and validates structured JSON output for user records.
*/ */
#include <format> #include <format>
#include <optional>
#include <stdexcept>
#include <string> #include <string>
#include <string_view>
#include "data_generation/json_grammars.h"
#include "data_generation/llama_generator.h" #include "data_generation/llama_generator.h"
#include "data_generation/llama_generator_helpers.h" #include "data_generation/llama_generator_helpers.h"
// TODO: Implement locale-aware user profile generation. static constexpr int kUserInitialMaxTokens = 1200;
// Current implementation returns a hardcoded test value and ignores the
// locale parameter. Future implementation should: UserResult LlamaGenerator::GenerateUser(const EnrichedCity& city,
// 1. Load a USER_GENERATION.md prompt template with locale context const UserPersona& persona,
// 2. Perform LLM inference with locale-specific username/bio generation const Name& name) {
// 3. Parse and validate JSON output with retry handling (similar to brewery) std::string style_affinities;
// 4. Return locale-aware username and biography for (const std::string& style : persona.style_affinities) {
UserResult LlamaGenerator::GenerateUser(const std::string& locale) { if (!style_affinities.empty()) {
return {.username = "test_user", style_affinities += ", ";
.bio = std::format("This is a test user profile from {}.", locale)}; }
style_affinities += style;
}
const std::string system_prompt = prompt_directory_->Load("USER_GENERATION");
std::string user_prompt = std::format(
"## NAME:\n{} {}\n\n## GENDER:\n{}\n\n## CITY:\n{}\n\n## "
"COUNTRY:\n{}\n\n"
"## PERSONA:\n{}\n\n## PERSONA DESCRIPTION:\n{}\n\n## STYLE "
"AFFINITIES:\n{}",
name.first_name, name.last_name, name.gender, city.location.city,
city.location.country, persona.name, persona.description,
style_affinities);
const std::string retry_context = std::format(
"Name: {} {}\nCity: {}, {}\nPersona: {}", name.first_name, name.last_name,
city.location.city, city.location.country, persona.name);
constexpr int max_attempts = 3;
std::string raw;
std::string last_error;
int max_tokens = kUserInitialMaxTokens;
for (int attempt = 0; attempt < max_attempts; ++attempt) {
raw = this->Infer(system_prompt, user_prompt, max_tokens, kUserJsonGrammar);
if (logger_) {
logger_->Log(
{.level = LogLevel::Debug,
.phase = PipelinePhase::UserGeneration,
.message = std::format("LlamaGenerator: raw output (attempt {}): {}",
attempt + 1, raw)});
}
UserResult user;
const std::optional<std::string> validation_error =
ValidateUserJson(raw, user);
if (!validation_error.has_value()) {
if (logger_) {
logger_->Log(
{.level = LogLevel::Info,
.phase = PipelinePhase::UserGeneration,
.message = std::format("LlamaGenerator: successfully "
"generated user data on attempt {}",
attempt + 1)});
}
user.first_name = name.first_name;
user.last_name = name.last_name;
user.gender = name.gender;
return user;
}
last_error = *validation_error;
if (logger_) {
logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration,
.message = std::format(
"LlamaGenerator: malformed user JSON (attempt {}): {}",
attempt + 1, *validation_error)});
}
user_prompt = std::format(
"Your previous response was invalid. Error: {}\nReturn the thought "
"process before the JSON if needed, then return ONLY valid JSON "
"with "
"exactly these keys, in this exact order: {{\"username\": "
"\"<handle "
"derived from the given name>\", \"bio\": \"<first-person bio "
"grounded in the persona>\", \"activity_weight\": <number between "
"0 "
"and 1>}}.\nDo not include markdown, comments, extra keys, or "
"literal placeholder values.\n\n{}",
*validation_error, retry_context);
}
if (logger_) {
logger_->Log({.level = LogLevel::Error,
.phase = PipelinePhase::UserGeneration,
.message = std::format(
"LlamaGenerator: malformed user response "
"after {} attempts: {}",
max_attempts, last_error.empty() ? raw : last_error)});
}
throw std::runtime_error("LlamaGenerator: malformed user response");
} }

View File

@@ -9,6 +9,7 @@
#include <boost/json.hpp> #include <boost/json.hpp>
#include <cctype> #include <cctype>
#include <optional> #include <optional>
#include <span>
#include <stdexcept> #include <stdexcept>
#include <string> #include <string>
#include <string_view> #include <string_view>
@@ -76,7 +77,7 @@ bool ReadRequiredTrimmedStringField(const boost::json::object& obj,
return !out.empty(); return !out.empty();
} }
bool HasSchemaPlaceholder(const std::array<std::string*, 4>& values) { bool HasSchemaPlaceholder(std::span<std::string* const> values) {
for (const std::string* value : values) { for (const std::string* value : values) {
std::string lowered = *value; std::string lowered = *value;
std::ranges::transform(lowered, lowered.begin(), std::ranges::transform(lowered, lowered.begin(),
@@ -207,3 +208,58 @@ std::optional<std::string> ValidateBreweryJson(const std::string& raw,
return std::nullopt; return std::nullopt;
} }
std::optional<std::string> ValidateUserJson(const std::string& raw,
UserResult& user_out) {
boost::system::error_code error_code;
const std::string_view raw_view(raw);
const size_t opening_brace = raw_view.find('{');
if (opening_brace == std::string_view::npos) {
return "JSON parse error: missing opening brace '{'";
}
const std::string_view json_payload = raw_view.substr(opening_brace);
boost::json::value json_value = boost::json::parse(json_payload, error_code);
if (error_code) {
return "JSON parse error: " + error_code.message();
}
if (!json_value.is_object()) {
return "JSON root must be an object";
}
const auto& obj = json_value.get_object();
if (obj.size() != 3) {
return "JSON object must contain exactly three keys";
}
std::string validation_error;
if (!ReadRequiredTrimmedStringField(obj, "username", user_out.username,
&validation_error)) {
return validation_error;
}
if (!ReadRequiredTrimmedStringField(obj, "bio", user_out.bio,
&validation_error)) {
return validation_error;
}
const boost::json::value* activity_weight_field =
obj.if_contains("activity_weight");
if (activity_weight_field == nullptr || !activity_weight_field->is_number()) {
return "Missing or invalid numeric field: activity_weight";
}
const double activity_weight = activity_weight_field->to_number<double>();
if (activity_weight < 0.0 || activity_weight > 1.0) {
return "activity_weight must be between 0 and 1";
}
user_out.activity_weight = static_cast<float>(activity_weight);
const std::array schema_placeholders = {&user_out.username, &user_out.bio};
if (HasSchemaPlaceholder(schema_placeholders)) {
return "JSON appears to be a schema placeholder, not content";
}
return std::nullopt;
}

View File

@@ -14,3 +14,13 @@ size_t MockGenerator::DeterministicHash(const Location& location) {
boost::hash_combine(seed, location.country); boost::hash_combine(seed, location.country);
return seed; return seed;
} }
size_t MockGenerator::DeterministicHash(const Location& location,
const UserPersona& persona,
const Name& name) {
size_t seed = DeterministicHash(location);
boost::hash_combine(seed, persona.name);
boost::hash_combine(seed, name.first_name);
boost::hash_combine(seed, name.last_name);
return seed;
}

View File

@@ -1,22 +1,31 @@
/** /**
* @file data_generation/mock/generate_user.cc * @file data_generation/mock/generate_user.cc
* @brief Generates deterministic mock user profiles by hashing locale values * @brief Generates deterministic mock user profiles by hashing city, persona,
* into predefined username and bio collections. * and sampled name into predefined username and bio collections.
*/ */
#include <functional>
#include <string>
#include <string_view> #include <string_view>
#include "data_generation/mock_generator.h" #include "data_generation/mock_generator.h"
UserResult MockGenerator::GenerateUser(const std::string& locale) { UserResult MockGenerator::GenerateUser(const EnrichedCity& city,
const size_t hash = std::hash<std::string>{}(locale); const UserPersona& persona,
const Name& name) {
const size_t hash = DeterministicHash(city.location, persona, name);
UserResult result;
const std::string_view username = kUsernames[hash % kUsernames.size()]; const std::string_view username = kUsernames[hash % kUsernames.size()];
const std::string_view bio = kBios[hash / kBioHashStride % kBios.size()]; const std::string_view bio = kBios[hash / kBioHashStride % kBios.size()];
result.username = username; const float activity_weight =
result.bio = bio; static_cast<float>(hash / kActivityWeightHashStride %
return result; kActivityWeightHashRange) /
static_cast<float>(kActivityWeightHashRange);
return {
.first_name = name.first_name,
.last_name = name.last_name,
.gender = name.gender,
.username = std::string(username),
.bio = std::string(bio),
.activity_weight = activity_weight,
};
} }

View File

@@ -0,0 +1,40 @@
/**
* @file data_model/names_by_country.cc
* @brief NamesByCountry::SampleName() implementation.
*/
#include "data_model/names_by_country.h"
#include <utility>
NamesByCountry::NamesByCountry(
std::unordered_map<std::string, std::vector<ForenameEntry>>
forenames_by_country,
std::unordered_map<std::string, std::vector<std::string>>
surnames_by_country)
: forenames_by_country_(std::move(forenames_by_country)),
surnames_by_country_(std::move(surnames_by_country)) {}
std::optional<Name> NamesByCountry::SampleName(const std::string& iso3166_1,
std::mt19937& rng) const {
const auto forenames_it = forenames_by_country_.find(iso3166_1);
const auto surnames_it = surnames_by_country_.find(iso3166_1);
if (forenames_it == forenames_by_country_.end() ||
surnames_it == surnames_by_country_.end() ||
forenames_it->second.empty() || surnames_it->second.empty()) {
return std::nullopt;
}
const std::vector<ForenameEntry>& forenames = forenames_it->second;
const std::vector<std::string>& surnames = surnames_it->second;
std::uniform_int_distribution<size_t> forename_dist(0, forenames.size() - 1);
std::uniform_int_distribution<size_t> surname_dist(0, surnames.size() - 1);
const ForenameEntry& forename = forenames[forename_dist(rng)];
return Name{.first_name = forename.name,
.last_name = surnames[surname_dist(rng)],
.gender = forename.gender};
}

View File

@@ -6,16 +6,18 @@
#include "json_handling/json_loader.h" #include "json_handling/json_loader.h"
#include <format>
#include "services/logging/logger.h"
#include <iostream>
#include <boost/json.hpp> #include <boost/json.hpp>
#include <format>
#include <fstream> #include <fstream>
#include <iostream>
#include <sstream> #include <sstream>
#include <stdexcept> #include <stdexcept>
#include <string> #include <string>
#include <string_view> #include <string_view>
#include <unordered_map>
#include <utility>
#include "services/logging/logger.h"
static std::string ReadRequiredString(const boost::json::object& object, static std::string ReadRequiredString(const boost::json::object& object,
const char* key) { const char* key) {
@@ -43,7 +45,7 @@ static std::vector<std::string> ReadRequiredStringArray(
const boost::json::value* value = object.if_contains(key); const boost::json::value* value = object.if_contains(key);
if (value == nullptr || !value->is_array()) { if (value == nullptr || !value->is_array()) {
throw std::runtime_error( throw std::runtime_error(
std::format("Missing or invalid string array field: {}", key)); std::format("Missing or invalid string array field: {}", key));
} }
const auto& array = value->as_array(); const auto& array = value->as_array();
@@ -59,24 +61,54 @@ static std::vector<std::string> ReadRequiredStringArray(
return items; return items;
} }
std::vector<Location> JsonLoader::LoadLocations( namespace {
const std::filesystem::path& filepath, std::shared_ptr<ILogger> logger) {
boost::json::value ParseJsonFile(const std::filesystem::path& filepath,
const char* what) {
std::ifstream input(filepath); std::ifstream input(filepath);
if (!input.is_open()) { if (!input.is_open()) {
throw std::runtime_error("Failed to open locations file: " + throw std::runtime_error(
filepath.string()); std::format("Failed to open {} file: {}", what, filepath.string()));
} }
std::stringstream buffer; std::stringstream buffer;
buffer << input.rdbuf(); buffer << input.rdbuf();
const std::string content = buffer.str();
boost::system::error_code error; boost::system::error_code error;
boost::json::value root = boost::json::parse(content, error); boost::json::value root = boost::json::parse(buffer.str(), error);
if (error) { if (error) {
throw std::runtime_error("Failed to parse locations JSON: " + throw std::runtime_error(
error.message()); std::format("Failed to parse {} JSON: {}", what, error.message()));
} }
return root;
}
/**
* @brief Returns the first element of a string array field, falling back to
* `fallback_key` if `key` is missing/empty (some source entries only have a
* "localized" form and no "romanized" form).
*/
std::string ReadFirstOfStringArray(const boost::json::object& object,
const char* key, const char* fallback_key) {
for (const char* candidate_key : {key, fallback_key}) {
const boost::json::value* value = object.if_contains(candidate_key);
if (value == nullptr || !value->is_array()) {
continue;
}
const auto& array = value->as_array();
if (!array.empty() && array.front().is_string()) {
return std::string(array.front().as_string());
}
}
throw std::runtime_error(
std::format("Missing or invalid string array field: {}", key));
}
} // namespace
std::vector<Location> JsonLoader::LoadLocations(
const std::filesystem::path& filepath) {
const boost::json::value root = ParseJsonFile(filepath, "locations");
if (!root.is_array()) { if (!root.is_array()) {
throw std::runtime_error( throw std::runtime_error(
@@ -108,3 +140,97 @@ std::vector<Location> JsonLoader::LoadLocations(
return locations; return locations;
} }
std::vector<UserPersona> JsonLoader::LoadPersonas(
const std::filesystem::path& filepath) {
const boost::json::value root = ParseJsonFile(filepath, "personas");
if (!root.is_array()) {
throw std::runtime_error(
"Invalid personas JSON: root element must be an array");
}
std::vector<UserPersona> personas;
const auto& items = root.as_array();
personas.reserve(items.size());
for (const auto& item : items) {
if (!item.is_object()) {
throw std::runtime_error(
"Invalid personas JSON: each entry must be an object");
}
const auto& object = item.as_object();
personas.push_back(UserPersona{
.name = ReadRequiredString(object, "name"),
.description = ReadRequiredString(object, "description"),
.style_affinities = ReadRequiredStringArray(object, "style_affinities"),
});
}
return personas;
}
NamesByCountry JsonLoader::LoadNamesByCountry(
const std::filesystem::path& forenames_filepath,
const std::filesystem::path& surnames_filepath) {
const boost::json::value forenames_root =
ParseJsonFile(forenames_filepath, "forenames-by-country");
const boost::json::value surnames_root =
ParseJsonFile(surnames_filepath, "surnames-by-country");
if (!forenames_root.is_object() || !surnames_root.is_object()) {
throw std::runtime_error(
"Invalid names-by-country JSON: root element must be an object "
"keyed by ISO 3166-1 country code");
}
std::unordered_map<std::string, std::vector<ForenameEntry>>
forenames_by_country;
for (const auto& [country_code, regions] : forenames_root.as_object()) {
if (!regions.is_array()) {
continue;
}
std::vector<ForenameEntry> entries;
for (const auto& region : regions.as_array()) {
if (!region.is_object()) {
continue;
}
const boost::json::value* names = region.as_object().if_contains("names");
if (names == nullptr || !names->is_array()) {
continue;
}
for (const auto& name_value : names->as_array()) {
if (!name_value.is_object()) {
continue;
}
const auto& name_object = name_value.as_object();
entries.push_back(ForenameEntry{
.name =
ReadFirstOfStringArray(name_object, "romanized", "localized"),
.gender = ReadRequiredString(name_object, "gender"),
});
}
}
forenames_by_country.emplace(country_code, std::move(entries));
}
std::unordered_map<std::string, std::vector<std::string>> surnames_by_country;
for (const auto& [country_code, name_entries] : surnames_root.as_object()) {
if (!name_entries.is_array()) {
continue;
}
std::vector<std::string> surnames;
for (const auto& name_value : name_entries.as_array()) {
if (!name_value.is_object()) {
continue;
}
surnames.push_back(ReadFirstOfStringArray(name_value.as_object(),
"romanized", "localized"));
}
surnames_by_country.emplace(country_code, std::move(surnames));
}
return NamesByCountry(std::move(forenames_by_country),
std::move(surnames_by_country));
}

View File

@@ -24,7 +24,9 @@
#include "data_generation/mock_generator.h" #include "data_generation/mock_generator.h"
#include "data_generation/prompt_formatting/gemma4_jinja_prompt_formatter.h" #include "data_generation/prompt_formatting/gemma4_jinja_prompt_formatter.h"
#include "data_model/models.h" #include "data_model/models.h"
#include "json_handling/json_loader.h"
#include "llama_backend_state.h" #include "llama_backend_state.h"
#include "services/curated_data/curated_data_service.h"
#include "services/database/export_service.h" #include "services/database/export_service.h"
#include "services/database/sqlite_export_service.h" #include "services/database/sqlite_export_service.h"
#include "services/datetime/timer.h" #include "services/datetime/timer.h"
@@ -47,12 +49,13 @@ int main(const int argc, char** argv) {
spdlog::set_pattern("│ %Y-%m-%d %H:%M:%S.%e │ %^%-7l%$ │ %v"); spdlog::set_pattern("│ %Y-%m-%d %H:%M:%S.%e │ %^%-7l%$ │ %v");
BoundedChannel<LogEntry> log_channel(kLogMaxCount); BoundedChannel<LogEntry> log_channel(kLogMaxCount);
auto log_dispatcher = // auto log_dispatcher = //
std::make_unique<LogDispatcher>(log_channel); std::make_unique<LogDispatcher>(log_channel);
std::shared_ptr<ILogger> log_producer = std::shared_ptr<ILogger> log_producer =
std::make_shared<LogProducer>(log_channel); std::make_shared<LogProducer>(log_channel);
std::thread log_thread([&log_dispatcher] { log_dispatcher->Run(); }); std::thread log_thread([&log_dispatcher] { log_dispatcher->Run(); });
auto shutdown = [&](const int exit_code) { auto shutdown = [&](const int exit_code) {
log_channel.Close(); log_channel.Close();
log_thread.join(); log_thread.join();
@@ -66,8 +69,8 @@ int main(const int argc, char** argv) {
const LlamaBackendState llama_backend_state; const LlamaBackendState llama_backend_state;
#endif #endif
log_producer->Log({.level = LogLevel::Info, log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = "STARTING PIPELINE"}); .message = "STARTING PIPELINE"});
const std::optional<ApplicationOptions> parsed_options = const std::optional<ApplicationOptions> parsed_options =
@@ -89,8 +92,8 @@ int main(const int argc, char** argv) {
prompt_directory = std::make_unique<PromptDirectory>( prompt_directory = std::make_unique<PromptDirectory>(
options.pipeline.prompt_dir, log_producer); options.pipeline.prompt_dir, log_producer);
} catch (const std::exception& dir_error) { } catch (const std::exception& dir_error) {
log_producer->Log({.level = LogLevel::Error, log_producer->Log({.level = LogLevel::Error,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = std::format("Invalid --prompt-dir: {}", .message = std::format("Invalid --prompt-dir: {}",
dir_error.what())}); dir_error.what())});
@@ -103,21 +106,22 @@ int main(const int argc, char** argv) {
di::bind<ApplicationOptions>().to(options), di::bind<ApplicationOptions>().to(options),
di::bind<std::string>().to(model_path), di::bind<std::string>().to(model_path),
di::bind<IExportService>().to<SqliteExportService>(), di::bind<IExportService>().to<SqliteExportService>(),
di::bind<ICuratedDataService>().to<JsonLoader>(),
di::bind<IPromptFormatter>().to([options, log_producer] { di::bind<IPromptFormatter>().to([options, log_producer] {
if (options.generator.use_mocked) { if (options.generator.use_mocked) {
{ {
log_producer->Log( log_producer->Log(
{.level = LogLevel::Info, {.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = "Prompt formatter: none (mock mode)"}); .message = "Prompt formatter: none (mock mode)"});
} }
return std::unique_ptr<IPromptFormatter>(nullptr); return std::unique_ptr<IPromptFormatter>(nullptr);
} }
{ {
log_producer->Log( log_producer->Log(
{.level = LogLevel::Info, {.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = "Prompt formatter: Gemma4JinjaPromptFormatter"}); .message = "Prompt formatter: Gemma4JinjaPromptFormatter"});
} }
return std::unique_ptr<IPromptFormatter>( return std::unique_ptr<IPromptFormatter>(
std::make_unique<Gemma4JinjaPromptFormatter>()); std::make_unique<Gemma4JinjaPromptFormatter>());
@@ -125,61 +129,60 @@ int main(const int argc, char** argv) {
di::bind<WebClient>().to([options, log_producer] { di::bind<WebClient>().to([options, log_producer] {
if (options.generator.use_mocked) { if (options.generator.use_mocked) {
{ {
log_producer->Log({.level = LogLevel::Info, log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = "Web client: none (mock mode)"}); .message = "Web client: none (mock mode)"});
} }
return std::unique_ptr<WebClient>(nullptr); return std::unique_ptr<WebClient>(nullptr);
} }
{
log_producer->Log({.level = LogLevel::Info, log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = "Web client: HttpWebClient"}); .message = "Web client: HttpWebClient"});
}
return std::unique_ptr<WebClient>( return std::unique_ptr<WebClient>(
std::make_unique<HttpWebClient>(log_producer)); std::make_unique<HttpWebClient>(log_producer));
}), }),
di::bind<IEnrichmentService>().to( di::bind<IEnrichmentService>().to(
[options, &log_producer]( [options, &log_producer](
const auto& inj) -> std::unique_ptr<IEnrichmentService> { const auto& inj) -> std::unique_ptr<IEnrichmentService> {
if (options.generator.use_mocked) { if (options.generator.use_mocked) {
{ log_producer->Log({.level = LogLevel::Info,
log_producer->Log({.level = LogLevel::Info, .phase = PipelinePhase::Startup,
.phase = PipelinePhase::Startup, .message = "Enrichment: mock"});
.message = "Enrichment: mock"});
}
return std::make_unique<MockEnrichmentService>(); return std::make_unique<MockEnrichmentService>();
} }
{
log_producer->Log({.level = LogLevel::Info, log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = "Enrichment: Wikipedia"}); .message = "Enrichment: Wikipedia"});
}
return std::make_unique<WikipediaEnrichmentService>( return std::make_unique<WikipediaEnrichmentService>(
inj.template create<std::unique_ptr<WebClient>>(), inj.template create<std::unique_ptr<WebClient>>(),
log_producer); log_producer);
}), }),
di::bind<DataGenerator>().to( di::bind<DataGenerator>().to(
[&options, &model_path, &sampling, &prompt_directory, [&options, &model_path, &sampling, &prompt_directory,
&log_producer](const auto& inj) -> std::unique_ptr<DataGenerator> { &log_producer
](const auto& inj) -> std::unique_ptr<DataGenerator> {
if (options.generator.use_mocked) { if (options.generator.use_mocked) {
{ log_producer->Log({.level = LogLevel::Info,
log_producer->Log({.level = LogLevel::Info, .phase = PipelinePhase::Startup,
.phase = PipelinePhase::Startup, .message = "Generator: mock"});
.message = "Generator: mock"});
}
return std::make_unique<MockGenerator>(); return std::make_unique<MockGenerator>();
} }
{
log_producer->Log( log_producer->Log(
{.level = LogLevel::Info, {.level = LogLevel::Info,
.phase = PipelinePhase::Startup, .phase = PipelinePhase::Startup,
.message = std::format( .message = std::format(
"Generator: LlamaGenerator | model={} | temp={:.2f} " "Generator: LlamaGenerator | model={} | "
"top_p={:.2f} top_k={} n_ctx={} seed={}", "temp={:.2f} top_p={:.2f} top_k={} n_ctx={} seed={}",
model_path, sampling.temperature, sampling.top_p, model_path, sampling.temperature, sampling.top_p,
sampling.top_k, sampling.n_ctx, sampling.seed)}); sampling.top_k, sampling.n_ctx, sampling.seed)});
}
return std::make_unique<LlamaGenerator>( return std::make_unique<LlamaGenerator>(
options, model_path, log_producer, options, model_path, log_producer,
inj.template create<std::unique_ptr<IPromptFormatter>>(), inj.template create<std::unique_ptr<IPromptFormatter>>(),
@@ -190,23 +193,22 @@ int main(const int argc, char** argv) {
injector.create<std::unique_ptr<BiergartenPipelineOrchestrator>>(); injector.create<std::unique_ptr<BiergartenPipelineOrchestrator>>();
if (!orchestrator->Run()) { if (!orchestrator->Run()) {
log_producer->Log({.level = LogLevel::Error, log_producer->Log({.level = LogLevel::Error,
.phase = PipelinePhase::Teardown, .phase = PipelinePhase::Teardown,
.message = "Pipeline execution failed"}); .message = "Pipeline execution failed"});
return shutdown(EXIT_FAILURE); return shutdown(EXIT_FAILURE);
} }
log_producer->Log({.level = LogLevel::Info, log_producer->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Teardown, .phase = PipelinePhase::Teardown,
.message = std::format("Pipeline complete in {} ms", .message = std::format("Pipeline complete in {} ms",
timer.Elapsed())}); timer.Elapsed())});
return shutdown(EXIT_SUCCESS); return shutdown(EXIT_SUCCESS);
} catch (const std::exception& exception) { } catch (const std::exception& exception) {
const LogDTO log_entry{.level = LogLevel::Error, const LogDTO log_entry{.level = LogLevel::Error,
.phase = PipelinePhase::Teardown, .phase = PipelinePhase::Teardown,
.message = exception.what()}; .message = exception.what()};
if (log_producer) { if (log_producer) {
log_producer->Log(log_entry); log_producer->Log(log_entry);
} else { } else {
@@ -215,4 +217,4 @@ int main(const int argc, char** argv) {
return shutdown(EXIT_FAILURE); return shutdown(EXIT_FAILURE);
} }
} }

View File

@@ -1,5 +1,6 @@
/** /**
* @file wikipedia/fetch_extract.cc * @file wikipedia/fetch_extract.cc
* @brief WikipediaEnrichmentService::FetchExtract() implementation.
*/ */
#include <boost/json.hpp> #include <boost/json.hpp>
@@ -20,9 +21,10 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
if (const auto cache_it = this->extract_cache_.find(cache_key); if (const auto cache_it = this->extract_cache_.find(cache_key);
cache_it != this->extract_cache_.end()) { cache_it != this->extract_cache_.end()) {
if (logger_) { if (logger_) {
logger_->Log({.level = LogLevel::Debug, logger_->Log(
.phase = PipelinePhase::UserGeneration, {.level = LogLevel::Debug,
.message = std::format("Wikipedia: Cache hit for {}!", cache_key)}); .phase = PipelinePhase::Enrichment,
.message = std::format("Wikipedia: Cache hit for {}!", cache_key)});
} }
return cache_it->second; return cache_it->second;
} }
@@ -30,7 +32,8 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
const std::string encoded = this->client_->EncodeURL(cache_key); const std::string encoded = this->client_->EncodeURL(cache_key);
const std::string url = std::format( const std::string url = std::format(
"https://en.wikipedia.org/w/" "https://en.wikipedia.org/w/"
"api.php?action=query&titles={}&prop=extracts&explaintext=1&format=json", "api.php?action=query&titles={}&prop=extracts&explaintext=1&format="
"json",
encoded); encoded);
const std::string body = this->client_->Get(url); const std::string body = this->client_->Get(url);
@@ -45,11 +48,11 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
if (ec) { if (ec) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Warn,
{.level = LogLevel::Warn, .phase = PipelinePhase::Enrichment,
.phase = PipelinePhase::UserGeneration, .message = std::format(
.message = std::format("WikipediaService: JSON parse error for '{}': {}", "WikipediaService: JSON parse error for '{}': {}",
std::string(query), ec.message())}); std::string(query), ec.message())});
} }
return {}; return {};
} }
@@ -58,12 +61,11 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
const json::object* obj = doc.if_object(); const json::object* obj = doc.if_object();
if (obj == nullptr) { if (obj == nullptr) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Warn,
{.level = LogLevel::Warn, .phase = PipelinePhase::Enrichment,
.phase = PipelinePhase::UserGeneration, .message = std::format(
.message = "WikipediaService: Expected root object for '{}'",
std::format("WikipediaService: Expected root object for '{}'", std::string(query))});
std::string(query))});
} }
return {}; return {};
} }
@@ -76,12 +78,11 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
if ((pages_ptr == nullptr) || !pages_ptr->is_object()) { if ((pages_ptr == nullptr) || !pages_ptr->is_object()) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Warn,
{.level = LogLevel::Warn, .phase = PipelinePhase::Enrichment,
.phase = PipelinePhase::UserGeneration, .message = std::format(
.message = "WikipediaService: Missing query.pages for '{}'",
std::format("WikipediaService: Missing query.pages for '{}'", std::string(query))});
std::string(query))});
} }
return {}; return {};
} }
@@ -90,11 +91,11 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
if (pages.empty()) { if (pages.empty()) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Warn,
{.level = LogLevel::Warn, .phase = PipelinePhase::Enrichment,
.phase = PipelinePhase::UserGeneration, .message = std::format(
.message = std::format("WikipediaService: No pages returned for '{}'", "WikipediaService: No pages returned for '{}'",
std::string(query))}); std::string(query))});
} }
this->extract_cache_.emplace(cache_key, ""); this->extract_cache_.emplace(cache_key, "");
return {}; return {};
@@ -106,12 +107,11 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
if (!page_val.is_object()) { if (!page_val.is_object()) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Warn,
{.level = LogLevel::Warn, .phase = PipelinePhase::Enrichment,
.phase = PipelinePhase::UserGeneration, .message = std::format(
.message = "WikipediaService: Unexpected page format for '{}'",
std::format("WikipediaService: Unexpected page format for '{}'", std::string(query))});
std::string(query))});
} }
return {}; return {};
} }
@@ -121,10 +121,11 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
// Handle 404/Missing status // Handle 404/Missing status
if (page.contains("missing")) { if (page.contains("missing")) {
if (logger_) { if (logger_) {
logger_->Log({.level = LogLevel::Warn, logger_->Log(
.phase = PipelinePhase::UserGeneration, {.level = LogLevel::Warn,
.message = std::format("WikipediaService: Page '{}' does not exist", .phase = PipelinePhase::Enrichment,
std::string(query))}); .message = std::format("WikipediaService: Page '{}' does not exist",
std::string(query))});
} }
this->extract_cache_.emplace(cache_key, ""); this->extract_cache_.emplace(cache_key, "");
return {}; return {};
@@ -134,12 +135,11 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
if ((extract_ptr == nullptr) || !extract_ptr->is_string()) { if ((extract_ptr == nullptr) || !extract_ptr->is_string()) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Warn,
{.level = LogLevel::Warn, .phase = PipelinePhase::Enrichment,
.phase = PipelinePhase::UserGeneration, .message = std::format(
.message = "WikipediaService: No extract string found for '{}'",
std::format("WikipediaService: No extract string found for '{}'", std::string(query))});
std::string(query))});
} }
this->extract_cache_.emplace(cache_key, ""); this->extract_cache_.emplace(cache_key, "");
return {}; return {};
@@ -148,10 +148,11 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
// 4. Success // 4. Success
std::string extract(extract_ptr->as_string()); std::string extract(extract_ptr->as_string());
if (logger_) { if (logger_) {
logger_->Log({.level = LogLevel::Info, logger_->Log(
.phase = PipelinePhase::UserGeneration, {.level = LogLevel::Info,
.message = std::format("WikipediaService: Fetched {} chars for '{}'", .phase = PipelinePhase::Enrichment,
extract.size(), std::string(query))}); .message = std::format("WikipediaService: Fetched {} chars for '{}'",
extract.size(), std::string(query))});
} }
this->extract_cache_.insert_or_assign(cache_key, extract); this->extract_cache_.insert_or_assign(cache_key, extract);

View File

@@ -1,6 +1,6 @@
/** /**
* @file wikipedia/get_summary.cc * @file wikipedia/get_summary.cc
* @brief WikipediaService::GetLocationContext() implementation. * @brief WikipediaEnrichmentService::GetLocationContext() implementation.
*/ */
#include <chrono> #include <chrono>
@@ -16,7 +16,7 @@ std::string WikipediaEnrichmentService::GetLocationContext(
if (!this->client_) { if (!this->client_) {
if (logger_) { if (logger_) {
logger_->Log({.level = LogLevel::Warn, logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::Enrichment,
.message = "Wikipedia client is nullptr."}); .message = "Wikipedia client is nullptr."});
} }
return {}; return {};
@@ -24,13 +24,6 @@ std::string WikipediaEnrichmentService::GetLocationContext(
std::string result; std::string result;
// std::string region_query(loc.city);
// if (!loc.country.empty()) {
// region_query += loc.state_province,
// region_query += ", ";
// region_query += loc.country;
// }
constexpr std::string_view brewing_query = "brewing"; constexpr std::string_view brewing_query = "brewing";
const std::string location_query = const std::string location_query =
std::format("{}, {}", loc.city, loc.iso3166_2); std::format("{}, {}", loc.city, loc.iso3166_2);
@@ -51,9 +44,10 @@ std::string WikipediaEnrichmentService::GetLocationContext(
append_extract(FetchExtract(beer_query)); append_extract(FetchExtract(beer_query));
if (logger_) { if (logger_) {
logger_->Log({.level = LogLevel::Info, logger_->Log({.level = LogLevel::Info,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::Enrichment,
.message = std::format("Done fetching for {}. Sleeping for 10 seconds.", .message = std::format(
location_query)}); "Done fetching for {}. Sleeping for 10 seconds.",
location_query)});
} }
std::this_thread::sleep_for(10s); std::this_thread::sleep_for(10s);
@@ -61,9 +55,9 @@ std::string WikipediaEnrichmentService::GetLocationContext(
if (logger_) { if (logger_) {
logger_->Log( logger_->Log(
{.level = LogLevel::Debug, {.level = LogLevel::Debug,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::Enrichment,
.message = std::format("WikipediaService lookup failed for '{}': {}", .message = std::format("WikipediaService lookup failed for '{}': {}",
location_query, e.what())}); location_query, e.what())});
} }
} }
return result; return result;

View File

@@ -1,6 +1,6 @@
/** /**
* @file services/wikipedia/wikipedia_service.cc * @file services/wikipedia/wikipedia_service.cc
* @brief WikipediaService constructor implementation. * @brief WikipediaEnrichmentService constructor implementation.
*/ */
#include "services/enrichment/wikipedia_service.h" #include "services/enrichment/wikipedia_service.h"

View File

@@ -19,6 +19,8 @@ namespace {
switch (phase) { switch (phase) {
case PipelinePhase::Startup: case PipelinePhase::Startup:
return "Startup"; return "Startup";
case PipelinePhase::Enrichment:
return "Enrichment";
case PipelinePhase::UserGeneration: case PipelinePhase::UserGeneration:
return "User Generation"; return "User Generation";
case PipelinePhase::BreweryAndBeerGeneration: case PipelinePhase::BreweryAndBeerGeneration:

View File

@@ -44,8 +44,8 @@ PromptDirectory::PromptDirectory(const std::filesystem::path& prompt_dir,
// Scenario 4: directory must be readable (probe with directory_iterator). // Scenario 4: directory must be readable (probe with directory_iterator).
std::filesystem::directory_iterator probe(prompt_dir_, ec); std::filesystem::directory_iterator probe(prompt_dir_, ec);
if (ec) { if (ec) {
throw std::runtime_error( throw std::runtime_error(std::format(
std::format("PromptDirectory: prompt directory is not readable: {} ({})", "PromptDirectory: prompt directory is not readable: {} ({})",
prompt_dir_.string(), ec.message())); prompt_dir_.string(), ec.message()));
} }
@@ -76,7 +76,7 @@ std::string PromptDirectory::Load(std::string_view key) {
if (!file.is_open()) { if (!file.is_open()) {
throw std::runtime_error( throw std::runtime_error(
std::format("PromptDirectory: prompt file not found for key '{}': {}", std::format("PromptDirectory: prompt file not found for key '{}': {}",
key_str, file_path.string())); key_str, file_path.string()));
} }
std::string content((std::istreambuf_iterator<char>(file)), std::string content((std::istreambuf_iterator<char>(file)),
@@ -84,15 +84,18 @@ std::string PromptDirectory::Load(std::string_view key) {
file.close(); file.close();
if (content.empty()) { if (content.empty()) {
throw std::runtime_error(std::format("PromptDirectory: prompt file for key '{}' is empty: {}", throw std::runtime_error(
key_str, file_path.string())); std::format("PromptDirectory: prompt file for key '{}' is empty: {}",
key_str, file_path.string()));
} }
if (logger_) { if (logger_) {
logger_->Log({.level = LogLevel::Info, logger_->Log(
.phase = PipelinePhase::Startup, {.level = LogLevel::Info,
.message = std::format("[PromptDirectory] Loaded prompt '{}' from '{}' ({} chars)", .phase = PipelinePhase::Startup,
key_str, file_path.string(), content.size())}); .message = std::format(
"[PromptDirectory] Loaded prompt '{}' from '{}' ({} chars)",
key_str, file_path.string(), content.size())});
} }
cache_.emplace(key_str, content); cache_.emplace(key_str, content);

View File

@@ -14,6 +14,7 @@ void SqliteExportService::Finalize() {
} }
try { try {
insert_user_stmt_.reset();
insert_brewery_stmt_.reset(); insert_brewery_stmt_.reset();
insert_location_stmt_.reset(); insert_location_stmt_.reset();
if (transaction_open_) { if (transaction_open_) {

View File

@@ -18,9 +18,9 @@ std::filesystem::path SqliteExportService::BuildDatabasePath() const {
std::filesystem::path candidate = output_path_ / base_filename; std::filesystem::path candidate = output_path_ / base_filename;
for (int suffix = 1; std::filesystem::exists(candidate); ++suffix) { for (int suffix = 1; std::filesystem::exists(candidate); ++suffix) {
candidate = output_path_ / candidate = output_path_ / std::filesystem::path(
std::filesystem::path(std::format("biergarten_seed_{}-{}.sqlite", std::format("biergarten_seed_{}-{}.sqlite",
run_timestamp_utc_, suffix)); run_timestamp_utc_, suffix));
} }
return candidate; return candidate;
@@ -33,6 +33,9 @@ void SqliteExportService::InitializeSchema() const {
sqlite_export_service_internal::ExecSql( sqlite_export_service_internal::ExecSql(
db_handle_, sqlite_export_service_internal::kCreateBreweriesTableSql, db_handle_, sqlite_export_service_internal::kCreateBreweriesTableSql,
"Failed to create SQLite breweries table"); "Failed to create SQLite breweries table");
sqlite_export_service_internal::ExecSql(
db_handle_, sqlite_export_service_internal::kCreateUsersTableSql,
"Failed to create SQLite users table");
} }
void SqliteExportService::PrepareStatements() { void SqliteExportService::PrepareStatements() {
@@ -42,6 +45,9 @@ void SqliteExportService::PrepareStatements() {
insert_brewery_stmt_ = sqlite_export_service_internal::PrepareStatement( insert_brewery_stmt_ = sqlite_export_service_internal::PrepareStatement(
db_handle_, sqlite_export_service_internal::kInsertBrewerySql, db_handle_, sqlite_export_service_internal::kInsertBrewerySql,
"Failed to prepare SQLite brewery insert statement"); "Failed to prepare SQLite brewery insert statement");
insert_user_stmt_ = sqlite_export_service_internal::PrepareStatement(
db_handle_, sqlite_export_service_internal::kInsertUserSql,
"Failed to prepare SQLite user insert statement");
} }
void SqliteExportService::RollbackAndCloseNoThrow() noexcept { void SqliteExportService::RollbackAndCloseNoThrow() noexcept {
@@ -54,6 +60,7 @@ void SqliteExportService::RollbackAndCloseNoThrow() noexcept {
transaction_open_ = false; transaction_open_ = false;
} }
insert_user_stmt_.reset();
insert_brewery_stmt_.reset(); insert_brewery_stmt_.reset();
insert_location_stmt_.reset(); insert_location_stmt_.reset();
db_handle_.reset(); db_handle_.reset();

View File

@@ -1,6 +1,7 @@
/** /**
* @file services/sqlite/process_record.cc * @file services/sqlite/process_record.cc
* @brief SqliteExportService::ProcessRecord() implementation. * @brief SqliteExportService::ProcessRecord() implementation
* and the shared location-resolution helper.
*/ */
#include <iomanip> #include <iomanip>
@@ -29,82 +30,84 @@ std::string SqliteExportService::BuildLocationKey(const Location& location) {
return key_stream.str(); return key_stream.str();
} }
uint64_t SqliteExportService::ProcessRecord(const GeneratedBrewery& brewery) { sqlite3_int64 SqliteExportService::ResolveLocationId(const Location& location) {
const std::string location_key = BuildLocationKey(location);
const auto cached_location = location_cache_.find(location_key);
if (cached_location != location_cache_.end()) {
return cached_location->second;
}
const std::string local_languages_json =
sqlite_export_service_internal::SerializeVector(location.local_languages);
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationCityBindIndex,
.value = location.city,
.action = "Failed to bind SQLite location city"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index =
sqlite_export_service_internal::kLocationStateProvinceBindIndex,
.value = location.state_province,
.action = "Failed to bind SQLite location state/province"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationIso31662BindIndex,
.value = location.iso3166_2,
.action = "Failed to bind SQLite location ISO 3166-2 code"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationCountryBindIndex,
.value = location.country,
.action = "Failed to bind SQLite location country"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationIso31661BindIndex,
.value = location.iso3166_1,
.action = "Failed to bind SQLite location ISO 3166-1 code"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationLanguagesBindIndex,
.value = local_languages_json,
.action = "Failed to bind SQLite location languages"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam{
.index = sqlite_export_service_internal::kLocationLatitudeBindIndex,
.value = location.latitude,
.action = "Failed to bind SQLite location latitude"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam{
.index = sqlite_export_service_internal::kLocationLongitudeBindIndex,
.value = location.longitude,
.action = "Failed to bind SQLite location longitude"});
sqlite_export_service_internal::StepStatement(
db_handle_, insert_location_stmt_,
"Failed to insert SQLite location row");
const sqlite3_int64 location_id =
sqlite_export_service_internal::LastInsertRowId(db_handle_);
location_cache_.emplace(location_key, location_id);
sqlite_export_service_internal::ResetStatement(insert_location_stmt_);
return location_id;
}
uint64_t SqliteExportService::ProcessRecord(const BreweryRecord& brewery) {
if (db_handle_ == nullptr || !transaction_open_) { if (db_handle_ == nullptr || !transaction_open_) {
throw std::runtime_error("SQLite export service is not initialized"); throw std::runtime_error("SQLite export service is not initialized");
} }
const std::string location_key = BuildLocationKey(brewery.location); const sqlite3_int64 location_id = ResolveLocationId(brewery.location);
const auto cached_location = location_cache_.find(location_key);
sqlite3_int64 location_id = 0;
if (cached_location != location_cache_.end()) {
location_id = cached_location->second;
} else {
const std::string local_languages_json =
sqlite_export_service_internal::SerializeVector(
brewery.location.local_languages);
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationCityBindIndex,
.value = brewery.location.city,
.action = "Failed to bind SQLite location city"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index =
sqlite_export_service_internal::kLocationStateProvinceBindIndex,
.value = brewery.location.state_province,
.action = "Failed to bind SQLite location state/province"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationIso31662BindIndex,
.value = brewery.location.iso3166_2,
.action = "Failed to bind SQLite location ISO 3166-2 code"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationCountryBindIndex,
.value = brewery.location.country,
.action = "Failed to bind SQLite location country"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationIso31661BindIndex,
.value = brewery.location.iso3166_1,
.action = "Failed to bind SQLite location ISO 3166-1 code"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index =
sqlite_export_service_internal::kLocationLanguagesBindIndex,
.value = local_languages_json,
.action = "Failed to bind SQLite location languages"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam{
.index = sqlite_export_service_internal::kLocationLatitudeBindIndex,
.value = brewery.location.latitude,
.action = "Failed to bind SQLite location latitude"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam{
.index =
sqlite_export_service_internal::kLocationLongitudeBindIndex,
.value = brewery.location.longitude,
.action = "Failed to bind SQLite location longitude"});
sqlite_export_service_internal::StepStatement(
db_handle_, insert_location_stmt_,
"Failed to insert SQLite location row");
location_id = sqlite_export_service_internal::LastInsertRowId(db_handle_);
location_cache_.emplace(location_key, location_id);
sqlite_export_service_internal::ResetStatement(insert_location_stmt_);
}
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_stmt_, insert_brewery_stmt_,
@@ -112,12 +115,14 @@ uint64_t SqliteExportService::ProcessRecord(const GeneratedBrewery& brewery) {
.index = sqlite_export_service_internal::kBreweryLocationIdBindIndex, .index = sqlite_export_service_internal::kBreweryLocationIdBindIndex,
.value = location_id, .value = location_id,
.action = "Failed to bind SQLite brewery location id"}); .action = "Failed to bind SQLite brewery location id"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_stmt_, insert_brewery_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{ sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kBreweryEnglishNameBindIndex, .index = sqlite_export_service_internal::kBreweryEnglishNameBindIndex,
.value = brewery.brewery.name_en, .value = brewery.brewery.name_en,
.action = "Failed to bind SQLite brewery English name"}); .action = "Failed to bind SQLite brewery English name"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_stmt_, insert_brewery_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{ sqlite_export_service_internal::BindParam<std::string_view>{
@@ -125,12 +130,14 @@ uint64_t SqliteExportService::ProcessRecord(const GeneratedBrewery& brewery) {
kBreweryEnglishDescriptionBindIndex, kBreweryEnglishDescriptionBindIndex,
.value = brewery.brewery.description_en, .value = brewery.brewery.description_en,
.action = "Failed to bind SQLite brewery English description"}); .action = "Failed to bind SQLite brewery English description"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_stmt_, insert_brewery_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{ sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kBreweryLocalNameBindIndex, .index = sqlite_export_service_internal::kBreweryLocalNameBindIndex,
.value = brewery.brewery.name_local, .value = brewery.brewery.name_local,
.action = "Failed to bind SQLite brewery local name"}); .action = "Failed to bind SQLite brewery local name"});
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_stmt_, insert_brewery_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{ sqlite_export_service_internal::BindParam<std::string_view>{

View File

@@ -0,0 +1,79 @@
/**
* @file services/sqlite/process_user_record.cc
* @brief SqliteExportService::ProcessRecord(UserRecord) implementation.
*/
#include <stdexcept>
#include "services/database/sqlite_export_service.h"
#include "services/database/sqlite_export_service_helpers.h"
uint64_t SqliteExportService::ProcessRecord(const UserRecord& user) {
if (db_handle_ == nullptr || !transaction_open_) {
throw std::runtime_error("SQLite export service is not initialized");
}
const sqlite3_int64 location_id = ResolveLocationId(user.location);
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BindParam<sqlite3_int64>{
.index = sqlite_export_service_internal::kUserLocationIdBindIndex,
.value = location_id,
.action = "Failed to bind SQLite user location id"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserFirstNameBindIndex,
.value = user.user.first_name,
.action = "Failed to bind SQLite user first name"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserLastNameBindIndex,
.value = user.user.last_name,
.action = "Failed to bind SQLite user last name"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserGenderBindIndex,
.value = user.user.gender,
.action = "Failed to bind SQLite user gender"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserUsernameBindIndex,
.value = user.user.username,
.action = "Failed to bind SQLite user username"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserBioBindIndex,
.value = user.user.bio,
.action = "Failed to bind SQLite user bio"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BindParam<double>{
.index = sqlite_export_service_internal::kUserActivityWeightBindIndex,
.value = static_cast<double>(user.user.activity_weight),
.action = "Failed to bind SQLite user activity weight"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserEmailBindIndex,
.value = user.email,
.action = "Failed to bind SQLite user email"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserDateOfBirthBindIndex,
.value = user.date_of_birth,
.action = "Failed to bind SQLite user date of birth"});
sqlite_export_service_internal::StepStatement(
db_handle_, insert_user_stmt_, "Failed to insert SQLite user row");
sqlite_export_service_internal::ResetStatement(insert_user_stmt_);
return sqlite_export_service_internal::LastInsertRowId(db_handle_);
}

View File

@@ -48,21 +48,20 @@ std::string HttpWebClient::Get(const std::string& url) {
const httplib::Result result = client.Get(path); const httplib::Result result = client.Get(path);
if (!result) { if (!result) {
throw std::runtime_error(std::format( throw std::runtime_error(
"[HttpWebClient] Request failed for URL: {} — {}", url, std::format("[HttpWebClient] Request failed for URL: {} — {}", url,
httplib::to_string(result.error()))); httplib::to_string(result.error())));
} }
if (result->status < kSuccessMin || result->status >= kSuccessMax) { if (result->status < kSuccessMin || result->status >= kSuccessMax) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log({.level = LogLevel::Error,
{.level = LogLevel::Error, .phase = PipelinePhase::Enrichment,
.phase = PipelinePhase::UserGeneration, .message = std::format(
.message = "[HttpWebClient] Request failed for URL: {}", url)});
std::format("[HttpWebClient] Request failed for URL: {}", url)});
} }
throw std::runtime_error(std::format("[HttpWebClient] HTTP {} for URL: {}", throw std::runtime_error(std::format("[HttpWebClient] HTTP {} for URL: {}",
result->status, url)); result->status, url));
} }
return result->body; return result->body;

File diff suppressed because it is too large Load Diff

View File

@@ -1,5 +1,5 @@
{ {
"printWidth": 80, "printWidth": 100,
"useTabs": false, "useTabs": false,
"indentSize": 4, "indentSize": 4,
"endOfLine": "lf", "endOfLine": "lf",

View File

@@ -8,33 +8,27 @@
</PropertyGroup> </PropertyGroup>
<ItemGroup> <ItemGroup>
<PackageReference <PackageReference Include="Microsoft.AspNetCore.OpenApi" Version="9.0.11" />
Include="Microsoft.AspNetCore.OpenApi" <PackageReference Include="Swashbuckle.AspNetCore" Version="6.6.2" />
Version="9.0.11" <PackageReference Include="FluentValidation.AspNetCore" Version="11.3.0" />
/> <PackageReference Include="MediatR" Version="12.4.1" />
<PackageReference Include="Swashbuckle.AspNetCore" Version="6.6.2"/>
<PackageReference
Include="FluentValidation.AspNetCore"
Version="11.3.0"
/>
<PackageReference Include="MediatR" Version="12.4.1"/>
</ItemGroup> </ItemGroup>
<ItemGroup> <ItemGroup>
<Folder Include="Infrastructure\"/> <Folder Include="Infrastructure\" />
</ItemGroup> </ItemGroup>
<ItemGroup> <ItemGroup>
<ProjectReference Include="..\..\Domain\Domain.Entities\Domain.Entities.csproj"/> <ProjectReference Include="..\..\Domain\Domain.Entities\Domain.Entities.csproj" />
<ProjectReference Include="..\..\Domain\Domain.Exceptions\Domain.Exceptions.csproj"/> <ProjectReference Include="..\..\Domain\Domain.Exceptions\Domain.Exceptions.csproj" />
<ProjectReference Include="..\..\Infrastructure\Infrastructure.Sql\Infrastructure.Sql.csproj"/> <ProjectReference Include="..\..\Infrastructure\Infrastructure.Sql\Infrastructure.Sql.csproj" />
<ProjectReference Include="..\..\Infrastructure\Infrastructure.Jwt\Infrastructure.Jwt.csproj"/> <ProjectReference Include="..\..\Infrastructure\Infrastructure.Jwt\Infrastructure.Jwt.csproj" />
<ProjectReference Include="..\..\Features\Features.Breweries\Features.Breweries.csproj"/> <ProjectReference Include="..\..\Features\Features.Breweries\Features.Breweries.csproj" />
<ProjectReference Include="..\..\Features\Features.UserManagement\Features.UserManagement.csproj"/> <ProjectReference Include="..\..\Features\Features.UserManagement\Features.UserManagement.csproj" />
<ProjectReference Include="..\..\Features\Features.Auth\Features.Auth.csproj"/> <ProjectReference Include="..\..\Features\Features.Auth\Features.Auth.csproj" />
<ProjectReference Include="..\..\Features\Features.Emails\Features.Emails.csproj"/> <ProjectReference Include="..\..\Features\Features.Emails\Features.Emails.csproj" />
<ProjectReference Include="..\..\Shared\Shared.Contracts\Shared.Contracts.csproj"/> <ProjectReference Include="..\..\Shared\Shared.Contracts\Shared.Contracts.csproj" />
<ProjectReference Include="..\..\Shared\Shared.Application\Shared.Application.csproj"/> <ProjectReference Include="..\..\Shared\Shared.Application\Shared.Application.csproj" />
</ItemGroup> </ItemGroup>
<ItemGroup> <ItemGroup>

View File

@@ -44,29 +44,19 @@ public class JwtAuthenticationHandler(
{ {
// Use the same access-token secret source as TokenService to avoid mismatched validation. // Use the same access-token secret source as TokenService to avoid mismatched validation.
string? secret = Environment.GetEnvironmentVariable("ACCESS_TOKEN_SECRET"); string? secret = Environment.GetEnvironmentVariable("ACCESS_TOKEN_SECRET");
if (string.IsNullOrWhiteSpace(secret)) secret = configuration["Jwt:SecretKey"]; if (string.IsNullOrWhiteSpace(secret))
secret = configuration["Jwt:SecretKey"];
if (string.IsNullOrWhiteSpace(secret)) return AuthenticateResult.Fail("JWT secret is not configured"); if (string.IsNullOrWhiteSpace(secret))
return AuthenticateResult.Fail("JWT secret is not configured");
// Check if Authorization header exists // Check if Authorization header exists
if ( if (!Request.Headers.TryGetValue("Authorization", out StringValues authHeaderValue))
!Request.Headers.TryGetValue(
"Authorization",
out StringValues authHeaderValue
)
)
return AuthenticateResult.Fail("Authorization header is missing"); return AuthenticateResult.Fail("Authorization header is missing");
string authHeader = authHeaderValue.ToString(); string authHeader = authHeaderValue.ToString();
if ( if (!authHeader.StartsWith("Bearer ", StringComparison.OrdinalIgnoreCase))
!authHeader.StartsWith( return AuthenticateResult.Fail("Invalid authorization header format");
"Bearer ",
StringComparison.OrdinalIgnoreCase
)
)
return AuthenticateResult.Fail(
"Invalid authorization header format"
);
string token = authHeader.Substring("Bearer ".Length).Trim(); string token = authHeader.Substring("Bearer ".Length).Trim();
@@ -81,9 +71,7 @@ public class JwtAuthenticationHandler(
} }
catch (Exception ex) catch (Exception ex)
{ {
return AuthenticateResult.Fail( return AuthenticateResult.Fail($"Token validation failed: {ex.Message}");
$"Token validation failed: {ex.Message}"
);
} }
} }
@@ -97,7 +85,10 @@ public class JwtAuthenticationHandler(
Response.ContentType = "application/json"; Response.ContentType = "application/json";
Response.StatusCode = 401; Response.StatusCode = 401;
ResponseBody response = new() { Message = "Unauthorized: Invalid or missing authentication token" }; ResponseBody response = new()
{
Message = "Unauthorized: Invalid or missing authentication token",
};
await Response.WriteAsJsonAsync(response); await Response.WriteAsJsonAsync(response);
} }
} }
@@ -106,6 +97,4 @@ public class JwtAuthenticationHandler(
/// Options for the <c>JWT</c> authentication scheme handled by <see cref="JwtAuthenticationHandler" />. /// Options for the <c>JWT</c> authentication scheme handled by <see cref="JwtAuthenticationHandler" />.
/// </summary> /// </summary>
/// <remarks>No additional options are defined beyond those provided by <see cref="AuthenticationSchemeOptions" />.</remarks> /// <remarks>No additional options are defined beyond those provided by <see cref="AuthenticationSchemeOptions" />.</remarks>
public class JwtAuthenticationOptions : AuthenticationSchemeOptions public class JwtAuthenticationOptions : AuthenticationSchemeOptions { }
{
}

View File

@@ -24,4 +24,4 @@ public class NotFoundController : ControllerBase
{ {
return NotFound(new { message = "Route not found." }); return NotFound(new { message = "Route not found." });
} }
} }

View File

@@ -31,8 +31,8 @@ public class ProtectedController : ControllerBase
new ResponseBody<object> new ResponseBody<object>
{ {
Message = "Protected endpoint accessed successfully", Message = "Protected endpoint accessed successfully",
Payload = new { userId, username } Payload = new { userId, username },
} }
); );
} }
} }

View File

@@ -14,8 +14,7 @@ namespace API.Core;
/// consistent JSON error responses with appropriate HTTP status codes. /// consistent JSON error responses with appropriate HTTP status codes.
/// </summary> /// </summary>
/// <param name="logger">Logger used to record unhandled exceptions before they are translated into a response.</param> /// <param name="logger">Logger used to record unhandled exceptions before they are translated into a response.</param>
public class GlobalExceptionFilter(ILogger<GlobalExceptionFilter> logger) public class GlobalExceptionFilter(ILogger<GlobalExceptionFilter> logger) : IExceptionFilter
: IExceptionFilter
{ {
/// <summary> /// <summary>
/// Logs the exception and sets <see cref="ExceptionContext.Result" /> to an appropriate error response, /// Logs the exception and sets <see cref="ExceptionContext.Result" /> to an appropriate error response,
@@ -84,10 +83,7 @@ public class GlobalExceptionFilter(ILogger<GlobalExceptionFilter> logger)
case ValidationException fluentValidationException: case ValidationException fluentValidationException:
Dictionary<string, string[]> errors = fluentValidationException Dictionary<string, string[]> errors = fluentValidationException
.Errors.GroupBy(e => e.PropertyName) .Errors.GroupBy(e => e.PropertyName)
.ToDictionary( .ToDictionary(g => g.Key, g => g.Select(e => e.ErrorMessage).ToArray());
g => g.Key,
g => g.Select(e => e.ErrorMessage).ToArray()
);
context.Result = new BadRequestObjectResult( context.Result = new BadRequestObjectResult(
new { message = "Validation failed", errors } new { message = "Validation failed", errors }
@@ -96,41 +92,33 @@ public class GlobalExceptionFilter(ILogger<GlobalExceptionFilter> logger)
break; break;
case ConflictException ex: case ConflictException ex:
context.Result = new ObjectResult( context.Result = new ObjectResult(new ResponseBody { Message = ex.Message })
new ResponseBody { Message = ex.Message }
)
{ {
StatusCode = 409 StatusCode = 409,
}; };
context.ExceptionHandled = true; context.ExceptionHandled = true;
break; break;
case NotFoundException ex: case NotFoundException ex:
context.Result = new ObjectResult( context.Result = new ObjectResult(new ResponseBody { Message = ex.Message })
new ResponseBody { Message = ex.Message }
)
{ {
StatusCode = 404 StatusCode = 404,
}; };
context.ExceptionHandled = true; context.ExceptionHandled = true;
break; break;
case UnauthorizedException ex: case UnauthorizedException ex:
context.Result = new ObjectResult( context.Result = new ObjectResult(new ResponseBody { Message = ex.Message })
new ResponseBody { Message = ex.Message }
)
{ {
StatusCode = 401 StatusCode = 401,
}; };
context.ExceptionHandled = true; context.ExceptionHandled = true;
break; break;
case ForbiddenException ex: case ForbiddenException ex:
context.Result = new ObjectResult( context.Result = new ObjectResult(new ResponseBody { Message = ex.Message })
new ResponseBody { Message = ex.Message }
)
{ {
StatusCode = 403 StatusCode = 403,
}; };
context.ExceptionHandled = true; context.ExceptionHandled = true;
break; break;
@@ -140,33 +128,28 @@ public class GlobalExceptionFilter(ILogger<GlobalExceptionFilter> logger)
new ResponseBody { Message = "A database error occurred." } new ResponseBody { Message = "A database error occurred." }
) )
{ {
StatusCode = 503 StatusCode = 503,
}; };
context.ExceptionHandled = true; context.ExceptionHandled = true;
break; break;
case Domain.Exceptions.ValidationException ex: case Domain.Exceptions.ValidationException ex:
context.Result = new ObjectResult( context.Result = new ObjectResult(new ResponseBody { Message = ex.Message })
new ResponseBody { Message = ex.Message }
)
{ {
StatusCode = 400 StatusCode = 400,
}; };
context.ExceptionHandled = true; context.ExceptionHandled = true;
break; break;
default: default:
context.Result = new ObjectResult( context.Result = new ObjectResult(
new ResponseBody new ResponseBody { Message = "An unexpected error occurred" }
{
Message = "An unexpected error occurred"
}
) )
{ {
StatusCode = 500 StatusCode = 500,
}; };
context.ExceptionHandled = true; context.ExceptionHandled = true;
break; break;
} }
} }
} }

View File

@@ -17,7 +17,11 @@ using Shared.Application.Behaviors;
WebApplicationBuilder builder = WebApplication.CreateBuilder(args); WebApplicationBuilder builder = WebApplication.CreateBuilder(args);
// Global Exception Filter // Global Exception Filter
builder.Services.AddControllers(options => { options.Filters.Add<GlobalExceptionFilter>(); }) builder
.Services.AddControllers(options =>
{
options.Filters.Add<GlobalExceptionFilter>();
})
.AddApplicationPart(typeof(BreweryController).Assembly) .AddApplicationPart(typeof(BreweryController).Assembly)
.AddApplicationPart(typeof(UserController).Assembly) .AddApplicationPart(typeof(UserController).Assembly)
.AddApplicationPart(typeof(AuthController).Assembly); .AddApplicationPart(typeof(AuthController).Assembly);
@@ -51,14 +55,12 @@ builder.Services.AddHealthChecks();
// Configure logging for container output // Configure logging for container output
builder.Logging.ClearProviders(); builder.Logging.ClearProviders();
builder.Logging.AddConsole(); builder.Logging.AddConsole();
if (!builder.Environment.IsProduction()) builder.Logging.AddDebug(); if (!builder.Environment.IsProduction())
builder.Logging.AddDebug();
// Configure Dependency Injection ------------------------------------------------------------------------------------- // Configure Dependency Injection -------------------------------------------------------------------------------------
builder.Services.AddSingleton< builder.Services.AddSingleton<ISqlConnectionFactory, DefaultSqlConnectionFactory>();
ISqlConnectionFactory,
DefaultSqlConnectionFactory
>();
builder.Services.AddFeaturesBreweries(); builder.Services.AddFeaturesBreweries();
builder.Services.AddFeaturesUserManagement(); builder.Services.AddFeaturesUserManagement();
@@ -75,10 +77,7 @@ builder.Services.AddScoped<GlobalExceptionFilter>();
// Configure JWT Authentication // Configure JWT Authentication
builder builder
.Services.AddAuthentication("JWT") .Services.AddAuthentication("JWT")
.AddScheme<JwtAuthenticationOptions, JwtAuthenticationHandler>( .AddScheme<JwtAuthenticationOptions, JwtAuthenticationHandler>("JWT", options => { });
"JWT",
options => { }
);
builder.Services.AddAuthorization(); builder.Services.AddAuthorization();
@@ -106,4 +105,4 @@ lifetime.ApplicationStopping.Register(() =>
app.Logger.LogInformation("Application is shutting down gracefully..."); app.Logger.LogInformation("Application is shutting down gracefully...");
}); });
app.Run(); app.Run();

View File

@@ -8,40 +8,37 @@
</PropertyGroup> </PropertyGroup>
<ItemGroup> <ItemGroup>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.12.0"/> <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.12.0" />
<PackageReference Include="xunit" Version="2.9.2"/> <PackageReference Include="xunit" Version="2.9.2" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.8.2"/> <PackageReference Include="xunit.runner.visualstudio" Version="2.8.2" />
<PackageReference Include="FluentAssertions" Version="6.9.0"/> <PackageReference Include="FluentAssertions" Version="6.9.0" />
<PackageReference Include="dbup" Version="5.0.41"/> <PackageReference Include="dbup" Version="5.0.41" />
<!-- Reqnroll core, xUnit adapter and code-behind generator --> <!-- Reqnroll core, xUnit adapter and code-behind generator -->
<PackageReference Include="Reqnroll" Version="3.3.3"/> <PackageReference Include="Reqnroll" Version="3.3.3" />
<PackageReference Include="Reqnroll.xUnit" Version="3.3.3"/> <PackageReference Include="Reqnroll.xUnit" Version="3.3.3" />
<PackageReference <PackageReference
Include="Reqnroll.Tools.MsBuild.Generation" Include="Reqnroll.Tools.MsBuild.Generation"
Version="3.3.3" Version="3.3.3"
PrivateAssets="all" PrivateAssets="all"
/> />
<!-- ASP.NET Core integration testing --> <!-- ASP.NET Core integration testing -->
<PackageReference <PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" Version="9.0.1" />
Include="Microsoft.AspNetCore.Mvc.Testing"
Version="9.0.1"
/>
</ItemGroup> </ItemGroup>
<ItemGroup> <ItemGroup>
<!-- Ensure feature files are included in the project --> <!-- Ensure feature files are included in the project -->
<None Include="Features\**\*.feature"/> <None Include="Features\**\*.feature" />
</ItemGroup> </ItemGroup>
<ItemGroup> <ItemGroup>
<Using Include="Xunit"/> <Using Include="Xunit" />
</ItemGroup> </ItemGroup>
<ItemGroup> <ItemGroup>
<ProjectReference Include="..\API.Core\API.Core.csproj"/> <ProjectReference Include="..\API.Core\API.Core.csproj" />
<ProjectReference Include="..\..\Infrastructure\Infrastructure.Email\Infrastructure.Email.csproj"/> <ProjectReference Include="..\..\Infrastructure\Infrastructure.Email\Infrastructure.Email.csproj" />
<ProjectReference Include="..\..\Features\Features.Emails\Features.Emails.csproj"/> <ProjectReference Include="..\..\Features\Features.Emails\Features.Emails.csproj" />
</ItemGroup> </ItemGroup>
</Project> </Project>

View File

@@ -8,11 +8,7 @@ public class MockEmailDispatcher : IEmailDispatcher
public List<ResendConfirmationEmail> SentResendConfirmationEmails { get; } = new(); public List<ResendConfirmationEmail> SentResendConfirmationEmails { get; } = new();
public Task SendRegistrationEmailAsync( public Task SendRegistrationEmailAsync(string firstName, string email, string confirmationToken)
string firstName,
string email,
string confirmationToken
)
{ {
SentRegistrationEmails.Add( SentRegistrationEmails.Add(
new RegistrationEmail new RegistrationEmail
@@ -20,7 +16,7 @@ public class MockEmailDispatcher : IEmailDispatcher
FirstName = firstName, FirstName = firstName,
Email = email, Email = email,
ConfirmationToken = confirmationToken, ConfirmationToken = confirmationToken,
SentAt = DateTime.UtcNow SentAt = DateTime.UtcNow,
} }
); );
@@ -39,7 +35,7 @@ public class MockEmailDispatcher : IEmailDispatcher
FirstName = firstName, FirstName = firstName,
Email = email, Email = email,
ConfirmationToken = confirmationToken, ConfirmationToken = confirmationToken,
SentAt = DateTime.UtcNow SentAt = DateTime.UtcNow,
} }
); );
@@ -67,4 +63,4 @@ public class MockEmailDispatcher : IEmailDispatcher
public string ConfirmationToken { get; init; } = string.Empty; public string ConfirmationToken { get; init; } = string.Empty;
public DateTime SentAt { get; init; } public DateTime SentAt { get; init; }
} }
} }

View File

@@ -10,12 +10,7 @@ public class MockEmailProvider : IEmailProvider
{ {
public List<SentEmail> SentEmails { get; } = new(); public List<SentEmail> SentEmails { get; } = new();
public Task SendAsync( public Task SendAsync(string to, string subject, string body, bool isHtml = true)
string to,
string subject,
string body,
bool isHtml = true
)
{ {
SentEmails.Add( SentEmails.Add(
new SentEmail new SentEmail
@@ -24,19 +19,14 @@ public class MockEmailProvider : IEmailProvider
Subject = subject, Subject = subject,
Body = body, Body = body,
IsHtml = isHtml, IsHtml = isHtml,
SentAt = DateTime.UtcNow SentAt = DateTime.UtcNow,
} }
); );
return Task.CompletedTask; return Task.CompletedTask;
} }
public Task SendAsync( public Task SendAsync(IEnumerable<string> to, string subject, string body, bool isHtml = true)
IEnumerable<string> to,
string subject,
string body,
bool isHtml = true
)
{ {
SentEmails.Add( SentEmails.Add(
new SentEmail new SentEmail
@@ -45,7 +35,7 @@ public class MockEmailProvider : IEmailProvider
Subject = subject, Subject = subject,
Body = body, Body = body,
IsHtml = isHtml, IsHtml = isHtml,
SentAt = DateTime.UtcNow SentAt = DateTime.UtcNow,
} }
); );
@@ -65,4 +55,4 @@ public class MockEmailProvider : IEmailProvider
public bool IsHtml { get; init; } public bool IsHtml { get; init; }
public DateTime SentAt { get; init; } public DateTime SentAt { get; init; }
} }
} }

View File

@@ -15,7 +15,8 @@ public class ApiGeneralSteps(ScenarioContext scenario)
private HttpClient GetClient() private HttpClient GetClient()
{ {
if (scenario.TryGetValue<HttpClient>(ClientKey, out HttpClient? client)) return client; if (scenario.TryGetValue<HttpClient>(ClientKey, out HttpClient? client))
return client;
TestApiFactory? factory = scenario.TryGetValue<TestApiFactory>( TestApiFactory? factory = scenario.TryGetValue<TestApiFactory>(
FactoryKey, FactoryKey,
@@ -47,11 +48,7 @@ public class ApiGeneralSteps(ScenarioContext scenario)
HttpRequestMessage requestMessage = new(new HttpMethod(method), url) HttpRequestMessage requestMessage = new(new HttpMethod(method), url)
{ {
Content = new StringContent( Content = new StringContent(jsonBody, Encoding.UTF8, "application/json"),
jsonBody,
Encoding.UTF8,
"application/json"
)
}; };
HttpResponseMessage response = await client.SendAsync(requestMessage); HttpResponseMessage response = await client.SendAsync(requestMessage);
@@ -62,16 +59,10 @@ public class ApiGeneralSteps(ScenarioContext scenario)
} }
[When("I send an HTTP request {string} to {string}")] [When("I send an HTTP request {string} to {string}")]
public async Task WhenISendAnHttpRequestStringToString( public async Task WhenISendAnHttpRequestStringToString(string method, string url)
string method,
string url
)
{ {
HttpClient client = GetClient(); HttpClient client = GetClient();
HttpRequestMessage requestMessage = new( HttpRequestMessage requestMessage = new(new HttpMethod(method), url);
new HttpMethod(method),
url
);
HttpResponseMessage response = await client.SendAsync(requestMessage); HttpResponseMessage response = await client.SendAsync(requestMessage);
string responseBody = await response.Content.ReadAsStringAsync(); string responseBody = await response.Content.ReadAsStringAsync();
@@ -100,19 +91,13 @@ public class ApiGeneralSteps(ScenarioContext scenario)
} }
[Then("the response JSON should have {string} equal {string}")] [Then("the response JSON should have {string} equal {string}")]
public void ThenTheResponseJsonShouldHaveStringEqualString( public void ThenTheResponseJsonShouldHaveStringEqualString(string field, string expected)
string field,
string expected
)
{ {
scenario scenario
.TryGetValue<HttpResponseMessage>(ResponseKey, out HttpResponseMessage? response) .TryGetValue<HttpResponseMessage>(ResponseKey, out HttpResponseMessage? response)
.Should() .Should()
.BeTrue(); .BeTrue();
scenario scenario.TryGetValue<string>(ResponseBodyKey, out string? responseBody).Should().BeTrue();
.TryGetValue<string>(ResponseBodyKey, out string? responseBody)
.Should()
.BeTrue();
using JsonDocument doc = JsonDocument.Parse(responseBody!); using JsonDocument doc = JsonDocument.Parse(responseBody!);
JsonElement root = doc.RootElement; JsonElement root = doc.RootElement;
@@ -125,25 +110,16 @@ public class ApiGeneralSteps(ScenarioContext scenario)
"Expected field '{0}' to be present either at the root or inside 'payload'", "Expected field '{0}' to be present either at the root or inside 'payload'",
field field
); );
payloadElem payloadElem.ValueKind.Should().Be(JsonValueKind.Object, "payload must be an object");
.ValueKind.Should()
.Be(JsonValueKind.Object, "payload must be an object");
payloadElem payloadElem
.TryGetProperty(field, out value) .TryGetProperty(field, out value)
.Should() .Should()
.BeTrue( .BeTrue("Expected field '{0}' to be present inside 'payload'", field);
"Expected field '{0}' to be present inside 'payload'",
field
);
} }
value value
.ValueKind.Should() .ValueKind.Should()
.Be( .Be(JsonValueKind.String, "Expected field '{0}' to be a string", field);
JsonValueKind.String,
"Expected field '{0}' to be a string",
field
);
value.GetString().Should().Be(expected); value.GetString().Should().Be(expected);
} }
@@ -157,10 +133,7 @@ public class ApiGeneralSteps(ScenarioContext scenario)
.TryGetValue<HttpResponseMessage>(ResponseKey, out HttpResponseMessage? response) .TryGetValue<HttpResponseMessage>(ResponseKey, out HttpResponseMessage? response)
.Should() .Should()
.BeTrue(); .BeTrue();
scenario scenario.TryGetValue<string>(ResponseBodyKey, out string? responseBody).Should().BeTrue();
.TryGetValue<string>(ResponseBodyKey, out string? responseBody)
.Should()
.BeTrue();
using JsonDocument doc = JsonDocument.Parse(responseBody!); using JsonDocument doc = JsonDocument.Parse(responseBody!);
JsonElement root = doc.RootElement; JsonElement root = doc.RootElement;
@@ -173,25 +146,16 @@ public class ApiGeneralSteps(ScenarioContext scenario)
"Expected field '{0}' to be present either at the root or inside 'payload'", "Expected field '{0}' to be present either at the root or inside 'payload'",
field field
); );
payloadElem payloadElem.ValueKind.Should().Be(JsonValueKind.Object, "payload must be an object");
.ValueKind.Should()
.Be(JsonValueKind.Object, "payload must be an object");
payloadElem payloadElem
.TryGetProperty(field, out value) .TryGetProperty(field, out value)
.Should() .Should()
.BeTrue( .BeTrue("Expected field '{0}' to be present inside 'payload'", field);
"Expected field '{0}' to be present inside 'payload'",
field
);
} }
value value
.ValueKind.Should() .ValueKind.Should()
.Be( .Be(JsonValueKind.String, "Expected field '{0}' to be a string", field);
JsonValueKind.String,
"Expected field '{0}' to be a string",
field
);
string? actualValue = value.GetString(); string? actualValue = value.GetString();
actualValue actualValue
.Should() .Should()
@@ -203,4 +167,4 @@ public class ApiGeneralSteps(ScenarioContext scenario)
actualValue actualValue
); );
} }
} }

View File

@@ -21,7 +21,8 @@ public class AuthSteps(ScenarioContext scenario)
private HttpClient GetClient() private HttpClient GetClient()
{ {
if (scenario.TryGetValue<HttpClient>(ClientKey, out HttpClient? client)) return client; if (scenario.TryGetValue<HttpClient>(ClientKey, out HttpClient? client))
return client;
TestApiFactory? factory = scenario.TryGetValue<TestApiFactory>( TestApiFactory? factory = scenario.TryGetValue<TestApiFactory>(
FactoryKey, FactoryKey,
@@ -39,9 +40,7 @@ public class AuthSteps(ScenarioContext scenario)
private static string GetRequiredEnvVar(string name) private static string GetRequiredEnvVar(string name)
{ {
return Environment.GetEnvironmentVariable(name) return Environment.GetEnvironmentVariable(name)
?? throw new InvalidOperationException( ?? throw new InvalidOperationException($"{name} environment variable is not set");
$"{name} environment variable is not set"
);
} }
private static string GenerateJwtToken( private static string GenerateJwtToken(
@@ -57,21 +56,13 @@ public class AuthSteps(ScenarioContext scenario)
private static Guid ParseRegisteredUserId(JsonElement root) private static Guid ParseRegisteredUserId(JsonElement root)
{ {
return root return root.GetProperty("payload").GetProperty("userAccountId").GetGuid();
.GetProperty("payload")
.GetProperty("userAccountId")
.GetGuid();
} }
private static string ParseRegisteredUsername(JsonElement root) private static string ParseRegisteredUsername(JsonElement root)
{ {
return root return root.GetProperty("payload").GetProperty("username").GetString()
.GetProperty("payload") ?? throw new InvalidOperationException("username missing from registration payload");
.GetProperty("username")
.GetString()
?? throw new InvalidOperationException(
"username missing from registration payload"
);
} }
private static string ParseTokenFromPayload( private static string ParseTokenFromPayload(
@@ -85,9 +76,7 @@ public class AuthSteps(ScenarioContext scenario)
|| payload.TryGetProperty(pascalCaseName, out tokenElem) || payload.TryGetProperty(pascalCaseName, out tokenElem)
) )
return tokenElem.GetString() return tokenElem.GetString()
?? throw new InvalidOperationException( ?? throw new InvalidOperationException($"{camelCaseName} is null");
$"{camelCaseName} is null"
);
throw new InvalidOperationException( throw new InvalidOperationException(
$"Could not find token field '{camelCaseName}' in payload" $"Could not find token field '{camelCaseName}' in payload"
@@ -113,22 +102,15 @@ public class AuthSteps(ScenarioContext scenario)
(string username, string password) = scenario.TryGetValue<( (string username, string password) = scenario.TryGetValue<(
string username, string username,
string password string password
)>(TestUserKey, out (string username, string password) user) )>(TestUserKey, out (string username, string password) user)
? user ? user
: ("test.user", "password"); : ("test.user", "password");
string body = JsonSerializer.Serialize(new { username, password }); string body = JsonSerializer.Serialize(new { username, password });
HttpRequestMessage requestMessage = new( HttpRequestMessage requestMessage = new(HttpMethod.Post, "/api/auth/login")
HttpMethod.Post,
"/api/auth/login"
)
{ {
Content = new StringContent( Content = new StringContent(body, Encoding.UTF8, "application/json"),
body,
Encoding.UTF8,
"application/json"
)
}; };
HttpResponseMessage response = await client.SendAsync(requestMessage); HttpResponseMessage response = await client.SendAsync(requestMessage);
@@ -144,16 +126,9 @@ public class AuthSteps(ScenarioContext scenario)
HttpClient client = GetClient(); HttpClient client = GetClient();
string body = JsonSerializer.Serialize(new { password = "test" }); string body = JsonSerializer.Serialize(new { password = "test" });
HttpRequestMessage requestMessage = new( HttpRequestMessage requestMessage = new(HttpMethod.Post, "/api/auth/login")
HttpMethod.Post,
"/api/auth/login"
)
{ {
Content = new StringContent( Content = new StringContent(body, Encoding.UTF8, "application/json"),
body,
Encoding.UTF8,
"application/json"
)
}; };
HttpResponseMessage response = await client.SendAsync(requestMessage); HttpResponseMessage response = await client.SendAsync(requestMessage);
@@ -169,16 +144,9 @@ public class AuthSteps(ScenarioContext scenario)
HttpClient client = GetClient(); HttpClient client = GetClient();
string body = JsonSerializer.Serialize(new { username = "test" }); string body = JsonSerializer.Serialize(new { username = "test" });
HttpRequestMessage requestMessage = new( HttpRequestMessage requestMessage = new(HttpMethod.Post, "/api/auth/login")
HttpMethod.Post,
"/api/auth/login"
)
{ {
Content = new StringContent( Content = new StringContent(body, Encoding.UTF8, "application/json"),
body,
Encoding.UTF8,
"application/json"
)
}; };
HttpResponseMessage response = await client.SendAsync(requestMessage); HttpResponseMessage response = await client.SendAsync(requestMessage);
@@ -192,16 +160,9 @@ public class AuthSteps(ScenarioContext scenario)
public async Task WhenISubmitALoginRequestWithBothUsernameAndPasswordMissing() public async Task WhenISubmitALoginRequestWithBothUsernameAndPasswordMissing()
{ {
HttpClient client = GetClient(); HttpClient client = GetClient();
HttpRequestMessage requestMessage = new( HttpRequestMessage requestMessage = new(HttpMethod.Post, "/api/auth/login")
HttpMethod.Post,
"/api/auth/login"
)
{ {
Content = new StringContent( Content = new StringContent("{}", Encoding.UTF8, "application/json"),
"{}",
Encoding.UTF8,
"application/json"
)
}; };
HttpResponseMessage response = await client.SendAsync(requestMessage); HttpResponseMessage response = await client.SendAsync(requestMessage);
@@ -218,10 +179,7 @@ public class AuthSteps(ScenarioContext scenario)
.TryGetValue<HttpResponseMessage>(ResponseKey, out HttpResponseMessage? response) .TryGetValue<HttpResponseMessage>(ResponseKey, out HttpResponseMessage? response)
.Should() .Should()
.BeTrue(); .BeTrue();
scenario scenario.TryGetValue<string>(ResponseBodyKey, out string? responseBody).Should().BeTrue();
.TryGetValue<string>(ResponseBodyKey, out string? responseBody)
.Should()
.BeTrue();
JsonDocument doc = JsonDocument.Parse(responseBody!); JsonDocument doc = JsonDocument.Parse(responseBody!);
JsonElement root = doc.RootElement; JsonElement root = doc.RootElement;
@@ -236,11 +194,7 @@ public class AuthSteps(ScenarioContext scenario)
payloadElem.TryGetProperty("accessToken", out tokenElem) payloadElem.TryGetProperty("accessToken", out tokenElem)
|| payloadElem.TryGetProperty("AccessToken", out tokenElem); || payloadElem.TryGetProperty("AccessToken", out tokenElem);
hasToken hasToken.Should().BeTrue("Expected an access token either at the root or inside 'payload'");
.Should()
.BeTrue(
"Expected an access token either at the root or inside 'payload'"
);
string? token = tokenElem.GetString(); string? token = tokenElem.GetString();
token.Should().NotBeNullOrEmpty(); token.Should().NotBeNullOrEmpty();
@@ -251,16 +205,9 @@ public class AuthSteps(ScenarioContext scenario)
{ {
HttpClient client = GetClient(); HttpClient client = GetClient();
// testing GET // testing GET
HttpRequestMessage requestMessage = new( HttpRequestMessage requestMessage = new(HttpMethod.Get, "/api/auth/login")
HttpMethod.Get,
"/api/auth/login"
)
{ {
Content = new StringContent( Content = new StringContent("{}", Encoding.UTF8, "application/json"),
"{}",
Encoding.UTF8,
"application/json"
)
}; };
HttpResponseMessage response = await client.SendAsync(requestMessage); HttpResponseMessage response = await client.SendAsync(requestMessage);
@@ -282,7 +229,8 @@ public class AuthSteps(ScenarioContext scenario)
string email = row["Email"] ?? ""; string email = row["Email"] ?? "";
string dateOfBirth = row["DateOfBirth"] ?? ""; string dateOfBirth = row["DateOfBirth"] ?? "";
if (dateOfBirth == "{underage_date}") dateOfBirth = DateTime.UtcNow.AddYears(-18).ToString("yyyy-MM-dd"); if (dateOfBirth == "{underage_date}")
dateOfBirth = DateTime.UtcNow.AddYears(-18).ToString("yyyy-MM-dd");
// Keep default registration fixture values unique across repeated runs. // Keep default registration fixture values unique across repeated runs.
if (email == "newuser@example.com") if (email == "newuser@example.com")
@@ -290,7 +238,8 @@ public class AuthSteps(ScenarioContext scenario)
string suffix = Guid.NewGuid().ToString("N")[..8]; string suffix = Guid.NewGuid().ToString("N")[..8];
email = $"newuser-{suffix}@example.com"; email = $"newuser-{suffix}@example.com";
if (username == "newuser") username = $"newuser-{suffix}"; if (username == "newuser")
username = $"newuser-{suffix}";
} }
string? password = row["Password"]; string? password = row["Password"];
@@ -302,21 +251,14 @@ public class AuthSteps(ScenarioContext scenario)
lastName, lastName,
email, email,
dateOfBirth, dateOfBirth,
password password,
}; };
string body = JsonSerializer.Serialize(registrationData); string body = JsonSerializer.Serialize(registrationData);
HttpRequestMessage requestMessage = new( HttpRequestMessage requestMessage = new(HttpMethod.Post, "/api/auth/register")
HttpMethod.Post,
"/api/auth/register"
)
{ {
Content = new StringContent( Content = new StringContent(body, Encoding.UTF8, "application/json"),
body,
Encoding.UTF8,
"application/json"
)
}; };
HttpResponseMessage response = await client.SendAsync(requestMessage); HttpResponseMessage response = await client.SendAsync(requestMessage);
@@ -330,16 +272,9 @@ public class AuthSteps(ScenarioContext scenario)
public async Task WhenISubmitARegistrationRequestUsingAGetRequest() public async Task WhenISubmitARegistrationRequestUsingAGetRequest()
{ {
HttpClient client = GetClient(); HttpClient client = GetClient();
HttpRequestMessage requestMessage = new( HttpRequestMessage requestMessage = new(HttpMethod.Get, "/api/auth/register")
HttpMethod.Get,
"/api/auth/register"
)
{ {
Content = new StringContent( Content = new StringContent("{}", Encoding.UTF8, "application/json"),
"{}",
Encoding.UTF8,
"application/json"
)
}; };
HttpResponseMessage response = await client.SendAsync(requestMessage); HttpResponseMessage response = await client.SendAsync(requestMessage);
@@ -361,20 +296,13 @@ public class AuthSteps(ScenarioContext scenario)
lastName = "User", lastName = "User",
email = $"newuser-{suffix}@example.com", email = $"newuser-{suffix}@example.com",
dateOfBirth = "1990-01-01", dateOfBirth = "1990-01-01",
password = "Password1!" password = "Password1!",
}; };
string body = JsonSerializer.Serialize(registrationData); string body = JsonSerializer.Serialize(registrationData);
HttpRequestMessage requestMessage = new( HttpRequestMessage requestMessage = new(HttpMethod.Post, "/api/auth/register")
HttpMethod.Post,
"/api/auth/register"
)
{ {
Content = new StringContent( Content = new StringContent(body, Encoding.UTF8, "application/json"),
body,
Encoding.UTF8,
"application/json"
)
}; };
HttpResponseMessage response = await client.SendAsync(requestMessage); HttpResponseMessage response = await client.SendAsync(requestMessage);
@@ -395,16 +323,9 @@ public class AuthSteps(ScenarioContext scenario)
var loginData = new { username = "test.user", password = "password" }; var loginData = new { username = "test.user", password = "password" };
string body = JsonSerializer.Serialize(loginData); string body = JsonSerializer.Serialize(loginData);
HttpRequestMessage requestMessage = new( HttpRequestMessage requestMessage = new(HttpMethod.Post, "/api/auth/login")
HttpMethod.Post,
"/api/auth/login"
)
{ {
Content = new StringContent( Content = new StringContent(body, Encoding.UTF8, "application/json"),
body,
Encoding.UTF8,
"application/json"
)
}; };
HttpResponseMessage response = await client.SendAsync(requestMessage); HttpResponseMessage response = await client.SendAsync(requestMessage);
@@ -445,17 +366,10 @@ public class AuthSteps(ScenarioContext scenario)
{ {
Guid userId = scenario.TryGetValue<Guid>(RegisteredUserIdKey, out Guid id) Guid userId = scenario.TryGetValue<Guid>(RegisteredUserIdKey, out Guid id)
? id ? id
: throw new InvalidOperationException( : throw new InvalidOperationException("registered user ID not found in scenario");
"registered user ID not found in scenario" string? username = scenario.TryGetValue<string>(RegisteredUsernameKey, out string? user)
);
string? username = scenario.TryGetValue<string>(
RegisteredUsernameKey,
out string? user
)
? user ? user
: throw new InvalidOperationException( : throw new InvalidOperationException("registered username not found in scenario");
"registered username not found in scenario"
);
string secret = GetRequiredEnvVar("ACCESS_TOKEN_SECRET"); string secret = GetRequiredEnvVar("ACCESS_TOKEN_SECRET");
scenario["accessToken"] = GenerateJwtToken( scenario["accessToken"] = GenerateJwtToken(
@@ -471,17 +385,10 @@ public class AuthSteps(ScenarioContext scenario)
{ {
Guid userId = scenario.TryGetValue<Guid>(RegisteredUserIdKey, out Guid id) Guid userId = scenario.TryGetValue<Guid>(RegisteredUserIdKey, out Guid id)
? id ? id
: throw new InvalidOperationException( : throw new InvalidOperationException("registered user ID not found in scenario");
"registered user ID not found in scenario" string? username = scenario.TryGetValue<string>(RegisteredUsernameKey, out string? user)
);
string? username = scenario.TryGetValue<string>(
RegisteredUsernameKey,
out string? user
)
? user ? user
: throw new InvalidOperationException( : throw new InvalidOperationException("registered username not found in scenario");
"registered username not found in scenario"
);
string secret = GetRequiredEnvVar("CONFIRMATION_TOKEN_SECRET"); string secret = GetRequiredEnvVar("CONFIRMATION_TOKEN_SECRET");
scenario["confirmationToken"] = GenerateJwtToken( scenario["confirmationToken"] = GenerateJwtToken(
@@ -497,17 +404,10 @@ public class AuthSteps(ScenarioContext scenario)
{ {
Guid userId = scenario.TryGetValue<Guid>(RegisteredUserIdKey, out Guid id) Guid userId = scenario.TryGetValue<Guid>(RegisteredUserIdKey, out Guid id)
? id ? id
: throw new InvalidOperationException( : throw new InvalidOperationException("registered user ID not found in scenario");
"registered user ID not found in scenario" string? username = scenario.TryGetValue<string>(RegisteredUsernameKey, out string? user)
);
string? username = scenario.TryGetValue<string>(
RegisteredUsernameKey,
out string? user
)
? user ? user
: throw new InvalidOperationException( : throw new InvalidOperationException("registered username not found in scenario");
"registered username not found in scenario"
);
string secret = GetRequiredEnvVar("CONFIRMATION_TOKEN_SECRET"); string secret = GetRequiredEnvVar("CONFIRMATION_TOKEN_SECRET");
scenario["confirmationToken"] = GenerateJwtToken( scenario["confirmationToken"] = GenerateJwtToken(
@@ -523,20 +423,12 @@ public class AuthSteps(ScenarioContext scenario)
{ {
Guid userId = scenario.TryGetValue<Guid>(RegisteredUserIdKey, out Guid id) Guid userId = scenario.TryGetValue<Guid>(RegisteredUserIdKey, out Guid id)
? id ? id
: throw new InvalidOperationException( : throw new InvalidOperationException("registered user ID not found in scenario");
"registered user ID not found in scenario" string? username = scenario.TryGetValue<string>(RegisteredUsernameKey, out string? user)
);
string? username = scenario.TryGetValue<string>(
RegisteredUsernameKey,
out string? user
)
? user ? user
: throw new InvalidOperationException( : throw new InvalidOperationException("registered username not found in scenario");
"registered username not found in scenario"
);
const string wrongSecret = const string wrongSecret = "wrong-confirmation-secret-that-is-very-long-1234567890";
"wrong-confirmation-secret-that-is-very-long-1234567890";
scenario["confirmationToken"] = GenerateJwtToken( scenario["confirmationToken"] = GenerateJwtToken(
userId, userId,
username, username,
@@ -545,9 +437,7 @@ public class AuthSteps(ScenarioContext scenario)
); );
} }
[When( [When("I submit a request to a protected endpoint with a valid access token")]
"I submit a request to a protected endpoint with a valid access token"
)]
public async Task WhenISubmitARequestToAProtectedEndpointWithAValidAccessToken() public async Task WhenISubmitARequestToAProtectedEndpointWithAValidAccessToken()
{ {
HttpClient client = GetClient(); HttpClient client = GetClient();
@@ -555,12 +445,9 @@ public class AuthSteps(ScenarioContext scenario)
? t ? t
: "invalid-token"; : "invalid-token";
HttpRequestMessage requestMessage = new( HttpRequestMessage requestMessage = new(HttpMethod.Get, "/api/protected")
HttpMethod.Get,
"/api/protected"
)
{ {
Headers = { { "Authorization", $"Bearer {token}" } } Headers = { { "Authorization", $"Bearer {token}" } },
}; };
HttpResponseMessage response = await client.SendAsync(requestMessage); HttpResponseMessage response = await client.SendAsync(requestMessage);
@@ -569,18 +456,13 @@ public class AuthSteps(ScenarioContext scenario)
scenario[ResponseBodyKey] = responseBody; scenario[ResponseBodyKey] = responseBody;
} }
[When( [When("I submit a request to a protected endpoint with an invalid access token")]
"I submit a request to a protected endpoint with an invalid access token"
)]
public async Task WhenISubmitARequestToAProtectedEndpointWithAnInvalidAccessToken() public async Task WhenISubmitARequestToAProtectedEndpointWithAnInvalidAccessToken()
{ {
HttpClient client = GetClient(); HttpClient client = GetClient();
HttpRequestMessage requestMessage = new( HttpRequestMessage requestMessage = new(HttpMethod.Get, "/api/protected")
HttpMethod.Get,
"/api/protected"
)
{ {
Headers = { { "Authorization", "Bearer invalid-token-format" } } Headers = { { "Authorization", "Bearer invalid-token-format" } },
}; };
HttpResponseMessage response = await client.SendAsync(requestMessage); HttpResponseMessage response = await client.SendAsync(requestMessage);
@@ -673,16 +555,9 @@ public class AuthSteps(ScenarioContext scenario)
: "valid-refresh-token"; : "valid-refresh-token";
string body = JsonSerializer.Serialize(new { refreshToken = token }); string body = JsonSerializer.Serialize(new { refreshToken = token });
HttpRequestMessage requestMessage = new( HttpRequestMessage requestMessage = new(HttpMethod.Post, "/api/auth/refresh")
HttpMethod.Post,
"/api/auth/refresh"
)
{ {
Content = new StringContent( Content = new StringContent(body, Encoding.UTF8, "application/json"),
body,
Encoding.UTF8,
"application/json"
)
}; };
HttpResponseMessage response = await client.SendAsync(requestMessage); HttpResponseMessage response = await client.SendAsync(requestMessage);
@@ -695,20 +570,11 @@ public class AuthSteps(ScenarioContext scenario)
public async Task WhenISubmitARefreshTokenRequestWithAnInvalidRefreshToken() public async Task WhenISubmitARefreshTokenRequestWithAnInvalidRefreshToken()
{ {
HttpClient client = GetClient(); HttpClient client = GetClient();
string body = JsonSerializer.Serialize( string body = JsonSerializer.Serialize(new { refreshToken = "invalid-refresh-token" });
new { refreshToken = "invalid-refresh-token" }
);
HttpRequestMessage requestMessage = new( HttpRequestMessage requestMessage = new(HttpMethod.Post, "/api/auth/refresh")
HttpMethod.Post,
"/api/auth/refresh"
)
{ {
Content = new StringContent( Content = new StringContent(body, Encoding.UTF8, "application/json"),
body,
Encoding.UTF8,
"application/json"
)
}; };
HttpResponseMessage response = await client.SendAsync(requestMessage); HttpResponseMessage response = await client.SendAsync(requestMessage);
@@ -722,20 +588,11 @@ public class AuthSteps(ScenarioContext scenario)
{ {
HttpClient client = GetClient(); HttpClient client = GetClient();
// Use an expired token // Use an expired token
string body = JsonSerializer.Serialize( string body = JsonSerializer.Serialize(new { refreshToken = "expired-refresh-token" });
new { refreshToken = "expired-refresh-token" }
);
HttpRequestMessage requestMessage = new( HttpRequestMessage requestMessage = new(HttpMethod.Post, "/api/auth/refresh")
HttpMethod.Post,
"/api/auth/refresh"
)
{ {
Content = new StringContent( Content = new StringContent(body, Encoding.UTF8, "application/json"),
body,
Encoding.UTF8,
"application/json"
)
}; };
HttpResponseMessage response = await client.SendAsync(requestMessage); HttpResponseMessage response = await client.SendAsync(requestMessage);
@@ -750,16 +607,9 @@ public class AuthSteps(ScenarioContext scenario)
HttpClient client = GetClient(); HttpClient client = GetClient();
string body = JsonSerializer.Serialize(new { }); string body = JsonSerializer.Serialize(new { });
HttpRequestMessage requestMessage = new( HttpRequestMessage requestMessage = new(HttpMethod.Post, "/api/auth/refresh")
HttpMethod.Post,
"/api/auth/refresh"
)
{ {
Content = new StringContent( Content = new StringContent(body, Encoding.UTF8, "application/json"),
body,
Encoding.UTF8,
"application/json"
)
}; };
HttpResponseMessage response = await client.SendAsync(requestMessage); HttpResponseMessage response = await client.SendAsync(requestMessage);
@@ -772,16 +622,9 @@ public class AuthSteps(ScenarioContext scenario)
public async Task WhenISubmitARefreshTokenRequestUsingAGETRequest() public async Task WhenISubmitARefreshTokenRequestUsingAGETRequest()
{ {
HttpClient client = GetClient(); HttpClient client = GetClient();
HttpRequestMessage requestMessage = new( HttpRequestMessage requestMessage = new(HttpMethod.Get, "/api/auth/refresh")
HttpMethod.Get,
"/api/auth/refresh"
)
{ {
Content = new StringContent( Content = new StringContent("{}", Encoding.UTF8, "application/json"),
"{}",
Encoding.UTF8,
"application/json"
)
}; };
HttpResponseMessage response = await client.SendAsync(requestMessage); HttpResponseMessage response = await client.SendAsync(requestMessage);
@@ -795,10 +638,7 @@ public class AuthSteps(ScenarioContext scenario)
public async Task WhenISubmitARequestToAProtectedEndpointWithoutAnAccessToken() public async Task WhenISubmitARequestToAProtectedEndpointWithoutAnAccessToken()
{ {
HttpClient client = GetClient(); HttpClient client = GetClient();
HttpRequestMessage requestMessage = new( HttpRequestMessage requestMessage = new(HttpMethod.Get, "/api/protected");
HttpMethod.Get,
"/api/protected"
);
HttpResponseMessage response = await client.SendAsync(requestMessage); HttpResponseMessage response = await client.SendAsync(requestMessage);
string responseBody = await response.Content.ReadAsStringAsync(); string responseBody = await response.Content.ReadAsStringAsync();
@@ -830,12 +670,9 @@ public class AuthSteps(ScenarioContext scenario)
? t ? t
: "expired-token"; : "expired-token";
HttpRequestMessage requestMessage = new( HttpRequestMessage requestMessage = new(HttpMethod.Get, "/api/protected")
HttpMethod.Get,
"/api/protected"
)
{ {
Headers = { { "Authorization", $"Bearer {token}" } } Headers = { { "Authorization", $"Bearer {token}" } },
}; };
HttpResponseMessage response = await client.SendAsync(requestMessage); HttpResponseMessage response = await client.SendAsync(requestMessage);
@@ -852,12 +689,9 @@ public class AuthSteps(ScenarioContext scenario)
? t ? t
: "tampered-token"; : "tampered-token";
HttpRequestMessage requestMessage = new( HttpRequestMessage requestMessage = new(HttpMethod.Get, "/api/protected")
HttpMethod.Get,
"/api/protected"
)
{ {
Headers = { { "Authorization", $"Bearer {token}" } } Headers = { { "Authorization", $"Bearer {token}" } },
}; };
HttpResponseMessage response = await client.SendAsync(requestMessage); HttpResponseMessage response = await client.SendAsync(requestMessage);
@@ -876,12 +710,9 @@ public class AuthSteps(ScenarioContext scenario)
? t ? t
: "refresh-token"; : "refresh-token";
HttpRequestMessage requestMessage = new( HttpRequestMessage requestMessage = new(HttpMethod.Get, "/api/protected")
HttpMethod.Get,
"/api/protected"
)
{ {
Headers = { { "Authorization", $"Bearer {token}" } } Headers = { { "Authorization", $"Bearer {token}" } },
}; };
HttpResponseMessage response = await client.SendAsync(requestMessage); HttpResponseMessage response = await client.SendAsync(requestMessage);
@@ -991,12 +822,9 @@ public class AuthSteps(ScenarioContext scenario)
? t ? t
: "confirmation-token"; : "confirmation-token";
HttpRequestMessage requestMessage = new( HttpRequestMessage requestMessage = new(HttpMethod.Get, "/api/protected")
HttpMethod.Get,
"/api/protected"
)
{ {
Headers = { { "Authorization", $"Bearer {token}" } } Headers = { { "Authorization", $"Bearer {token}" } },
}; };
HttpResponseMessage response = await client.SendAsync(requestMessage); HttpResponseMessage response = await client.SendAsync(requestMessage);
@@ -1049,54 +877,30 @@ public class AuthSteps(ScenarioContext scenario)
[Then("the response JSON should have a new access token")] [Then("the response JSON should have a new access token")]
public void ThenTheResponseJsonShouldHaveANewAccessToken() public void ThenTheResponseJsonShouldHaveANewAccessToken()
{ {
scenario scenario.TryGetValue<string>(ResponseBodyKey, out string? responseBody).Should().BeTrue();
.TryGetValue<string>(ResponseBodyKey, out string? responseBody)
.Should()
.BeTrue();
using JsonDocument doc = JsonDocument.Parse(responseBody!); using JsonDocument doc = JsonDocument.Parse(responseBody!);
JsonElement payload = doc.RootElement.GetProperty("payload"); JsonElement payload = doc.RootElement.GetProperty("payload");
string accessToken = ParseTokenFromPayload( string accessToken = ParseTokenFromPayload(payload, "accessToken", "AccessToken");
payload,
"accessToken",
"AccessToken"
);
accessToken.Should().NotBeNullOrWhiteSpace(); accessToken.Should().NotBeNullOrWhiteSpace();
if ( if (scenario.TryGetValue<string>(PreviousAccessTokenKey, out string? previousAccessToken))
scenario.TryGetValue<string>(
PreviousAccessTokenKey,
out string? previousAccessToken
)
)
accessToken.Should().NotBe(previousAccessToken); accessToken.Should().NotBe(previousAccessToken);
} }
[Then("the response JSON should have a new refresh token")] [Then("the response JSON should have a new refresh token")]
public void ThenTheResponseJsonShouldHaveANewRefreshToken() public void ThenTheResponseJsonShouldHaveANewRefreshToken()
{ {
scenario scenario.TryGetValue<string>(ResponseBodyKey, out string? responseBody).Should().BeTrue();
.TryGetValue<string>(ResponseBodyKey, out string? responseBody)
.Should()
.BeTrue();
using JsonDocument doc = JsonDocument.Parse(responseBody!); using JsonDocument doc = JsonDocument.Parse(responseBody!);
JsonElement payload = doc.RootElement.GetProperty("payload"); JsonElement payload = doc.RootElement.GetProperty("payload");
string refreshToken = ParseTokenFromPayload( string refreshToken = ParseTokenFromPayload(payload, "refreshToken", "RefreshToken");
payload,
"refreshToken",
"RefreshToken"
);
refreshToken.Should().NotBeNullOrWhiteSpace(); refreshToken.Should().NotBeNullOrWhiteSpace();
if ( if (scenario.TryGetValue<string>(PreviousRefreshTokenKey, out string? previousRefreshToken))
scenario.TryGetValue<string>(
PreviousRefreshTokenKey,
out string? previousRefreshToken
)
)
refreshToken.Should().NotBe(previousRefreshToken); refreshToken.Should().NotBe(previousRefreshToken);
} }
@@ -1186,4 +990,4 @@ public class AuthSteps(ScenarioContext scenario)
scenario[ResponseKey] = response; scenario[ResponseKey] = response;
scenario[ResponseBodyKey] = responseBody; scenario[ResponseBodyKey] = responseBody;
} }
} }

View File

@@ -20,7 +20,8 @@ public class TestApiFactory : WebApplicationFactory<Program>
d.ServiceType == typeof(IEmailProvider) d.ServiceType == typeof(IEmailProvider)
); );
if (emailProviderDescriptor != null) services.Remove(emailProviderDescriptor); if (emailProviderDescriptor != null)
services.Remove(emailProviderDescriptor);
services.AddScoped<IEmailProvider, MockEmailProvider>(); services.AddScoped<IEmailProvider, MockEmailProvider>();
@@ -29,9 +30,10 @@ public class TestApiFactory : WebApplicationFactory<Program>
d.ServiceType == typeof(IEmailDispatcher) d.ServiceType == typeof(IEmailDispatcher)
); );
if (emailDispatcherDescriptor != null) services.Remove(emailDispatcherDescriptor); if (emailDispatcherDescriptor != null)
services.Remove(emailDispatcherDescriptor);
services.AddScoped<IEmailDispatcher, MockEmailDispatcher>(); services.AddScoped<IEmailDispatcher, MockEmailDispatcher>();
}); });
} }
} }

View File

@@ -9,11 +9,11 @@
</PropertyGroup> </PropertyGroup>
<ItemGroup> <ItemGroup>
<PackageReference Include="dbup" Version="5.0.41"/> <PackageReference Include="dbup" Version="5.0.41" />
<PackageReference Include="Microsoft.Data.SqlClient" Version="6.1.2"/> <PackageReference Include="Microsoft.Data.SqlClient" Version="6.1.2" />
</ItemGroup> </ItemGroup>
<ItemGroup> <ItemGroup>
<EmbeddedResource Include="scripts/**/*.sql"/> <EmbeddedResource Include="scripts/**/*.sql" />
</ItemGroup> </ItemGroup>
<ItemGroup> <ItemGroup>
<Content Include="..\..\.dockerignore"> <Content Include="..\..\.dockerignore">

View File

@@ -35,21 +35,25 @@ public static class Program
/// </exception> /// </exception>
private static string BuildConnectionString(string? databaseName = null) private static string BuildConnectionString(string? databaseName = null)
{ {
string server = Environment.GetEnvironmentVariable("DB_SERVER") string server =
?? throw new InvalidOperationException("DB_SERVER environment variable is not set"); Environment.GetEnvironmentVariable("DB_SERVER")
?? throw new InvalidOperationException("DB_SERVER environment variable is not set");
string dbName = databaseName string dbName =
?? Environment.GetEnvironmentVariable("DB_NAME") databaseName
?? throw new InvalidOperationException("DB_NAME environment variable is not set"); ?? Environment.GetEnvironmentVariable("DB_NAME")
?? throw new InvalidOperationException("DB_NAME environment variable is not set");
string user = Environment.GetEnvironmentVariable("DB_USER") string user =
?? throw new InvalidOperationException("DB_USER environment variable is not set"); Environment.GetEnvironmentVariable("DB_USER")
?? throw new InvalidOperationException("DB_USER environment variable is not set");
string password = Environment.GetEnvironmentVariable("DB_PASSWORD") string password =
?? throw new InvalidOperationException("DB_PASSWORD environment variable is not set"); Environment.GetEnvironmentVariable("DB_PASSWORD")
?? throw new InvalidOperationException("DB_PASSWORD environment variable is not set");
string trustServerCertificate = Environment.GetEnvironmentVariable("DB_TRUST_SERVER_CERTIFICATE") string trustServerCertificate =
?? "True"; Environment.GetEnvironmentVariable("DB_TRUST_SERVER_CERTIFICATE") ?? "True";
SqlConnectionStringBuilder builder = new() SqlConnectionStringBuilder builder = new()
{ {
@@ -58,7 +62,7 @@ public static class Program
UserID = user, UserID = user,
Password = password, Password = password,
TrustServerCertificate = bool.Parse(trustServerCertificate), TrustServerCertificate = bool.Parse(trustServerCertificate),
Encrypt = true Encrypt = true,
}; };
return builder.ConnectionString; return builder.ConnectionString;
@@ -99,9 +103,10 @@ public static class Program
// First, set the database to single user mode to close all connections // First, set the database to single user mode to close all connections
SqlCommand setModeCommand = new( SqlCommand setModeCommand = new(
"IF EXISTS (SELECT 1 FROM sys.databases WHERE name = 'Biergarten') " + "IF EXISTS (SELECT 1 FROM sys.databases WHERE name = 'Biergarten') "
"ALTER DATABASE [Biergarten] SET SINGLE_USER WITH ROLLBACK IMMEDIATE;", + "ALTER DATABASE [Biergarten] SET SINGLE_USER WITH ROLLBACK IMMEDIATE;",
myConn); myConn
);
try try
{ {
setModeCommand.ExecuteNonQuery(); setModeCommand.ExecuteNonQuery();
@@ -132,7 +137,8 @@ public static class Program
} }
finally finally
{ {
if (myConn.State == ConnectionState.Open) myConn.Close(); if (myConn.State == ConnectionState.Open)
myConn.Close();
} }
return true; return true;
@@ -149,9 +155,9 @@ public static class Program
SqlConnection myConn = new(masterConnectionString); SqlConnection myConn = new(masterConnectionString);
const string str = """ const string str = """
IF NOT EXISTS (SELECT 1 FROM sys.databases WHERE name = 'Biergarten') IF NOT EXISTS (SELECT 1 FROM sys.databases WHERE name = 'Biergarten')
CREATE DATABASE [Biergarten] CREATE DATABASE [Biergarten]
"""; """;
SqlCommand myCommand = new(str, myConn); SqlCommand myCommand = new(str, myConn);
try try
@@ -166,7 +172,8 @@ public static class Program
} }
finally finally
{ {
if (myConn.State == ConnectionState.Open) myConn.Close(); if (myConn.State == ConnectionState.Open)
myConn.Close();
} }
return true; return true;
@@ -211,4 +218,4 @@ public static class Program
return 1; return 1;
} }
} }
} }

View File

@@ -8,16 +8,13 @@
</PropertyGroup> </PropertyGroup>
<ItemGroup> <ItemGroup>
<PackageReference Include="idunno.Password.Generator" Version="1.0.1"/> <PackageReference Include="idunno.Password.Generator" Version="1.0.1" />
<PackageReference <PackageReference Include="Konscious.Security.Cryptography.Argon2" Version="1.3.1" />
Include="Konscious.Security.Cryptography.Argon2" <PackageReference Include="Microsoft.Data.SqlClient" Version="6.1.4" />
Version="1.3.1" <PackageReference Include="dbup" Version="5.0.41" />
/>
<PackageReference Include="Microsoft.Data.SqlClient" Version="6.1.4"/>
<PackageReference Include="dbup" Version="5.0.41"/>
</ItemGroup> </ItemGroup>
<ItemGroup> <ItemGroup>
<ProjectReference Include="..\..\Domain\Domain.Entities\Domain.Entities.csproj"/> <ProjectReference Include="..\..\Domain\Domain.Entities\Domain.Entities.csproj" />
</ItemGroup> </ItemGroup>
</Project> </Project>

View File

@@ -16,4 +16,4 @@ internal interface ISeeder
/// <param name="connection">An open connection to the target database.</param> /// <param name="connection">An open connection to the target database.</param>
/// <returns>A task that completes when seeding is finished.</returns> /// <returns>A task that completes when seeding is finished.</returns>
Task SeedAsync(SqlConnection connection); Task SeedAsync(SqlConnection connection);
} }

View File

@@ -15,24 +15,22 @@ namespace Database.Seed;
internal class LocationSeeder : ISeeder internal class LocationSeeder : ISeeder
{ {
/// <summary>The set of countries to seed, identified by name and ISO 3166-1 code.</summary> /// <summary>The set of countries to seed, identified by name and ISO 3166-1 code.</summary>
private static readonly IReadOnlyList<( private static readonly IReadOnlyList<(string CountryName, string CountryCode)> Countries =
string CountryName,
string CountryCode
)> Countries =
[ [
("Canada", "CA"), ("Canada", "CA"),
("Mexico", "MX"), ("Mexico", "MX"),
("United States", "US") ("United States", "US"),
]; ];
/// <summary> /// <summary>
/// The set of states/provinces to seed, each identified by name, ISO 3166-2 code, /// The set of states/provinces to seed, each identified by name, ISO 3166-2 code,
/// and the ISO 3166-1 code of its parent country. /// and the ISO 3166-1 code of its parent country.
/// </summary> /// </summary>
private static IReadOnlyList<(string StateProvinceName, string StateProvinceCode, string CountryCode)> States private static IReadOnlyList<(
{ string StateProvinceName,
get; string StateProvinceCode,
} = string CountryCode
)> States { get; } =
[ [
("Alabama", "US-AL", "US"), ("Alabama", "US-AL", "US"),
("Alaska", "US-AK", "US"), ("Alaska", "US-AK", "US"),
@@ -134,7 +132,7 @@ internal class LocationSeeder : ISeeder
("Veracruz de Ignacio de la Llave", "MX-VER", "MX"), ("Veracruz de Ignacio de la Llave", "MX-VER", "MX"),
("Yucatán", "MX-YUC", "MX"), ("Yucatán", "MX-YUC", "MX"),
("Zacatecas", "MX-ZAC", "MX"), ("Zacatecas", "MX-ZAC", "MX"),
("Ciudad de México", "MX-CMX", "MX") ("Ciudad de México", "MX-CMX", "MX"),
]; ];
/// <summary> /// <summary>
@@ -255,7 +253,7 @@ internal class LocationSeeder : ISeeder
("MX-COA", "Saltillo"), ("MX-COA", "Saltillo"),
("MX-BCS", "La Paz"), ("MX-BCS", "La Paz"),
("MX-NAY", "Tepic"), ("MX-NAY", "Tepic"),
("MX-ZAC", "Zacatecas") ("MX-ZAC", "Zacatecas"),
]; ];
/// <summary> /// <summary>
@@ -269,9 +267,7 @@ internal class LocationSeeder : ISeeder
foreach ((string countryName, string countryCode) in Countries) foreach ((string countryName, string countryCode) in Countries)
await CreateCountryAsync(connection, countryName, countryCode); await CreateCountryAsync(connection, countryName, countryCode);
foreach ( foreach ((string stateProvinceName, string stateProvinceCode, string countryCode) in States)
(string stateProvinceName, string stateProvinceCode, string countryCode) in States
)
await CreateStateProvinceAsync( await CreateStateProvinceAsync(
connection, connection,
stateProvinceName, stateProvinceName,
@@ -294,10 +290,7 @@ internal class LocationSeeder : ISeeder
string countryCode string countryCode
) )
{ {
await using SqlCommand command = new( await using SqlCommand command = new("dbo.USP_CreateCountry", connection);
"dbo.USP_CreateCountry",
connection
);
command.CommandType = CommandType.StoredProcedure; command.CommandType = CommandType.StoredProcedure;
command.Parameters.AddWithValue("@CountryName", countryName); command.Parameters.AddWithValue("@CountryName", countryName);
command.Parameters.AddWithValue("@ISO3166_1", countryCode); command.Parameters.AddWithValue("@ISO3166_1", countryCode);
@@ -318,15 +311,9 @@ internal class LocationSeeder : ISeeder
string countryCode string countryCode
) )
{ {
await using SqlCommand command = new( await using SqlCommand command = new("dbo.USP_CreateStateProvince", connection);
"dbo.USP_CreateStateProvince",
connection
);
command.CommandType = CommandType.StoredProcedure; command.CommandType = CommandType.StoredProcedure;
command.Parameters.AddWithValue( command.Parameters.AddWithValue("@StateProvinceName", stateProvinceName);
"@StateProvinceName",
stateProvinceName
);
command.Parameters.AddWithValue("@ISO3166_2", stateProvinceCode); command.Parameters.AddWithValue("@ISO3166_2", stateProvinceCode);
command.Parameters.AddWithValue("@CountryCode", countryCode); command.Parameters.AddWithValue("@CountryCode", countryCode);
@@ -344,17 +331,11 @@ internal class LocationSeeder : ISeeder
string stateProvinceCode string stateProvinceCode
) )
{ {
await using SqlCommand command = new( await using SqlCommand command = new("dbo.USP_CreateCity", connection);
"dbo.USP_CreateCity",
connection
);
command.CommandType = CommandType.StoredProcedure; command.CommandType = CommandType.StoredProcedure;
command.Parameters.AddWithValue("@CityName", cityName); command.Parameters.AddWithValue("@CityName", cityName);
command.Parameters.AddWithValue( command.Parameters.AddWithValue("@StateProvinceCode", stateProvinceCode);
"@StateProvinceCode",
stateProvinceCode
);
await command.ExecuteNonQueryAsync(); await command.ExecuteNonQueryAsync();
} }
} }

View File

@@ -17,20 +17,24 @@ using Microsoft.Data.SqlClient;
/// </exception> /// </exception>
string BuildConnectionString() string BuildConnectionString()
{ {
string server = Environment.GetEnvironmentVariable("DB_SERVER") string server =
?? throw new InvalidOperationException("DB_SERVER environment variable is not set"); Environment.GetEnvironmentVariable("DB_SERVER")
?? throw new InvalidOperationException("DB_SERVER environment variable is not set");
string dbName = Environment.GetEnvironmentVariable("DB_NAME") string dbName =
?? throw new InvalidOperationException("DB_NAME environment variable is not set"); Environment.GetEnvironmentVariable("DB_NAME")
?? throw new InvalidOperationException("DB_NAME environment variable is not set");
string user = Environment.GetEnvironmentVariable("DB_USER") string user =
?? throw new InvalidOperationException("DB_USER environment variable is not set"); Environment.GetEnvironmentVariable("DB_USER")
?? throw new InvalidOperationException("DB_USER environment variable is not set");
string password = Environment.GetEnvironmentVariable("DB_PASSWORD") string password =
?? throw new InvalidOperationException("DB_PASSWORD environment variable is not set"); Environment.GetEnvironmentVariable("DB_PASSWORD")
?? throw new InvalidOperationException("DB_PASSWORD environment variable is not set");
string trustServerCertificate = Environment.GetEnvironmentVariable("DB_TRUST_SERVER_CERTIFICATE") string trustServerCertificate =
?? "True"; Environment.GetEnvironmentVariable("DB_TRUST_SERVER_CERTIFICATE") ?? "True";
SqlConnectionStringBuilder builder = new() SqlConnectionStringBuilder builder = new()
{ {
@@ -39,13 +43,12 @@ string BuildConnectionString()
UserID = user, UserID = user,
Password = password, Password = password,
TrustServerCertificate = bool.Parse(trustServerCertificate), TrustServerCertificate = bool.Parse(trustServerCertificate),
Encrypt = true Encrypt = true,
}; };
return builder.ConnectionString; return builder.ConnectionString;
} }
try try
{ {
string connectionString = BuildConnectionString(); string connectionString = BuildConnectionString();
@@ -74,17 +77,14 @@ try
connection = null; connection = null;
} }
if (connection == null) throw new Exception($"Failed to connect to database after {maxRetries} attempts."); if (connection == null)
throw new Exception($"Failed to connect to database after {maxRetries} attempts.");
Console.WriteLine("Starting seeding..."); Console.WriteLine("Starting seeding...");
using (connection) using (connection)
{ {
ISeeder[] seeders = ISeeder[] seeders = [new LocationSeeder(), new UserSeeder()];
[
new LocationSeeder(),
new UserSeeder()
];
foreach (ISeeder seeder in seeders) foreach (ISeeder seeder in seeders)
{ {
@@ -103,4 +103,4 @@ catch (Exception ex)
Console.Error.WriteLine("Seed failed:"); Console.Error.WriteLine("Seed failed:");
Console.Error.WriteLine(ex); Console.Error.WriteLine(ex);
return 1; return 1;
} }

View File

@@ -19,10 +19,7 @@ namespace Database.Seed;
internal class UserSeeder : ISeeder internal class UserSeeder : ISeeder
{ {
/// <summary>The first/last name pairs used to generate seed user accounts.</summary> /// <summary>The first/last name pairs used to generate seed user accounts.</summary>
private static readonly IReadOnlyList<( private static readonly IReadOnlyList<(string FirstName, string LastName)> SeedNames =
string FirstName,
string LastName
)> SeedNames =
[ [
("Aarya", "Mathews"), ("Aarya", "Mathews"),
("Aiden", "Wells"), ("Aiden", "Wells"),
@@ -123,7 +120,7 @@ internal class UserSeeder : ISeeder
("Zara", "Wilkinson"), ("Zara", "Wilkinson"),
("Zaria", "Gibson"), ("Zaria", "Gibson"),
("Zion", "Watkins"), ("Zion", "Watkins"),
("Zoie", "Armstrong") ("Zoie", "Armstrong"),
]; ];
/// <summary> /// <summary>
@@ -168,14 +165,9 @@ internal class UserSeeder : ISeeder
DateTime dob = GenerateDateOfBirth(rng); DateTime dob = GenerateDateOfBirth(rng);
// generate a password and hash it // generate a password and hash it
string pwd = generator.Generate( string pwd = generator.Generate(64, 10, 10);
64,
10,
10
);
string hash = GeneratePasswordHash(pwd); string hash = GeneratePasswordHash(pwd);
// register the user (creates account + credential) // register the user (creates account + credential)
Guid id = await RegisterUserAsync( Guid id = await RegisterUserAsync(
connection, connection,
@@ -189,9 +181,9 @@ internal class UserSeeder : ISeeder
createdUsers++; createdUsers++;
createdCredentials++; createdCredentials++;
// add user verification // add user verification
if (await HasUserVerificationAsync(connection, id)) continue; if (await HasUserVerificationAsync(connection, id))
continue;
await AddUserVerificationAsync(connection, id); await AddUserVerificationAsync(connection, id);
createdVerifications++; createdVerifications++;
@@ -227,7 +219,6 @@ internal class UserSeeder : ISeeder
await using SqlCommand command = new("dbo.USP_RegisterUser", connection); await using SqlCommand command = new("dbo.USP_RegisterUser", connection);
command.CommandType = CommandType.StoredProcedure; command.CommandType = CommandType.StoredProcedure;
command.Parameters.Add("@Username", SqlDbType.VarChar, 64).Value = username; command.Parameters.Add("@Username", SqlDbType.VarChar, 64).Value = username;
command.Parameters.Add("@FirstName", SqlDbType.NVarChar, 128).Value = firstName; command.Parameters.Add("@FirstName", SqlDbType.NVarChar, 128).Value = firstName;
command.Parameters.Add("@LastName", SqlDbType.NVarChar, 128).Value = lastName; command.Parameters.Add("@LastName", SqlDbType.NVarChar, 128).Value = lastName;
@@ -237,7 +228,6 @@ internal class UserSeeder : ISeeder
object? result = await command.ExecuteScalarAsync(); object? result = await command.ExecuteScalarAsync();
return (Guid)result!; return (Guid)result!;
} }
@@ -257,7 +247,7 @@ internal class UserSeeder : ISeeder
Salt = salt, Salt = salt,
DegreeOfParallelism = Math.Max(Environment.ProcessorCount, 1), DegreeOfParallelism = Math.Max(Environment.ProcessorCount, 1),
MemorySize = 65536, MemorySize = 65536,
Iterations = 4 Iterations = 4,
}; };
byte[] hash = argon2.GetBytes(32); byte[] hash = argon2.GetBytes(32);
@@ -274,10 +264,10 @@ internal class UserSeeder : ISeeder
) )
{ {
const string sql = """ const string sql = """
SELECT 1 SELECT 1
FROM dbo.UserVerification FROM dbo.UserVerification
WHERE UserAccountId = @UserAccountId; WHERE UserAccountId = @UserAccountId;
"""; """;
await using SqlCommand command = new(sql, connection); await using SqlCommand command = new(sql, connection);
command.Parameters.AddWithValue("@UserAccountId", userAccountId); command.Parameters.AddWithValue("@UserAccountId", userAccountId);
object? result = await command.ExecuteScalarAsync(); object? result = await command.ExecuteScalarAsync();
@@ -288,15 +278,9 @@ internal class UserSeeder : ISeeder
/// <param name="connection">An open connection to the target database.</param> /// <param name="connection">An open connection to the target database.</param>
/// <param name="userAccountId">The identifier of the user account to verify.</param> /// <param name="userAccountId">The identifier of the user account to verify.</param>
/// <returns>A task that completes when the verification record has been created.</returns> /// <returns>A task that completes when the verification record has been created.</returns>
private static async Task AddUserVerificationAsync( private static async Task AddUserVerificationAsync(SqlConnection connection, Guid userAccountId)
SqlConnection connection,
Guid userAccountId
)
{ {
await using SqlCommand command = new( await using SqlCommand command = new("dbo.USP_CreateUserVerification", connection);
"dbo.USP_CreateUserVerification",
connection
);
command.CommandType = CommandType.StoredProcedure; command.CommandType = CommandType.StoredProcedure;
command.Parameters.AddWithValue("@UserAccountID_", userAccountId); command.Parameters.AddWithValue("@UserAccountID_", userAccountId);
@@ -316,4 +300,4 @@ internal class UserSeeder : ISeeder
int offsetDays = random.Next(0, 365); int offsetDays = random.Next(0, 365);
return baseDate.AddDays(-offsetDays); return baseDate.AddDays(-offsetDays);
} }
} }

View File

@@ -47,4 +47,4 @@ public class BreweryPost
/// navigation property. /// navigation property.
/// </summary> /// </summary>
public BreweryPostLocation? Location { get; set; } public BreweryPostLocation? Location { get; set; }
} }

View File

@@ -7,46 +7,46 @@ namespace Domain.Entities;
/// </summary> /// </summary>
public class BreweryPostLocation public class BreweryPostLocation
{ {
/// <summary> /// <summary>
/// Primary key identifying this location. Maps to <c>BreweryPostLocationID</c>. /// Primary key identifying this location. Maps to <c>BreweryPostLocationID</c>.
/// </summary> /// </summary>
public Guid BreweryPostLocationId { get; set; } public Guid BreweryPostLocationId { get; set; }
/// <summary> /// <summary>
/// Foreign key referencing the owning <see cref="BreweryPost" />. Maps to <c>BreweryPostID</c>; unique, enforcing a /// Foreign key referencing the owning <see cref="BreweryPost" />. Maps to <c>BreweryPostID</c>; unique, enforcing a
/// one-to-one relationship. /// one-to-one relationship.
/// </summary> /// </summary>
public Guid BreweryPostId { get; set; } public Guid BreweryPostId { get; set; }
/// <summary> /// <summary>
/// The primary street address line. /// The primary street address line.
/// </summary> /// </summary>
public string AddressLine1 { get; set; } = string.Empty; public string AddressLine1 { get; set; } = string.Empty;
/// <summary> /// <summary>
/// An optional secondary address line (e.g., suite or unit number). /// An optional secondary address line (e.g., suite or unit number).
/// </summary> /// </summary>
public string? AddressLine2 { get; set; } public string? AddressLine2 { get; set; }
/// <summary> /// <summary>
/// The postal/ZIP code of the location. /// The postal/ZIP code of the location.
/// </summary> /// </summary>
public string PostalCode { get; set; } = string.Empty; public string PostalCode { get; set; } = string.Empty;
/// <summary> /// <summary>
/// Foreign key referencing the city in which the brewery is located. Maps to <c>CityID</c>. /// Foreign key referencing the city in which the brewery is located. Maps to <c>CityID</c>.
/// </summary> /// </summary>
public Guid CityId { get; set; } public Guid CityId { get; set; }
/// <summary> /// <summary>
/// Serialized SQL Server <c>GEOGRAPHY</c> value representing the geographic coordinates of the location, or /// Serialized SQL Server <c>GEOGRAPHY</c> value representing the geographic coordinates of the location, or
/// <c>null</c> if not set. /// <c>null</c> if not set.
/// </summary> /// </summary>
public byte[]? Coordinates { get; set; } public byte[]? Coordinates { get; set; }
/// <summary> /// <summary>
/// SQL Server <c>ROWVERSION</c> concurrency token used to detect concurrent updates. <c>null</c> until the row has /// SQL Server <c>ROWVERSION</c> concurrency token used to detect concurrent updates. <c>null</c> until the row has
/// been read from the database. /// been read from the database.
/// </summary> /// </summary>
public byte[]? Timer { get; set; } public byte[]? Timer { get; set; }
} }

View File

@@ -7,49 +7,49 @@ namespace Domain.Entities;
/// </summary> /// </summary>
public class UserAccount public class UserAccount
{ {
/// <summary> /// <summary>
/// Primary key identifying this user account. Maps to <c>UserAccountID</c>. /// Primary key identifying this user account. Maps to <c>UserAccountID</c>.
/// </summary> /// </summary>
public Guid UserAccountId { get; set; } public Guid UserAccountId { get; set; }
/// <summary> /// <summary>
/// The user's unique username, used for login and display. Enforced unique at the database level. /// The user's unique username, used for login and display. Enforced unique at the database level.
/// </summary> /// </summary>
public string Username { get; set; } = string.Empty; public string Username { get; set; } = string.Empty;
/// <summary> /// <summary>
/// The user's first name. /// The user's first name.
/// </summary> /// </summary>
public string FirstName { get; set; } = string.Empty; public string FirstName { get; set; } = string.Empty;
/// <summary> /// <summary>
/// The user's last name. /// The user's last name.
/// </summary> /// </summary>
public string LastName { get; set; } = string.Empty; public string LastName { get; set; } = string.Empty;
/// <summary> /// <summary>
/// The user's email address. Enforced unique at the database level. /// The user's email address. Enforced unique at the database level.
/// </summary> /// </summary>
public string Email { get; set; } = string.Empty; public string Email { get; set; } = string.Empty;
/// <summary> /// <summary>
/// The date and time the account was created. /// The date and time the account was created.
/// </summary> /// </summary>
public DateTime CreatedAt { get; set; } public DateTime CreatedAt { get; set; }
/// <summary> /// <summary>
/// The date and time the account was last updated, or <c>null</c> if it has never been updated. /// The date and time the account was last updated, or <c>null</c> if it has never been updated.
/// </summary> /// </summary>
public DateTime? UpdatedAt { get; set; } public DateTime? UpdatedAt { get; set; }
/// <summary> /// <summary>
/// The user's date of birth. /// The user's date of birth.
/// </summary> /// </summary>
public DateTime DateOfBirth { get; set; } public DateTime DateOfBirth { get; set; }
/// <summary> /// <summary>
/// SQL Server <c>ROWVERSION</c> concurrency token used to detect concurrent updates. <c>null</c> until the row has /// SQL Server <c>ROWVERSION</c> concurrency token used to detect concurrent updates. <c>null</c> until the row has
/// been read from the database. /// been read from the database.
/// </summary> /// </summary>
public byte[]? Timer { get; set; } public byte[]? Timer { get; set; }
} }

Some files were not shown because too many files have changed in this diff Show More