43 Commits

Author SHA1 Message Date
Aaron Po
58e476afdb Add user and brewery address tables 2026-07-13 19:59:18 -04:00
Aaron Po
d52dba904c Pipeline: rename Location to City, wire postal code into breweries 2026-07-13 02:30:09 -04:00
Aaron Po
fbcf438381 Fix mock-only builds (#231) 2026-07-03 02:47:54 -04:00
2b8a900d12 Pipeline: Add LLM and mocked user generation to the pipeline (#230) 2026-07-01 07:47:24 -04:00
Aaron Po
880f73e004 update pipeline docs 2026-06-21 21:58:03 -04:00
Aaron Po
3711591db1 commit csharpier config update 2026-06-20 15:54:53 -04:00
Aaron Po
194509f9c9 update mailkit 2026-06-20 15:54:53 -04:00
Aaron Po
67d6350425 Format ./web/backend/Shared/Shared.Contracts 2026-06-20 15:54:53 -04:00
Aaron Po
cee2917556 Format ./web/backend/Shared/Shared.Application 2026-06-20 15:54:53 -04:00
Aaron Po
6f8e5dc722 Format ./web/backend/API/API.Specs 2026-06-20 15:54:53 -04:00
Aaron Po
db3ed3581d Format ./web/backend/API/API.Core 2026-06-20 15:54:53 -04:00
Aaron Po
b54f70ccbe Format ./web/backend/Domain/Domain.Exceptions 2026-06-20 15:54:53 -04:00
Aaron Po
d4734ae9e7 Format ./web/backend/Domain/Domain.Entities 2026-06-20 15:54:53 -04:00
Aaron Po
27f28d5207 Format ./web/backend/Database/Database.Seed 2026-06-20 15:54:53 -04:00
Aaron Po
bf0d29fa54 Format ./web/backend/Database/Database.Migrations 2026-06-20 15:54:53 -04:00
Aaron Po
4772f59a68 Format ./web/backend/Infrastructure/Infrastructure.Sql 2026-06-20 15:54:53 -04:00
Aaron Po
2e5d4923c0 Format ./web/backend/Infrastructure/Infrastructure.PasswordHashing 2026-06-20 15:54:53 -04:00
Aaron Po
62766b7638 Format ./web/backend/Infrastructure/Infrastructure.Jwt 2026-06-20 15:54:53 -04:00
Aaron Po
744d4c7f8c Format ./web/backend/Infrastructure/Infrastructure.Email.Templates 2026-06-20 15:54:53 -04:00
Aaron Po
2f28ac9350 Format ./web/backend/Infrastructure/Infrastructure.Email 2026-06-20 15:54:53 -04:00
Aaron Po
c8bddae817 Format ./web/backend/Features/Features.UserManagement.Tests 2026-06-20 15:54:53 -04:00
Aaron Po
b298bd02d3 Format ./web/backend/Features/Features.UserManagement 2026-06-20 15:54:53 -04:00
Aaron Po
e2e5eb89e3 Format ./web/backend/Features/Features.Emails.Tests 2026-06-20 15:54:53 -04:00
Aaron Po
3dd44fca5a Format ./web/backend/Features/Features.Emails 2026-06-20 15:54:53 -04:00
Aaron Po
0c55d6c83f Format ./web/backend/Features/Features.Breweries.Tests 2026-06-20 15:54:53 -04:00
Aaron Po
58cbc5b5ba Format ./web/backend/Features/Features.Breweries 2026-06-20 15:54:53 -04:00
Aaron Po
5bef24696b Format ./web/backend/Features/Features.Auth.Tests 2026-06-20 15:54:53 -04:00
Aaron Po
79ae18b3e2 Format ./web/backend/Features/Features.Auth 2026-06-20 15:54:53 -04:00
Aaron Po
254431928f code style cleanup 2026-06-20 15:54:53 -04:00
Aaron Po
07aedcb866 Update documentation to reflect new architecture and previous refactors 2026-06-20 15:54:53 -04:00
Aaron Po
fd341e332c Migrate Auth and Emails to vertical slices (Features.Auth, Features.Emails)
Auth and Emails land together since Auth's registration/resend flows need
Emails to exist first. Service.Auth's four services (Register, Login,
Confirmation, Token) collapse into MediatR commands/queries in
Features.Auth; TokenService stays as a slice-internal service since
multiple handlers call it. Auth no longer references Emails directly —
it sends SendRegistrationEmailCommand/SendResendConfirmationEmailCommand
(defined in Shared.Application) which Features.Emails handles, so neither
slice has a project reference on the other.

Service.Emails' IEmailService becomes Features.Emails' IEmailDispatcher,
simplified to take (firstName, email, token) instead of a full UserAccount
since that's all it ever used. API.Specs' TestApiFactory/MockEmailService
are updated to swap in the relocated interface.

Also deletes Infrastructure.Repository now that Breweries, UserManagement,
and Auth have all moved their repos out of it, and replaces the two
now-dead docker-compose test services (repository.tests, service.auth.tests,
both pointing at deleted Dockerfiles) with a single unit.tests service that
runs every Features.*.Tests project via Dockerfile.tests.
2026-06-20 15:54:53 -04:00
Aaron Po
5cf4df1fd8 Migrate UserManagement to a vertical slice (Features.UserManagement)
Same pattern as the Breweries migration: Service.UserManagement and the
UserAccount repository fold into Features.UserManagement, with MediatR
queries replacing UserService. UpdateUserCommand/Handler carry forward
IUserService.UpdateAsync as-is (it has no HTTP route today, and adding one
is a separate decision from this migration). Adds handler/repository test
coverage that didn't exist before, since Service.UserManagement had no
test project.
2026-06-20 15:54:53 -04:00
Aaron Po
34ba7e8271 Migrate Breweries to a vertical slice (Features.Breweries)
Moves brewery CRUD from the Service.Breweries/Infrastructure.Repository
layered split into a single Features.Breweries project: MediatR
commands/queries replace BreweryService, and the repository moves in
alongside its own controller. Extracts Infrastructure.Sql out of
Infrastructure.Repository (just the generic ADO.NET connection/base-repo
plumbing) since Breweries no longer needs the rest of that project, while
Auth and UserAccount repos still do until their own migration.

Also fixes the API.Core and Infrastructure.Repository.Tests Dockerfiles,
which were restoring against COPY destinations that didn't match the
projects' actual relative paths (silently working only because of the
COPY . . that followed); both now restore against Core.slnx with correctly
nested paths, verified with a real docker build.
2026-06-20 15:54:53 -04:00
Aaron Po
60a7b790d4 Scaffold vertical-slice migration: Shared.Contracts, Shared.Application, MediatR
Begins the move to vertical-slice architecture. ResponseBody moves out of
API.Core into Shared.Contracts so slices won't have to depend on the API
host project for it. Shared.Application adds the MediatR pipeline's
ValidationBehavior plus the cross-slice email commands that Features.Auth
will send and Features.Emails will handle, keeping slices decoupled from
each other.
2026-06-20 15:54:53 -04:00
Aaron Po
4554fff534 add tests 2026-06-20 15:54:53 -04:00
Aaron Po
86a0c50f6e finish brewery feature set 2026-06-20 15:54:53 -04:00
Aaron Po
3034020d56 add xmldoc comments 2026-06-20 15:54:53 -04:00
6a66619c70 Add multithreaded logging infrastructure for preparation for future designs (#225)
* Update class diagrams

* Implement BoundedChannel and multithreaded logging infra

* Integrate logging channel system

* Update string concatenations to use std::format

* Add pretty print log
2026-05-22 22:00:38 -04:00
2ee7b3d2a2 Add timeout to wikipedia enrichment to avoid breaking rate limits, add mock enrichment (#224)
* Add timeout for enrichment, refactor json deserialization

* Add location count to application options and as a cli arg

* Add mock enrichment process
2026-05-14 19:15:51 -04:00
b7c0b1c8d4 Fix mistake in .gitattributes
archive/* is incorrect as it will ignore sub-dirs
2026-05-12 01:05:07 -04:00
b8ebe03921 Pipeline: Add Runpod docker configuration (#222)
* Begin work on Runpod docker config

* Reduce docker image size

* Create .dockerignore
2026-05-12 00:44:09 -04:00
26635ace84 Organize and consolidate header files (#220) 2026-05-03 21:44:37 -04:00
031be8ad5d Pipeline: Remove CURL as a dependency, add new HTTP module (#219)
Rationale: 

HTTP is a supporting concern in the pipeline, used only for Wikipedia enrichment calls. libcurl's C API required significant boilerplate to wrap safely. cpp-httplib is a header-only library that covers the same functionality with far less overhead and no manual resource management.
2026-05-03 13:35:58 -04:00
366 changed files with 77648 additions and 16352 deletions

2
.gitattributes vendored
View File

@@ -1 +1 @@
archive/* linguist-vendored archive/** linguist-vendored

View File

@@ -41,7 +41,7 @@ Data generation pipeline (C++):
Active areas in the repository: Active areas in the repository:
- .NET 10 backend (layered architecture) + SQL Server - .NET 10 backend (vertical-slice architecture with MediatR) + SQL Server
- React 19 website (React Router 7 + Vite) - React 19 website (React Router 7 + Vite)
- Shared Biergarten theme system + Storybook coverage - Shared Biergarten theme system + Storybook coverage
- Auth flows and account/email integration (local Mailpit in dev compose) - Auth flows and account/email integration (local Mailpit in dev compose)
@@ -105,7 +105,8 @@ npm run test:storybook:playwright
```text ```text
web/ web/
backend/ .NET API + domain/service/infrastructure + DB projects backend/ .NET API: API.Core (host), Features.* (vertical slices),
Shared.*, Domain.*, Infrastructure.*, Database.* projects
frontend/ React Router website + Storybook + Playwright/Vitest frontend/ React Router website + Storybook + Playwright/Vitest
tooling/ tooling/

View File

@@ -7,61 +7,65 @@ This document describes the active architecture of The Biergarten App.
The Biergarten App is a monorepo with a clear split between the backend and the The Biergarten App is a monorepo with a clear split between the backend and the
active website: active website:
- **Backend**: .NET 10 Web API with SQL Server and a layered architecture - **Backend**: .NET 10 Web API with SQL Server, organized as vertical feature
- **Frontend**: React 19 + React Router 7 website in `src/Website` slices with MediatR
- **Architecture Style**: Layered backend plus server-rendered React frontend - **Frontend**: React 19 + React Router 7 website in `web/frontend`
- **Architecture Style**: Vertical-slice backend plus server-rendered React
frontend
The legacy Next.js frontend has been retained in `src/Website-v1` for reference The legacy Next.js frontend has been retained in `archive/next-js-web-app/`
only and is documented in for reference only.
[archive/legacy-website-v1.md](archive/legacy-website-v1.md).
## Diagrams ## Diagrams
For visual representations, see: For visual representations, see:
- [architecture.svg](diagrams-out/architecture.svg) - Layered architecture - [architecture.svg](website/diagrams-out/architecture.svg) - Vertical-slice
architecture diagram
- [deployment.svg](website/diagrams-out/deployment.svg) - Docker deployment
diagram diagram
- [deployment.svg](diagrams-out/deployment.svg) - Docker deployment diagram - [authentication-flow.svg](website/diagrams-out/authentication-flow.svg) -
- [authentication-flow.svg](diagrams-out/authentication-flow.svg) -
Authentication workflow Authentication workflow
- [database-schema.svg](diagrams-out/database-schema.svg) - Database - [database-schema.svg](website/diagrams-out/database-schema.svg) - Database
relationships relationships
## Backend Architecture ## Backend Architecture
### Layered Architecture Pattern ### Vertical Slice Architecture Pattern
The backend follows a strict layered architecture: The backend organizes business capabilities as feature slices instead of
technical layers. Each feature (`Features.Auth`, `Features.Breweries`,
`Features.UserManagement`, `Features.Emails`) is a single project that owns
its own controller, MediatR commands/queries/handlers, validators, and
repository, end to end:
``` ```
┌─────────────────────────────────────┐ ┌───────────────────────────────────────────────────────────────────────
API Layer (Controllers) API.Core (thin host)
│ - HTTP Endpoints │ - Program.cs wiring: MediatR + AddApplicationPart per slice
│ - Request/Response mapping │ - Swagger/OpenAPI, JWT auth middleware, global exception filter
│ - Swagger/OpenAPI │ └───────────────────────────────────────────────────────────────────────┘
└─────────────────────────────────────┘ ↓ discovers controllers via AddApplicationPart
┌───────────────┬───────────────┬───────────────────┬───────────────────┐
│ Features.Auth │Features. │ Features. │ Features.Emails │
│ │Breweries │ UserManagement │ (no controller, │
│ Controller │ Controller │ Controller │ internal only) │
│ Commands/ │ Commands/ │ Commands/ │ Commands/ │
│ Queries + │ Queries + │ Queries + │ Handlers │
│ Handlers │ Handlers │ Handlers │ │
│ Repository │ Repository │ Repository │ EmailDispatcher │
└───────────────┴───────────────┴───────────────────┴───────────────────┘
↓ each slice depends only on shared/domain/infra, never on another slice
┌─────────────────────────┬─────────────────────────┬────────────────────┐
│ Shared.Contracts │ Shared.Application │ Domain.Entities / │
│ (ResponseBody envelope) │ (ValidationBehavior, │ Domain.Exceptions │
│ │ cross-slice email cmds) │ │
└─────────────────────────┴─────────────────────────┴────────────────────┘
┌─────────────────────────────────────┐ ┌─────────────────────────────────────────────────────────────────────────
Service Layer (Business Logic) Infrastructure.Sql, Infrastructure.Jwt, Infrastructure.PasswordHashing,
- Authentication logic Infrastructure.Email, Infrastructure.Email.Templates
│ - User management │ └─────────────────────────────────────────────────────────────────────────┘
│ - Validation & orchestration │
└─────────────────────────────────────┘
┌─────────────────────────────────────┐
│ Infrastructure Layer (Tools) │
│ - JWT token generation │
│ - Password hashing (Argon2id) │
│ - Email services │
│ - Repository implementations │
└─────────────────────────────────────┘
┌─────────────────────────────────────┐
│ Domain Layer (Entities) │
│ - UserAccount, UserCredential │
│ - Pure POCO classes │
│ - No external dependencies │
└─────────────────────────────────────┘
┌─────────────────────────────────────┐ ┌─────────────────────────────────────┐
│ Database (SQL Server) │ │ Database (SQL Server) │
@@ -70,75 +74,123 @@ The backend follows a strict layered architecture:
└─────────────────────────────────────┘ └─────────────────────────────────────┘
``` ```
Slices never reference each other's project. The one cross-slice
interaction (`Features.Auth` triggering a confirmation email handled by
`Features.Emails`) goes through a MediatR command (`SendRegistrationEmailCommand`)
whose contract lives in `Shared.Application`, so neither slice takes a
project reference on the other.
### Layer Responsibilities ### Layer Responsibilities
#### API Layer (`API.Core`) #### API Layer (`API.Core`)
**Purpose**: HTTP interface and request handling **Purpose**: Thin ASP.NET Core host: no business logic, no controllers of
its own
**Components**: **Components**:
- Controllers (`AuthController`, `UserController`) - `Program.cs`: registers MediatR (scanning every `Features.*` assembly),
- Middleware for error handling FluentValidation, and uses `AddApplicationPart` so each slice's controllers
- Swagger/OpenAPI documentation are discovered by MVC
- Health check endpoints - `GlobalException.cs`: global exception filter (host-level cross-cutting
concern)
- `Authentication/JwtAuthenticationHandler.cs`: JWT auth scheme middleware
- Swagger/OpenAPI documentation, health check endpoints
**Dependencies**: **Dependencies**:
- Service layer - Every `Features.*` project (for controller/MediatR discovery)
- ASP.NET Core framework - `Shared.Contracts`, `Shared.Application`
- `Infrastructure.Jwt` (for the auth middleware)
**Rules**: **Rules**:
- No business logic - No controllers, no business logic, no feature-specific contracts
- Only request/response transformation - Exists purely to host and wire up the feature slices
- Delegates all work to Service layer
#### Service Layer (`Service.Auth`, `Service.UserManagement`) #### Feature Slices (`Features.Auth`, `Features.Breweries`, `Features.UserManagement`, `Features.Emails`)
**Purpose**: Business logic and orchestration **Purpose**: Each slice is the complete vertical for one business capability
**Components** (per slice):
- `Controllers/`: HTTP endpoints, binding directly to Command/Query types
as the request contract
- `Commands/<Operation>/` and `Queries/<Operation>/`: one folder per
operation, each containing the Command/Query record, its
`IRequestHandler`, and (for commands) a FluentValidation validator
- `Repository/`: the slice's own stored-procedure repository
implementation
- `Dtos/`: response shapes returned by query handlers (never the raw
domain entity)
- `DependencyInjection/`: an `AddFeaturesX()` extension method registering
the slice's repository/services
`Features.Emails` has no `Controllers/` folder. It's invoked only via
MediatR commands sent from other slices, never over HTTP.
**Dependencies**:
- `Domain.Entities`, `Domain.Exceptions`
- `Infrastructure.Sql` (generic ADO.NET plumbing) plus whichever
infrastructure project the slice needs (`Infrastructure.Jwt`/
`Infrastructure.PasswordHashing` for Auth, `Infrastructure.Email`/
`Infrastructure.Email.Templates` for Emails)
- `Shared.Contracts`, `Shared.Application`
- **Never** another `Features.*` project
**Rules**:
- All business logic for that feature lives in its command/query handlers
- No direct controller-to-repository calls; everything flows through
MediatR
- Read endpoints return a dedicated `Dto`, never the domain entity directly
#### Shared Projects (`Shared.Contracts`, `Shared.Application`)
**Purpose**: The minimum cross-slice surface area required because every
slice needs it, or because duplicating it four times would be worse than
sharing it
**Components**: **Components**:
- Authentication services (login, registration) - `Shared.Contracts`: `ResponseBody<T>`/`ResponseBody`, the API response
- User management services envelope every controller returns
- Business rule validation - `Shared.Application`: `ValidationBehavior<TRequest,TResponse>` (the
- Transaction coordination MediatR pipeline behavior that runs FluentValidation before a handler
executes) and the cross-slice email commands
**Dependencies**: (`SendRegistrationEmailCommand`, `SendResendConfirmationEmailCommand`)
- Infrastructure layer (repositories, JWT, password hashing)
- Domain entities
**Rules**: **Rules**:
- Contains all business logic - Kept deliberately small: this is the exception to "no slice depends on
- Coordinates multiple infrastructure components another slice," not a general-purpose dumping ground
- No direct database access (uses repositories)
- Returns domain models, not DTOs
#### Infrastructure Layer #### Infrastructure Layer
**Purpose**: Technical capabilities and external integrations **Purpose**: Technical capabilities and external integrations, shared by
whichever slices need them
**Components**: **Components**:
- **Infrastructure.Repository**: Data access via stored procedures - **Infrastructure.Sql**: generic ADO.NET connection/command plumbing
(`ISqlConnectionFactory`, the abstract `Repository<T>` base class), not
domain-specific; each slice's own `Repository/` folder builds on this
- **Infrastructure.Jwt**: JWT token generation and validation - **Infrastructure.Jwt**: JWT token generation and validation
- **Infrastructure.PasswordHashing**: Argon2id password hashing - **Infrastructure.PasswordHashing**: Argon2id password hashing
- **Infrastructure.Email**: Email sending capabilities - **Infrastructure.Email**: Email sending capabilities (SMTP/MailKit)
- **Infrastructure.Email.Templates**: Email template rendering - **Infrastructure.Email.Templates**: Email template rendering (Razor
components)
**Dependencies**: **Dependencies**:
- Domain entities - Domain entities
- External libraries (ADO.NET, JWT, Argon2, etc.) - External libraries (ADO.NET, JWT, Argon2, MailKit, etc.)
**Rules**: **Rules**:
- Implements technical concerns - Implements technical concerns only, no business logic
- No business logic - Reusable across slices
- Reusable across services
#### Domain Layer (`Domain.Entities`) #### Domain Layer (`Domain.Entities`)
@@ -163,21 +215,55 @@ The backend follows a strict layered architecture:
### Design Patterns ### Design Patterns
#### Vertical Slice + MediatR
**Purpose**: Organize code by feature instead of by technical layer, so
everything needed to understand or change one capability lives in one
project
**Implementation**:
- Each HTTP write operation is a `Command` (e.g. `CreateBreweryCommand` in
`Features.Breweries/Commands/CreateBrewery/`); each read operation is a
`Query` (e.g. `GetBreweryByIdQuery`)
- Controllers bind directly to the Command/Query as the request body;
there is no separate request DTO + mapping step for writes
- A single shared `ValidationBehavior<TRequest,TResponse>`
(`Shared.Application/Behaviors/`) runs FluentValidation validators in the
MediatR pipeline before any handler executes
- Query handlers map to a dedicated response `Dto`, so domain entities never
leak over the wire
**Example**:
```csharp
public record CreateBreweryCommand(Guid PostedById, string BreweryName, string Description, ...)
: IRequest<BreweryDto>;
public class CreateBreweryHandler(IBreweryRepository repository) : IRequestHandler<CreateBreweryCommand, BreweryDto>
{
public async Task<BreweryDto> Handle(CreateBreweryCommand request, CancellationToken cancellationToken) { ... }
}
```
#### Repository Pattern #### Repository Pattern
**Purpose**: Abstract database access behind interfaces **Purpose**: Abstract database access behind interfaces
**Implementation**: **Implementation**: each slice owns its own repository, scoped to that
feature only:
- `IAuthRepository` - Authentication queries - `Features.Auth/Repository/IAuthRepository.cs`
- `IUserAccountRepository` - User account queries - `Features.Breweries/Repository/IBreweryRepository.cs`
- `DefaultSqlConnectionFactory` - Connection management - `Features.UserManagement/Repository/IUserAccountRepository.cs`
- `Infrastructure.Sql/DefaultSqlConnectionFactory.cs`: the generic
connection factory every slice's repository builds on
**Benefits**: **Benefits**:
- Testable (easy to mock) - Testable (easy to mock)
- SQL-first approach (stored procedures) - SQL-first approach (stored procedures)
- Centralized data access logic - Each slice's data access logic is self-contained
**Example**: **Example**:
@@ -193,17 +279,19 @@ public interface IAuthRepository
**Purpose**: Loose coupling and testability **Purpose**: Loose coupling and testability
**Configuration**: `Program.cs` registers all services **Configuration**: `Program.cs` wires up MediatR/FluentValidation across
every `Features.*` assembly; each slice exposes its own `AddFeaturesX()`
extension method that registers its repository and slice-internal services
**Lifetimes**: **Lifetimes**:
- Scoped: Repositories, Services (per request) - Scoped: Repositories, slice-internal services (per request)
- Singleton: Connection factories, JWT configuration - Singleton: `ISqlConnectionFactory`
- Transient: Utilities, helpers - Transient: Utilities, helpers
#### SQL-First Approach #### SQL-First Approach
**Purpose**: Leverage database capabilities **Purpose**: Push complex logic into the database
**Strategy**: **Strategy**:
@@ -220,13 +308,13 @@ public interface IAuthRepository
## Frontend Architecture ## Frontend Architecture
### Active Website (`src/Website`) ### Active Website (`web/frontend`)
The current website is a React Router 7 application with server-side rendering The current website is a React Router 7 application with server-side rendering
enabled. enabled.
```text ```text
src/Website/ web/frontend/
├── app/ ├── app/
│ ├── components/ Shared UI such as Navbar, FormField, SubmitButton, ToastProvider │ ├── components/ Shared UI such as Navbar, FormField, SubmitButton, ToastProvider
│ ├── lib/ Auth helpers, schemas, and theme metadata │ ├── lib/ Auth helpers, schemas, and theme metadata
@@ -262,10 +350,9 @@ All component styling should prefer semantic tokens such as `primary`,
### Legacy Frontend ### Legacy Frontend
The previous Next.js frontend has been archived at `src/Website-v1`. Active The previous Next.js frontend has been archived at `archive/next-js-web-app/`
product and engineering documentation should point to `src/Website`, while for reference only. Active product and engineering documentation should point
legacy notes live in to `web/frontend`.
[archive/legacy-website-v1.md](archive/legacy-website-v1.md).
## Security Architecture ## Security Architecture
@@ -390,7 +477,7 @@ scripts/
- Testing (`docker-compose.test.yaml`) - Testing (`docker-compose.test.yaml`)
- Production (`docker-compose.prod.yaml`) - Production (`docker-compose.prod.yaml`)
For details, see [Docker Guide](docker.md). For details, see [Docker Guide](website/docker.md).
### Health Checks ### Health Checks
@@ -416,11 +503,10 @@ healthcheck:
│ Integration │ ← API.Specs (Reqnroll) │ Integration │ ← API.Specs (Reqnroll)
│ Tests │ │ Tests │
├──────────────┤ ├──────────────┤
│ Unit Tests │ ← Service.Auth.Tests │ Unit Tests │ ← Features.Auth.Tests, Features.Breweries.Tests,
(Service)Repository.Tests (per slice,Features.UserManagement.Tests, Features.Emails.Tests
├──────────────┤ │ handlers + │ (commands/queries/handlers + that slice's own
Unit Tests │ repository) │ repository, mocked with Moq/DbMocker)
│ (Repository) │
└──────────────┘ └──────────────┘
``` ```
@@ -431,4 +517,4 @@ healthcheck:
- Mock external dependencies - Mock external dependencies
- Test database for integration tests - Test database for integration tests
For details, see [Testing Guide](testing.md). For details, see [Testing Guide](website/testing.md).

View File

@@ -13,6 +13,7 @@ low-resource language failures.
- [Model Bias and Language Quality](#model-bias-and-language-quality) - [Model Bias and Language Quality](#model-bias-and-language-quality)
- [Western and Eurocentric Lens](#western-and-eurocentric-lens) - [Western and Eurocentric Lens](#western-and-eurocentric-lens)
- [Wikipedia Enrichment](#wikipedia-enrichment) - [Wikipedia Enrichment](#wikipedia-enrichment)
- [Names-by-Country Dataset](#names-by-country-dataset)
- [The "Avoid AI Phrases" Prompt Instruction](#the-avoid-ai-phrases-prompt-instruction) - [The "Avoid AI Phrases" Prompt Instruction](#the-avoid-ai-phrases-prompt-instruction)
- [Known Issues](#known-issues) - [Known Issues](#known-issues)
- [Hallucinated Brewing Techniques](#hallucinated-brewing-techniques) - [Hallucinated Brewing Techniques](#hallucinated-brewing-techniques)
@@ -91,6 +92,33 @@ generated descriptions.
--- ---
## Names-by-Country Dataset
`tooling/pipeline/forenames-by-country.json` and `surnames-by-country.json`
(used to sample a `Name` per ISO 3166-1 country code for user generation) are
vendored verbatim, unmodified, from
[sigpwned/popular-names-by-country-dataset](https://github.com/sigpwned/popular-names-by-country-dataset)
(the `common-forenames-by-country.json` / `common-surnames-by-country.json`
release assets), released under **CC0** (public domain). That dataset's own
forename/surname lists are pulled from Wikipedia's "Lists of most common
surnames" and "List of most popular given names" as of the week of
2023-07-08 — see that project's README for full provenance. Names are not
LLM-generated; this is curated fixture data per ROADMAP.md §2. Per-forename
gender from the source data is preserved through to the sampled `Name`
(rather than discarded during loading) so it's available for gender-aware
persona/bio generation later.
The full multinational dataset is kept as-is (106 countries for forenames,
75 for surnames) rather than trimmed to `cities.json`'s current country
list, so it doesn't need re-sourcing if more countries are added later.
`SampleName()` (a free helper in `generate_users.cc`) returns no result for
a country present in neither file; of the countries in `cities.json`,
that's currently `KE`, `SE`, `SG`, `TH`, `VN`, and `ZA``GenerateUsers`
skips cities in those countries the same way brewery generation skips
cities whose enrichment lookup fails.
---
## The "Avoid AI Phrases" Prompt Instruction ## The "Avoid AI Phrases" Prompt Instruction
The system prompt instructs the model to avoid common AI-generated phrasing The system prompt instructs the model to avoid common AI-generated phrasing
@@ -245,7 +273,7 @@ but is semantically incoherent. This comes from limited training-data coverage
rather than prompt engineering. rather than prompt engineering.
Output sample: Output sample:
[./out-sample/french-cities.example](out-sample/french-cities.example) [./french-cities.example](french-cities.example)
#### Proposed Mitigations #### Proposed Mitigations
@@ -253,7 +281,7 @@ Output sample:
a location's code is unlisted, skip `description_local` generation and fall a location's code is unlisted, skip `description_local` generation and fall
back to English. back to English.
- **Upstream sanitization:** strip known low-resource language codes from the - **Upstream sanitization:** strip known low-resource language codes from the
`locations.json` payload before generation. `cities.json` payload before generation.
- **Downstream flagging:** add a `description_local_confidence` column to the - **Downstream flagging:** add a `description_local_confidence` column to the
SQLite schema so downstream applications can filter or flag potentially SQLite schema so downstream applications can filter or flag potentially
hallucinated text by language tier. hallucinated text by language tier.

View File

@@ -1,8 +1,9 @@
# Biergarten Pipeline # Biergarten Pipeline
A C++20 command-line pipeline that samples city records from local JSON, A C++20 command-line pipeline that samples city records from local JSON,
enriches each with Wikipedia context, and generates bilingual brewery names and enriches each with Wikipedia context, and generates bilingual brewery names
descriptions via a local GGUF model or a deterministic mock. and descriptions plus locale-grounded user profiles via a local GGUF model or
a deterministic mock.
> **This pipeline produces AI-generated data.** It is not a source of truth for > **This pipeline produces AI-generated data.** It is not a source of truth for
> brewing techniques, cultural representation, or local-language accuracy. See > brewing techniques, cultural representation, or local-language accuracy. See
@@ -18,6 +19,7 @@ descriptions via a local GGUF model or a deterministic mock.
- [Build](#build) - [Build](#build)
- [Model](#model) - [Model](#model)
- [Run](#run) - [Run](#run)
- [Docker / RunPod](#docker--runpod)
- [Architecture](#architecture) - [Architecture](#architecture)
- [Pipeline Stages](#pipeline-stages) - [Pipeline Stages](#pipeline-stages)
- [Key Components](#key-components) - [Key Components](#key-components)
@@ -44,6 +46,7 @@ step.
| Beer reviews and ratings | Stable brewery fixtures with enough context to anchor review pages | | Beer reviews and ratings | Stable brewery fixtures with enough context to anchor review pages |
| Social follow relationships | Repeatable brewery entities for feeds, follows, and saved lists | | Social follow relationships | Repeatable brewery entities for feeds, follows, and saved lists |
| Geospatial brewery experiences | Latitude, longitude, and country-level metadata | | Geospatial brewery experiences | Latitude, longitude, and country-level metadata |
| User accounts and profiles | Locale-grounded names, bios, and an auth-ready email/date-of-birth pair for seeding real accounts |
--- ---
@@ -51,7 +54,7 @@ step.
### Build ### Build
Requirements: C++20 compiler, CMake 3.24+, libcurl, Boost (JSON and Requirements: C++20 compiler, CMake 3.31+, OpenSSL, Boost (JSON and
ProgramOptions). SQLite is fetched from the upstream amalgamation, so no system ProgramOptions). SQLite is fetched from the upstream amalgamation, so no system
SQLite package is required. SQLite package is required.
@@ -60,6 +63,16 @@ cmake -S . -B build
cmake --build build cmake --build build
``` ```
CMake automatically detects whether a compatible llama.cpp installation is
present on the system (`libllama`, `libggml`, `libggml-base`, and `llama.h`
visible on the default search paths). If found, it links against those
libraries and skips the FetchContent build. If not found, it fetches and builds
llama.cpp from source at tag `b9012`. No additional flags are required in
either case.
Metal is enabled automatically on Apple Silicon. CUDA or HIP/ROCm is detected
automatically on Linux when the relevant toolkit is present.
### Model ### Model
> Skip this step if you only need `--mocked`. > Skip this step if you only need `--mocked`.
@@ -73,26 +86,36 @@ curl -L \
### Run ### Run
Run from `build/` so the copied `locations.json` and `prompts/` are available. Run from `build/` so the copied `cities.json` and `prompts/` are available.
Each run also writes a fresh dated SQLite file such as Each run writes a fresh dated SQLite file such as
`biergarten_seed_2026-04-19T15-30-45.123456Z.sqlite` into the working directory. `biergarten_seed_2026-04-19T15-30-45.123456Z.sqlite` into the working directory.
```bash ```bash
./biergarten-pipeline --mocked ./biergarten-pipeline --mocked
./biergarten-pipeline --model models/google_gemma-4-E4B-it-Q6_K.gguf --temperature 1.0 --top-p 0.95 --top-k 64 --n-ctx 8192 --seed -1
./biergarten-pipeline \
--model ../models/google_gemma-4-E4B-it-Q6_K.gguf \
--prompt-dir prompts \
--location-count 25 \
--temperature 1.0 --top-p 0.95 --top-k 64 --n-ctx 8192 --seed -1
``` ```
#### CLI Flags #### CLI Flags
| Flag | Purpose | | Flag | Purpose |
| --------------- | ------------------------------------------------------- | | ------------------ | ---------------------------------------------------------------------------------------------------- |
| `--mocked` | Deterministic mock generator, no model required. | | `--mocked` | Deterministic mock generator, no model required. |
| `--model, -m` | Path to a GGUF file. Required unless `--mocked` is set. | | `--model, -m` | Path to a GGUF file. Required unless `--mocked` is set. |
| `--prompt-dir` | Directory containing prompt files (e.g. `BREWERY_GENERATION.md`). Required unless `--mocked` is set. |
| `--output, -o` | Directory for generated SQLite artifacts. Default: `output`. |
| `--log-path` | Path for application logs. Default: `pipeline.log`. |
| `--location-count` | Number of cities to sample from `cities.json` per run. Default: `10`. |
| `--temperature` | Sampling temperature. Default: `1.0`. | | `--temperature` | Sampling temperature. Default: `1.0`. |
| `--top-p` | Nucleus sampling. Default: `0.95`. | | `--top-p` | Nucleus sampling. Default: `0.95`. |
| `--top-k` | Top-k sampling. Default: `64`. | | `--top-k` | Top-k sampling. Default: `64`. |
| `--n-ctx` | Context window size. Default: `8192`. | | `--n-ctx` | Context window size. Default: `8192`. |
| `--seed` | Random seed. Default: `-1` (random at runtime). | | `--seed` | Random seed. Default: `-1` (random at runtime). |
| `--n-gpu-layers` | Number of model layers to offload to GPU. Default: `0`. |
| `--help, -h` | Print usage and exit. | | `--help, -h` | Print usage and exit. |
`--mocked` and `--model` are mutually exclusive. Omitting both exits with an `--mocked` and `--model` are mutually exclusive. Omitting both exits with an
@@ -100,7 +123,96 @@ error before the pipeline starts. Sampling flags are ignored when `--mocked` is
set. set.
The post-build step copies `prompts/` into `build/prompts/`. Rebuild after The post-build step copies `prompts/` into `build/prompts/`. Rebuild after
editing `prompts/system.md`. editing any prompt file.
---
## Docker / RunPod
The `tooling/pipeline/runpod/` directory contains a GPU-ready container
configuration for running the pipeline on RunPod or any Docker host with an
NVIDIA GPU.
### How it works
The container uses a two-stage build. The builder stage installs CMake/Ninja,
clones the matching llama.cpp release tag for its headers only (installed into
`/usr/local/include`), and copies prebuilt shared libraries (`libllama`,
`libggml`, and CUDA/CPU backend plugins) from `ghcr.io/ggml-org/llama.cpp:full-cuda`
into `/usr/local/lib`. With both headers and libraries present, CMake's
system-library detection (see [Build](#build) above) finds them and skips the
FetchContent source build, keeping image build times short.
The runtime stage copies the compiled binary, the same prebuilt shared
libraries, and config/prompt assets into a slim CUDA runtime image. It sets
`LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH` so the dynamic linker
resolves `libllama`/`libggml` at startup, and also co-locates
`libggml-cuda.so` and the CPU backend plugins next to the binary for
`ggml_backend_load_all()`'s `dlopen` scan.
### Build the image
Run from the `tooling/pipeline/` directory (the CMake project root), not from
inside `runpod/`, so the `COPY . .` step picks up the full project context.
```bash
docker build -t biergarten-pipeline:latest -f runpod/Dockerfile .
```
To monitor the full build output and confirm CMake selects the system llama.cpp:
```bash
docker build \
--progress=plain \
--no-cache \
-t biergarten-pipeline:latest \
-f runpod/Dockerfile \
. 2>&1 | tee build.log
```
Look for `[biergarten] Found system llama.cpp — skipping FetchContent` in the
output to confirm the fast path was taken.
### Run the container
The container always runs the model-backed path; there is no `--mocked`
container mode (use a native build for that — see [Quick Start](#quick-start)).
The entrypoint, `runpod/start.sh`, downloads the GGUF model automatically if
it is not already present at the configured path.
```bash
docker run --rm \
--runtime=nvidia \
-v "$PWD/models:/workspace/models" \
-v "$PWD/output:/workspace/output" \
-v "$PWD/logs:/workspace/logs" \
biergarten-pipeline:latest
```
By default this downloads `google_gemma-4-E4B-it-Q6_K.gguf` to
`./models/` on first run if it isn't already there. To use a pre-downloaded
model, place it at that path first — see [Model](#model) above.
#### Environment variables
| Variable | Purpose |
| ------------------------- | ----------------------------------------------------------------- |
| `BIERGARTEN_MODEL_PATH` | GGUF model path. Default: `/workspace/models/google_gemma-4-E4B-it-Q6_K.gguf`. |
| `BIERGARTEN_OUTPUT_DIR` | SQLite output directory. Default: `/workspace/output`. |
| `BIERGARTEN_LOG_PATH` | Log file path. Default: `/workspace/logs/pipeline.log`. |
| `BIERGARTEN_GL_LAYERS` | GPU layers to offload (`--n-gpu-layers`). Default: `40`. |
| `BIERGARTEN_TEMPERATURE`, `BIERGARTEN_TOP_P`, `BIERGARTEN_TOP_K`, `BIERGARTEN_N_CTX`, `BIERGARTEN_SEED` | Optional sampling overrides, unset by default (binary defaults apply). |
| `BIERGARTEN_EXTRA_ARGS` | Additional raw CLI args appended verbatim. |
`--prompt-dir` is hardcoded to `/app/prompts` inside the container and is not
configurable via environment variable.
### RunPod deployment
Use a GPU pod template. Mount persistent storage for `/workspace/models`,
`/workspace/output`, and `/workspace/logs`. See
`tooling/pipeline/runpod/pod-template.yaml` for a starter template — set the
environment variables listed above to match your run.
--- ---
@@ -109,51 +221,91 @@ editing `prompts/system.md`.
### Pipeline Stages ### Pipeline Stages
| Stage | Implementation | | Stage | Implementation |
| -------- | --------------------------------------------------------------------------------------------------------------------------------------- | | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| Load | `JsonLoader::LoadLocations()` reads `locations.json` into typed `Location` records. | | Load | `ICuratedDataService` (`CuratedJsonDataService`) reads `cities.json`, `personas.json`, `forenames-by-country.json`, and `surnames-by-country.json` (paths supplied via a `CuratedDataFilePaths` DTO at construction) into typed records, caching each after its first load. `--mocked` runs use `MockCuratedDataService`'s fixed in-memory dataset instead. |
| Sample | `BiergartenDataGenerator::QueryCitiesWithCountries()` samples up to 50 locations per run. | | Sample | `BiergartenPipelineOrchestrator::QueryCitiesWithCountries()` samples `--location-count` locations per run (default `10`). |
| Enrich | `WikipediaService` fetches city and beer context. Keeps going when a lookup fails. | | Enrich | `WikipediaEnrichmentService` fetches brewing and beer-related context. Keeps going when a lookup fails. `--mocked` runs use `MockEnrichmentService` instead and skip Wikipedia entirely. |
| Generate | `MockGenerator` or `LlamaGenerator` produces brewery names and descriptions in English and the local language. | | Generate Users | `GenerateUsers()` samples a persona and a forename/surname pair per enriched city (skipping countries with no name data), then `MockGenerator` or `LlamaGenerator` produces a username, bio, and activity weight around the sampled name. |
| Store | `SqliteExportService` writes each successful brewery into a fresh dated `.sqlite` database with normalized location and brewery tables. | | Generate Breweries | `MockGenerator` or `LlamaGenerator` produces brewery names and descriptions in English and the local language. |
| Store | `SqliteExportService` writes each successful user and brewery into a fresh dated `.sqlite` database with normalized `cities`, `users`, and `breweries` tables. |
| Log | `spdlog` writes results and warnings to the console. | | Log | `spdlog` writes results and warnings to the console. |
If enrichment or generation fails for a city, that city is skipped and the If name sampling, enrichment, or generation fails for a city, that city is
pipeline continues. skipped and the pipeline continues. `GenerateUsers()` runs before
`GenerateBreweries()` in `BiergartenPipelineOrchestrator::Run()`.
### Key Components ### Key Components
- `src/main.cc` — argument parsing and Boost.DI composition root. - `src/main.cc` — argument parsing and Boost.DI composition root.
- `JsonLoader` — validates curated location input. - `CuratedJsonDataService` — implements `ICuratedDataService`; takes a
- `WikipediaService` — queries Wikipedia extracts, caches results, returns empty `CuratedDataFilePaths` DTO (locations/personas/forenames/surnames paths) in
context on failure. its constructor, then parses and validates curated location, persona, and
- `LlamaGenerator` — formats prompts for Gemma 4, validates JSON output, retries forename/surname JSON, memoizing each result after its first load on a
malformed responses up to three times. If output looks truncated, the retry given instance. Each `cities.json` entry's `postal_code.city_regex` and
raises the token budget before trying again. `postal_code.examples` are parsed into `City::postal_regex` and
- `MockGenerator` — stable hash-based output so the same city input always `City::postal_code_examples`. `MockCuratedDataService` is the in-memory
produces the same brewery. substitute (4 fixed locations, 3 personas, and name data for
- `SqliteExportService` — creates a dated SQLite file per run and persists each `US`/`DE`/`FR`/`BE`) used in `--mocked` runs, and carries matching
successful brewery into normalized tables. `postal_regex`/`postal_code_examples` values for its 4 locations.
- `WikipediaEnrichmentService` — queries Wikipedia extracts, caches results,
returns empty context on failure. `MockEnrichmentService` is the no-op
substitute used in `--mocked` runs.
- `IPostalCodeService` — generates a postal code for a `City`, consumed by
`GenerateBreweries()` and stored on `BreweryRecord::address` (an `Address`
struct, currently just `postal_code`, mirroring the web backend's
`BreweryPostLocation`). Only `MockPostalCodeService` exists today, which
ignores `postal_regex` and returns `postal_code_examples.front()` — it's
wired into the Boost.DI graph unconditionally (no `--mocked` branch yet,
since there's no real implementation to switch to). A real implementation
still needs a **xeger**-style generator — turning a `postal_regex` pattern
into a random matching string — instead of always replaying a fixed
example; see [ROADMAP.md §9](./ROADMAP.md#9-postal-code-generation).
Street-address generation (`Address::address_line1`) has no fixture data or
service yet and remains future work.
- `LlamaGenerator` — formats prompts for Gemma 4, validates JSON output for
both `GenerateBrewery` and `GenerateUser`, retries malformed responses up
to three times with corrective feedback in the retry prompt. The token
budget is fixed across attempts; it is not raised automatically on
truncation.
- `MockGenerator` — stable hash-based output so the same city/persona/name
input always produces the same brewery or user.
- `SqliteExportService` — creates a dated SQLite file per run and persists
each successful user and brewery into normalized tables.
- Brewery payloads include English and local-language name and description - Brewery payloads include English and local-language name and description
fields. fields. User payloads carry a sampled first/last name and gender, an
LLM-generated username/bio/activity weight, and a programmatically
generated (not LLM-authored) unique email and date of birth.
### Runtime Behaviour ### Runtime Behaviour
`WikipediaService` queries city, country, and beer-related Wikipedia extracts `WikipediaEnrichmentService` fetches two Wikipedia extracts per city: a
using its configured lookup, then caches the first successful response per query generic "brewing" extract and a "beer in `{country}`" extract. It does not
string. The fetched extract text is included in the prompt as context for currently query a city- or region-specific page. Each query string is cached
generation. after its first successful (or empty) lookup.
`GetLocationContext()` returns an empty string when the web client is `GetLocationContext()` returns an empty string when the web client is
unavailable or when lookup/parsing fails. unavailable or when lookup/parsing fails.
`LlamaGenerator` validates model output as structured JSON. The retry path `LlamaGenerator` validates model output as structured JSON. On validation
exists as a safety hatch for cases where the reasoning block consumes available failure it retries up to three times, replaying the previous error message in
token budget and compresses the JSON output space. All runs to date have the next prompt so the model can self-correct. All runs to date have produced
produced valid output on the first pass; the path is kept for resilience. valid output on the first pass; the retry path is kept for resilience.
`MockGenerator` uses stable hashes for repeatable output in demos and Storybook `MockGenerator` uses stable hashes for repeatable output in demos and Storybook
runs. runs.
`CuratedJsonDataService` memoizes each of `LoadCities()`, `LoadPersonas()`,
`LoadForenamesByCountry()`, and `LoadSurnamesByCountry()` independently the
first time each is called, since `BiergartenPipelineOrchestrator` owns a
single `ICuratedDataService` instance for the whole run — later calls return
the cached result instead of re-parsing.
`GenerateUsers()` samples a forename/surname pair per city via `SampleName()`,
keyed by the city's ISO 3166-1 code. Countries present in `cities.json`
but absent from either name fixture (currently `KE`, `SE`, `SG`, `TH`, `VN`,
`ZA`) are skipped, the same way a failed enrichment or generation call skips
a city — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section.
### Process Flow - Activity Diagram ### Process Flow - Activity Diagram
![An activity diagram](./diagrams/current/output/activity.svg) ![An activity diagram](./diagrams/current/output/activity.svg)
@@ -166,9 +318,10 @@ runs.
## Generated Output ## Generated Output
Each successful run stores a `GeneratedBrewery` pair with the source location Each successful run stores a `BreweryRecord` (source `City`, an `Address`,
and a `BreweryResult` payload. The same generated records are also written to a and a `BreweryResult` payload), and a `UserRecord` pair with the source
fresh SQLite export file named with the current UTC timestamp. `City` and a `UserResult` payload. The same generated records are also
written to a fresh SQLite export file named with the current UTC timestamp.
| Field | Meaning | | Field | Meaning |
| ------------------- | ------------------------------------------ | | ------------------- | ------------------------------------------ |
@@ -176,6 +329,18 @@ fresh SQLite export file named with the current UTC timestamp.
| `description_en` | Brewery description in English. | | `description_en` | Brewery description in English. |
| `name_local` | Brewery name in the local language. | | `name_local` | Brewery name in the local language. |
| `description_local` | Brewery description in the local language. | | `description_local` | Brewery description in the local language. |
| `postal_code` | Postal code generated for the brewery's city (see `IPostalCodeService`, above). |
| Field | Meaning |
| ----------------- | ----------------------------------------------------------------- |
| `first_name` | Sampled forename, copied from the curated name data (not LLM-invented). |
| `last_name` | Sampled surname, copied from the curated name data (not LLM-invented). |
| `gender` | Gender associated with the sampled forename in the source dataset. |
| `username` | LLM-generated handle. |
| `bio` | LLM-generated short biography. |
| `activity_weight` | Relative check-in/activity weight, reserved for a future J-curve activity profile. |
| `email` | Unique `@thebiergarten.app` address, generated programmatically from the sampled name. |
| `date_of_birth` | Randomized date of birth (age 19-48), generated programmatically. |
The log dump also includes city, country, state or province, ISO subdivision The log dump also includes city, country, state or province, ISO subdivision
code, latitude, and longitude for each entry. code, latitude, and longitude for each entry.
@@ -190,23 +355,25 @@ code, latitude, and longitude for each entry.
| `local_languages` | Locale-aware copy selection | | `local_languages` | Locale-aware copy selection |
| `name_en`, `description_en` | Default English display content | | `name_en`, `description_en` | Default English display content |
| `name_local`, `description_local` | Local-language display content | | `name_local`, `description_local` | Local-language display content |
| `region_context` | Richer copy for cards and detail pages | | `postal_code` | Brewery address matching, mirrors the web backend's `BreweryPostLocation.PostalCode` |
--- ---
## Tech Stack ## Tech Stack
- C++20 - C++20
- CMake 3.24+ - CMake 3.31+
- Boost.JSON, Boost.ProgramOptions, Boost.DI - Boost.JSON, Boost.ProgramOptions, Boost.DI
- spdlog - spdlog
- libcurl - cpp-httplib (with OpenSSL)
- SQLite amalgamation fetched and compiled via CMake FetchContent - SQLite amalgamation fetched and compiled via CMake FetchContent
- llama.cpp - llama.cpp (auto-detected from system install or fetched via FetchContent)
- Docker with NVIDIA CUDA 12.6 base image for GPU container builds
- RunPod for cloud GPU inference
The build fetches Boost.DI, spdlog, llama.cpp, and SQLite via CMake. Metal is The build fetches Boost.DI, spdlog, and SQLite via CMake. llama.cpp is fetched
enabled on Apple Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit only when a system installation is not detected. Metal is enabled on Apple
is present. Silicon; CUDA or HIP/ROCm is detected on Linux when the toolkit is present.
> **Code Style:** Modern C++20 throughout — RAII for ownership, > **Code Style:** Modern C++20 throughout — RAII for ownership,
> `std::unique_ptr` for injected dependencies, `std::optional` for parse > `std::unique_ptr` for injected dependencies, `std::optional` for parse
@@ -218,7 +385,7 @@ is present.
## Tested Hardware ## Tested Hardware
### ARM macOS - M1 Pro ### ARM macOS M1 Pro
| | | | | |
| --------- | --------------------------------- | | --------- | --------------------------------- |
@@ -229,7 +396,7 @@ is present.
| Model | Gemma 4 E4B | | Model | Gemma 4 E4B |
| Inference | llama.cpp with Metal | | Inference | llama.cpp with Metal |
### x86_64 Linux - NVIDIA RTX 2000 ### x86_64 Linux NVIDIA RTX 2000
| | | | | |
| --------- | ------------------------------ | | --------- | ------------------------------ |
@@ -240,50 +407,82 @@ is present.
| Model | Gemma 4 E4B | | Model | Gemma 4 E4B |
| Inference | llama.cpp with CUDA 12.x | | Inference | llama.cpp with CUDA 12.x |
### x86_64 Linux — Docker / RunPod (NVIDIA CUDA)
| | |
| --------- | ------------------------------------------- |
| Host | RunPod GPU pod |
| Base | nvidia/cuda:12.6.3-devel-ubuntu24.04 |
| Model | Gemma 4 E4B Q6_K |
| Inference | llama.cpp prebuilt CUDA backends via dlopen |
--- ---
## Fixture Strategy ## Fixture Strategy
- `--mocked` for stable fixtures, repeatable screenshots, and Storybook runs. - `--mocked` for stable fixtures, repeatable screenshots, and Storybook runs.
`MockCuratedDataService` swaps in for `CuratedJsonDataService`, so no
fixture files need to be present on disk.
- `--model` when geographically grounded content matters for demos. - `--model` when geographically grounded content matters for demos.
- Keep `locations.json` structured enough to support discovery and future - Keep `cities.json` structured enough to support discovery and future
filtering. filtering.
- Treat SQLite output as seed material for the app's brewery domain, not - `personas.json`, `forenames-by-country.json`, and
production data. `surnames-by-country.json` are curated/vendored fixture data, not
LLM-generated — see ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset
section for provenance.
- Treat SQLite output as seed material for the app's brewery and user
domains, not production data.
--- ---
## Repo Layout ## Repo Layout
| Path | Purpose | | Path | Purpose |
| ---------------------------- | -------------------------------------------------- | | -------------------------------------- | -------------------------------------------------- |
| `includes/` | Public headers and shared models. | | `tooling/pipeline/includes/` | Public headers and shared models. |
| `src/` | Implementation files. | | `tooling/pipeline/src/` | Implementation files. |
| `locations.json` | Curated city input copied into the build tree. | | `tooling/pipeline/cities.json` | Curated city input copied into the build tree. |
| `prompts/` | System prompt used by the model-backed path. | | `tooling/pipeline/personas.json` | Curated user persona archetypes copied into the build tree. |
| `diagrams/` | Architecture and pipeline diagrams. | | `tooling/pipeline/forenames-by-country.json` | Vendored (CC0) forename data by ISO 3166-1 country code. |
| `ETHICS-AND-KNOWN-ISSUES.md` | Ethics, bias, hallucination analysis, mitigations. | | `tooling/pipeline/surnames-by-country.json` | Vendored (CC0) surname data by ISO 3166-1 country code. |
| `tooling/pipeline/prompts/` | System prompts used by the model-backed path. |
| `tooling/pipeline/runpod/` | Dockerfile, launcher, and RunPod pod template. |
| `docs/pipeline/diagrams/` | Architecture and pipeline diagrams. |
| `docs/pipeline/ETHICS-AND-KNOWN-ISSUES.md` | Ethics, bias, hallucination analysis, mitigations. |
--- ---
## Code Tour ## Code Tour
Paths below are relative to `tooling/pipeline/`.
- `src/main.cc` — argument parsing and DI composition root. - `src/main.cc` — argument parsing and DI composition root.
- `src/biergarten_data_generator/` — orchestration, sampling, logging, and - `src/biergarten_pipeline_orchestrator/` — orchestration, sampling, logging,
export. and export.
- `src/services/wikipedia/` — enrichment service and cache. - `src/services/curated_data/``CuratedJsonDataService`, the file-backed
`ICuratedDataService`, and `MockCuratedDataService`, the in-memory
`ICuratedDataService` used in `--mocked` runs.
- `src/services/enrichment/wikipedia/` — enrichment service and cache.
- `includes/services/postal_code/``IPostalCodeService` and
`MockPostalCodeService` (header-only), consumed by `GenerateBreweries()`.
The real xeger-based implementation and its `--mocked`-aware DI wiring are
still to come.
- `src/services/sqlite/` — SQLite export implementation. - `src/services/sqlite/` — SQLite export implementation.
- `src/data_generation/llama/` — local inference, prompt loading, output - `src/data_generation/llama/` — local inference, prompt loading, output
validation. validation.
- `src/data_generation/mock/` — deterministic fallback. - `src/data_generation/mock/` — deterministic fallback.
- `runpod/` — container build and runtime launcher.
--- ---
## Next Steps ## Next Steps
The pipeline currently produces city-aware brewery records and dated SQLite The pipeline currently produces city-aware brewery and user records and
exports. The next passes add additional fixture types so the app can exercise dated SQLite exports. The next passes add additional fixture types so the
the full brewery domain without live data. app can exercise the full brewery and social domains without live data. For
the detailed engineering breakdown of what's needed to reach the
architecture in [`diagrams/planned/`](./diagrams/planned/), see
[ROADMAP.md](./ROADMAP.md).
### Testing — Very High Priority ### Testing — Very High Priority
@@ -305,12 +504,6 @@ Generate catalog entries with style, ABV, IBU, color, aroma notes, and food
pairing hints. Link beers back to breweries and cities. Keep style coverage wide pairing hints. Link beers back to breweries and cities. Keep style coverage wide
enough to exercise search, sort, and category filters. enough to exercise search, sort, and category filters.
### User Generation
Generate user profiles with stable names, bios, locale hints, and preference
signals. Include stable IDs for downstream fixture joins. Keep output
deterministic for screenshots while allowing larger randomized batches.
### Check-In System ### Check-In System
Produce timestamped check-in events between users and breweries. Use a J-curve Produce timestamped check-in events between users and breweries. Use a J-curve

321
docs/pipeline/ROADMAP.md Normal file
View File

@@ -0,0 +1,321 @@
# Pipeline Roadmap — Reaching the Planned Architecture
This is the engineering breakdown for closing the gap between the **current**
pipeline and the **planned** architecture documented in
[`diagrams/planned/class.puml`](./diagrams/planned/class.puml) and
[`diagrams/planned/activity.puml`](./diagrams/planned/activity.puml). Nothing
in `diagrams/planned/` is implemented yet — this file tracks what it would
take to get there. For the current implementation, see
[README.md](./README.md).
Items are grouped by layer and roughly ordered by dependency: later groups
build on types and services introduced earlier.
The only concurrency in the current pipeline is the log dispatcher thread
(`main.cc` spawns it to drain a `BoundedChannel<LogEntry>` into spdlog for the
whole run). Sampling, enrichment, generation, and export all happen
synchronously on the main thread inside `BiergartenPipelineOrchestrator::Run()`
— §6 below is what introduces the additional worker threads the planned
architecture needs.
---
## 1. Domain Models
`tooling/pipeline/includes/data_model/generated_models.h` and `models.h`.
- [ ] Add `Completeness` enum (`Full`, `Partial`, `Absent`) and a
`LocationContext` struct (`text`, `completeness`, `char_count`).
Replace `EnrichedCity::region_context` (currently a plain
`std::string`) with `context : LocationContext`.
- [ ] Add `BeerStyle` (`name`, `description`, `min_abv`, `max_abv`,
`min_ibu`, `max_ibu`).
- [ ] Add `BeerResult`, `CheckinResult`, `RatingResult` result payloads.
- [ ] Add `GenerationMetadata` (`generation_id`, `generated_time`,
`context_provided`, `generated_with`).
- [x] Add `first_name`, `last_name`, `gender`, and `activity_weight` to
`UserResult` (currently just `username`, `bio`). `first_name`/
`last_name`/`gender` are copied from the sampled `Name` (see below),
not LLM-invented.
- [x] Add `Name` (`first_name`, `last_name`, `gender`) — the sampled result
handed to `DataGenerator::GenerateUser`. Add `ForenameEntry` (`name`,
`gender`). Landed as a flatter shape than originally planned here: no
`NamesByCountry` wrapper class. `curated_data_service.h` instead
declares `ForenameList = std::vector<ForenameEntry>` and
`SurnameList = std::vector<std::string>`, and
`ICuratedDataService` exposes two maps keyed by ISO 3166-1 code —
`ForenamesByCountryMap = unordered_map<string, ForenameList>` and
`SurnamesByCountryMap = unordered_map<string, SurnameList>`
directly (see §2). Sampling is
a free function, `SampleName(forenames_by_country,
surnames_by_country, iso3166_1, rng)`, in
`generate_users.cc`'s anonymous namespace, not a method on a class.
Pairing a forename with a surname happens at sample time so gender
(present per-forename in the source data) is never discarded the way a
pre-flattened `{first_name, last_name}` list would lose it; see §2.
- [ ] Extend `GeneratedBrewery` with `brewery_id`, `context_completeness`,
`metadata` (currently `{location, address, brewery}` — see the
`Location``City` rename and `Address` struct added in the
`postal_regex`/§9 entries below).
- [ ] Add `GeneratedBeer`, `GeneratedCheckin`, `GeneratedRating`,
`GeneratedFollow` aggregate structs.
- [x] Add `UserRecord` (`location`, `user : UserResult`, `email`,
`date_of_birth`) as the current stand-in for the planned
`GeneratedUser``email` and `date_of_birth` are programmatically
generated by the orchestrator, never LLM-authored, so a downstream
auth-account seeding consumer can register real accounts from the
pipeline's SQLite export. No `password`, `user_id`, or `metadata`
field yet, matching `BreweryRecord`'s equally pre-`metadata` shape.
Already wired into `IExportService`/`SqliteExportService`: a `users`
table exists and `GenerateUsers()` exports each successful record via
`ProcessRecord(const UserRecord&)` (see §5) in addition to logging and
holding `generated_users_` in memory. Still missing vs. the planned
`GeneratedUser`: `user_id`, `password`, and `metadata`.
- [x] Add `UserPersona` (`name`, `description`, `style_affinities`).
- [x] Add `postal_regex` (`std::vector<std::string>`) and
`postal_code_examples` (`std::vector<std::string>`) to `Location`
(now `City` — see below), sourced from each `cities.json` entry's
`postal_code.city_regex` / `postal_code.examples`. Not part of the
originally planned `Location` shape in `diagrams/planned/class.puml`
added ahead of postal code generation work; see §9.
- [x] Renamed `Location``City` throughout the pipeline (struct, all
parameter/return types, `LocationsList``CityList`,
`LoadLocations()``LoadCities()`) so the domain vocabulary matches the
web backend's `City`/`CityId`. Field/variable names that merely hold a
`City` instance (e.g. `EnrichedCity::location`, `BreweryRecord::location`)
were left as-is — `City` itself has a `city` field (the city name
string), so renaming the container field too would read as
`enriched_city.city.city`. The SQLite export layer's own vocabulary
(`locations` table, `location_id` column, `location_cache_`, etc.) was
renamed to `cities`/`city_id` to match, since it's this pipeline's own
throwaway export, not the backend's SQL Server schema (see §5).
- [x] Added a new `Address` struct (`postal_code` only, for now) and a
`BreweryRecord::address` member, mirroring the web backend's
`BreweryPostLocation` (`AddressLine1`, `AddressLine2`, `PostalCode`,
`CityId`). Only `postal_code` is populated — `address_line1`/
`address_line2` generation has no fixture data or service yet. Users
do not get an `Address`: the backend's `UserAccount` has no address
fields, only `BreweryPostLocation` does.
## 2. Data Preloading
`tooling/pipeline/includes/services/curated_data/`, fixture files.
- [x] Extract an interface; have `CuratedJsonDataService` implement it.
Landed as `ICuratedDataService`
(`services/curated_data/curated_data_service.h`), not `DataPreloader`
`LoadCities()`, `LoadPersonas()`, `LoadForenamesByCountry()`, and
`LoadSurnamesByCountry()` are all virtual methods returning `const&`
and take no arguments. `CuratedJsonDataService`
(`services/curated_data/curated_json_data_service.h`, originally
landed as `JsonLoader` under `json_handling/` before being renamed and
moved) takes a `CuratedDataFilePaths` DTO (`locations_path`,
`personas_path`, `forenames_path`, `surnames_path`) in its
constructor rather than a path per `Load*()` call, and memoizes each
result in a private `cache` struct (`locations`, `personas`,
`forenames_by_country`, `surnames_by_country`), so a second call on
the same instance returns the cached result instead of re-parsing —
safe because `CuratedJsonDataService` outlives every call site (owned
by `BiergartenPipelineOrchestrator` via
`unique_ptr<ICuratedDataService>` for the whole run). `main.cc`'s DI
injector also gained a `MockCuratedDataService` (fixed in-memory
data: 4 locations across `US`/`DE`/`FR`/`BE`, 3 personas, and
matching forename/surname sets for those four countries), bound in
place of `CuratedJsonDataService` under `--mocked`, mirroring the
existing `MockEnrichmentService`/`MockGenerator` pattern.
- [ ] Add `LoadBeerStyles()`. `beer-styles.json` already exists in the repo
and is already copied into the Docker image
(`runpod/Dockerfile`), but no loader reads it yet, and the native
CMake build doesn't copy it into `build/` at all — `CMakeLists.txt`'s
"Runtime Assets" step only copies `cities.json` and `prompts/`.
- [x] Add `LoadPersonas()` and author `personas.json` (doesn't exist yet).
- [x] Add name-by-country loading, parsing
`tooling/pipeline/forenames-by-country.json` and
`surnames-by-country.json`. Landed as two separate methods —
`LoadForenamesByCountry()` and `LoadSurnamesByCountry()` — each
returning a flat `ForenamesByCountryMap` / `SurnamesByCountryMap`
(aliases for `unordered_map<string, ForenameList>` /
`unordered_map<string, SurnameList>`) keyed by ISO 3166-1 code, rather
than one combined `LoadNamesByCountry() : NamesByCountry` call. Both
files are vendored verbatim (unmodified,
full multinational coverage) from
`sigpwned/popular-names-by-country-dataset` (CC0) — see
ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for
provenance. Deliberately not pre-paired or filtered down to just the
countries in `cities.json`: keeping the source shape (including
per-forename gender) intact means the loader can support more
countries later, or gender-aware persona/bio generation, without
re-sourcing data.
- [x] Parse `postal_code.city_regex` and `postal_code.examples` out of each
`cities.json` entry's nested `postal_code` object in
`CuratedJsonDataService::LoadCities()`, into `City::postal_regex`
/ `City::postal_code_examples` (see §1). `MockCuratedDataService`'s
4 fixed locations carry matching values pulled from the corresponding
`cities.json` entries.
## 3. Policy / Strategy Layer
Entirely new — no `includes/policy/` (or equivalent) directory exists today.
- [ ] `ContextStrategy` interface + `BreweryContextStrategy` /
`BeerContextStrategy`. Today `WikipediaEnrichmentService::GetLocationContext`
hardcodes a generic `"brewing"` query and a `"beer in {country}"`
query directly — no per-phase strategy selection.
- [ ] `SamplingStrategy` interface + `UniformSamplingStrategy`, replacing
the inline `std::ranges::sample(...)` call in
`BiergartenPipelineOrchestrator::QueryCitiesWithCountries()`.
- [ ] `BeerSelectionStrategy` interface + `RandomBeerSelectionStrategy`, to
pick styles per brewery from the `BeerStyle` palette (depends on §1
and §2).
- [ ] `CheckinDistributionStrategy` interface + `JCurveCheckinStrategy` /
`RandomCheckinStrategy` — this is the "Check-In System" item already
called out in README.md's Next Steps, made concrete.
- [ ] `FollowGenerationStrategy` interface + `RandomFollowStrategy` /
`ActivityWeightedFollowStrategy`.
## 4. Data Generation
`tooling/pipeline/includes/data_generation/`, `src/data_generation/`.
- [ ] Extend the `DataGenerator` interface with `GenerateBeer`,
`GenerateCheckin`, `GenerateRating` (today: only `GenerateBrewery` and
`GenerateUser`).
- [x] Implement `LlamaGenerator::GenerateUser` for real, with a retry loop
mirroring `GenerateBrewery` (GBNF grammar, `ValidateUserJson`, up to 3
attempts with corrective feedback) and a new
`prompts/USER_GENERATION.md`. Changed the `DataGenerator::GenerateUser`
signature to
`(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult`,
matching what the activity diagram actually passes. `MockGenerator::GenerateUser`
updated to match the new signature too.
- [ ] Implement `MockGenerator::GenerateBeer` / `GenerateCheckin` /
`GenerateRating`.
- [ ] Add `IPromptFormatter::ExpectedArchitecture()` and
`LlamaGenerator::ValidateModelArchitecture()` so loading a GGUF that
doesn't match the configured chat template fails fast instead of
silently producing degraded output.
- [ ] Add prompt template files for beer/checkin/rating generation next to
the existing `prompts/BREWERY_GENERATION.md`.
## 5. Export Service
`tooling/pipeline/includes/services/database/`, `src/services/sqlite/`.
- [x] `IExportService::ProcessRecord(const UserRecord&)` and
`SqliteExportService`'s `users` table (see
`kCreateUsersTableSql` / `kInsertUserSql` in
`includes/services/database/sqlite_statement_helpers.h`) are
implemented — `GenerateUsers()` exports every successful user
alongside its resolved `city_id`.
- [x] `ProcessRecord(const BreweryRecord&)` now also binds
`BreweryRecord::address.postal_code` into a `postal_code` column on
`breweries` (see §9).
- [ ] Extend `IExportService` with `ProcessBeer`, `ProcessCheckin`,
`ProcessRating`, `ProcessFollow` (today `ProcessRecord(const
BreweryRecord&)` and `ProcessRecord(const UserRecord&)` exist).
- [ ] Add `beers`, `checkins`, `ratings`, `follows` tables to
`SqliteExportService::InitializeSchema()` (`cities`, `breweries`,
and `users` already exist) — see `kCreateCitiesTableSql` /
`kCreateBreweriesTableSql` / `kCreateUsersTableSql` in
`includes/services/database/sqlite_statement_helpers.h` for the
existing pattern to follow.
- [ ] Add a `brewery_cache_` alongside the existing `city_cache_`, and
move from one open transaction for the whole run to per-phase batched
commits (`BEGIN` / `COMMIT & BEGIN` on a batch-size threshold), as
shown in the planned activity diagram.
## 6. Concurrency & Orchestration
`tooling/pipeline/includes/concurrency/`,
`src/biergarten_pipeline_orchestrator/`.
- [ ] `BoundedChannel<T>` already exists and is production-tested — but
it's only wired to the single log channel today. Stand up the
per-phase producer/consumer channels (`loc_ch`, `exp_ch`, etc.) the
planned activity diagram describes, each with a dedicated LLM worker
thread and SQLite worker thread.
- [ ] Rewrite `BiergartenPipelineOrchestrator::Run()` from one synchronous
pass over `generated_breweries_` into phased methods —
`RunUserPhase``RunBreweryAndBeerPhase` (with a `RunBeerPhase`
sub-step once `brewery_pool_` is populated) → `RunCheckinPhase` /
`RunFollowPhase` (these two can run in parallel) → `RunRatingPhase`
backed by `user_pool_`, `brewery_pool_`, `beer_pool_`,
`checkin_pool_`, `follow_pool_`.
- [ ] Honor the structural-concurrency requirement already called out in a
comment on `BiergartenPipelineOrchestrator::Run()` in
`biergarten_pipeline_orchestrator.h`: once real worker threads exist,
they must be structurally joined (e.g. via `std::jthread`) before
`Run()` returns, so no worker logs to a closed channel during
teardown.
## 7. Enrichment
- [ ] Decide whether to restore the city/region-specific Wikipedia query
that's currently commented out in
`src/services/enrichment/wikipedia/get_summary.cc`
(`GetLocationContext`). The `ContextStrategy` work in §3 is a natural
place to reintroduce it via `BreweryContextStrategy::QueriesFor()`.
- [ ] Pre-warm caches at startup (`PreWarmBeerStyleCache`,
`PreWarmLocationCache` in the planned activity diagram) instead of
fetching lazily per record, so the streaming phases never block on a
cold cache mid-run.
## 8. Fixtures / Build
- [x] Author `personas.json`, `forenames-by-country.json`, and
`surnames-by-country.json` (see §2).
- [ ] Fix the `beer-styles.json` build-tree gap: add it to the "Runtime
Assets" `configure_file` step in `CMakeLists.txt` so native builds and
the Docker image agree on what's available at runtime.
## 9. Postal Code Generation
`tooling/pipeline/includes/services/postal_code/`. Not part of the original
`diagrams/planned/class.puml` sketch — added once `Location` (now `City`)
gained `postal_regex`/`postal_code_examples` (§1, §2).
- [x] Add the `IPostalCodeService` interface —
`GeneratePostalCode(location : const City&) : std::string`.
- [x] Add `MockPostalCodeService`: ignores `location.postal_regex` entirely
and returns `location.postal_code_examples.front()`.
- [ ] **Implement a real, xeger-based `IPostalCodeService`.** A xeger
generator turns a regex pattern (here, one of
`City::postal_regex`) into a random string that matches it — the
inverse of matching a string against a regex. Without it, every
generated postal code is either a hardcoded mock value or absent
entirely; a real implementation is required before postal codes can
vary per generated brewery instead of always reusing the same
curated example.
- [x] Wire `IPostalCodeService` into `main.cc`'s Boost.DI injector — bound
unconditionally to `MockPostalCodeService` (no `--mocked` branch yet,
since a real implementation doesn't exist to switch to; add one once
the xeger-based generator above lands, mirroring `IEnrichmentService`)
— and into `BiergartenPipelineOrchestrator::GenerateBreweries()`,
which calls `GeneratePostalCode(location)` per brewery and stores the
result on `BreweryRecord::address` (a new `Address` struct — see §1),
exported to the `breweries.postal_code` SQLite column (§5).
Postal-code generation is per-brewery, not per-city or per-user: users
don't have an address concept in the web backend, only breweries do
(`BreweryPostLocation`), so `UserRecord` was deliberately not extended.
---
## Suggested build order
The planned activity diagram's own phase notes ("brewery*pool* is now fully
populated. Phase 1b may begin.", etc.) imply a dependency order. Roughly:
1. Domain models (§1) and data preloading (§2) — nothing else compiles
without these.
2. Export schema (§5) — so every later generation phase has somewhere to
land its output.
3. Policy/strategy layer (§3) and generator interface (§4).
4. Concurrency/orchestration rewrite (§6), which is the only piece that
actually wires §1§5 into the phased, parallel pipeline shown in the
planned diagrams.
5. Enrichment cache pre-warming and the city/region query decision (§7) —
useful at any point, but most valuable once phases run concurrently.
6. Postal code generation (§9) is independent of the phases above — the
xeger-based generator just needs `City::postal_regex`, already in
place. Wiring the mock into brewery output has already landed (§9); the
real generator can be swapped in once built.

View File

@@ -18,23 +18,40 @@ skinparam ActivityBarColor #628A5B
skinparam SwimlaneBorderColor #547461 skinparam SwimlaneBorderColor #547461
skinparam SwimlaneBorderThickness 0.3 skinparam SwimlaneBorderThickness 0.3
title The Biergarten Data Pipeline (Streaming Architecture) title The Biergarten Data Pipeline (Current — Synchronous Data Path)
|#F2F6F0|main.cc| |#F2F6F0|main.cc|
start start
:Create BoundedChannel<LogEntry> log_channel;
:Spawn log dispatcher thread\n(LogDispatcher::Run());
note right
The only concurrency in the current pipeline:
a dedicated thread drains log_channel and
forwards entries to spdlog for the entire run.
Joined during shutdown after log_channel closes.
end note
:ParseArguments(argc, argv); :ParseArguments(argc, argv);
if (Are arguments valid?) then (no) if (Are arguments valid?) then (no)
:spdlog::error usage info; :Log error / usage info;
stop stop
else (yes) else (yes)
endif endif
:Init CurlGlobalState & LlamaBackendState; :Init OpenSSL global state & LlamaBackendState;
:di::make_injector(...); :di::make_injector(...);
:injector.create<std::unique_ptr<BiergartenDataGenerator>>(); note right
:BiergartenDataGenerator::Run(); Binds ILogger, IEnrichmentService, DataGenerator,
IExportService, IPromptFormatter, WebClient based
on --mocked vs. model-backed mode.
end note
:injector.create<std::unique_ptr<BiergartenPipelineOrchestrator>>();
:BiergartenPipelineOrchestrator::Run();
note right
Run() itself is synchronous — sampling, enrichment,
generation, and export all happen on this thread.
end note
|#EAF0E8|BiergartenDataGenerator| |#EAF0E8|BiergartenPipelineOrchestrator|
:Initialize SQLite export; :Initialize SQLite export;
|#E0EAE0|SqliteExportService| |#E0EAE0|SqliteExportService|
@@ -48,23 +65,106 @@ note right
Begins Transaction Begins Transaction
end note end note
|#EAF0E8|BiergartenDataGenerator| |#EAF0E8|BiergartenPipelineOrchestrator|
:QueryCitiesWithCountries(); :QueryCitiesWithCountries();
|#E2EBDC|JsonLoader| |#E2EBDC|JsonLoader|
:JsonLoader::LoadLocations("locations.json"); :JsonLoader::LoadCities("cities.json");
:std::ranges::sample(all_locations, 50); :std::ranges::sample(all_locations, --location-count);
note right
--location-count defaults to 10.
end note
|#EAF0E8|BiergartenDataGenerator| |#EAF0E8|BiergartenPipelineOrchestrator|
while (For each sampled Location?) is (Remaining cities) while (For each sampled City?) is (Remaining cities)
|#DCE8D8|WikipediaService| |#DCE8D8|IEnrichmentService|
:GetLocationContext(loc); :GetLocationContext(loc);
:FetchExtracts(City, Country, Beer); note right
|#EAF0E8|BiergartenDataGenerator| WikipediaEnrichmentService fetches a generic
:Store EnrichedCity{Location, region_context}; "brewing" extract and a "beer in {country}" extract
(not a city/region-specific page). MockEnrichmentService
(--mocked) returns an empty string instead.
end note
|#EAF0E8|BiergartenPipelineOrchestrator|
if (Lookup failed?) then (yes)
:Log warning, skip city;
else (no)
:Store EnrichedCity{City, region_context};
endif
endwhile (Done) endwhile (Done)
|#EAF0E8|BiergartenDataGenerator| |#EAF0E8|BiergartenPipelineOrchestrator|
:GenerateUsers(enriched_cities);
|#E2EBDC|JsonLoader|
:LoadPersonas("personas.json");
:LoadForenamesByCountry("forenames-by-country.json");
:LoadSurnamesByCountry("surnames-by-country.json");
note right
Each call is cached after its first parse.
MockCuratedDataService (--mocked) returns a
fixed in-memory dataset instead and skips
these three JSON files entirely.
end note
|#EAF0E8|BiergartenPipelineOrchestrator|
while (For each EnrichedCity?) is (Remaining cities)
:SampleName(forenames_by_country,\n surnames_by_country, location.iso3166_1);
if (Names available for country?) then (no)
:Log warning, skip city;
else (yes)
:Sample UserPersona (uniform random);
|#E5EDE1|DataGenerator|
if (Generator Mode) then (MockGenerator)
:DeterministicHash & Format;
else (LlamaGenerator)
:prompt_directory_->Load("USER_GENERATION");
note right
Resolves to USER_GENERATION.md inside --prompt-dir.
end note
repeat
:Infer(system_prompt, user_prompt, max_tokens, kUserJsonGrammar);
:ValidateUserJson(raw, user);
if (Is JSON Valid?) then (yes)
break
else (no)
:Attempt++;
:Append validation error to retry prompt;
endif
repeat while (Attempt < 3?) is (yes)
endif
|#EAF0E8|BiergartenPipelineOrchestrator|
if (Generation successful?) then (yes)
:BuildEmail(name, used_email_local_parts);
:GenerateDateOfBirth(rng);
note right
email and date_of_birth are generated
programmatically, never LLM-authored.
end note
|#E0EAE0|SqliteExportService|
:ProcessRecord(UserRecord);
if (City in cache?) then (yes)
:Reuse city_id;
else (no)
:Insert City & Cache ID;
endif
:Insert User (FK: city_id);
if (Exception caught during insert?) then (yes)
|#EAF0E8|BiergartenPipelineOrchestrator|
:Log warning "SQLite export failed";
else (no)
endif
else (no)
:Log warning "Generation failed, skipping...";
endif
endif
endwhile (Done)
|#EAF0E8|BiergartenPipelineOrchestrator|
:GenerateBreweries(enriched_cities); :GenerateBreweries(enriched_cities);
|#E5EDE1|DataGenerator| |#E5EDE1|DataGenerator|
@@ -73,7 +173,10 @@ while (For each EnrichedCity?) is (Remaining cities)
:DeterministicHash & Format; :DeterministicHash & Format;
else (LlamaGenerator) else (LlamaGenerator)
:PrepareRegionContext; :PrepareRegionContext;
:LoadBrewerySystemPrompt("prompts/system.md"); :prompt_directory_->Load("BREWERY_GENERATION");
note right
Resolves to BREWERY_GENERATION.md inside --prompt-dir.
end note
repeat repeat
:Infer(system_prompt, user_prompt, max_tokens, kBreweryJsonGrammar); :Infer(system_prompt, user_prompt, max_tokens, kBreweryJsonGrammar);
:ValidateBreweryJson(raw, brewery); :ValidateBreweryJson(raw, brewery);
@@ -81,24 +184,29 @@ while (For each EnrichedCity?) is (Remaining cities)
break break
else (no) else (no)
:Attempt++; :Attempt++;
:Append validation error to retry prompt;
endif endif
repeat while (Attempt < 3?) is (yes) repeat while (Attempt < 3?) is (yes)
note right
max_tokens is fixed across retries —
it is not raised on truncation.
end note
endif endif
|#EAF0E8|BiergartenDataGenerator| |#EAF0E8|BiergartenPipelineOrchestrator|
if (Generation successful?) then (yes) if (Generation successful?) then (yes)
|#E0EAE0|SqliteExportService| |#E0EAE0|SqliteExportService|
:ProcessRecord(GeneratedBrewery); :ProcessRecord(GeneratedBrewery);
if (Location in cache?) then (yes) if (City in cache?) then (yes)
:Reuse location_id; :Reuse city_id;
else (no) else (no)
:Insert Location & Cache ID; :Insert City & Cache ID;
endif endif
:Insert Brewery (FK: location_id); :Insert Brewery (FK: city_id);
if (Exception caught during insert?) then (yes) if (Exception caught during insert?) then (yes)
|#EAF0E8|BiergartenDataGenerator| |#EAF0E8|BiergartenPipelineOrchestrator|
:spdlog::warn "Failed to stream record to SQLite export"; :Log warning "SQLite export failed";
note right note right
Data loss is prevented per-record. Data loss is prevented per-record.
The pipeline continues running. The pipeline continues running.
@@ -106,7 +214,7 @@ while (For each EnrichedCity?) is (Remaining cities)
else (no) else (no)
endif endif
else (no) else (no)
:spdlog::warn "Generation failed, skipping..."; :Log warning "Generation failed, skipping...";
endif endif
|#E5EDE1|DataGenerator| |#E5EDE1|DataGenerator|
endwhile (Done) endwhile (Done)
@@ -119,6 +227,8 @@ note right
end note end note
|#F2F6F0|main.cc| |#F2F6F0|main.cc|
:Close log_channel;
:Join log dispatcher thread;
:Return 0; :Return 0;
stop stop

View File

@@ -25,25 +25,83 @@ skinparam note {
title The Biergarten Data Pipeline - Class Diagram title The Biergarten Data Pipeline - Class Diagram
class BiergartenDataGenerator { class BiergartenPipelineOrchestrator {
- logger_ : std::shared_ptr<ILogger>
- context_service_ : std::unique_ptr<IEnrichmentService> - context_service_ : std::unique_ptr<IEnrichmentService>
- generator_ : std::unique_ptr<DataGenerator> - generator_ : std::unique_ptr<DataGenerator>
- exporter_ : std::unique_ptr<IExportService> - exporter_ : std::unique_ptr<IExportService>
- generated_breweries_ : std::vector<GeneratedBrewery> - curated_data_service_ : std::unique_ptr<ICuratedDataService>
- postal_code_service_ : std::unique_ptr<IPostalCodeService>
- application_options_ : ApplicationOptions
- generated_breweries_ : std::vector<BreweryRecord>
- generated_users_ : std::vector<UserRecord>
+ Run() : bool + Run() : bool
- QueryCitiesWithCountries() : std::vector<Location> - QueryCitiesWithCountries() : std::vector<City>
- GenerateBreweries(cities : std::span<const EnrichedCity>) : void - GenerateBreweries(cities : std::span<const EnrichedCity>) : void
- GenerateUsers(cities : std::span<const EnrichedCity>) : void
- LogResults() : void - LogResults() : void
} }
interface IEnrichmentService <<interface>> { class LogLevel <<enumeration>> {
+ GetLocationContext(loc : const Location&) : std::string Debug
Info
Warn
Error
} }
class WikipediaService { class PipelinePhase <<enumeration>> {
Startup
UserGeneration
BreweryAndBeerGeneration
CheckinGeneration
RatingGeneration
FollowGeneration
Teardown
}
struct LogDTO {
+ level : LogLevel
+ phase : PipelinePhase
+ message : std::string
}
struct LogEntry {
+ timestamp : std::chrono::system_clock::time_point
+ origin : std::source_location
+ thread_id : std::thread::id
+ level : LogLevel
+ phase : PipelinePhase
+ message : std::string
}
interface ILogger <<interface>> {
+ Log(payload : LogDTO) : void
- {abstract} DoLog(entry : LogEntry) : void
}
class LogProducer {
- channel_ : BoundedChannel<LogEntry>&
- DoLog(entry : LogEntry) : void
}
class LogDispatcher {
- channel_ : BoundedChannel<LogEntry>&
+ Run() : void
- ToSpdlogLevel(level) : spdlog::level::level_enum
}
interface IEnrichmentService <<interface>> {
+ GetLocationContext(loc : const City&) : std::string
}
class MockEnrichmentService {
+ GetLocationContext(loc : const City&) : std::string
}
class WikipediaEnrichmentService {
- client_ : std::unique_ptr<WebClient> - client_ : std::unique_ptr<WebClient>
- extract_cache_ : std::unordered_map<std::string, std::string> - extract_cache_ : std::unordered_map<std::string, std::string>
+ GetLocationContext(loc : const Location&) : std::string + GetLocationContext(loc : const City&) : std::string
- FetchExtract(query : std::string_view) : std::string - FetchExtract(query : std::string_view) : std::string
} }
@@ -52,33 +110,34 @@ interface WebClient <<interface>> {
+ UrlEncode(value : const std::string&) : std::string + UrlEncode(value : const std::string&) : std::string
} }
class CURLWebClient { class HttpWebClient {
+ Get(url : const std::string&) : std::string + Get(url : const std::string&) : std::string
+ UrlEncode(value : const std::string&) : std::string + UrlEncode(value : const std::string&) : std::string
} }
interface DataGenerator <<interface>> { interface DataGenerator <<interface>> {
+ GenerateBrewery(location : const Location&, region_context : const std::string&) : BreweryResult + GenerateBrewery(location : const City&, region_context : const std::string&) : BreweryResult
+ GenerateUser(locale : const std::string&) : UserResult + GenerateUser(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult
} }
class MockGenerator { class MockGenerator {
+ GenerateBrewery(...) : BreweryResult + GenerateBrewery(location : const City&, region_context : const std::string&) : BreweryResult
+ GenerateUser(...) : UserResult + GenerateUser(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult
- DeterministicHash(location : const Location&) : size_t - DeterministicHash(location : const City&) : size_t
- DeterministicHash(location : const City&, persona : const UserPersona&, name : const Name&) : size_t
} }
class LlamaGenerator { class LlamaGenerator {
- model_ : ModelHandle - model_ : ModelHandle
- context_ : ContextHandle - context_ : ContextHandle
- prompt_formatter_ : std::unique_ptr<IPromptFormatter> - prompt_formatter_ : std::unique_ptr<IPromptFormatter>
- prompt_directory_ : std::unique_ptr<IPromptDirectory>
- rng_ : std::mt19937 - rng_ : std::mt19937
+ GenerateBrewery(...) : BreweryResult + GenerateBrewery(...) : BreweryResult
+ GenerateUser(...) : UserResult + GenerateUser(...) : UserResult
- Load(model_path : const std::string&) : void - Load(model_path : const std::string&) : void
- Infer(...) : std::string - Infer(...) : std::string
- InferFormatted(...) : std::string - InferFormatted(...) : std::string
- LoadBrewerySystemPrompt(...) : std::string
} }
interface IPromptFormatter <<interface>> { interface IPromptFormatter <<interface>> {
@@ -89,13 +148,80 @@ class Gemma4JinjaPromptFormatter {
+ Format(system_prompt : std::string_view, user_prompt : std::string_view) : std::string + Format(system_prompt : std::string_view, user_prompt : std::string_view) : std::string
} }
class JsonLoader { interface IPromptDirectory <<interface>> {
+ {static} LoadLocations(filepath : const std::filesystem::path&) : std::vector<Location> + Load(key : std::string_view) : std::string
} }
class PromptDirectory {
- prompt_dir_ : std::filesystem::path
- cache_ : std::unordered_map<std::string, std::string>
+ Load(key : std::string_view) : std::string
}
interface ICuratedDataService <<interface>> {
+ LoadCities(filepath : const std::filesystem::path&) : const std::vector<City>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
}
class JsonLoader {
- cache_ : cache
+ LoadCities(filepath : const std::filesystem::path&) : const std::vector<City>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
}
note right of JsonLoader
Each Load* method memoizes its result in
cache_ on first call; later calls on the
same instance return the cached value
without re-parsing.
end note
class MockCuratedDataService {
- locations_ : std::vector<City>
- personas_ : std::vector<UserPersona>
- forenames_by_country_ : std::unordered_map<std::string, forename_list>
- surnames_by_country_ : std::unordered_map<std::string, surname_list>
+ LoadCities(filepath : const std::filesystem::path&) : const std::vector<City>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
}
note right of MockCuratedDataService
Fixed 4-location / 3-persona dataset
(US, DE, FR, BE) used in --mocked runs;
filepath arguments are ignored.
end note
interface IPostalCodeService <<interface>> {
+ GeneratePostalCode(location : const City&) : std::string
}
class MockPostalCodeService {
+ GeneratePostalCode(location : const City&) : std::string
}
note right of MockPostalCodeService
Ignores location.postal_regex and returns
location.postal_code_examples.front(). Bound
unconditionally in main.cc's DI graph (no
--mocked branch yet -- no real implementation
exists to switch to) and called from
GenerateBreweries() per brewery, stored on
BreweryRecord::address. A real implementation
needs a xeger-style generator to turn
postal_regex into a synthesized matching
string. See ROADMAP.md §9.
end note
interface IExportService <<interface>> { interface IExportService <<interface>> {
+ Initialize() : void + Initialize() : void
+ ProcessRecord(brewery : const GeneratedBrewery&) : void + ProcessRecord(brewery : const BreweryRecord&) : uint64_t
+ ProcessRecord(user : const UserRecord&) : uint64_t
+ Finalize() : void + Finalize() : void
} }
@@ -103,15 +229,18 @@ class SqliteExportService {
- date_time_provider_ : std::unique_ptr<IDateTimeProvider> - date_time_provider_ : std::unique_ptr<IDateTimeProvider>
- run_timestamp_utc_ : std::string - run_timestamp_utc_ : std::string
- database_path_ : std::filesystem::path - database_path_ : std::filesystem::path
- db_handle_ : sqlite3* - db_handle_ : SqliteDatabaseHandle
- insert_location_stmt_ : sqlite3_stmt* - insert_city_stmt_ : SqliteStatementHandle
- insert_brewery_stmt_ : sqlite3_stmt* - insert_brewery_stmt_ : SqliteStatementHandle
- insert_user_stmt_ : SqliteStatementHandle
- transaction_open_ : bool - transaction_open_ : bool
- location_cache_ : std::unordered_map<std::string, sqlite3_int64> - city_cache_ : std::unordered_map<std::string, sqlite3_int64>
+ Initialize() : void + Initialize() : void
+ ProcessRecord(brewery : const GeneratedBrewery&) : void + ProcessRecord(brewery : const BreweryRecord&) : uint64_t
+ ProcessRecord(user : const UserRecord&) : uint64_t
+ Finalize() : void + Finalize() : void
- InitializeSchema() : void - InitializeSchema() : void
- ResolveCityId(location : const City&) : sqlite3_int64
} }
interface IDateTimeProvider <<interface>> { interface IDateTimeProvider <<interface>> {
@@ -123,23 +252,40 @@ class SystemDateTimeProvider {
} }
' Structural Relationships / Dependency Injection ' Structural Relationships / Dependency Injection
BiergartenDataGenerator *-- IEnrichmentService : owns BiergartenPipelineOrchestrator *-- ILogger : owns
BiergartenDataGenerator *-- DataGenerator : owns BiergartenPipelineOrchestrator *-- IEnrichmentService : owns
BiergartenDataGenerator *-- IExportService : owns BiergartenPipelineOrchestrator *-- DataGenerator : owns
BiergartenPipelineOrchestrator *-- IExportService : owns
BiergartenPipelineOrchestrator *-- ICuratedDataService : owns
BiergartenPipelineOrchestrator *-- IPostalCodeService : owns
IEnrichmentService <|.. WikipediaService : implements LogEntry *-- LogLevel
WikipediaService *-- WebClient : owns LogEntry *-- PipelinePhase
LogDTO *-- LogLevel
LogDTO *-- PipelinePhase
ILogger <|.. LogProducer : implements
LogProducer ..> LogEntry : emits
LogDispatcher ..> LogEntry : consumes
WebClient <|.. CURLWebClient : implements IEnrichmentService <|.. WikipediaEnrichmentService : implements
IEnrichmentService <|.. MockEnrichmentService : implements
WikipediaEnrichmentService *-- WebClient : owns
WebClient <|.. HttpWebClient : implements
DataGenerator <|.. MockGenerator : implements DataGenerator <|.. MockGenerator : implements
DataGenerator <|.. LlamaGenerator : implements DataGenerator <|.. LlamaGenerator : implements
LlamaGenerator *-- IPromptFormatter : uses LlamaGenerator *-- IPromptFormatter : uses
LlamaGenerator *-- IPromptDirectory : uses
IPromptFormatter <|.. Gemma4JinjaPromptFormatter : implements IPromptFormatter <|.. Gemma4JinjaPromptFormatter : implements
IPromptDirectory <|.. PromptDirectory : implements
BiergartenDataGenerator ..> JsonLoader : uses ICuratedDataService <|.. JsonLoader : implements
ICuratedDataService <|.. MockCuratedDataService : implements
IPostalCodeService <|.. MockPostalCodeService : implements
IExportService <|.. SqliteExportService : implements IExportService <|.. SqliteExportService : implements
SqliteExportService *-- IDateTimeProvider : owns SqliteExportService *-- IDateTimeProvider : owns

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 44 KiB

After

Width:  |  Height:  |  Size: 86 KiB

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 44 KiB

After

Width:  |  Height:  |  Size: 94 KiB

View File

@@ -13,7 +13,7 @@ if (Invalid args?) then (yes)
stop stop
else (no) else (no)
endif endif
:Init CurlGlobalState & LlamaBackendState; :Init OpenSSL global state & LlamaBackendState;
:Build DI injector; :Build DI injector;
:Initialize SqliteExportService; :Initialize SqliteExportService;
@@ -39,14 +39,22 @@ fork
:JsonLoader::LoadBeerStyles("beer-styles.json"); :JsonLoader::LoadBeerStyles("beer-styles.json");
:EnrichmentService::PreWarmBeerStyleCache(beer_styles); :EnrichmentService::PreWarmBeerStyleCache(beer_styles);
fork again fork again
:JsonLoader::LoadLocations("locations.json"); :JsonLoader::LoadCities("cities.json");
:EnrichmentService::PreWarmLocationCache(sampled_locations); :EnrichmentService::PreWarmLocationCache(sampled_locations);
end fork end fork
fork fork
:JsonLoader::LoadNamesByCountry("names-by-country.json"); :JsonLoader::LoadForenamesByCountry(\n "forenames-by-country.json");
fork again
:JsonLoader::LoadSurnamesByCountry(\n "surnames-by-country.json");
fork again fork again
:JsonLoader::LoadPersonas("personas.json"); :JsonLoader::LoadPersonas("personas.json");
end fork end fork
note right
Each call is memoized after its first parse
(implemented today). MockCuratedDataService
returns a fixed in-memory dataset instead
under --mocked, skipping these JSON files.
end note
' ═══════════════════════════════════════════ ' ═══════════════════════════════════════════
' PHASE 0 — USER GENERATION ' PHASE 0 — USER GENERATION
@@ -67,7 +75,7 @@ fork
fork again fork again
|LLM Worker| |LLM Worker|
while (loc_ch has items?) is (yes) while (loc_ch has items?) is (yes)
:Receive Location; :Receive City;
:GetLocationContextFromCache(location); :GetLocationContextFromCache(location);
note right note right
@@ -82,11 +90,15 @@ fork again
ibu_preference, checkin_weight. ibu_preference, checkin_weight.
end note end note
:NamesByCountry::SampleName(\n location.iso3166_1); :SampleName(forenames_by_country,\n surnames_by_country, location.iso3166_1, rng);
note right note right
Deterministic lookup -- no LLM involved. Deterministic lookup -- no LLM involved.
Name selected from pre-keyed table Free function (implemented today in
generate_users.cc), not a class method.
Name selected from pre-keyed maps
and passed into the generation prompt. and passed into the generation prompt.
Skips the city if either map has no
entry for iso3166_1.
end note end note
:GenerateUser(enriched_city, persona, sampled_name)\nvia DataGenerator; :GenerateUser(enriched_city, persona, sampled_name)\nvia DataGenerator;
@@ -134,13 +146,13 @@ end fork
fork fork
|Orchestrator| |Orchestrator|
:Loop: Sample User from user_pool_ :Loop: Sample User from user_pool_
and pair with Location; and pair with City;
:Send BreweryTask(Location, User) -> loc_ch; :Send BreweryTask(City, User) -> loc_ch;
:Close loc_ch; :Close loc_ch;
fork again fork again
|LLM Worker| |LLM Worker|
while (loc_ch has items?) is (yes) while (loc_ch has items?) is (yes)
:Receive BreweryTask(Location, User); :Receive BreweryTask(City, User);
:GetLocationContextFromCache(task.location); :GetLocationContextFromCache(task.location);
note right note right

View File

@@ -1,4 +1,4 @@
@startuml @startuml class_diagram
' ========================================== ' ==========================================
' CONFIGURATION & STYLING ' CONFIGURATION & STYLING
@@ -8,19 +8,44 @@ skinparam classAttributeFontSize 9
skinparam defaultFontSize 25 skinparam defaultFontSize 25
skinparam titleFontSize 30 skinparam titleFontSize 30
title Biergarten Data Pipeline — Class Diagram
package "Domain: Models" { package "Domain: Models" {
class Location { class City {
+ city : std::string + city : std::string
+ state_province : std::string + state_province : std::string
+ iso3166_2 : std::string + iso3166_2 : std::string
+ country : std::string + country : std::string
+ iso3166_1 : std::string + iso3166_1 : std::string
+ postal_regex : std::vector<std::string>
+ postal_code_examples : std::vector<std::string>
+ local_languages : std::vector<std::string> + local_languages : std::vector<std::string>
+ latitude : double + latitude : double
+ longitude : double + longitude : double
} }
note right of City
postal_regex / postal_code_examples are
sourced from each cities.json entry's
postal_code.city_regex / postal_code.examples
(implemented today, ahead of the rest of this
diagram -- see ROADMAP.md §1/§2). Renamed from
Location to City to match the web backend's
City/CityId vocabulary -- see ROADMAP.md §1.
end note
class Address {
+ postal_code : std::string
}
note right of Address
Mirrors the web backend's BreweryPostLocation.
Only postal_code is populated today --
address_line1/address_line2 have no fixture
data or generator yet. See ROADMAP.md §9.
end note
class LocationContext { class LocationContext {
+ text : std::string + text : std::string
+ completeness : Completeness + completeness : Completeness
@@ -34,7 +59,7 @@ package "Domain: Models" {
} }
class EnrichedCity { class EnrichedCity {
+ location : Location + location : City
+ context : LocationContext + context : LocationContext
} }
@@ -65,11 +90,50 @@ package "Domain: Models" {
} }
class UserResult { class UserResult {
+ first_name : std::string
+ last_name : std::string
+ gender : std::string
+ username : std::string + username : std::string
+ bio : std::string + bio : std::string
+ activity_weight : float + activity_weight : float
} }
class Name {
+ first_name : std::string
+ last_name : std::string
+ gender : std::string
}
class ForenameEntry {
+ name : std::string
+ gender : std::string
}
note right of ForenameEntry
forename_list = std::unordered_set<ForenameEntry>
surname_list = std::unordered_set<std::string>
(aliases declared in curated_data_service.h).
ICuratedDataService::LoadForenamesByCountry() /
LoadSurnamesByCountry() each return a flat
std::unordered_map<std::string, forename_list>
or <std::string, surname_list> keyed by ISO
3166-1 country code -- no NamesByCountry
wrapper class. SampleName(forenames_by_country,
surnames_by_country, iso3166_1, rng) is a free
function, not a method, so it can be called by
any per-city worker without owning the maps.
Sourced from popular-names-by-country-dataset
(CC0) -- see ETHICS-AND-KNOWN-ISSUES.md. Gender
is preserved per forename so it can be threaded
through to persona/bio generation later; surnames
carry no gender in the source data. Pairing
happens at sample time, not at fixture-authoring
time, so no information from the source dataset
is discarded up front.
end note
ForenameEntry ..> Name : SampleName() produces
class CheckinResult { class CheckinResult {
+ checked_in_at : std::string + checked_in_at : std::string
+ note : std::string + note : std::string
@@ -89,7 +153,8 @@ package "Domain: Models" {
class GeneratedBrewery { class GeneratedBrewery {
+ brewery_id : uint64_t + brewery_id : uint64_t
+ location : Location + location : City
+ address : Address
+ brewery : BreweryResult + brewery : BreweryResult
+ context_completeness : LocationContext::Completeness + context_completeness : LocationContext::Completeness
+ metadata : GenerationMetadata + metadata : GenerationMetadata
@@ -98,7 +163,7 @@ package "Domain: Models" {
class GeneratedBeer { class GeneratedBeer {
+ beer_id : uint64_t + beer_id : uint64_t
+ brewery_id : uint64_t + brewery_id : uint64_t
+ location : Location + location : City
+ style : BeerStyle + style : BeerStyle
+ beer : BeerResult + beer : BeerResult
+ metadata : GenerationMetadata + metadata : GenerationMetadata
@@ -106,11 +171,22 @@ package "Domain: Models" {
class GeneratedUser { class GeneratedUser {
+ user_id : uint64_t + user_id : uint64_t
+ location : Location + location : City
+ user : UserResult + user : UserResult
+ email : std::string
+ date_of_birth : std::string
+ password : std::string
+ metadata : GenerationMetadata + metadata : GenerationMetadata
} }
note right of GeneratedUser
email, date_of_birth, and password are
programmatically generated by the orchestrator
(not LLM-authored) so a downstream auth-account
seeding consumer can register real accounts from
this export. See ROADMAP.md §1.
end note
class GeneratedCheckin { class GeneratedCheckin {
+ checkin_id : uint64_t + checkin_id : uint64_t
+ user_id : uint64_t + user_id : uint64_t
@@ -141,7 +217,7 @@ package "Domain: Models" {
LocationContext *-- Completeness LocationContext *-- Completeness
} }
@startuml
package "Domain: Application Configuration" { package "Domain: Application Configuration" {
class SamplingOptions { class SamplingOptions {
+ temperature: float = 1.0F + temperature: float = 1.0F
@@ -167,37 +243,35 @@ package "Domain: Application Configuration" {
+ pipeline: PipelineOptions + pipeline: PipelineOptions
} }
' --- Domain Model Relationships ---
ApplicationOptions *-- GeneratorOptions ApplicationOptions *-- GeneratorOptions
ApplicationOptions *-- PipelineOptions ApplicationOptions *-- PipelineOptions
GeneratorOptions o-- SamplingOptions GeneratorOptions o-- SamplingOptions
} }
@endum
package "Domain: Policy" { package "Domain: Policy" {
interface ContextStrategy <<interface>> { interface ContextStrategy <<interface>> {
+ QueriesFor(loc : const Location&) : std::vector<std::string> + QueriesFor(loc : const City&) : std::vector<std::string>
+ MaxContextChars() : size_t + MaxContextChars() : size_t
} }
class BreweryContextStrategy { class BreweryContextStrategy {
+ QueriesFor(loc : const Location&) : std::vector<std::string> + QueriesFor(loc : const City&) : std::vector<std::string>
+ MaxContextChars() : size_t + MaxContextChars() : size_t
} }
class BeerContextStrategy { class BeerContextStrategy {
+ QueriesFor(loc : const Location&) : std::vector<std::string> + QueriesFor(loc : const City&) : std::vector<std::string>
+ MaxContextChars() : size_t + MaxContextChars() : size_t
} }
interface SamplingStrategy <<interface>> { interface SamplingStrategy <<interface>> {
+ Sample(locations : const std::vector<Location>&) : std::vector<Location> + Sample(locations : const std::vector<City>&) : std::vector<City>
} }
class UniformSamplingStrategy { class UniformSamplingStrategy {
- sample_size_ : size_t - sample_size_ : size_t
+ Sample(locations : const std::vector<Location>&) : std::vector<Location> + Sample(locations : const std::vector<City>&) : std::vector<City>
} }
interface BeerSelectionStrategy <<interface>> { interface BeerSelectionStrategy <<interface>> {
@@ -275,33 +349,29 @@ package "Infrastructure: Logging" {
+ level : LogLevel + level : LogLevel
+ phase : PipelinePhase + phase : PipelinePhase
+ message : std::string + message : std::string
+ city : std::optional<std::string>
+ entity_id : std::optional<std::string>
+ worker : std::optional<std::string> + worker : std::optional<std::string>
} }
interface Logger <<interface>> { interface ILogger <<interface>> {
+ Log(level, phase, message,\n city, entity_id, worker) : void + Log(entry : const LogEntry&) : void
} }
class PipelineLogger { class LogProducer {
- log_ch_ : BoundedChannel<LogEntry>& - channel_ : BoundedChannel<LogEntry>&
+ Log(level, phase, message,\n city, entity_id, worker) : void + Log(entry : const LogEntry&) : void
} }
class LogWorker { class LogDispatcher {
- log_ch_ : BoundedChannel<LogEntry>& - channel_ : BoundedChannel<LogEntry>&
+ Run() : void + Run() : void
- FormatTimestamp(tp) : std::string
- ToSpdlogLevel(level) : spdlog::level::level_enum - ToSpdlogLevel(level) : spdlog::level::level_enum
- ToString(phase) : std::string
} }
' --- Logging Relationships ---
LogEntry *-- LogLevel LogEntry *-- LogLevel
LogEntry *-- PipelinePhase LogEntry *-- PipelinePhase
PipelineLogger ..> LogEntry : emits ILogger <|.. LogProducer
LogWorker ..> LogEntry : consumes LogProducer ..> LogEntry : emits
LogDispatcher ..> LogEntry : consumes
} }
package "Infrastructure: Pipeline Channel" { package "Infrastructure: Pipeline Channel" {
@@ -322,32 +392,55 @@ package "Infrastructure: Pipeline Channel" {
package "Infrastructure: Data Preloading" { package "Infrastructure: Data Preloading" {
interface DataPreloader <<interface>> { interface ICuratedDataService <<interface>> {
+ LoadLocations(filepath : const std::filesystem::path&) : std::vector<Location> + LoadCities(filepath : const std::filesystem::path&) : const std::vector<City>&
+ LoadBeerStyles(filepath : const std::filesystem::path&) : std::vector<BeerStyle> + LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&
+ LoadPersonas(filepath : const std::filesystem::path&) : std::vector<Persona> + LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadNamesByCountry(filepath : const std::filesystem::path&) : NamesByCountry + LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
} }
class JsonLoader { class JsonLoader {
+ LoadLocations(filepath : const std::filesystem::path&) : std::vector<Location> + LoadCities(filepath : const std::filesystem::path&) : const std::vector<City>&
+ LoadBeerStyles(filepath : const std::filesystem::path&) : std::vector<BeerStyle> + LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&
+ LoadPersonas(filepath : const std::filesystem::path&) : std::vector<Persona> + LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadNamesByCountry(filepath : const std::filesystem::path&) : NamesByCountry + LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
} }
note right of JsonLoader
Each Load* call is memoized after its first
parse -- implemented today, ahead of the rest
of this package (LoadBeerStyles is still planned).
end note
class MockCuratedDataService {
+ LoadCities(filepath : const std::filesystem::path&) : const std::vector<City>&
+ LoadBeerStyles(filepath : const std::filesystem::path&) : const std::vector<BeerStyle>&
+ LoadPersonas(filepath : const std::filesystem::path&) : const std::vector<UserPersona>&
+ LoadForenamesByCountry(forenames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, forename_list>&
+ LoadSurnamesByCountry(surnames_filepath : const std::filesystem::path&) : const std::unordered_map<std::string, surname_list>&
}
note right of MockCuratedDataService
Fixed in-memory dataset used in --mocked mode,
implemented today for Locations/Personas/
Forenames/Surnames; would need a LoadBeerStyles
fixture too once that method is implemented.
end note
} }
package "Infrastructure: Enrichment" { package "Infrastructure: Enrichment" {
interface EnrichmentService <<interface>> { interface EnrichmentService <<interface>> {
+ GetLocationContext(loc : const Location&,\n strategy : const ContextStrategy&) : LocationContext + GetLocationContext(loc : const City&,\n strategy : const ContextStrategy&) : LocationContext
} }
class WikipediaService { class WikipediaService {
- client_ : std::unique_ptr<WebClient> - client_ : std::unique_ptr<WebClient>
- extract_cache_ : std::unordered_map<std::string, std::string> - extract_cache_ : std::unordered_map<std::string, std::string>
+ GetLocationContext(loc : const Location&,\n strategy : const ContextStrategy&) : LocationContext + GetLocationContext(loc : const City&,\n strategy : const ContextStrategy&) : LocationContext
- FetchExtract(query : std::string_view) : std::string - FetchExtract(query : std::string_view) : std::string
} }
@@ -356,19 +449,68 @@ package "Infrastructure: Enrichment" {
+ UrlEncode(value : const std::string&) : std::string + UrlEncode(value : const std::string&) : std::string
} }
class CURLWebClient { class HttpWebClient {
+ Get(url : const std::string&) : std::string + Get(url : const std::string&) : std::string
+ UrlEncode(value : const std::string&) : std::string + UrlEncode(value : const std::string&) : std::string
} }
} }
package "Infrastructure: Postal Code Generation" {
interface IPostalCodeService <<interface>> {
+ GeneratePostalCode(location : const City&) : std::string
}
class MockPostalCodeService {
+ GeneratePostalCode(location : const City&) : std::string
}
class XegerPostalCodeService {
+ GeneratePostalCode(location : const City&) : std::string
}
note right of MockPostalCodeService
Implemented today: ignores postal_regex and
returns postal_code_examples.front(). Not yet
bound in main.cc's DI graph or used by
BiergartenPipelineOrchestrator.
end note
note right of XegerPostalCodeService
Not implemented yet. Picks one of
location.postal_regex and generates a random
string matching it (a "xeger" -- the inverse of
regex matching) instead of always replaying a
fixed example. See ROADMAP.md §9.
end note
IPostalCodeService <|.. MockPostalCodeService
IPostalCodeService <|.. XegerPostalCodeService
}
package "Infrastructure: Prompting" {
interface IPromptDirectory <<interface>> {
+ Load(key : std::string_view) : std::string
}
class PromptDirectory {
- prompt_dir_ : std::filesystem::path
- cache_ : std::unordered_map<std::string, std::string>
+ PromptDirectory(prompt_dir : const std::filesystem::path&)
+ Load(key : std::string_view) : std::string
}
IPromptDirectory <|.. PromptDirectory
}
package "Infrastructure: Data Generation" { package "Infrastructure: Data Generation" {
interface DataGenerator <<interface>> { interface DataGenerator <<interface>> {
+ GenerateBrewery(location : const Location&,\n context : const LocationContext&) : BreweryResult + GenerateBrewery(location : const City&,\n context : const LocationContext&) : BreweryResult
+ GenerateBeer(brewery_id : uint64_t,\n location : const Location&,\n context : const LocationContext&,\n style : const BeerStyle&) : BeerResult + GenerateBeer(brewery_id : uint64_t,\n location : const City&,\n context : const LocationContext&,\n style : const BeerStyle&) : BeerResult
+ GenerateUser(location : const Location&) : UserResult + GenerateUser(city : const EnrichedCity&,\n persona : const UserPersona&,\n name : const Name&) : UserResult
+ GenerateCheckin(user : const GeneratedUser&,\n brewery : const GeneratedBrewery&,\n timestamp : const std::string&) : CheckinResult + GenerateCheckin(user : const GeneratedUser&,\n brewery : const GeneratedBrewery&,\n timestamp : const std::string&) : CheckinResult
+ GenerateRating(user : const GeneratedUser&,\n beer : const GeneratedBeer&,\n checkin_id : uint64_t) : RatingResult + GenerateRating(user : const GeneratedUser&,\n beer : const GeneratedBeer&,\n checkin_id : uint64_t) : RatingResult
} }
@@ -379,13 +521,14 @@ package "Infrastructure: Data Generation" {
+ GenerateUser(...) : UserResult + GenerateUser(...) : UserResult
+ GenerateCheckin(...) : CheckinResult + GenerateCheckin(...) : CheckinResult
+ GenerateRating(...) : RatingResult + GenerateRating(...) : RatingResult
- DeterministicHash(location : const Location&) : size_t - DeterministicHash(location : const City&) : size_t
} }
class LlamaGenerator { class LlamaGenerator {
- model_ : ModelHandle - model_ : ModelHandle
- context_ : ContextHandle - context_ : ContextHandle
- prompt_formatter_ : std::unique_ptr<PromptFormatter> - prompt_formatter_ : std::unique_ptr<PromptFormatter>
- prompt_directory_ : std::unique_ptr<IPromptDirectory>
- rng_ : std::mt19937 - rng_ : std::mt19937
+ GenerateBrewery(...) : BreweryResult + GenerateBrewery(...) : BreweryResult
+ GenerateBeer(...) : BeerResult + GenerateBeer(...) : BeerResult
@@ -459,10 +602,8 @@ package "Infrastructure: Data Export" {
} }
class BiergartenPipelineOrchestrator { class BiergartenPipelineOrchestrator {
- preloader_ : std::unique_ptr<DataPreloader> - curated_data_service_ : std::unique_ptr<ICuratedDataService>
- enrichment_service_ : std::unique_ptr<EnrichmentService> - enrichment_service_ : std::unique_ptr<EnrichmentService>
- generator_ : std::unique_ptr<DataGenerator> - generator_ : std::unique_ptr<DataGenerator>
- logger_ : std::unique_ptr<Logger> - logger_ : std::unique_ptr<Logger>
@@ -482,15 +623,15 @@ class BiergartenPipelineOrchestrator {
- follow_pool_ : std::vector<GeneratedFollow> - follow_pool_ : std::vector<GeneratedFollow>
-- --
+ Run() : bool + Run() : bool
- RunUserPhase(locations : const std::vector<Location>&) : void - RunUserPhase(locations : const std::vector<City>&) : void
- RunBreweryAndBeerPhase(locations : const std::vector<Location>&) : void - RunBreweryAndBeerPhase(locations : const std::vector<City>&) : void
- RunCheckinPhase() : void - RunCheckinPhase() : void
- RunRatingPhase() : void - RunRatingPhase() : void
- RunFollowPhase() : void - RunFollowPhase() : void
} }
' --- Orchestration Aggregations (Services & Strategies) --- ' --- Orchestration Aggregations (Services & Strategies) ---
BiergartenPipelineOrchestrator *-- DataPreloader BiergartenPipelineOrchestrator *-- ICuratedDataService
BiergartenPipelineOrchestrator *-- EnrichmentService BiergartenPipelineOrchestrator *-- EnrichmentService
BiergartenPipelineOrchestrator *-- DataGenerator BiergartenPipelineOrchestrator *-- DataGenerator
BiergartenPipelineOrchestrator *-- ExportService BiergartenPipelineOrchestrator *-- ExportService
@@ -509,7 +650,8 @@ BiergartenPipelineOrchestrator *-- "0..*" GeneratedCheckin : checkin_pool_
BiergartenPipelineOrchestrator *-- "0..*" GeneratedFollow : follow_pool_ BiergartenPipelineOrchestrator *-- "0..*" GeneratedFollow : follow_pool_
' --- Interfaces & Implementations --- ' --- Interfaces & Implementations ---
DataPreloader <|.. JsonLoader ICuratedDataService <|.. JsonLoader
ICuratedDataService <|.. MockCuratedDataService
Logger <|.. PipelineLogger Logger <|.. PipelineLogger
ContextStrategy <|.. BreweryContextStrategy ContextStrategy <|.. BreweryContextStrategy
ContextStrategy <|.. BeerContextStrategy ContextStrategy <|.. BeerContextStrategy
@@ -520,7 +662,7 @@ CheckinDistributionStrategy <|.. RandomCheckinStrategy
FollowGenerationStrategy <|.. RandomFollowStrategy FollowGenerationStrategy <|.. RandomFollowStrategy
FollowGenerationStrategy <|.. ActivityWeightedFollowStrategy FollowGenerationStrategy <|.. ActivityWeightedFollowStrategy
EnrichmentService <|.. WikipediaService EnrichmentService <|.. WikipediaService
WebClient <|.. CURLWebClient WebClient <|.. HttpWebClient
DataGenerator <|.. MockGenerator DataGenerator <|.. MockGenerator
DataGenerator <|.. LlamaGenerator DataGenerator <|.. LlamaGenerator
PromptFormatter <|.. Gemma4JinjaPromptFormatter PromptFormatter <|.. Gemma4JinjaPromptFormatter
@@ -531,6 +673,7 @@ DateTimeProvider <|.. SystemDateTimeProvider
WikipediaService *-- WebClient WikipediaService *-- WebClient
WikipediaService ..> ContextStrategy WikipediaService ..> ContextStrategy
LlamaGenerator *-- PromptFormatter LlamaGenerator *-- PromptFormatter
LlamaGenerator *-- IPromptDirectory
LlamaGenerator ..> GeneratorOptions LlamaGenerator ..> GeneratorOptions
SqliteExportService *-- DateTimeProvider SqliteExportService *-- DateTimeProvider
@@ -539,16 +682,17 @@ PipelineLogger o-- BoundedChannel : logs to
LogWorker o-- BoundedChannel : drains from LogWorker o-- BoundedChannel : drains from
' --- Domain Containment --- ' --- Domain Containment ---
EnrichedCity *-- Location EnrichedCity *-- City
EnrichedCity *-- LocationContext EnrichedCity *-- LocationContext
GeneratedBrewery *-- Location GeneratedBrewery *-- City
GeneratedBrewery *-- Address
GeneratedBrewery *-- BreweryResult GeneratedBrewery *-- BreweryResult
GeneratedBrewery *-- GenerationMetadata GeneratedBrewery *-- GenerationMetadata
GeneratedBeer *-- Location GeneratedBeer *-- City
GeneratedBeer *-- BeerStyle GeneratedBeer *-- BeerStyle
GeneratedBeer *-- BeerResult GeneratedBeer *-- BeerResult
GeneratedBeer *-- GenerationMetadata GeneratedBeer *-- GenerationMetadata
GeneratedUser *-- Location GeneratedUser *-- City
GeneratedUser *-- UserResult GeneratedUser *-- UserResult
GeneratedUser *-- GenerationMetadata GeneratedUser *-- GenerationMetadata
GeneratedCheckin *-- CheckinResult GeneratedCheckin *-- CheckinResult

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 106 KiB

After

Width:  |  Height:  |  Size: 142 KiB

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 184 KiB

After

Width:  |  Height:  |  Size: 257 KiB

File diff suppressed because one or more lines are too long

View File

@@ -4,41 +4,67 @@ skinparam backgroundColor #FFFFFF
skinparam defaultFontName Arial skinparam defaultFontName Arial
skinparam packageStyle rectangle skinparam packageStyle rectangle
title The Biergarten App - Layered Architecture title The Biergarten App - Vertical Slice Architecture
package "API Layer" #E3F2FD { package "API.Core (thin host)" #E3F2FD {
[API.Core\nASP.NET Core Web API] as API [API.Core\nASP.NET Core Web API host] as API
note right of API note right of API
- Controllers (Auth, User) - Program.cs wiring only
- Swagger/OpenAPI - MediatR + AddApplicationPart
- Middleware per Features.* assembly
- Health Checks - Swagger/OpenAPI, health checks
- JWT auth middleware
- Global exception filter
end note end note
} }
package "Service Layer" #F3E5F5 { package "Feature Slices" #F3E5F5 {
[Service.Auth] as AuthSvc [Features.Auth] as AuthSlice
[Service.UserManagement] as UserSvc [Features.Breweries] as BrewerySlice
note right of AuthSvc [Features.UserManagement] as UserSlice
- Business Logic [Features.Emails] as EmailsSlice
- Validation note right of AuthSlice
- Orchestration Each slice owns its own:
- Controller (HTTP endpoints)
- Commands/Queries + Handlers
- Validators
- Repository
end note
note right of EmailsSlice
No controller - invoked only
via MediatR commands sent
from other slices
end note
}
package "Shared" #FCE4EC {
[Shared.Contracts] as SharedContracts
[Shared.Application] as SharedApp
note right of SharedApp
- ValidationBehavior
(MediatR pipeline)
- Cross-slice email commands
(SendRegistrationEmailCommand,
SendResendConfirmationEmailCommand)
end note end note
} }
package "Infrastructure Layer" #FFF3E0 { package "Infrastructure Layer" #FFF3E0 {
[Infrastructure.Repository] as Repo [Infrastructure.Sql] as Sql
[Infrastructure.Jwt] as JWT [Infrastructure.Jwt] as JWT
[Infrastructure.PasswordHashing] as PwdHash [Infrastructure.PasswordHashing] as PwdHash
[Infrastructure.Email] as Email [Infrastructure.Email] as Email
[Infrastructure.Email.Templates] as EmailTemplates
} }
package "Domain Layer" #E8F5E9 { package "Domain Layer" #E8F5E9 {
[Domain.Entities] as Domain [Domain.Entities] as Domain
[Domain.Exceptions] as DomainExceptions
note right of Domain note right of Domain
- UserAccount - UserAccount
- UserCredential - UserCredential
- UserVerification - UserVerification
- BreweryPost
end note end note
} }
@@ -48,28 +74,53 @@ database "SQL Server" {
} }
' Relationships ' Relationships
API --> AuthSvc API ..> AuthSlice : AddApplicationPart\n+ MediatR scan
API --> UserSvc API ..> BrewerySlice : AddApplicationPart\n+ MediatR scan
API ..> UserSlice : AddApplicationPart\n+ MediatR scan
API ..> EmailsSlice : MediatR scan only\n(no controller)
AuthSvc --> Repo AuthSlice --> Sql
AuthSvc --> JWT AuthSlice --> JWT
AuthSvc --> PwdHash AuthSlice --> PwdHash
AuthSvc --> Email AuthSlice --> SharedContracts
AuthSlice --> SharedApp
UserSvc --> Repo BrewerySlice --> Sql
BrewerySlice --> SharedContracts
BrewerySlice --> SharedApp
Repo --> SP UserSlice --> Sql
Repo --> Domain UserSlice --> SharedContracts
UserSlice --> SharedApp
EmailsSlice --> Email
EmailsSlice --> EmailTemplates
EmailsSlice --> SharedApp
' The one cross-slice interaction: a MediatR command contract, not a project reference
AuthSlice ..> SharedApp : sends\nSendRegistrationEmailCommand
EmailsSlice ..> SharedApp : handles\nSendRegistrationEmailCommand
Sql --> SP
Sql --> Domain
SP --> Tables SP --> Tables
AuthSvc --> Domain AuthSlice --> Domain
UserSvc --> Domain BrewerySlice --> Domain
UserSlice --> Domain
AuthSlice --> DomainExceptions
' Notes ' Notes
note left of Repo note left of Sql
SQL-first approach SQL-first approach
All queries via All queries via
stored procedures stored procedures
end note end note
note bottom of AuthSlice
No Features.* project
ever references another
Features.* project directly
end note
@enduml @enduml

View File

@@ -112,38 +112,32 @@ package "Test Environment\n(docker-compose.test.yaml)" #FFF3E0 {
end note end note
} }
node "Infrastructure.Repository.Tests\n(Unit Tests)" as RepoTests { node "unit.tests\n(Features.*.Tests, one container)" as UnitTests {
component "xUnit + DbMocker" as RepoComp component "xUnit + Moq + DbMocker" as UnitComp
note right note right
Tests: Builds Dockerfile.tests, which
- AuthRepository restores/builds against Core.slnx
- UserAccountRepository then runs `dotnet test` for every
- SQL command building Features/*.Tests project in turn:
- Features.Auth.Tests
- Features.Breweries.Tests
- Features.UserManagement.Tests
- Features.Emails.Tests
Uses: Mock connections Uses: Moq for handlers,
DbMocker for repositories
No real database needed No real database needed
end note end note
} }
node "Service.Auth.Tests\n(Unit Tests)" as SvcTests {
component "xUnit + Moq" as SvcComp
note right
Tests:
- RegisterService
- LoginService
- Token generation
Uses: Mocked dependencies
No database or infrastructure
end note
}
} }
folder "test-results/\n(mounted volume)" as Results { folder "test-results/\n(mounted volume)" as Results {
file "api-specs/\n results.trx" as Result1 file "api-specs/\n results.trx" as Result1
file "repository-tests/\n results.trx" as Result2 file "Features.Auth.Tests.trx" as Result2
file "service-auth-tests/\n results.trx" as Result3 file "Features.Breweries.Tests.trx" as Result3
file "Features.UserManagement.Tests.trx" as Result4
file "Features.Emails.Tests.trx" as Result5
note bottom note bottom
TRX format TRX format
@@ -170,17 +164,16 @@ DevAPI .up.> DevSeed : depends_on
TestMig --> TestDB : 1. Migrate TestMig --> TestDB : 1. Migrate
TestSeed --> TestDB : 2. Seed TestSeed --> TestDB : 2. Seed
Specs --> TestDB : 3. Integration test Specs --> TestDB : 3. Integration test
RepoTests ..> TestDB : Mock (no connection) UnitTests ..> TestDB : depends_on for startup\nordering only (Moq/DbMocker,\nno real connection)
SvcTests ..> TestDB : Mock (no connection)
TestMig .up.> TestDB : depends_on TestMig .up.> TestDB : depends_on
TestSeed .up.> TestMig : depends_on TestSeed .up.> TestMig : depends_on
Specs .up.> TestSeed : depends_on Specs .up.> TestSeed : depends_on
UnitTests .up.> TestSeed : depends_on
' Test results export ' Test results export
Specs --> Results : Export TRX Specs --> Results : Export TRX
RepoTests --> Results : Export TRX UnitTests --> Results : Export TRX (one .trx per slice)
SvcTests --> Results : Export TRX
' Network notes ' Network notes
note bottom of DevDB note bottom of DevDB

View File

@@ -13,7 +13,7 @@ The project uses Docker Compose to orchestrate multiple services:
- .NET API - .NET API
- Test runners - Test runners
See the [deployment diagram](diagrams/pdf/deployment.pdf) for visual See the [deployment diagram](diagrams-out/deployment.svg) for visual
representation. representation.
## Docker Compose Environments ## Docker Compose Environments
@@ -25,9 +25,10 @@ representation.
**Features**: **Features**:
- Persistent SQL Server volume - Persistent SQL Server volume
- Hot reload support - NuGet package cache volume (speeds up rebuilds)
- Swagger UI enabled - Swagger UI enabled
- Seed data included - Seed data included
- Local Mailpit SMTP server for dev email testing
- `CLEAR_DATABASE=true` (drops and recreates schema) - `CLEAR_DATABASE=true` (drops and recreates schema)
**Services**: **Services**:
@@ -37,28 +38,30 @@ sqlserver # SQL Server 2022 (port 1433)
database.migrations # DbUp migrations database.migrations # DbUp migrations
database.seed # Seed initial data database.seed # Seed initial data
api.core # Web API (ports 8080, 8081) api.core # Web API (ports 8080, 8081)
mailpit # Local dev SMTP server + UI (port 8025)
``` ```
**Start Development Environment**: **Start Development Environment**:
```bash ```bash
docker compose -f docker-compose.dev.yaml up -d docker compose --env-file web/.env.dev -f web/docker-compose.dev.yaml up -d
``` ```
**Access**: **Access**:
- API Swagger: http://localhost:8080/swagger - API Swagger: http://localhost:8080/swagger
- Health Check: http://localhost:8080/health - Health Check: http://localhost:8080/health
- Mailpit UI: http://localhost:8025
- SQL Server: localhost:1433 (sa credentials from .env.dev) - SQL Server: localhost:1433 (sa credentials from .env.dev)
**Stop Environment**: **Stop Environment**:
```bash ```bash
# Stop services (keep volumes) # Stop services (keep volumes)
docker compose -f docker-compose.dev.yaml down docker compose --env-file web/.env.dev -f web/docker-compose.dev.yaml down
# Stop and remove volumes (fresh start) # Stop and remove volumes (fresh start)
docker compose -f docker-compose.dev.yaml down -v docker compose --env-file web/.env.dev -f web/docker-compose.dev.yaml down -v
``` ```
### 2. Testing (`docker-compose.test.yaml`) ### 2. Testing (`docker-compose.test.yaml`)
@@ -80,24 +83,22 @@ sqlserver # Test database
database.migrations # Fresh schema database.migrations # Fresh schema
database.seed # Test data database.seed # Test data
api.specs # Reqnroll BDD tests api.specs # Reqnroll BDD tests
repository.tests # Repository unit tests unit.tests # All Features.*.Tests unit test projects
service.auth.tests # Service unit tests
``` ```
**Run Tests**: **Run Tests**:
```bash ```bash
# Run all tests # Run all tests
docker compose -f docker-compose.test.yaml up --abort-on-container-exit docker compose --env-file web/.env.test -f web/docker-compose.test.yaml up --abort-on-container-exit
# View results # View results
ls -la test-results/ ls -la test-results/
cat test-results/api-specs/results.trx cat test-results/api-specs/results.trx
cat test-results/repository-tests/results.trx cat test-results/Features.Auth.Tests.trx
cat test-results/service-auth-tests/results.trx
# Clean up # Clean up
docker compose -f docker-compose.test.yaml down -v docker compose --env-file web/.env.test -f web/docker-compose.test.yaml down -v
``` ```
### 3. Production (`docker-compose.prod.yaml`) ### 3. Production (`docker-compose.prod.yaml`)
@@ -124,7 +125,7 @@ api.core # Production API
**Deploy Production**: **Deploy Production**:
```bash ```bash
docker compose -f docker-compose.prod.yaml up -d docker compose --env-file web/.env.prod -f web/docker-compose.prod.yaml up -d
``` ```
## Service Dependencies ## Service Dependencies
@@ -194,13 +195,6 @@ volumes:
Test results are written to host filesystem for CI/CD integration. Test results are written to host filesystem for CI/CD integration.
**Code Volumes** (development only):
```yaml
volumes:
- ./src:/app/src # Hot reload for development
```
## Networks ## Networks
Each environment uses isolated bridge networks: Each environment uses isolated bridge networks:
@@ -223,7 +217,9 @@ environment:
DB_NAME: "${DB_NAME}" DB_NAME: "${DB_NAME}"
DB_USER: "${DB_USER}" DB_USER: "${DB_USER}"
DB_PASSWORD: "${DB_PASSWORD}" DB_PASSWORD: "${DB_PASSWORD}"
JWT_SECRET: "${JWT_SECRET}" ACCESS_TOKEN_SECRET: "${ACCESS_TOKEN_SECRET}"
REFRESH_TOKEN_SECRET: "${REFRESH_TOKEN_SECRET}"
CONFIRMATION_TOKEN_SECRET: "${CONFIRMATION_TOKEN_SECRET}"
``` ```
For complete list, see [Environment Variables](environment-variables.md). For complete list, see [Environment Variables](environment-variables.md).
@@ -234,7 +230,7 @@ For complete list, see [Environment Variables](environment-variables.md).
```bash ```bash
# Running services # Running services
docker compose -f docker-compose.dev.yaml ps docker compose --env-file web/.env.dev -f web/docker-compose.dev.yaml ps
# All containers (including stopped) # All containers (including stopped)
docker ps -a docker ps -a
@@ -244,13 +240,13 @@ docker ps -a
```bash ```bash
# All services # All services
docker compose -f docker-compose.dev.yaml logs -f docker compose --env-file web/.env.dev -f web/docker-compose.dev.yaml logs -f
# Specific service # Specific service
docker compose -f docker-compose.dev.yaml logs -f api.core docker compose --env-file web/.env.dev -f web/docker-compose.dev.yaml logs -f api.core
# Last 100 lines # Last 100 lines
docker compose -f docker-compose.dev.yaml logs --tail=100 api.core docker compose --env-file web/.env.dev -f web/docker-compose.dev.yaml logs --tail=100 api.core
``` ```
### Execute Commands in Container ### Execute Commands in Container
@@ -267,39 +263,39 @@ docker exec dev-env-sqlserver /opt/mssql-tools18/bin/sqlcmd -S localhost -U sa -
```bash ```bash
# Restart all services # Restart all services
docker compose -f docker-compose.dev.yaml restart docker compose --env-file web/.env.dev -f web/docker-compose.dev.yaml restart
# Restart specific service # Restart specific service
docker compose -f docker-compose.dev.yaml restart api.core docker compose --env-file web/.env.dev -f web/docker-compose.dev.yaml restart api.core
# Rebuild and restart # Rebuild and restart
docker compose -f docker-compose.dev.yaml up -d --build api.core docker compose --env-file web/.env.dev -f web/docker-compose.dev.yaml up -d --build api.core
``` ```
### Build Images ### Build Images
```bash ```bash
# Build all images # Build all images
docker compose -f docker-compose.dev.yaml build docker compose --env-file web/.env.dev -f web/docker-compose.dev.yaml build
# Build specific service # Build specific service
docker compose -f docker-compose.dev.yaml build api.core docker compose --env-file web/.env.dev -f web/docker-compose.dev.yaml build api.core
# Build without cache # Build without cache
docker compose -f docker-compose.dev.yaml build --no-cache docker compose --env-file web/.env.dev -f web/docker-compose.dev.yaml build --no-cache
``` ```
### Clean Up ### Clean Up
```bash ```bash
# Stop and remove containers # Stop and remove containers
docker compose -f docker-compose.dev.yaml down docker compose --env-file web/.env.dev -f web/docker-compose.dev.yaml down
# Remove containers and volumes # Remove containers and volumes
docker compose -f docker-compose.dev.yaml down -v docker compose --env-file web/.env.dev -f web/docker-compose.dev.yaml down -v
# Remove containers, volumes, and images # Remove containers, volumes, and images
docker compose -f docker-compose.dev.yaml down -v --rmi all docker compose --env-file web/.env.dev -f web/docker-compose.dev.yaml down -v --rmi all
# System-wide cleanup # System-wide cleanup
docker system prune -af --volumes docker system prune -af --volumes

View File

@@ -17,7 +17,7 @@ The application uses environment variables for:
Direct environment variable access via `Environment.GetEnvironmentVariable()`. Direct environment variable access via `Environment.GetEnvironmentVariable()`.
### Frontend (`src/Website`) ### Frontend (`web/frontend`)
The active website reads runtime values from the server environment for its auth The active website reads runtime values from the server environment for its auth
and API integration. and API integration.
@@ -130,7 +130,7 @@ ASPNETCORE_URLS=http://0.0.0.0:8080 # Binding address and port
DOTNET_RUNNING_IN_CONTAINER=true # Flag for container execution DOTNET_RUNNING_IN_CONTAINER=true # Flag for container execution
``` ```
## Frontend Variables (`src/Website`) ## Frontend Variables (`web/frontend`)
The active website does not use the old Next.js/Prisma environment model. Its The active website does not use the old Next.js/Prisma environment model. Its
core runtime variables are: core runtime variables are:
@@ -147,7 +147,7 @@ NODE_ENV=development # Standard Node runtime mode
- **Required**: Yes for local development - **Required**: Yes for local development
- **Default in code**: `http://localhost:8080` - **Default in code**: `http://localhost:8080`
- **Used by**: `src/Website/app/lib/auth.server.ts` - **Used by**: `web/frontend/app/lib/auth.server.ts`
- **Purpose**: Routes website auth actions to the .NET API - **Purpose**: Routes website auth actions to the .NET API
#### `SESSION_SECRET` #### `SESSION_SECRET`
@@ -163,15 +163,27 @@ NODE_ENV=development # Standard Node runtime mode
- **Typical values**: `development`, `production`, `test` - **Typical values**: `development`, `production`, `test`
- **Purpose**: Controls secure cookie behavior and runtime mode - **Purpose**: Controls secure cookie behavior and runtime mode
### Admin Account (Seeding) ### SMTP Configuration (Backend)
Read by `Infrastructure.Email/SmtpEmailProvider.cs` for sending confirmation
and account emails.
```bash ```bash
ADMIN_PASSWORD=SecureAdminPassword123! # Initial admin password for seeding SMTP_HOST=mailpit # Required, no default
SMTP_PORT=1025 # Optional, defaults to 587
SMTP_USERNAME=<username> # Optional, no default
SMTP_PASSWORD=<password> # Optional, no default
SMTP_USE_SSL=false # Optional, defaults to true
SMTP_FROM_EMAIL=noreply@thebiergarten.app # Required, no default
SMTP_FROM_NAME=The Biergarten App # Optional, defaults to "The Biergarten"
``` ```
- **Required**: No (only needed for seeding) - **Implementation**: `Infrastructure.Email/SmtpEmailProvider.cs` throws on
- **Purpose**: Sets admin account password during database seeding startup if `SMTP_HOST` or `SMTP_FROM_EMAIL` is missing
- **Security**: Use strong password, change immediately in production - **Local dev**: point at the `mailpit` Docker service (SMTP on port 1025, web
UI on http://localhost:8025)
- **Production**: point at a real provider (SendGrid, Mailgun, Amazon SES,
etc.)
## Docker-Specific Variables ## Docker-Specific Variables
@@ -191,34 +203,33 @@ MSSQL_PID=Express # SQL Server edition (Express, Developer, Ente
## Environment File Structure ## Environment File Structure
### Backend/Docker (Root Directory) ### Backend/Docker (`web/` Directory)
``` ```
.env.example # Template (tracked in Git) web/.env.example # Template (tracked in Git)
.env.dev # Development config (gitignored) web/.env.dev # Development config (gitignored)
.env.test # Testing config (gitignored) web/.env.test # Testing config (gitignored)
.env.prod # Production config (gitignored) web/.env.prod # Production config (gitignored)
``` ```
**Setup**: **Setup**:
```bash ```bash
cp .env.example .env.dev cp web/.env.example web/.env.dev
# Edit .env.dev with your values # Edit web/.env.dev with your values
``` ```
## Legacy Frontend Variables ## Legacy Frontend Variables
Variables for the archived Next.js frontend (`src/Website-v1`) have been removed Variables for the archived Next.js frontend (`archive/next-js-web-app/`) have
from this active reference. See been removed from this active reference, since that app is retained for
[archive/legacy-website-v1.md](archive/legacy-website-v1.md) if you need the reference only and is not run as part of the active stack.
legacy Prisma, Cloudinary, Mapbox, or SparkPost notes.
**Docker Compose Mapping**: **Docker Compose Mapping**:
- `docker-compose.dev.yaml``.env.dev` - `web/docker-compose.dev.yaml``web/.env.dev`
- `docker-compose.test.yaml``.env.test` - `web/docker-compose.test.yaml``web/.env.test`
- `docker-compose.prod.yaml``.env.prod` - `web/docker-compose.prod.yaml``web/.env.prod`
## Variable Reference Table ## Variable Reference Table
@@ -234,6 +245,13 @@ legacy Prisma, Cloudinary, Mapbox, or SparkPost notes.
| `REFRESH_TOKEN_SECRET` | ✓ | | ✓ | Yes | Refresh token signing | | `REFRESH_TOKEN_SECRET` | ✓ | | ✓ | Yes | Refresh token signing |
| `CONFIRMATION_TOKEN_SECRET` | ✓ | | ✓ | Yes | Confirmation token signing | | `CONFIRMATION_TOKEN_SECRET` | ✓ | | ✓ | Yes | Confirmation token signing |
| `WEBSITE_BASE_URL` | ✓ | | | Yes | Website URL for emails | | `WEBSITE_BASE_URL` | ✓ | | | Yes | Website URL for emails |
| `SMTP_HOST` | ✓ | | ✓ | Yes | SMTP server host |
| `SMTP_PORT` | ✓ | | ✓ | No | Defaults to `587` |
| `SMTP_USERNAME` | ✓ | | ✓ | No | SMTP auth username |
| `SMTP_PASSWORD` | ✓ | | ✓ | No | SMTP auth password |
| `SMTP_USE_SSL` | ✓ | | ✓ | No | Defaults to `true` |
| `SMTP_FROM_EMAIL` | ✓ | | ✓ | Yes | Email sender address |
| `SMTP_FROM_NAME` | ✓ | | ✓ | No | Defaults to `The Biergarten` |
| `API_BASE_URL` | | ✓ | | Yes | Website-to-API base URL | | `API_BASE_URL` | | ✓ | | Yes | Website-to-API base URL |
| `SESSION_SECRET` | | ✓ | | Yes | Website session signing | | `SESSION_SECRET` | | ✓ | | Yes | Website session signing |
| `NODE_ENV` | | ✓ | | No | Runtime mode | | `NODE_ENV` | | ✓ | | No | Runtime mode |
@@ -254,9 +272,12 @@ legacy Prisma, Cloudinary, Mapbox, or SparkPost notes.
Variables are validated at startup: Variables are validated at startup:
- Missing required variables cause application to fail - Missing required variables (`ACCESS_TOKEN_SECRET`, `REFRESH_TOKEN_SECRET`,
- JWT_SECRET length is enforced (min 32 chars) `CONFIRMATION_TOKEN_SECRET`, `SMTP_HOST`, `SMTP_FROM_EMAIL`, DB connection
- Connection string format is validated values) cause the application to fail with an `InvalidOperationException`
- No minimum length is enforced on the token secrets in code; the
"minimum 32 characters" guidance above is a recommendation, not an
enforced check
### Frontend Validation ### Frontend Validation
@@ -284,6 +305,13 @@ REFRESH_TOKEN_SECRET=<generated-with-openssl>
CONFIRMATION_TOKEN_SECRET=<generated-with-openssl> CONFIRMATION_TOKEN_SECRET=<generated-with-openssl>
WEBSITE_BASE_URL=http://localhost:3000 WEBSITE_BASE_URL=http://localhost:3000
# SMTP (Mailpit in dev)
SMTP_HOST=mailpit
SMTP_PORT=1025
SMTP_USE_SSL=false
SMTP_FROM_EMAIL=noreply@thebiergarten.app
SMTP_FROM_NAME=The Biergarten App
# Migration # Migration
CLEAR_DATABASE=true CLEAR_DATABASE=true

View File

@@ -1,7 +1,7 @@
# Getting Started # Getting Started
This guide covers local setup for the current Biergarten stack: the .NET backend This guide covers local setup for the current Biergarten stack: the .NET backend
in `src/Core` and the active React Router frontend in `src/Website`. in `web/backend` and the active React Router frontend in `web/frontend`.
## Prerequisites ## Prerequisites
@@ -22,7 +22,7 @@ cd the-biergarten-app
### 2. Configure Backend Environment Variables ### 2. Configure Backend Environment Variables
```bash ```bash
cp .env.example .env.dev cp web/.env.example web/.env.dev
``` ```
At minimum, ensure `.env.dev` includes valid database and token values: At minimum, ensure `.env.dev` includes valid database and token values:
@@ -43,20 +43,21 @@ See [Environment Variables](environment-variables.md) for the full list.
### 3. Start the Backend Stack ### 3. Start the Backend Stack
```bash ```bash
docker compose -f docker-compose.dev.yaml up -d docker compose --env-file web/.env.dev -f web/docker-compose.dev.yaml up -d
``` ```
This starts SQL Server, migrations, seeding, and the API. This starts SQL Server, migrations, seeding, the API, and Mailpit (local dev SMTP).
Available endpoints: Available endpoints:
- API Swagger: http://localhost:8080/swagger - API Swagger: http://localhost:8080/swagger
- Health Check: http://localhost:8080/health - Health Check: http://localhost:8080/health
- Mailpit UI: http://localhost:8025
### 4. Start the Active Frontend ### 4. Start the Active Frontend
```bash ```bash
cd src/Website cd web/frontend
npm install npm install
API_BASE_URL=http://localhost:8080 SESSION_SECRET=dev-secret-change-me npm run dev API_BASE_URL=http://localhost:8080 SESSION_SECRET=dev-secret-change-me npm run dev
``` ```
@@ -71,7 +72,7 @@ Required frontend runtime variables for local work:
### 5. Optional: Run Storybook ### 5. Optional: Run Storybook
```bash ```bash
cd src/Website cd web/frontend
npm run storybook npm run storybook
``` ```
@@ -82,15 +83,15 @@ Storybook runs at http://localhost:6006 by default.
### Backend ### Backend
```bash ```bash
docker compose -f docker-compose.dev.yaml logs -f docker compose --env-file web/.env.dev -f web/docker-compose.dev.yaml logs -f
docker compose -f docker-compose.dev.yaml down docker compose --env-file web/.env.dev -f web/docker-compose.dev.yaml down
docker compose -f docker-compose.dev.yaml down -v docker compose --env-file web/.env.dev -f web/docker-compose.dev.yaml down -v
``` ```
### Frontend ### Frontend
```bash ```bash
cd src/Website cd web/frontend
npm run lint npm run lint
npm run typecheck npm run typecheck
npm run format:check npm run format:check
@@ -115,7 +116,7 @@ export WEBSITE_BASE_URL="http://localhost:3000"
### 2. Run Migrations and Seed ### 2. Run Migrations and Seed
```bash ```bash
cd src/Core cd web/backend
dotnet run --project Database/Database.Migrations/Database.Migrations.csproj dotnet run --project Database/Database.Migrations/Database.Migrations.csproj
dotnet run --project Database/Database.Seed/Database.Seed.csproj dotnet run --project Database/Database.Seed/Database.Seed.csproj
``` ```
@@ -128,12 +129,11 @@ dotnet run --project API/API.Core/API.Core.csproj
## Legacy Frontend Note ## Legacy Frontend Note
The previous Next.js frontend now lives in `src/Website-v1` and is not the The previous Next.js frontend lives in `archive/next-js-web-app/` for reference
active website. Legacy setup details have been moved to only and is not the active website.
[docs/archive/legacy-website-v1.md](archive/legacy-website-v1.md).
## Next Steps ## Next Steps
- Review [Architecture](architecture.md) - Review [Architecture](../architecture.md)
- Run backend and frontend checks from [Testing](testing.md) - Run backend and frontend checks from [Testing](testing.md)
- Use [Docker Guide](docker.md) for container troubleshooting - Use [Docker Guide](docker.md) for container troubleshooting

View File

@@ -7,9 +7,13 @@ Biergarten App.
The project uses a multi-layered testing approach across backend and frontend: The project uses a multi-layered testing approach across backend and frontend:
- **API.Specs** - BDD integration tests using Reqnroll (Gherkin) - **API.Specs** - BDD integration tests using Reqnroll (Gherkin), run against
- **Infrastructure.Repository.Tests** - Unit tests for data access layer a live, seeded database
- **Service.Auth.Tests** - Unit tests for authentication business logic - **Features.\*.Tests** - One unit test project per backend feature slice
(`Features.Auth.Tests`, `Features.Breweries.Tests`,
`Features.UserManagement.Tests`, `Features.Emails.Tests`), covering that
slice's command/query handlers and its own repository (mocked with Moq /
DbMocker, no real database required)
- **Storybook Vitest project** - Browser-based interaction tests for shared - **Storybook Vitest project** - Browser-based interaction tests for shared
website stories website stories
- **Storybook Playwright suite** - Browser checks against Storybook-rendered - **Storybook Playwright suite** - Browser checks against Storybook-rendered
@@ -21,7 +25,7 @@ The easiest way to run all tests is using Docker Compose, which sets up an
isolated test environment: isolated test environment:
```bash ```bash
docker compose -f docker-compose.test.yaml up --abort-on-container-exit docker compose --env-file web/.env.test -f web/docker-compose.test.yaml up --abort-on-container-exit
``` ```
This command: This command:
@@ -41,15 +45,17 @@ ls -la test-results/
# View specific test results # View specific test results
cat test-results/api-specs/results.trx cat test-results/api-specs/results.trx
cat test-results/repository-tests/results.trx cat test-results/Features.Auth.Tests.trx
cat test-results/service-auth-tests/results.trx cat test-results/Features.Breweries.Tests.trx
cat test-results/Features.UserManagement.Tests.trx
cat test-results/Features.Emails.Tests.trx
``` ```
### Clean Up ### Clean Up
```bash ```bash
# Remove test containers and volumes # Remove test containers and volumes
docker compose -f docker-compose.test.yaml down -v docker compose --env-file web/.env.test -f web/docker-compose.test.yaml down -v
``` ```
## Running Tests Locally ## Running Tests Locally
@@ -59,7 +65,7 @@ You can run individual test projects locally without Docker:
### Integration Tests (API.Specs) ### Integration Tests (API.Specs)
```bash ```bash
cd src/Core cd web/backend
dotnet test API/API.Specs/API.Specs.csproj dotnet test API/API.Specs/API.Specs.csproj
``` ```
@@ -69,32 +75,31 @@ dotnet test API/API.Specs/API.Specs.csproj
- Database migrated and seeded - Database migrated and seeded
- Environment variables set (DB connection, JWT secret) - Environment variables set (DB connection, JWT secret)
### Repository Tests ### Feature Slice Unit Tests
Each feature slice has its own test project, covering its command/query
handlers and repository:
```bash ```bash
cd src/Core cd web/backend
dotnet test Infrastructure/Infrastructure.Repository.Tests/Infrastructure.Repository.Tests.csproj dotnet test Features/Features.Auth.Tests/Features.Auth.Tests.csproj
dotnet test Features/Features.Breweries.Tests/Features.Breweries.Tests.csproj
dotnet test Features/Features.UserManagement.Tests/Features.UserManagement.Tests.csproj
dotnet test Features/Features.Emails.Tests/Features.Emails.Tests.csproj
# Or run all of them at once via the solution:
for proj in Features/*.Tests; do dotnet test "$proj"; done
``` ```
**Requirements**: **Requirements**:
- SQL Server instance running (uses mock data) - No database required (handlers use Moq; repository tests use DbMocker to
simulate SQL Server responses)
### Service Tests
```bash
cd src/Core
dotnet test Service/Service.Auth.Tests/Service.Auth.Tests.csproj
```
**Requirements**:
- No database required (uses Moq for mocking)
### Frontend Storybook Tests ### Frontend Storybook Tests
```bash ```bash
cd src/Website cd web/frontend
npm install npm install
npm run test:storybook npm run test:storybook
``` ```
@@ -108,7 +113,7 @@ npm run test:storybook
### Frontend Playwright Storybook Tests ### Frontend Playwright Storybook Tests
```bash ```bash
cd src/Website cd web/frontend
npm install npm install
npm run test:storybook:playwright npm run test:storybook:playwright
``` ```
@@ -124,27 +129,28 @@ npm run test:storybook:playwright
### Current Coverage ### Current Coverage
**Authentication & User Management**: **Features.Auth.Tests**:
- User registration with validation - User registration with validation
- User login with JWT token generation - User login with JWT token generation
- Password hashing and verification (Argon2id) - Password hashing and verification (Argon2id)
- JWT token generation and claims - Email confirmation and confirmation-email resend
- Invalid credentials handling - Refresh token exchange
- 404 error responses - JWT token generation, validation, and claims handling
- `IAuthRepository` data access (DbMocker-backed)
- Invalid credentials and 404 error responses (via API.Specs)
**Repository Layer**: **Features.Breweries.Tests**:
- User account creation - Brewery create/update/delete commands and get-by-id/get-all queries
- User credential management
- GetUserByUsername queries
- Stored procedure execution
**Service Layer**: **Features.UserManagement.Tests**:
- Login service with password verification - User get-by-id/get-all queries and the (currently unrouted) update command
- Register service with validation
- Business logic for authentication flow **Features.Emails.Tests**:
- Registration and resend-confirmation email dispatch handlers
**Frontend UI Coverage**: **Frontend UI Coverage**:
@@ -156,10 +162,7 @@ npm run test:storybook:playwright
### Planned Coverage ### Planned Coverage
- [ ] Email verification workflow
- [ ] Password reset functionality - [ ] Password reset functionality
- [ ] Token refresh mechanism
- [ ] Brewery data management
- [ ] Beer post operations - [ ] Beer post operations
- [ ] User follow/unfollow - [ ] User follow/unfollow
- [ ] Image upload service - [ ] Image upload service
@@ -204,21 +207,28 @@ npm run test:storybook:playwright
``` ```
API.Specs/ API.Specs/
├── Features/ ├── Features/
│ ├── Authentication.feature # Login/register scenarios │ ├── Registration.feature # Registration scenarios
── UserManagement.feature # User CRUD scenarios ── Login.feature # Login scenarios
│ ├── Confirmation.feature # Email confirmation scenarios
│ ├── ResendConfirmation.feature # Resend-confirmation scenarios
│ ├── TokenRefresh.feature # Refresh token scenarios
│ ├── AccessTokenValidation.feature # Protected endpoint access scenarios
│ └── NotFound.feature # 404 handling
├── Steps/ ├── Steps/
│ ├── AuthenticationSteps.cs # Step definitions │ ├── AuthSteps.cs # Step definitions for the Auth features
│ └── UserManagementSteps.cs │ └── ApiGeneralSteps.cs # Shared/general step definitions
── Mocks/ ── Mocks/
── TestApiFactory.cs # Test server setup ── MockEmailDispatcher.cs # Substitutes Features.Emails' IEmailDispatcher
│ └── MockEmailProvider.cs # Substitutes Infrastructure.Email's IEmailProvider
└── TestApiFactory.cs # Test server setup (swaps in the mocks above)
``` ```
**Example Feature**: **Example Feature**:
```gherkin ```gherkin
Feature: User Authentication Feature: User Registration
As a user As a user
I want to register and login I want to register
So that I can access the platform So that I can access the platform
Scenario: Successful user registration Scenario: Successful user registration
@@ -228,45 +238,51 @@ Scenario: Successful user registration
And my account should be created And my account should be created
``` ```
### Infrastructure.Repository.Tests ### Features.Auth.Tests
``` ```
Infrastructure.Repository.Tests/ Features.Auth.Tests/
├── AuthRepositoryTests.cs # Auth repository tests ├── Commands/
├── UserAccountRepositoryTests.cs # User account tests │ ├── RegisterUserHandlerTests.cs
└── TestFixtures/ │ ├── ConfirmUserHandlerTests.cs
── DatabaseFixture.cs # Shared test setup ── ResendConfirmationEmailHandlerTests.cs
│ └── RefreshTokenHandlerTests.cs
├── Queries/
│ └── LoginHandlerTests.cs
├── Repository/
│ └── AuthRepositoryTests.cs # DbMocker-backed repository tests
└── Services/
├── TokenServiceRefreshTests.cs
└── TokenServiceValidationTests.cs
``` ```
### Service.Auth.Tests Each of the other three slices (`Features.Breweries.Tests`,
`Features.UserManagement.Tests`, `Features.Emails.Tests`) follows the same
``` shape: a `Commands/`/`Queries/` folder with one test file per handler.
Service.Auth.Tests/
├── LoginService.test.cs # Login business logic tests
└── RegisterService.test.cs # Registration business logic tests
```
## Writing Tests ## Writing Tests
### Unit Test Example (xUnit) ### Unit Test Example (xUnit)
```csharp ```csharp
public class LoginServiceTests public class LoginHandlerTests
{ {
[Fact] [Fact]
public async Task LoginAsync_ValidCredentials_ReturnsToken() public async Task Handle_WithValidData_ReturnsPayloadWithMatchingUsername()
{ {
// Arrange // Arrange
var mockRepo = new Mock<IAuthRepository>(); var authRepoMock = new Mock<IAuthRepository>();
var mockJwt = new Mock<IJwtService>(); var passwordInfraMock = new Mock<IPasswordInfrastructure>();
var service = new AuthService(mockRepo.Object, mockJwt.Object); var tokenServiceMock = new Mock<ITokenService>();
var handler = new LoginHandler(authRepoMock.Object, passwordInfraMock.Object, tokenServiceMock.Object);
// ...set up mocks for a known user/credential...
// Act // Act
var result = await service.LoginAsync("testuser", "password123"); var result = await handler.Handle(new LoginQuery("testuser", "password123"), CancellationToken.None);
// Assert // Assert
result.Should().NotBeNull(); result.Should().NotBeNull();
result.Token.Should().NotBeNullOrEmpty(); result.AccessToken.Should().NotBeNullOrEmpty();
} }
} }
``` ```
@@ -288,9 +304,9 @@ configuration:
```bash ```bash
# CI/CD command # CI/CD command
docker compose -f docker-compose.test.yaml build docker compose --env-file web/.env.test -f web/docker-compose.test.yaml build
docker compose -f docker-compose.test.yaml up --abort-on-container-exit docker compose --env-file web/.env.test -f web/docker-compose.test.yaml up --abort-on-container-exit
docker compose -f docker-compose.test.yaml down -v docker compose --env-file web/.env.test -f web/docker-compose.test.yaml down -v
``` ```
Exit codes: Exit codes:
@@ -302,7 +318,7 @@ Frontend UI checks should also be included in CI for the active website
workspace: workspace:
```bash ```bash
cd src/Website cd web/frontend
npm ci npm ci
npm run test:storybook npm run test:storybook
npm run test:storybook:playwright npm run test:storybook:playwright
@@ -315,7 +331,7 @@ npm run test:storybook:playwright
Ensure SQL Server is running and environment variables are set: Ensure SQL Server is running and environment variables are set:
```bash ```bash
docker compose -f docker-compose.test.yaml ps docker compose --env-file web/.env.test -f web/docker-compose.test.yaml ps
``` ```
### Port Conflicts ### Port Conflicts
@@ -328,13 +344,13 @@ If port 1433 is in use, stop other SQL Server instances or modify the port in
Clean up test database: Clean up test database:
```bash ```bash
docker compose -f docker-compose.test.yaml down -v docker compose --env-file web/.env.test -f web/docker-compose.test.yaml down -v
``` ```
### View Container Logs ### View Container Logs
```bash ```bash
docker compose -f docker-compose.test.yaml logs <service-name> docker compose --env-file web/.env.test -f web/docker-compose.test.yaml logs <service-name>
``` ```
## Best Practices ## Best Practices

View File

@@ -2,8 +2,7 @@
## Overview ## Overview
The Core project implements comprehensive JWT token validation across three The Core project validates JWTs across three token types:
token types:
- **Access Tokens**: Short-lived (1 hour) tokens for API authentication - **Access Tokens**: Short-lived (1 hour) tokens for API authentication
- **Refresh Tokens**: Long-lived (21 days) tokens for obtaining new access - **Refresh Tokens**: Long-lived (21 days) tokens for obtaining new access
@@ -15,7 +14,7 @@ token types:
### Infrastructure Layer ### Infrastructure Layer
#### [ITokenInfrastructure](Infrastructure.Jwt/ITokenInfrastructure.cs) #### [ITokenInfrastructure](../../web/backend/Infrastructure/Infrastructure.Jwt/ITokenInfrastructure.cs)
Low-level JWT operations. Low-level JWT operations.
@@ -25,77 +24,92 @@ Low-level JWT operations.
- `ValidateJwtAsync()` - Validates token signature, expiration, and format - `ValidateJwtAsync()` - Validates token signature, expiration, and format
**Implementation:** **Implementation:**
[JwtInfrastructure.cs](Infrastructure.Jwt/JwtInfrastructure.cs) [JwtInfrastructure.cs](../../web/backend/Infrastructure/Infrastructure.Jwt/JwtInfrastructure.cs)
- Uses Microsoft.IdentityModel.JsonWebTokens.JsonWebTokenHandler - Uses Microsoft.IdentityModel.JsonWebTokens.JsonWebTokenHandler
- Algorithm: HS256 (HMAC-SHA256) - Algorithm: HS256 (HMAC-SHA256)
- Validates token lifetime, signature, and well-formedness - Validates token lifetime, signature, and well-formedness
### Service Layer ### Features.Auth Slice
#### [ITokenValidationService](Service.Auth/ITokenValidationService.cs) #### [ITokenService](../../web/backend/Features/Features.Auth/Services/ITokenService.cs)
High-level token validation with context (token type, user extraction). Both token generation and validation live on the same slice-internal
service (there is no separate validation service/class).
**Methods:** **Generation methods:**
- `ValidateAccessTokenAsync(string token)` - Validates access tokens
- `ValidateRefreshTokenAsync(string token)` - Validates refresh tokens
- `ValidateConfirmationTokenAsync(string token)` - Validates confirmation tokens
**Returns:** `ValidatedToken` record containing:
- `UserId` (Guid)
- `Username` (string)
- `Principal` (ClaimsPrincipal) - Full JWT claims
**Implementation:**
[TokenValidationService.cs](Service.Auth/TokenValidationService.cs)
- Reads token secrets from environment variables
- Extracts and validates claims (Sub, UniqueName)
- Throws `UnauthorizedException` on validation failure
#### [ITokenService](Service.Auth/ITokenService.cs)
Token generation (existing service extended).
**Methods:**
- `GenerateAccessToken(UserAccount)` - Creates 1-hour access token - `GenerateAccessToken(UserAccount)` - Creates 1-hour access token
- `GenerateRefreshToken(UserAccount)` - Creates 21-day refresh token - `GenerateRefreshToken(UserAccount)` - Creates 21-day refresh token
- `GenerateConfirmationToken(UserAccount)` - Creates 30-minute confirmation - `GenerateConfirmationToken(UserAccount)` - Creates 30-minute confirmation
token token
**Validation methods:**
- `ValidateAccessTokenAsync(string token)` - Validates access tokens
- `ValidateRefreshTokenAsync(string token)` - Validates refresh tokens
- `ValidateConfirmationTokenAsync(string token)` - Validates confirmation tokens
**Returns (validation):** `ValidatedToken` record containing:
- `UserId` (Guid)
- `Username` (string)
- `Principal` (ClaimsPrincipal) - Full JWT claims
**Implementation:** [TokenService.cs](../../web/backend/Features/Features.Auth/Services/TokenService.cs)
- Reads token secrets from environment variables
- Extracts and validates claims (Sub, UniqueName)
- Throws `UnauthorizedException` on validation failure
`TokenService` is registered by `Features.Auth`'s own
`AddFeaturesAuth()` extension method, except for the lower-level
`ITokenInfrastructure` (JWT signing/verification) it depends on, which is
registered by the host (`API.Core/Program.cs`) since
`JwtAuthenticationHandler` (host-level auth middleware) also depends on
it directly.
### Integration Points ### Integration Points
#### [ConfirmationService](Service.Auth/IConfirmationService.cs) #### [ConfirmUserHandler](../../web/backend/Features/Features.Auth/Commands/ConfirmUser/ConfirmUserHandler.cs)
**Flow:** **Flow:**
1. Receives confirmation token from user 1. Receives confirmation token from user via `ConfirmUserCommand`
2. Calls `TokenValidationService.ValidateConfirmationTokenAsync()` 2. Calls `ITokenService.ValidateConfirmationTokenAsync()`
3. Extracts user ID from validated token 3. Extracts user ID from validated token
4. Calls `AuthRepository.ConfirmUserAccountAsync()` to update database 4. Calls `IAuthRepository.ConfirmUserAccountAsync()` to update database
5. Returns confirmation result 5. Returns confirmation result
#### [RefreshTokenService](Service.Auth/RefreshTokenService.cs) #### [ResendConfirmationEmailHandler](../../web/backend/Features/Features.Auth/Commands/ResendConfirmationEmail/ResendConfirmationEmailHandler.cs)
**Flow:** **Flow:**
1. Receives refresh token from user 1. Receives a user ID via `ResendConfirmationEmailCommand`
2. Calls `TokenValidationService.ValidateRefreshTokenAsync()` 2. Looks up the user and checks they aren't already verified
3. Retrieves user account via `AuthRepository.GetUserByIdAsync()` 3. Generates a fresh confirmation token via `ITokenService`
4. Issues new access and refresh tokens via `TokenService` 4. Sends `SendResendConfirmationEmailCommand` over MediatR, handled by the
5. Returns new token pair `Features.Emails` slice, with no direct project reference between the two
slices
#### [AuthController](API.Core/Controllers/AuthController.cs) #### [RefreshTokenHandler](../../web/backend/Features/Features.Auth/Commands/RefreshToken/RefreshTokenHandler.cs)
**Flow:**
1. Receives a refresh token via `RefreshTokenCommand`
2. Delegates to `ITokenService.RefreshTokenAsync()`, which validates the
token, retrieves the user account via `IAuthRepository.GetUserByIdAsync()`,
and issues a new access/refresh token pair
3. Maps the result onto `LoginPayload`
#### [AuthController](../../web/backend/Features/Features.Auth/Controllers/AuthController.cs)
**Endpoints:** **Endpoints:**
- `POST /api/auth/register` - Register new user - `POST /api/auth/register` - Register new user
- `POST /api/auth/login` - Authenticate user - `POST /api/auth/login` - Authenticate user
- `POST /api/auth/confirm?token=...` - Confirm email - `POST /api/auth/confirm?token=...` - Confirm email
- `POST /api/auth/confirm/resend?userId=...` - Resend the confirmation email
- `POST /api/auth/refresh` - Refresh access token - `POST /api/auth/refresh` - Refresh access token
## Validation Security ## Validation Security
@@ -158,32 +172,44 @@ Validation failures return HTTP 401 Unauthorized:
1. **Generation**: During user registration (30-minute validity) 1. **Generation**: During user registration (30-minute validity)
2. **Delivery**: Emailed to user in confirmation link 2. **Delivery**: Emailed to user in confirmation link
3. **Usage**: User clicks link, token posted to `/api/auth/confirm` 3. **Usage**: User clicks link, token posted to `/api/auth/confirm`
4. **Validation**: Validated by ConfirmationService 4. **Validation**: Validated by `ConfirmUserHandler`
5. **Completion**: User account marked as confirmed 5. **Completion**: User account marked as confirmed
6. **Expiration**: Token becomes invalid after 30 minutes 6. **Expiration**: Token becomes invalid after 30 minutes
## Testing ## Testing
All of the following live in `Features.Auth.Tests`.
### Unit Tests ### Unit Tests
**TokenValidationService.test.cs** **Services/TokenServiceValidationTests.cs**
- Happy path: Valid token extraction - Happy path: Valid token extraction
- Error cases: Invalid, expired, malformed tokens - Error cases: Invalid, expired, malformed tokens
- Missing/invalid claims scenarios - Missing/invalid claims scenarios
**RefreshTokenService.test.cs** **Services/TokenServiceRefreshTests.cs**
- Successful refresh with valid token - Successful refresh with valid token
- Invalid/expired refresh token rejection - Invalid/expired refresh token rejection
- Non-existent user handling - Non-existent user handling
**ConfirmationService.test.cs** **Commands/RefreshTokenHandlerTests.cs**
- Verifies the handler maps `ITokenService.RefreshTokenAsync()`'s result onto
`LoginPayload`
**Commands/ConfirmUserHandlerTests.cs**
- Successful confirmation with valid token - Successful confirmation with valid token
- Token validation failures - Token validation failures
- User not found scenarios - User not found scenarios
**Commands/ResendConfirmationEmailHandlerTests.cs**
- Sends a fresh confirmation email when the user exists and is unverified
- No-ops when the user doesn't exist or is already verified
### BDD Tests (Reqnroll) ### BDD Tests (Reqnroll)
**TokenRefresh.feature** **TokenRefresh.feature**

View File

@@ -0,0 +1,24 @@
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
lerna-debug.log*
node_modules
dist
dist-ssr
*.local
# Editor directories and files
.vscode/*
!.vscode/extensions.json
.idea
.DS_Store
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?

View File

@@ -0,0 +1,74 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<link rel="icon" type="image/svg+xml" href="/favicon.svg" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Pipeline Results Viewer</title>
</head>
<body>
<div class="dialog" id="window">
<div class="dialog-header">
<p>Pipeline Results Viewer</p>
</div>
<div class="tab-strip" role="tablist">
<button
type="button"
class="tab is-active"
id="tab-tables"
role="tab"
aria-selected="true"
aria-controls="panel-tables"
>
Tables
</button>
<button
type="button"
class="tab"
id="tab-query"
role="tab"
aria-selected="false"
aria-controls="panel-query"
>
Query
</button>
</div>
<div class="dialog-body">
<div
class="tab-panel"
id="panel-tables"
role="tabpanel"
aria-labelledby="tab-tables"
>
<div class="field-group">
<label for="table-select">Table</label>
<select id="table-select"></select>
</div>
<p id="status"></p>
<div id="table-container"></div>
</div>
<div
class="tab-panel"
id="panel-query"
role="tabpanel"
aria-labelledby="tab-query"
hidden
>
<div class="field-group field-group--stacked">
<label for="query-input">SQL Query</label>
<textarea
id="query-input"
rows="4"
spellcheck="false"
placeholder="SELECT * FROM ..."
></textarea>
</div>
<button type="button" id="query-run" class="btn">Run Query</button>
<p id="query-status"></p>
<div id="query-container"></div>
</div>
</div>
</div>
<script type="module" src="/src/main.ts"></script>
</body>
</html>

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,20 @@
{
"name": "pipeline-results-viewer",
"version": "0.0.0",
"private": true,
"type": "module",
"scripts": {
"dev": "vite",
"build": "tsc && vite build",
"preview": "vite preview"
},
"devDependencies": {
"@types/sql.js": "^1.4.11",
"sass": "^1.101.0",
"typescript": "^6.0.3",
"vite": "^8.0.12"
},
"dependencies": {
"sql.js": "^1.14.1"
}
}

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 9.3 KiB

View File

@@ -0,0 +1,147 @@
import initSqlJs, { type Database, type SqlValue } from "sql.js";
import sqlWasmUrl from "sql.js/dist/sql-wasm.wasm?url";
import "./style.scss";
const DB_URL = "/biergarten.sqlite";
const tabs = document.querySelectorAll<HTMLButtonElement>("[role='tab']");
const panels = document.querySelectorAll<HTMLDivElement>("[role='tabpanel']");
const tableSelect = document.querySelector<HTMLSelectElement>("#table-select")!;
const tableContainer =
document.querySelector<HTMLDivElement>("#table-container")!;
const status = document.querySelector<HTMLParagraphElement>("#status")!;
const queryInput = document.querySelector<HTMLTextAreaElement>("#query-input")!;
const queryRun = document.querySelector<HTMLButtonElement>("#query-run")!;
const queryStatus = document.querySelector<HTMLParagraphElement>("#query-status")!;
const queryContainer =
document.querySelector<HTMLDivElement>("#query-container")!;
for (const tab of tabs) {
tab.addEventListener("click", () => {
for (const otherTab of tabs) {
const isActive = otherTab === tab;
otherTab.classList.toggle("is-active", isActive);
otherTab.setAttribute("aria-selected", String(isActive));
}
for (const panel of panels) {
panel.hidden = panel.id !== tab.getAttribute("aria-controls");
}
});
}
const renderResultTable = (
container: HTMLDivElement,
columns: string[],
values: SqlValue[][],
) => {
const table = document.createElement("table");
const head = table.createTHead().insertRow();
for (const column of columns) {
const th = document.createElement("th");
th.textContent = column;
head.append(th);
}
const body = table.createTBody();
for (const row of values) {
const tr = body.insertRow();
for (const cell of row) {
const td = document.createElement("td");
td.textContent = String(cell ?? "");
tr.append(td);
}
}
container.append(table);
};
const renderTable = (db: Database, name: string) => {
const result = db.exec(`SELECT * FROM "${name}"`);
tableContainer.replaceChildren();
if (result.length === 0) {
tableContainer.append(
Object.assign(document.createElement("p"), { textContent: "No rows." }),
);
return;
}
const { columns, values } = result[0];
renderResultTable(tableContainer, columns, values);
};
const runQuery = (db: Database) => {
const sql = queryInput.value.trim();
queryContainer.replaceChildren();
if (!sql) {
queryStatus.textContent = "Enter a query to run.";
return;
}
try {
const results = db.exec(sql);
if (results.length === 0) {
queryStatus.textContent = "Query executed. No rows returned.";
return;
}
const { columns, values } = results[results.length - 1];
renderResultTable(queryContainer, columns, values);
queryStatus.textContent = `${values.length} row${values.length === 1 ? "" : "s"} returned.`;
} catch (error) {
queryStatus.textContent = `Error: ${error instanceof Error ? error.message : String(error)}`;
}
};
async function main(): Promise<void> {
status.textContent = "Loading database...";
const sqlJs = await initSqlJs({ locateFile: () => sqlWasmUrl });
const response = await fetch(DB_URL);
const buffer = await response.arrayBuffer();
const db = new sqlJs.Database(new Uint8Array(buffer));
const tables =
db
.exec(
`SELECT name
FROM sqlite_master
WHERE
type = 'table'
AND
name NOT LIKE 'sqlite_%'
ORDER BY name`,
)[0]
?.values.map((row) => String(row[0])) ?? [];
for (const name of tables) {
tableSelect.append(new Option(name, name));
}
tableSelect.addEventListener("change", () =>
renderTable(db, tableSelect.value),
);
if (tables.length > 0) {
renderTable(db, tables[0]);
}
status.textContent = "";
queryRun.addEventListener("click", () => runQuery(db));
queryInput.addEventListener("keydown", (event) => {
if ((event.metaKey || event.ctrlKey) && event.key === "Enter") {
event.preventDefault();
runQuery(db);
}
});
}
main().catch((error: unknown) => {
status.textContent = `Failed to load database: ${String(error)}`;
});

View File

@@ -0,0 +1,267 @@
$gray-100: #dedede;
$gray-300: #c3c3c3;
$gray-500: #808080;
$white: #ffffff;
$black: #000000;
$navy: #000080;
$blue: #1084d0;
$teal: #008080;
$color-surface: $gray-300;
$color-surface-alt: $gray-100;
$color-surface-sunken: $white;
$color-border-highlight: $gray-100;
$color-border-lowlight: $gray-500;
$color-border-light: $white;
$color-border-dark: $black;
$color-text: $black;
$color-text-on-accent: $white;
$color-text-status: $navy;
$color-accent-start: $navy;
$color-accent-end: $blue;
$color-desktop-bg: $teal;
$color-row-stripe: $gray-100;
$color-shadow: rgba($black, 0.4);
$color-focus-ring: $black;
$bevel-size: 2px;
$font-stack: Tahoma, "MS Sans Serif", Verdana, sans-serif;
@mixin bevel-raised($size: $bevel-size) {
border-style: solid;
border-width: $size;
border-top-color: $color-border-highlight;
border-left-color: $color-border-highlight;
border-right-color: $color-border-dark;
border-bottom-color: $color-border-dark;
box-shadow:
inset $size $size 0 0 $color-border-light,
inset (-$size) (-$size) 0 0 $color-border-lowlight;
}
@mixin bevel-sunken($size: $bevel-size) {
border-style: solid;
border-width: $size;
border-top-color: $color-border-lowlight;
border-left-color: $color-border-lowlight;
border-right-color: $color-border-light;
border-bottom-color: $color-border-light;
box-shadow:
inset $size $size 0 0 $color-border-dark,
inset (-$size) (-$size) 0 0 $color-surface-alt;
}
* {
box-sizing: border-box;
}
body {
margin: 0;
padding: 24px;
background: $color-desktop-bg;
font-family: $font-stack;
color: $color-text;
min-height: 100vh;
}
#window {
width: 100%;
max-width: 1600px;
margin: 0 auto;
background: $color-surface;
padding: 6px;
display: flex;
flex-direction: column;
gap: 6px;
@include bevel-raised(3px);
box-shadow: 8px 8px 0 0 $color-shadow;
}
.dialog-header {
background: linear-gradient(90deg, $color-accent-start, $color-accent-end);
color: $color-text-on-accent;
padding: 6px 8px;
display: flex;
align-items: center;
p {
margin: 0;
font-size: 14px;
font-weight: bold;
letter-spacing: 0.5px;
text-transform: uppercase;
}
}
.tab-strip {
display: flex;
gap: 4px;
padding: 8px 8px 0;
}
.tab {
position: relative;
top: 3px;
margin-bottom: -2px;
font-family: $font-stack;
font-size: 12px;
font-weight: bold;
text-transform: uppercase;
padding: 6px 14px;
background: $color-surface;
color: $color-text;
border-style: solid;
border-width: 2px;
border-top-color: $color-border-light;
border-left-color: $color-border-light;
border-right-color: $color-border-dark;
border-bottom-color: $color-border-dark;
border-radius: 0;
cursor: pointer;
&.is-active {
top: 0;
z-index: 1;
border-bottom-color: $color-surface;
}
&:focus {
outline: 1px dotted $color-focus-ring;
outline-offset: -4px;
}
}
.dialog-body {
border-top: 2px solid $color-border-dark;
padding: 12px;
}
.tab-panel {
display: flex;
flex-direction: column;
gap: 12px;
&[hidden] {
display: none;
}
}
.field-group {
display: flex;
align-items: center;
gap: 8px;
label {
font-size: 12px;
font-weight: bold;
text-transform: uppercase;
}
&.field-group--stacked {
flex-direction: column;
align-items: stretch;
}
}
select#table-select {
font-family: $font-stack;
font-size: 13px;
padding: 3px 6px;
background: $color-surface-sunken;
color: $color-text;
border-radius: 0;
@include bevel-sunken();
&:focus {
outline: 1px dotted $color-focus-ring;
outline-offset: 2px;
}
}
textarea#query-input {
font-family: $font-stack;
font-size: 13px;
padding: 6px;
background: $color-surface-sunken;
color: $color-text;
border-radius: 0;
resize: vertical;
@include bevel-sunken();
&:focus {
outline: 1px dotted $color-focus-ring;
outline-offset: 2px;
}
}
.btn {
align-self: flex-start;
font-family: $font-stack;
font-size: 12px;
font-weight: bold;
text-transform: uppercase;
padding: 6px 16px;
background: $color-surface;
color: $color-text;
cursor: pointer;
@include bevel-raised();
&:active {
@include bevel-sunken();
}
&:focus {
outline: 1px dotted $color-focus-ring;
outline-offset: -4px;
}
}
#status,
#query-status {
margin: 0;
font-size: 12px;
font-style: italic;
color: $color-text-status;
}
#table-container,
#query-container {
background: $color-surface-sunken;
padding: 4px;
overflow-x: auto;
@include bevel-sunken();
}
table {
border-collapse: collapse;
margin: 0 auto;
font-family: $font-stack;
}
th,
td {
border: 1px solid $color-border-lowlight;
padding: 4px 8px;
text-align: left;
font-size: 13px;
}
th {
background: $color-surface;
white-space: nowrap;
font-weight: bold;
text-transform: uppercase;
@include bevel-raised(1px);
}
td {
min-width: 120px;
max-width: 800px;
white-space: normal;
word-break: break-word;
background: $color-surface-sunken;
}
tbody tr:nth-child(even) td {
background: $color-row-stripe;
}

View File

@@ -0,0 +1,24 @@
{
"compilerOptions": {
"target": "es2023",
"module": "esnext",
"lib": ["ES2023", "DOM"],
"types": ["vite/client"],
"skipLibCheck": true,
/* Bundler mode */
"moduleResolution": "bundler",
"esModuleInterop": true,
"allowImportingTsExtensions": true,
"verbatimModuleSyntax": true,
"moduleDetection": "force",
"noEmit": true,
/* Linting */
"noUnusedLocals": true,
"noUnusedParameters": true,
"erasableSyntaxOnly": true,
"noFallthroughCasesInSwitch": true
},
"include": ["src"]
}

View File

@@ -0,0 +1,9 @@
build/
cmake-build-debug/
.git/
.idea/
**/*.sqlite
**/*.log
**/*.sqlite3
**/*.db

View File

@@ -1,42 +1,45 @@
# CMakeLists.txt (project root)
cmake_minimum_required(VERSION 3.31) cmake_minimum_required(VERSION 3.31)
project(biergarten-pipeline) project(biergarten-pipeline)
# Set policy to allow FetchContent_Populate for header-only libraries
# that have outdated CMakeLists.txt files
cmake_policy(SET CMP0169 OLD)
# 1. Build Options # 1. Build Options
option(BIERGARTEN_MOCK_ONLY "Build with mock data generators only — skips llama.cpp" OFF) option(BIERGARTEN_MOCK_ONLY "Build with mock data generators only — skips llama.cpp" OFF)
if (BIERGARTEN_MOCK_ONLY) if(BIERGARTEN_MOCK_ONLY)
message(STATUS "[biergarten] MOCK_ONLY build — llama.cpp will not be compiled.") message(STATUS "[biergarten] MOCK_ONLY build — llama.cpp will not be compiled.")
endif () endif()
# 2. Platform & GPU Detection # 2. Platform & GPU Detection
if (NOT UNIX) if(NOT UNIX)
message(FATAL_ERROR "[biergarten] Windows is not supported. Please use Linux (Fedora 43) or macOS (M1 Pro).") message(FATAL_ERROR "[biergarten] Windows is not supported. Please use Linux (Fedora 43) or macOS (M1 Pro).")
endif () endif()
if (APPLE) if(APPLE)
if (CMAKE_SYSTEM_PROCESSOR MATCHES "arm64") if(CMAKE_SYSTEM_PROCESSOR MATCHES "arm64")
message(STATUS "[biergarten] Apple Silicon detected — enabling Metal acceleration.") message(STATUS "[biergarten] Apple Silicon detected — enabling Metal acceleration.")
set(GGML_METAL ON CACHE BOOL "Enable Metal for Apple Silicon" FORCE) set(GGML_METAL ON CACHE BOOL "Enable Metal for Apple Silicon" FORCE)
else () else()
message(STATUS "[biergarten] Intel Mac detected — using CPU / Accelerate framework.") message(STATUS "[biergarten] Intel Mac detected — using CPU / Accelerate framework.")
set(GGML_METAL OFF CACHE BOOL "Disable Metal for Intel Macs" FORCE) set(GGML_METAL OFF CACHE BOOL "Disable Metal for Intel Macs" FORCE)
endif () endif()
else () else()
find_package(CUDAToolkit QUIET) find_package(CUDAToolkit QUIET)
find_package(hip CONFIG QUIET) find_package(hip CONFIG QUIET)
if (CUDAToolkit_FOUND) if(CUDAToolkit_FOUND)
message(STATUS "[biergarten] NVIDIA GPU detected — enabling CUDA acceleration.") message(STATUS "[biergarten] NVIDIA GPU detected — enabling CUDA acceleration.")
set(GGML_CUDA ON CACHE BOOL "Enable CUDA for NVIDIA GPUs" FORCE) set(GGML_CUDA ON CACHE BOOL "Enable CUDA for NVIDIA GPUs" FORCE)
set(CMAKE_CUDA_ARCHITECTURES native) set(CMAKE_CUDA_ARCHITECTURES native)
elseif (hip_FOUND OR DEFINED ENV{ROCM_PATH} OR EXISTS "/opt/rocm") elseif(hip_FOUND OR DEFINED ENV{ROCM_PATH} OR EXISTS "/opt/rocm")
message(STATUS "[biergarten] AMD GPU detected — enabling HIP/ROCm acceleration.") message(STATUS "[biergarten] AMD GPU detected — enabling HIP/ROCm acceleration.")
set(GGML_HIPBLAS ON CACHE BOOL "Enable HIP for AMD GPUs" FORCE) set(GGML_HIPBLAS ON CACHE BOOL "Enable HIP for AMD GPUs" FORCE)
else () else()
message(STATUS "[biergarten] No NVIDIA or AMD GPU found — falling back to CPU.") message(STATUS "[biergarten] No NVIDIA or AMD GPU found — falling back to CPU.")
endif () endif()
endif () endif()
# 3. Project-wide Settings # 3. Project-wide Settings
set(CMAKE_CXX_STANDARD 20) set(CMAKE_CXX_STANDARD 20)
@@ -48,16 +51,27 @@ set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -Og -g")
# 4. Dependencies # 4. Dependencies
include(FetchContent) include(FetchContent)
# DEPRECATED: libcurl — to be removed once all usages are migrated to cpp-httplib. # Boost (system install — via dnf/brew)
# Tracked in: web_client/curl_web_client_get.cc, web_client/curl_web_client_url_encode.cc,
# web_client/curl_global_state.cc
find_package(CURL QUIET)
if (NOT CURL_FOUND)
message(FATAL_ERROR "[biergarten] libcurl not found. Install it (e.g. 'sudo dnf install libcurl-devel').")
endif ()
find_package(Boost REQUIRED COMPONENTS json program_options) find_package(Boost REQUIRED COMPONENTS json program_options)
# Boost.DI (unofficial Boost extension, must declare separately from main Boost dependency)
# Header-only library, so we only fetch without invoking its CMakeLists.txt
FetchContent_Declare(
boost-di
GIT_REPOSITORY https://github.com/boost-ext/di.git
GIT_TAG v1.3.0
GIT_SHALLOW TRUE
)
FetchContent_GetProperties(boost-di)
if(NOT boost-di_POPULATED)
FetchContent_Populate(boost-di)
endif()
add_library(boost_di INTERFACE)
add_library(boost::di ALIAS boost_di)
target_include_directories(boost_di INTERFACE
$<BUILD_INTERFACE:${boost-di_SOURCE_DIR}/include>
)
# SQLite amalgamation # SQLite amalgamation
FetchContent_Declare( FetchContent_Declare(
sqlite_amalgamation sqlite_amalgamation
@@ -66,32 +80,38 @@ FetchContent_Declare(
EXCLUDE_FROM_ALL EXCLUDE_FROM_ALL
) )
FetchContent_MakeAvailable(sqlite_amalgamation) FetchContent_MakeAvailable(sqlite_amalgamation)
if (NOT TARGET sqlite3) if(NOT TARGET sqlite3)
add_library(sqlite3 STATIC ${sqlite_amalgamation_SOURCE_DIR}/sqlite3.c) add_library(sqlite3 STATIC ${sqlite_amalgamation_SOURCE_DIR}/sqlite3.c)
target_include_directories(sqlite3 PUBLIC ${sqlite_amalgamation_SOURCE_DIR}) target_include_directories(sqlite3 PUBLIC ${sqlite_amalgamation_SOURCE_DIR})
target_compile_definitions(sqlite3 PUBLIC SQLITE_THREADSAFE=1) target_compile_definitions(sqlite3 PUBLIC SQLITE_THREADSAFE=1)
endif () endif()
# llama.cpp — skipped for mock-only builds # llama.cpp — skipped for mock-only builds
if (NOT BIERGARTEN_MOCK_ONLY) if(NOT BIERGARTEN_MOCK_ONLY)
find_library(LLAMA_LIB NAMES llama)
find_library(GGML_LIB NAMES ggml)
find_library(GGML_BASE_LIB NAMES ggml-base)
find_path(LLAMA_INC_DIR NAMES llama.h PATH_SUFFIXES include)
if(LLAMA_LIB AND GGML_LIB AND GGML_BASE_LIB AND LLAMA_INC_DIR)
message(STATUS "[biergarten] Found system llama.cpp — skipping FetchContent")
add_library(llama SHARED IMPORTED)
set_target_properties(llama PROPERTIES
IMPORTED_LOCATION "${LLAMA_LIB}"
INTERFACE_INCLUDE_DIRECTORIES "${LLAMA_INC_DIR}"
INTERFACE_LINK_LIBRARIES "${GGML_LIB};${GGML_BASE_LIB}"
)
else()
message(STATUS "[biergarten] System llama.cpp not found — fetching via FetchContent")
FetchContent_Declare( FetchContent_Declare(
llama-cpp llama-cpp
GIT_REPOSITORY https://github.com/ggml-org/llama.cpp.git GIT_REPOSITORY https://github.com/ggml-org/llama.cpp.git
GIT_TAG b8742 GIT_TAG b9012
) )
FetchContent_MakeAvailable(llama-cpp) FetchContent_MakeAvailable(llama-cpp)
endif () endif()
endif()
# Boost.DI (unofficial Boost extension, must declare separately from main Boost dependency)
FetchContent_Declare(
boost-di
GIT_REPOSITORY https://github.com/boost-ext/di.git
GIT_TAG v1.3.0
)
FetchContent_MakeAvailable(boost-di)
if (TARGET Boost.DI AND NOT TARGET boost::di)
add_library(boost::di ALIAS Boost.DI)
endif ()
# spdlog # spdlog
FetchContent_Declare( FetchContent_Declare(
@@ -101,16 +121,15 @@ FetchContent_Declare(
) )
FetchContent_MakeAvailable(spdlog) FetchContent_MakeAvailable(spdlog)
# cpp-httplib — replaces direct libcurl usage in web_client. # cpp-httplib — header-only HTTP/HTTPS client replacing libcurl.
# OpenSSL is required for HTTPS (Wikipedia). find_package is called first so # OpenSSL is required for HTTPS (Wikipedia API). find_package locates
# CMake can locate libssl/libcrypto; cpp-httplib itself is header-only so the # libssl/libcrypto; HTTPLIB_REQUIRE_OPENSSL causes a hard build failure
# CPPHTTPLIB_OPENSSL_SUPPORT compile definition is propagated via the target. # if OpenSSL is absent rather than silently producing an HTTP-only binary.
find_package(OpenSSL REQUIRED) find_package(OpenSSL REQUIRED)
FetchContent_Declare( FetchContent_Declare(
cpp-httplib cpp-httplib
GIT_REPOSITORY https://github.com/yhirose/cpp-httplib.git GIT_REPOSITORY https://github.com/yhirose/cpp-httplib.git
GIT_TAG v0.41.0 GIT_TAG v0.43.2
GIT_SHALLOW TRUE GIT_SHALLOW TRUE
SYSTEM SYSTEM
) )
@@ -118,7 +137,12 @@ set(HTTPLIB_REQUIRE_OPENSSL ON CACHE BOOL "Require OpenSSL for cpp-httplib" FORC
FetchContent_MakeAvailable(cpp-httplib) FetchContent_MakeAvailable(cpp-httplib)
# 5. Executable & Sources # 5. Executable & Sources
add_executable(${PROJECT_NAME}) add_executable(${PROJECT_NAME}
includes/services/enrichment/mock_enrichment.h
includes/services/curated_data/mock_curated_data_service.h
includes/services/postal_code/postal_code_service.h
includes/services/postal_code/mock_postal_code_service.h
includes/json_handling/pretty_print.h)
# --- Entry point --- # --- Entry point ---
target_sources(${PROJECT_NAME} PRIVATE target_sources(${PROJECT_NAME} PRIVATE
@@ -127,7 +151,12 @@ target_sources(${PROJECT_NAME} PRIVATE
# --- json_handling --- # --- json_handling ---
target_sources(${PROJECT_NAME} PRIVATE target_sources(${PROJECT_NAME} PRIVATE
src/json_handling/json_loader.cc src/services/curated_data/curated_json_data_service.cc
)
# --- services: curated_data ---
target_sources(${PROJECT_NAME} PRIVATE
src/services/curated_data/mock_curated_data_service.cc
) )
# --- application_options --- # --- application_options ---
@@ -135,23 +164,17 @@ target_sources(${PROJECT_NAME} PRIVATE
src/application_options/parse_arguments.cc src/application_options/parse_arguments.cc
) )
# --- biergarten_data_generator --- # --- biergarten_pipeline_orchestrator ---
target_sources(${PROJECT_NAME} PRIVATE target_sources(${PROJECT_NAME} PRIVATE
src/biergarten_data_generator/log_results.cc src/biergarten_pipeline_orchestrator/log_results.cc
src/biergarten_data_generator/biergarten_data_generator.cc src/biergarten_pipeline_orchestrator/biergarten_pipeline_orchestrator.cc
src/biergarten_data_generator/generate_breweries.cc src/biergarten_pipeline_orchestrator/generate_breweries.cc
src/biergarten_data_generator/run.cc src/biergarten_pipeline_orchestrator/generate_users.cc
src/biergarten_data_generator/query_cities_with_countries.cc src/biergarten_pipeline_orchestrator/run.cc
src/biergarten_pipeline_orchestrator/query_cities_with_countries.cc
) )
# --- web_client --- # --- web_client ---
# DEPRECATED: curl_web_client_* — to be replaced with cpp-httplib equivalents.
target_sources(${PROJECT_NAME} PRIVATE
src/web_client/curl_web_client_url_encode.cc
src/web_client/curl_web_client_get.cc
src/web_client/curl_global_state.cc
)
target_sources(${PROJECT_NAME} PRIVATE target_sources(${PROJECT_NAME} PRIVATE
src/web_client/http_web_client.cc src/web_client/http_web_client.cc
) )
@@ -169,7 +192,7 @@ target_sources(${PROJECT_NAME} PRIVATE
) )
# --- data_generation: llama (skipped for mock-only builds) --- # --- data_generation: llama (skipped for mock-only builds) ---
if (NOT BIERGARTEN_MOCK_ONLY) if(NOT BIERGARTEN_MOCK_ONLY)
target_sources(${PROJECT_NAME} PRIVATE target_sources(${PROJECT_NAME} PRIVATE
src/data_generation/llama/load.cc src/data_generation/llama/load.cc
src/data_generation/llama/helpers.cc src/data_generation/llama/helpers.cc
@@ -178,18 +201,19 @@ if (NOT BIERGARTEN_MOCK_ONLY)
src/data_generation/llama/llama_generator.cc src/data_generation/llama/llama_generator.cc
src/data_generation/llama/generate_user.cc src/data_generation/llama/generate_user.cc
) )
endif () endif()
# --- services: wikipedia --- # --- services: wikipedia ---
target_sources(${PROJECT_NAME} PRIVATE target_sources(${PROJECT_NAME} PRIVATE
src/services/wikipedia/wikipedia_service.cc src/services/enrichment/wikipedia/wikipedia_service.cc
src/services/wikipedia/fetch_extract.cc src/services/enrichment/wikipedia/fetch_extract.cc
src/services/wikipedia/get_summary.cc src/services/enrichment/wikipedia/get_summary.cc
) )
# --- services: sqlite --- # --- services: sqlite ---
target_sources(${PROJECT_NAME} PRIVATE target_sources(${PROJECT_NAME} PRIVATE
src/services/sqlite/process_record.cc src/services/sqlite/process_record.cc
src/services/sqlite/process_user_record.cc
src/services/sqlite/sqlite_export_service.cc src/services/sqlite/sqlite_export_service.cc
src/services/sqlite/finalize.cc src/services/sqlite/finalize.cc
src/services/sqlite/initialize.cc src/services/sqlite/initialize.cc
@@ -197,16 +221,20 @@ target_sources(${PROJECT_NAME} PRIVATE
src/services/sqlite/helpers/sqlite_statement_helpers.cc src/services/sqlite/helpers/sqlite_statement_helpers.cc
) )
# --- services: logging ---
target_sources(${PROJECT_NAME} PRIVATE
"src/services/logging/log_producer.cc"
src/services/logging/log_dispatcher.cc
)
# --- services (top-level) --- # --- services (top-level) ---
target_sources(${PROJECT_NAME} PRIVATE target_sources(${PROJECT_NAME} PRIVATE
src/services/prompt_directory.cc src/services/prompt_directory.cc
) )
# 6. Include Directories & Link Libraries # 6. Include Directories, Link Libraries & Compile Definitions
target_include_directories(${PROJECT_NAME} PRIVATE target_include_directories(${PROJECT_NAME} PRIVATE
includes includes
$<$<NOT:$<BOOL:${BIERGARTEN_MOCK_ONLY}>>:${llama-cpp_SOURCE_DIR}/include>
$<$<NOT:$<BOOL:${BIERGARTEN_MOCK_ONLY}>>:${llama-cpp_SOURCE_DIR}/common>
) )
target_link_libraries(${PROJECT_NAME} PRIVATE target_link_libraries(${PROJECT_NAME} PRIVATE
@@ -219,17 +247,42 @@ target_link_libraries(${PROJECT_NAME} PRIVATE
httplib::httplib httplib::httplib
OpenSSL::SSL OpenSSL::SSL
OpenSSL::Crypto OpenSSL::Crypto
CURL::libcurl # DEPRECATED: remove once web_client is migrated to cpp-httplib
) )
if (BIERGARTEN_MOCK_ONLY) target_compile_definitions(${PROJECT_NAME} PRIVATE
target_compile_definitions(${PROJECT_NAME} PRIVATE BIERGARTEN_MOCK_ONLY) # Defined when -DBIERGARTEN_MOCK_ONLY=ON — skips llama.cpp entirely.
endif () # Use #ifdef BIERGARTEN_MOCK_ONLY in source to guard llama-specific code.
$<$<BOOL:${BIERGARTEN_MOCK_ONLY}>:BIERGARTEN_MOCK_ONLY>
# Defined for Debug configuration builds.
# Use #ifdef DEBUG in source to enable debug-only behaviour (e.g. verbose logging).
$<$<CONFIG:Debug>:DEBUG>
)
target_compile_options(biergarten-pipeline PRIVATE
-fmacro-prefix-map=${CMAKE_SOURCE_DIR}/tooling/pipeline/src/=
)
# 7. Runtime Assets # 7. Runtime Assets
configure_file( configure_file(
${CMAKE_SOURCE_DIR}/locations.json ${CMAKE_SOURCE_DIR}/cities.json
${CMAKE_BINARY_DIR}/locations.json ${CMAKE_BINARY_DIR}/cities.json
COPYONLY
)
configure_file(
${CMAKE_SOURCE_DIR}/personas.json
${CMAKE_BINARY_DIR}/personas.json
COPYONLY
)
configure_file(
${CMAKE_SOURCE_DIR}/forenames-by-country.json
${CMAKE_BINARY_DIR}/forenames-by-country.json
COPYONLY
)
configure_file(
${CMAKE_SOURCE_DIR}/surnames-by-country.json
${CMAKE_BINARY_DIR}/surnames-by-country.json
COPYONLY COPYONLY
) )
add_custom_command(TARGET ${PROJECT_NAME} POST_BUILD add_custom_command(TARGET ${PROJECT_NAME} POST_BUILD

1718
tooling/pipeline/cities.json Normal file

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,72 @@
#!/bin/bash
#
# Walks every directory under src/ and includes/ (including nested
# subdirectories), runs clang-format on the C/C++ files in that directory
# only (non-recursive), and commits the result with its own commit.
#
# Usage: ./format-per-directory.sh [-y]
# -y skip the confirmation prompt
set -euo pipefail
SKIP_CONFIRM=false
if [[ "${1:-}" == "-y" ]]; then
SKIP_CONFIRM=true
fi
cd "$(dirname "$0")"
if [ ! -f .clang-format ]; then
echo "ERROR: .clang-format file not found."
exit 1
fi
if ! command -v clang-format &>/dev/null; then
echo "ERROR: clang-format not found."
exit 1
fi
if ! command -v git &>/dev/null; then
echo "ERROR: git not found."
exit 1
fi
echo "WARNING: This script will format .cpp, .h, .cxx, .cc, .c, .hpp files"
echo "directory-by-directory under src/ and includes/, committing after each"
echo "directory is formatted."
if [[ "$SKIP_CONFIRM" == false ]]; then
read -p "Do you want to continue? (y/n) " -n 1 -r
echo
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
echo "Aborted."
exit 1
fi
fi
if [[ -n "$(git status --porcelain)" ]]; then
echo "ERROR: working tree is not clean. Commit or stash your changes first."
exit 1
fi
dirs=$(find src includes -type d | sort)
for dir in $dirs; do
files=$(find "$dir" -maxdepth 1 -type f \( -name "*.cpp" -o -name "*.hpp" -o -name "*.h" -o -name "*.c" -o -name "*.cc" -o -name "*.cxx" \))
if [[ -z "$files" ]]; then
continue
fi
echo "Formatting $dir..."
echo "$files" | xargs clang-format -i
if [[ -n "$(git status --porcelain -- "$dir")" ]]; then
git add -- "$dir"
git commit -m "Formatted $dir"
else
echo "No changes in $dir, skipping commit."
fi
done
echo "Done."

View File

@@ -1,83 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_DATA_GENERATOR_H_
#define BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_DATA_GENERATOR_H_
/**
* @file biergarten_data_generator.h
* @brief Core orchestration class for pipeline data generation.
*/
#include <memory>
#include <span>
#include <vector>
#include "data_generation/data_generator.h"
#include "data_model/enriched_city.h"
#include "data_model/generated_brewery.h"
#include "data_model/location.h"
#include "services/enrichment_service.h"
#include "services/export_service.h"
/**
* @brief Main data generator class for the Biergarten pipeline.
*
* This class encapsulates the core logic for generating brewery data.
* It handles location loading, city enrichment, and brewery generation.
*/
class BiergartenDataGenerator {
public:
/**
* @brief Construct a BiergartenDataGenerator with injected dependencies.
*
* @param context_service Context provider for sampled locations.
* @param generator Brewery and user data generator.
* @param exporter Storage backend for generated brewery data.
*/
BiergartenDataGenerator(std::unique_ptr<IEnrichmentService> context_service,
std::unique_ptr<DataGenerator> generator,
std::unique_ptr<IExportService> exporter);
/**
* @brief Run the data generation pipeline.
*
* Performs the following steps:
* 1. Load curated locations from JSON
* 2. Resolve context for each city using the injected context service
* 3. Generate brewery data for sampled cities
*
* @return true if successful, false if not
*/
bool Run();
private:
/// @brief Owning context provider dependency.
std::unique_ptr<IEnrichmentService> context_service_;
/// @brief Generator dependency selected in the composition root.
std::unique_ptr<DataGenerator> generator_;
/// @brief Storage backend for generated brewery records.
std::unique_ptr<IExportService> exporter_;
/**
* @brief Load locations from JSON and sample cities.
*
* @return Vector of sampled locations capped at 50 entries.
*/
static std::vector<Location> QueryCitiesWithCountries();
/**
* @brief Generate breweries for enriched cities.
*
* @param cities Span of enriched city data.
*/
void GenerateBreweries(std::span<const EnrichedCity> cities);
/**
* @brief Log the generated brewery results.
*/
void LogResults() const;
/// @brief Stores generated brewery data.
std::vector<GeneratedBrewery> generated_breweries_;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_DATA_GENERATOR_H_

View File

@@ -0,0 +1,78 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_H_
#define BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_H_
/**
* @file biergarten_pipeline.h
* @brief Master umbrella header — includes every public header in the
* Biergarten pipeline.
*/
// --- orchestrator ---
#include "biergarten_pipeline_orchestrator.h"
// --- concurrency ---
#include "concurrency/bounded_channel.h"
// --- data_generation ---
#include "data_generation/data_generator.h"
#include "data_generation/json_grammars.h"
#ifndef BIERGARTEN_MOCK_ONLY
#include "data_generation/llama_generator.h"
#include "data_generation/llama_generator_helpers.h"
#endif
#include "data_generation/mock_generator.h"
#include "data_generation/prompt_formatting/gemma4_jinja_prompt_formatter.h"
#include "data_generation/prompt_formatting/prompt_formatter.h"
// --- data_model ---
#include "data_model/generated_models.h"
#include "data_model/models.h"
// --- json_handling ---
#include "json_handling/pretty_print.h"
#include "services/curated_data/curated_json_data_service.h"
// --- llama backend ---
#ifndef BIERGARTEN_MOCK_ONLY
#include "llama_backend_state.h"
#endif
// --- services: curated_data ---
#include "services/curated_data/curated_data_service.h"
#include "services/curated_data/mock_curated_data_service.h"
// --- services: database ---
#include "services/database/export_service.h"
#include "services/database/sqlite_connection_helpers.h"
#include "services/database/sqlite_export_service.h"
#include "services/database/sqlite_export_service_helpers.h"
#include "services/database/sqlite_handle_types.h"
#include "services/database/sqlite_statement_helpers.h"
// --- services: datetime ---
#include "services/datetime/date_time_provider.h"
#include "services/datetime/timer.h"
// --- services: enrichment ---
#include "services/enrichment/enrichment_service.h"
#include "services/enrichment/mock_enrichment.h"
#include "services/enrichment/wikipedia_service.h"
// --- services: logging ---
#include "services/logging/log_dispatcher.h"
#include "services/logging/log_entry.h"
#include "services/logging/log_producer.h"
#include "services/logging/logger.h"
// --- services: postal_code ---
#include "services/postal_code/mock_postal_code_service.h"
#include "services/postal_code/postal_code_service.h"
// --- services: prompting ---
#include "services/prompting/prompt_directory.h"
// --- web_client ---
#include "web_client/http_web_client.h"
#include "web_client/web_client.h"
#endif // BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_H_

View File

@@ -0,0 +1,119 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_ORCHESTRATOR_H_
#define BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_ORCHESTRATOR_H_
/**
* @file biergarten_pipeline_orchestrator.h
* @brief Orchestration for end-to-end brewery and user data generation.
*/
#include <memory>
#include <span>
#include <vector>
#include "data_generation/data_generator.h"
#include "data_model/generated_models.h"
#include "services/curated_data/curated_data_service.h"
#include "services/database/export_service.h"
#include "services/enrichment/enrichment_service.h"
#include "services/logging/logger.h"
#include "services/postal_code/postal_code_service.h"
/**
* @brief Main orchestrator for the Biergarten data generation pipeline.
*
* Handles location loading, city enrichment, and brewery/user generation.
*/
class BiergartenPipelineOrchestrator {
public:
/**
* @brief Constructs the orchestrator with injected pipeline dependencies.
*
* @param logger Sink for pipeline diagnostics.
* @param context_service Provides regional context for locations.
* @param generator Implementation (Llama or Mock) for brewery/user
* generation.
* @param exporter Database backend for persisting generated records.
* @param curated_data_service Loads curated location, persona, and name
* data used to seed generation.
* @param postal_code_service
* @param application_options CLI configuration and paths.
*/
BiergartenPipelineOrchestrator(
std::shared_ptr<ILogger> logger,
std::unique_ptr<IEnrichmentService> context_service,
std::unique_ptr<DataGenerator> generator,
std::unique_ptr<IExportService> exporter,
std::unique_ptr<ICuratedDataService> curated_data_service,
std::unique_ptr<IPostalCodeService> postal_code_service,
const ApplicationOptions& application_options);
/**
* @brief Run the data generation pipeline.
*
* Performs the following steps:
* 1. Load curated locations from JSON
* 2. Resolve context for each city using the injected context service
* 3. Generate brewery and user data for sampled cities
*
* @note STRUCTURAL CONCURRENCY REQUIREMENT:
* When transitioned to a multithreaded design, this method MUST
* structurally enforce that all deployed worker threads are joined before
* returning (e.g. by using std::jthread or a structured concurrency
* primitive). This ensures workers do not attempt to log to a closed
* channel during application teardown.
*
* @return true if successful, false if not
*/
bool Run();
private:
std::shared_ptr<ILogger> logger_;
std::unique_ptr<IEnrichmentService> context_service_;
/**
* @brief Generator implementation selected at the composition root (Llama
* or Mock).
*/
std::unique_ptr<DataGenerator> generator_;
/**
* @brief Storage backend for generated brewery and user records.
*/
std::unique_ptr<IExportService> exporter_;
std::unique_ptr<ICuratedDataService> curated_data_service_;
std::unique_ptr<IPostalCodeService> postal_code_service_;
ApplicationOptions application_options_;
/**
* @brief Load locations from JSON and sample cities.
*
* @return Vector of locations randomly sampled per
* PipelineOptions::location_count.
*/
std::vector<City> QueryCitiesWithCountries();
/**
* @brief Generate breweries for enriched cities.
*
* @param cities Span of enriched city data.
*/
void GenerateBreweries(std::span<const EnrichedCity> cities);
/**
* @brief Generate users grounded in sampled names and personas for
* enriched cities.
*
* @param cities Span of enriched city data.
*/
void GenerateUsers(std::span<const EnrichedCity> cities);
/**
* @brief Log the generated brewery and user results.
*/
void LogResults() const;
std::vector<BreweryRecord> generated_breweries_;
std::vector<UserRecord> generated_users_;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_PIPELINE_ORCHESTRATOR_H_

View File

@@ -0,0 +1,75 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_CONCURRENCY_BOUNDED_CHANNEL_H_
#define BIERGARTEN_PIPELINE_INCLUDES_CONCURRENCY_BOUNDED_CHANNEL_H_
#include <condition_variable>
#include <cstddef>
#include <mutex>
#include <optional>
#include <queue>
/**
* @file bounded_channel.h
* @brief Thread-safe, bounded multi-producer/multi-consumer synchronous
* channel.
*
* Intent: Enables asynchronous inter-thread communication with backpressure.
* Models a synchronous channel where producers/consumers block on capacity
* limits.
*/
/**
* @class BoundedChannel
* @brief MPMC channel with fixed capacity and blocking semantics.
*
* Producers block when buffer is full; consumers block when empty.
* Close() unblocks all waiters and signals channel exhaustion.
*/
template <typename T>
class BoundedChannel {
// -------------------------------------------------------------------------
// Internal state — all access must be guarded by mutex_.
// -------------------------------------------------------------------------
std::queue<T> queue_;
std::mutex mutex_;
std::condition_variable not_full_;
std::condition_variable not_empty_;
std::size_t capacity_;
bool closed_ = false;
public:
/**
* @brief Construct a bounded channel with the given capacity.
* @param capacity Maximum number of items the channel may hold.
*/
explicit BoundedChannel(std::size_t capacity) : capacity_(capacity) {}
/**
* @brief Send an item into the channel. Blocks when the channel is full.
* @param item Move-only item to enqueue.
*/
void Send(T item);
/**
* @brief Receive an item from the channel. Blocks when the channel is
* empty.
* @return std::optional<T> containing the item, or std::nullopt when the
* channel is closed and drained.
*/
std::optional<T> Receive();
/**
* @brief Close the channel and unblock all waiting threads. Idempotent.
*/
void Close();
};
// Include the template implementation
#include "bounded_channel.tcc"
#endif // BIERGARTEN_PIPELINE_INCLUDES_CONCURRENCY_BOUNDED_CHANNEL_H_

View File

@@ -0,0 +1,57 @@
#include "concurrency/bounded_channel.h"
template <typename T>
void BoundedChannel<T>::Send(T item) {
// Acquire exclusive ownership of the mutex; released automatically on scope exit.
std::unique_lock lock(mutex_);
// Block until there is space in the queue or the channel has been closed.
// The predicate guards against spurious wakeups.
not_full_.wait(lock, [&] { return queue_.size() < capacity_ || closed_; });
// If the channel was closed while waiting, discard the item and return.
if (closed_) return;
// Move the item into the queue to avoid an unnecessary copy.
queue_.push(std::move(item));
// Wake one blocked Receive() call to signal that data is now available.
not_empty_.notify_one();
}
template <typename T>
std::optional<T> BoundedChannel<T>::Receive() {
// Acquire exclusive ownership of the mutex.
std::unique_lock lock(mutex_);
// Block until the queue is non-empty or the channel has been closed.
// The predicate guards against spurious wakeups.
not_empty_.wait(lock, [&] { return !queue_.empty() || closed_; });
// If woken due to closure and no items remain, signal exhaustion via nullopt.
if (queue_.empty()) return std::nullopt;
// Move the front item out of the queue to avoid an unnecessary copy.
T item = std::move(queue_.front());
queue_.pop();
// Wake one blocked Send() call to signal that a slot has opened.
not_full_.notify_one();
return item;
}
template <typename T>
void BoundedChannel<T>::Close() {
// Acquire exclusive ownership of the mutex to ensure visibility of the flag.
std::unique_lock lock(mutex_);
// Mark the channel as closed; subsequent Send() calls will be dropped.
closed_ = true;
// Wake all blocked Send() callers so they can observe the closed flag and exit.
not_full_.notify_all();
// Wake all blocked Receive() callers so they can drain remaining items or return nullopt.
not_empty_.notify_all();
}

View File

@@ -8,9 +8,7 @@
#include <string> #include <string>
#include "data_model/brewery_result.h" #include "data_model/generated_models.h"
#include "data_model/location.h"
#include "data_model/user_result.h"
/** /**
* @brief Interface for data generator implementations. * @brief Interface for data generator implementations.
@@ -26,16 +24,20 @@ class DataGenerator {
* @param region_context Additional regional context text. * @param region_context Additional regional context text.
* @return Brewery generation result. * @return Brewery generation result.
*/ */
virtual BreweryResult GenerateBrewery(const Location& location, virtual BreweryResult GenerateBrewery(const City& location,
const std::string& region_context) = 0; const std::string& region_context) = 0;
/** /**
* @brief Generates a user profile for a locale. * @brief Generates a user profile grounded in a sampled name and persona.
* *
* @param locale Locale hint used by generator. * @param city Enriched city the user is associated with.
* @param persona Persona archetype grounding the generated bio.
* @param name Sampled first/last name (and gender)
* @return User generation result. * @return User generation result.
*/ */
virtual UserResult GenerateUser(const std::string& locale) = 0; virtual UserResult GenerateUser(const EnrichedCity& city,
const UserPersona& persona,
const Name& name) = 0;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_DATA_GENERATOR_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_DATA_GENERATOR_H_

View File

@@ -0,0 +1,52 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_JSON_GRAMMARS_H_
#define BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_JSON_GRAMMARS_H_
/**
* @file data_generation/json_grammars.h
* @brief GBNF grammars constraining structured JSON output from
* LlamaGenerator inference calls.
*/
#include <string_view>
// GBNF grammar for structured user JSON output.
// thought-block permits the model to emit free-form reasoning before the
// JSON object (the prompts explicitly invite this); only the "{...}" tail is
// constrained to the expected shape.
inline constexpr std::string_view kUserJsonGrammar = R"json_user(
root ::= thought-block (
"{" ws
"\"username\"" ws ":" ws string ws "," ws
"\"bio\"" ws ":" ws string ws "," ws
"\"activity_weight\"" ws ":" ws number ws
"}" ws
)
thought-block ::= [^{]*
ws ::= [ \t\n\r]*
string ::= "\"" char+ "\""
char ::= [^"\\\x7F\x00-\x1F] | [\\] escape
escape ::= ["\\/bfnrt] | "u" hex hex hex hex
hex ::= [0-9a-fA-F]
number ::= "-"? ("0" | [1-9] [0-9]*) ("." [0-9]+)?
)json_user";
// GBNF grammar for structured brewery JSON output (see thought-block note
// above).
inline constexpr std::string_view kBreweryJsonGrammar = R"json_brewery(
root ::= thought-block (
"{" ws
"\"name_en\"" ws ":" ws string ws "," ws
"\"description_en\"" ws ":" ws string ws "," ws
"\"name_local\"" ws ":" ws string ws "," ws
"\"description_local\"" ws ":" ws string ws
"}" ws
)
thought-block ::= [^{]*
ws ::= [ \t\n\r]*
string ::= "\"" char+ "\""
char ::= [^"\\\x7F\x00-\x1F] | [\\] escape
escape ::= ["\\/bfnrt] | "u" hex hex hex hex
hex ::= [0-9a-fA-F]
)json_brewery";
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_JSON_GRAMMARS_H_

View File

@@ -1,14 +1,13 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_LLAMA_GENERATOR_H_ #ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_LLAMA_GENERATOR_H_
#define BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_LLAMA_GENERATOR_H_ #define BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_LLAMA_GENERATOR_H_
#include <filesystem>
/** /**
* @file data_generation/llama_generator.h * @file data_generation/llama_generator.h
* @brief llama.cpp-backed implementation of DataGenerator. * @brief llama.cpp-backed implementation of DataGenerator.
*/ */
#include <cstdint> #include <cstdint>
#include <filesystem>
#include <memory> #include <memory>
#include <random> #include <random>
#include <string> #include <string>
@@ -16,8 +15,9 @@
#include "data_generation/data_generator.h" #include "data_generation/data_generator.h"
#include "data_generation/prompt_formatting/prompt_formatter.h" #include "data_generation/prompt_formatting/prompt_formatter.h"
#include "data_model/application_options.h" #include "data_model/models.h"
#include "services/prompt_directory.h" #include "services/logging/logger.h"
#include "services/prompting/prompt_directory.h"
struct llama_model; struct llama_model;
struct llama_context; struct llama_context;
@@ -37,20 +37,15 @@ class LlamaGenerator final : public DataGenerator {
* @param prompt_directory Directory service for loading named prompt files. * @param prompt_directory Directory service for loading named prompt files.
*/ */
LlamaGenerator(const ApplicationOptions& options, LlamaGenerator(const ApplicationOptions& options,
const std::string& model_path, const std::string& model_path, std::shared_ptr<ILogger> logger,
std::unique_ptr<IPromptFormatter> prompt_formatter, std::unique_ptr<IPromptFormatter> prompt_formatter,
std::unique_ptr<IPromptDirectory> prompt_directory); std::unique_ptr<IPromptDirectory> prompt_directory);
~LlamaGenerator() override; ~LlamaGenerator() override;
// disable copy constructor
LlamaGenerator(const LlamaGenerator&) = delete; LlamaGenerator(const LlamaGenerator&) = delete;
// disable copy assignment operator
LlamaGenerator& operator=(const LlamaGenerator&) = delete; LlamaGenerator& operator=(const LlamaGenerator&) = delete;
// disable move constructor
LlamaGenerator(LlamaGenerator&&) = delete; LlamaGenerator(LlamaGenerator&&) = delete;
// disable move assignment operator
LlamaGenerator& operator=(LlamaGenerator&&) = delete; LlamaGenerator& operator=(LlamaGenerator&&) = delete;
/** /**
@@ -60,16 +55,19 @@ class LlamaGenerator final : public DataGenerator {
* @param region_context Additional regional context. * @param region_context Additional regional context.
* @return Generated brewery result. * @return Generated brewery result.
*/ */
BreweryResult GenerateBrewery(const Location& location, BreweryResult GenerateBrewery(const City& location,
const std::string& region_context) override; const std::string& region_context) override;
/** /**
* @brief Generates a user profile for the provided locale. * @brief Generates a user profile grounded in a sampled name and persona.
* *
* @param locale Locale hint. * @param city Enriched city the user is associated with.
* @param persona Persona archetype grounding the generated bio.
* @param name Sampled first/last name -- not LLM-invented.
* @return Generated user profile. * @return Generated user profile.
*/ */
UserResult GenerateUser(const std::string& locale) override; UserResult GenerateUser(const EnrichedCity& city, const UserPersona& persona,
const Name& name) override;
private: private:
static constexpr int32_t kDefaultMaxTokens = 10000; static constexpr int32_t kDefaultMaxTokens = 10000;
@@ -129,6 +127,8 @@ class LlamaGenerator final : public DataGenerator {
uint32_t sampling_top_k_ = kDefaultSamplingTopK; uint32_t sampling_top_k_ = kDefaultSamplingTopK;
std::mt19937 rng_; std::mt19937 rng_;
uint32_t n_ctx_ = kDefaultContextSize; uint32_t n_ctx_ = kDefaultContextSize;
int n_gpu_layers_ = 0;
std::shared_ptr<ILogger> logger_;
std::unique_ptr<IPromptFormatter> prompt_formatter_; std::unique_ptr<IPromptFormatter> prompt_formatter_;
std::unique_ptr<IPromptDirectory> prompt_directory_; std::unique_ptr<IPromptDirectory> prompt_directory_;
}; };

View File

@@ -12,7 +12,7 @@
#include <string> #include <string>
#include <string_view> #include <string_view>
#include "data_model/brewery_result.h" #include "data_model/generated_models.h"
struct llama_vocab; struct llama_vocab;
using llama_token = int32_t; using llama_token = int32_t;
@@ -47,4 +47,18 @@ void AppendTokenPiece(const llama_vocab* vocab, llama_token token,
std::optional<std::string> ValidateBreweryJson(const std::string& raw, std::optional<std::string> ValidateBreweryJson(const std::string& raw,
BreweryResult& brewery_out); BreweryResult& brewery_out);
/**
* @brief Validates and parses user JSON output.
*
* Only populates `username`, `bio`, and `activity_weight` -- `first_name`
* and `last_name` are not LLM-authored and are set separately from the
* sampled Name.
*
* @param raw Raw model output.
* @param user_out Parsed user payload.
* @return Validation error message if invalid, or std::nullopt on success.
*/
std::optional<std::string> ValidateUserJson(const std::string& raw,
UserResult& user_out);
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_LLAMA_GENERATOR_HELPERS_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_LLAMA_GENERATOR_HELPERS_H_

View File

@@ -24,16 +24,20 @@ class MockGenerator final : public DataGenerator {
* @param region_context Unused for mock generation. * @param region_context Unused for mock generation.
* @return Generated brewery result. * @return Generated brewery result.
*/ */
BreweryResult GenerateBrewery(const Location& location, BreweryResult GenerateBrewery(const City& location,
const std::string& region_context) override; const std::string& region_context) override;
/** /**
* @brief Generates deterministic user data for a locale. * @brief Generates deterministic user data grounded in a sampled name and
* persona.
* *
* @param locale Locale hint. * @param city Enriched city the user is associated with.
* @param persona Persona archetype.
* @param name Sampled first/last name, copied directly into the result.
* @return Generated user result. * @return Generated user result.
*/ */
UserResult GenerateUser(const std::string& locale) override; UserResult GenerateUser(const EnrichedCity& city, const UserPersona& persona,
const Name& name) override;
private: private:
/** /**
@@ -42,7 +46,18 @@ class MockGenerator final : public DataGenerator {
* @param location City and country names. * @param location City and country names.
* @return Deterministic hash value. * @return Deterministic hash value.
*/ */
static size_t DeterministicHash(const Location& location); static size_t DeterministicHash(const City& location);
/**
* @brief Combines city, persona, and name into a stable hash value.
*
* @param location City and country names.
* @param persona Persona archetype.
* @param name Sampled first/last name.
* @return Deterministic hash value.
*/
static size_t DeterministicHash(const City& location,
const UserPersona& persona, const Name& name);
// Hash stride constants for deterministic distribution across fixed-size // Hash stride constants for deterministic distribution across fixed-size
// arrays. These coprime strides spread hash values uniformly without // arrays. These coprime strides spread hash values uniformly without
@@ -50,6 +65,8 @@ class MockGenerator final : public DataGenerator {
static constexpr size_t kNounHashStride = 7; static constexpr size_t kNounHashStride = 7;
static constexpr size_t kDescriptionHashStride = 13; static constexpr size_t kDescriptionHashStride = 13;
static constexpr size_t kBioHashStride = 11; static constexpr size_t kBioHashStride = 11;
static constexpr size_t kActivityWeightHashStride = 17;
static constexpr size_t kActivityWeightHashRange = 1000;
static constexpr std::array<std::string_view, 18> kBreweryAdjectives = { static constexpr std::array<std::string_view, 18> kBreweryAdjectives = {
"Craft", "Heritage", "Local", "Artisan", "Pioneer", "Golden", "Craft", "Heritage", "Local", "Artisan", "Pioneer", "Golden",
@@ -124,7 +141,8 @@ class MockGenerator final : public DataGenerator {
"Visits breweries for the stories, stays for the flagship pours.", "Visits breweries for the stories, stays for the flagship pours.",
"Craft beer fan mapping tasting notes and favorite brew routes.", "Craft beer fan mapping tasting notes and favorite brew routes.",
"Always ready to trade recommendations for underrated local breweries.", "Always ready to trade recommendations for underrated local breweries.",
"Keeping a running list of must-try collab releases and tap takeovers."}; "Keeping a running list of must-try collab releases and tap "
"takeovers."};
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_MOCK_GENERATOR_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_MOCK_GENERATOR_H_

View File

@@ -1,4 +1,5 @@
#pragma once #ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_PROMPT_FORMATTING_GEMMA4_JINJA_PROMPT_FORMATTER_H_
#define BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_PROMPT_FORMATTING_GEMMA4_JINJA_PROMPT_FORMATTER_H_
#include <string> #include <string>
#include <string_view> #include <string_view>
@@ -13,3 +14,5 @@ class Gemma4JinjaPromptFormatter final : public IPromptFormatter {
[[nodiscard]] std::string Format(std::string_view system_prompt, [[nodiscard]] std::string Format(std::string_view system_prompt,
std::string_view user_prompt) const override; std::string_view user_prompt) const override;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_PROMPT_FORMATTING_GEMMA4_JINJA_PROMPT_FORMATTER_H_

View File

@@ -1,4 +1,5 @@
#pragma once #ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_PROMPT_FORMATTING_PROMPT_FORMATTER_H_
#define BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_PROMPT_FORMATTING_PROMPT_FORMATTER_H_
#include <string> #include <string>
#include <string_view> #include <string_view>
@@ -15,3 +16,5 @@ class IPromptFormatter {
[[nodiscard]] virtual std::string Format( [[nodiscard]] virtual std::string Format(
std::string_view system_prompt, std::string_view user_prompt) const = 0; std::string_view system_prompt, std::string_view user_prompt) const = 0;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_PROMPT_FORMATTING_PROMPT_FORMATTER_H_

View File

@@ -1,76 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_APPLICATION_OPTIONS_H_
#define BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_APPLICATION_OPTIONS_H_
/**
* @file data_model/application_options.h
* @brief Program options for the Biergarten pipeline application.
*/
#include <boost/program_options.hpp>
#include <cstdint>
#include <filesystem>
#include <optional>
#include <string>
namespace prog_opts = boost::program_options;
/**
* @brief LLM sampling parameters.
*/
struct SamplingOptions {
/// @brief LLM sampling temperature (0.0 to 1.0, higher = more random).
float temperature = 1.0F;
/// @brief LLM nucleus sampling top-p parameter.
float top_p = 0.95F;
/// @brief LLM top-k sampling parameter.
uint32_t top_k = 64;
/// @brief Context window size (tokens).
uint32_t n_ctx = 8192;
/// @brief Random seed (-1 for random, otherwise non-negative).
int seed = -1;
};
/**
* @brief Configuration for the LLM generator component.
*/
struct GeneratorOptions {
/// @brief Path to the LLM model file (gguf format).
std::filesystem::path model_path;
/// @brief Use mocked generator instead of actual LLM inference.
bool use_mocked = false;
/// @brief Specific sampling parameters for this generator.
/// If nullopt, the application should use global defaults.
std::optional<SamplingOptions> sampling;
};
/**
* @brief Configuration for the pipeline execution and output.
*/
struct PipelineOptions {
/// @brief Directory for generated artifacts.
std::filesystem::path output_path;
/// @brief Directory that contains named prompt files (e.g.
/// BREWERY_GENERATION.md).
std::filesystem::path prompt_dir;
/// @brief Path for application logs.
std::filesystem::path log_path;
};
/**
* @brief Root configuration object for the Biergarten pipeline.
*/
struct ApplicationOptions {
GeneratorOptions generator;
PipelineOptions pipeline;
};
std::optional<ApplicationOptions> ParseArguments(const int argc, char** argv);
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_APPLICATION_OPTIONS_H_

View File

@@ -1,22 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_BREWERY_LOCATION_H_
#define BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_BREWERY_LOCATION_H_
/**
* @file data_model/brewery_location.h
* @brief Non-owning brewery location input.
*/
#include <string_view>
/**
* @brief Non-owning brewery location input.
*/
struct BreweryLocation {
/// @brief City name.
std::string_view city_name;
/// @brief Country name.
std::string_view country_name;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_BREWERY_LOCATION_H_

View File

@@ -1,28 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_BREWERY_RESULT_H_
#define BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_BREWERY_RESULT_H_
/**
* @file data_model/brewery_result.h
* @brief Generated brewery payload.
*/
#include <string>
/**
* @brief Generated brewery payload.
*/
struct BreweryResult {
/// @brief Brewery display name in English.
std::string name_en;
/// @brief Brewery description text in English.
std::string description_en;
/// @brief Brewery display name in the local language.
std::string name_local;
/// @brief Brewery description text in the local language.
std::string description_local;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_BREWERY_RESULT_H_

View File

@@ -1,21 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_ENRICHED_CITY_H_
#define BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_ENRICHED_CITY_H_
/**
* @file data_model/enriched_city.h
* @brief Enriched city data with Wikipedia context.
*/
#include <string>
#include "data_model/location.h"
/**
* @brief Enriched city data with Wikipedia context.
*/
struct EnrichedCity {
Location location;
std::string region_context{};
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_ENRICHED_CITY_H_

View File

@@ -1,20 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_GENERATED_BREWERY_H_
#define BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_GENERATED_BREWERY_H_
/**
* @file data_model/generated_brewery.h
* @brief Helper struct to store generated brewery data.
*/
#include "data_model/brewery_result.h"
#include "data_model/location.h"
/**
* @brief Helper struct to store generated brewery data.
*/
struct GeneratedBrewery {
Location location;
BreweryResult brewery;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_GENERATED_BREWERY_H_

View File

@@ -0,0 +1,147 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_GENERATED_MODELS_H_
#define BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_GENERATED_MODELS_H_
/**
* @file data_model/generated_models.h
* @brief Generated output models from the pipeline: brewery/user results,
* enriched data, and complete generation results.
*/
#include <optional>
#include <string>
#include "data_model/models.h"
// ============================================================================
// Generation Output Models
// ============================================================================
/**
* @brief Generated brewery payload.
*/
struct BreweryResult {
/**
* @brief Brewery display name in English.
*/
std::string name_en;
/**
* @brief Brewery description text in English.
*/
std::string description_en;
/**
* @brief Brewery display name in the local language.
*/
std::string name_local;
/**
* @brief Brewery description text in the local language.
*/
std::string description_local;
};
/**
* @brief Generated user profile payload.
*/
struct UserResult {
/**
* @brief First (given) name, copied from the sampled Name -- not
* LLM-invented.
*/
std::string first_name{};
/**
* @brief Last (family) name, copied from the sampled Name -- not
* LLM-invented.
*/
std::string last_name{};
/**
* @brief Gender associated with the sampled first name, copied from the
* sampled Name -- not LLM-invented.
*/
std::string gender{};
/**
* @brief Username handle.
*/
std::string username{};
/**
* @brief Short user biography.
*/
std::string bio{};
/**
* @brief Relative check-in/activity weight for this user, used to drive
* a J-curve activity profile in later pipeline phases.
*/
float activity_weight{};
};
// ============================================================================
// Pipeline Data Models
// ============================================================================
/**
* @brief Enriched city data with Wikipedia context.
*/
struct EnrichedCity {
City location;
std::string region_context{};
};
/**
* @brief A brewery's address. Owns the `City` it belongs to alongside its
* street-level fields. Mirrors the `brewery_addresses` table and the
* backend's `BreweryPostLocation` shape.
*
* Only `city` and `postal_code` are populated today -- street-address
* generation has no fixture data or service yet, so the address lines stay
* empty.
*/
struct BreweryAddress {
City city{};
std::optional<std::string> address_line_1{};
std::optional<std::string> address_line_2{};
std::string postal_code{};
};
/**
* @brief A user's address. Owns the `City` it belongs to alongside its
* street-level fields. Mirrors the `user_addresses` table.
*
* Only `city` is populated today; the street-level columns exist for future
* enrichment.
*/
struct UserAddress {
City city{};
std::optional<std::string> address_line_1{};
std::optional<std::string> address_line_2{};
std::optional<std::string> postal_code{};
};
/**
* @brief Helper struct to store generated brewery data.
*/
struct BreweryRecord {
BreweryAddress address;
BreweryResult brewery;
};
/**
* @brief Helper struct to store generated user data.
*
* `email` and `date_of_birth` are programmatically generated by the
* orchestrator (never LLM-authored) so a downstream auth-account seeding
* consumer can register real accounts from the pipeline's SQLite export.
*/
struct UserRecord {
UserAddress address{};
UserResult user;
std::string email{};
std::string date_of_birth{};
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_GENERATED_MODELS_H_

View File

@@ -1,13 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_GENERATION_MODELS_H_
#define BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_GENERATION_MODELS_H_
/**
* @file data_model/generation_models.h
* @brief Convenience include for shared generation payload models.
*/
#include "data_model/brewery_location.h"
#include "data_model/brewery_result.h"
#include "data_model/user_result.h"
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_GENERATION_MODELS_H_

View File

@@ -1,41 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_LOCATION_H_
#define BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_LOCATION_H_
/**
* @file data_model/location.h
* @brief Location data model used throughout generation pipeline.
*/
#include <string>
#include <vector>
/**
* @brief Canonical location record for city-level generation.
*/
struct Location {
/// @brief City name.
std::string city{};
/// @brief State or province name.
std::string state_province{};
/// @brief ISO 3166-2 subdivision code.
std::string iso3166_2{};
/// @brief Country name.
std::string country{};
/// @brief ISO 3166-1 country code.
std::string iso3166_1{};
/// @brief Local language codes in priority order.
std::vector<std::string> local_languages{};
/// @brief Latitude in decimal degrees.
double latitude{};
/// @brief Longitude in decimal degrees.
double longitude{};
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_LOCATION_H_

View File

@@ -0,0 +1,235 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_MODELS_H_
#define BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_MODELS_H_
/**
* @file data_model/models.h
* @brief Core data models: locations, application configuration, and generation
* inputs.
*/
#include <boost/program_options.hpp>
#include <cstdint>
#include <filesystem>
#include <functional>
#include <memory>
#include <optional>
#include <string>
#include <string_view>
#include <vector>
class ILogger;
namespace prog_opts = boost::program_options;
// ============================================================================
// City Models
// ============================================================================
/**
* @brief Canonical location record for city-level generation.
*/
struct City {
std::string city{};
std::string state_province{};
/**
* @brief ISO 3166-2 subdivision code.
*/
std::string iso3166_2{};
std::string country{};
/**
* @brief ISO 3166-1 country code.
*/
std::string iso3166_1{};
/**
* @brief A list of regular expressions used for valid postal codes in the
* region.
*
*/
std::vector<std::string> postal_regex{};
/**
* @brief Example postal codes for the region.
*/
std::vector<std::string> postal_code_examples{};
/**
* @brief Local language codes in priority order.
*/
std::vector<std::string> local_languages{};
};
// ============================================================================
// Name / Persona Models
// ============================================================================
/**
* @brief A sampled first/last name pair, with the source forename's gender.
*
* Produced by the SampleName() helper in generate_users.cc.
*/
struct Name {
std::string first_name{};
std::string last_name{};
/**
* @brief Gender associated with the sampled forename (e.g. "M", "F"), as
* reported by the source dataset.
*/
std::string gender{};
};
/**
* @brief A single forename entry from the names-by-country fixture data.
*/
struct ForenameEntry {
/**
* @brief Romanized forename.
*/
std::string name{};
/**
* @brief Gender associated with this forename, as reported by the source
* dataset (e.g. "M", "F").
*/
std::string gender{};
bool operator==(const ForenameEntry& other) const {
return name == other.name && gender == other.gender;
}
};
namespace std {
template <>
struct hash<ForenameEntry> {
size_t operator()(const ForenameEntry& entry) const noexcept {
const size_t name_hash = std::hash<std::string>{}(entry.name);
const size_t gender_hash = std::hash<std::string>{}(entry.gender);
return name_hash ^ (gender_hash << 1);
}
};
} // namespace std
/**
* @brief A persona archetype used to ground LLM-generated user bios.
*/
struct UserPersona {
/**
* @brief Persona display name (e.g. "Hophead Explorer").
*/
std::string name{};
/**
* @brief Short description of the persona's interests and voice.
*/
std::string description{};
/**
* @brief Beer styles this persona gravitates toward.
*/
std::vector<std::string> style_affinities{};
};
// ============================================================================
// Configuration Models
// ============================================================================
/**
* @brief LLM sampling parameters.
*/
struct SamplingOptions {
/**
* @brief LLM sampling temperature (higher = more random).
*/
float temperature = 1.0F;
/**
* @brief LLM nucleus sampling top-p parameter.
*/
float top_p = 0.95F;
/**
* @brief LLM top-k sampling parameter.
*/
uint32_t top_k = 64;
/**
* @brief Context window size (tokens).
*/
uint32_t n_ctx = 8192;
/**
* @brief Random seed (-1 for random, otherwise non-negative).
*/
int seed = -1;
/**
* @brief Number of layers to offload to GPU.
*/
int n_gpu_layers = 0;
};
/**
* @brief Configuration for the LLM generator component.
*/
struct GeneratorOptions {
/**
* @brief Path to the LLM model file (gguf format).
*/
std::filesystem::path model_path;
/**
* @brief Use mocked generator instead of actual LLM inference.
*/
bool use_mocked = false;
/**
* @brief Specific sampling parameters for this generator.
* If nullopt, the application should use global defaults.
*/
std::optional<SamplingOptions> sampling;
};
/**
* @brief Configuration for the pipeline execution and output.
*/
struct PipelineOptions {
/**
* @brief Directory for generated artifacts.
*/
std::filesystem::path output_path;
/**
* @brief Directory that contains named prompt files (e.g.
* BREWERY_GENERATION.md).
*/
std::filesystem::path prompt_dir;
std::filesystem::path log_path;
/**
* @brief Number of locations to sample from the dataset
* More locations -> more users/more breweries
*/
uint32_t location_count;
};
/**
* @brief Root configuration object for the Biergarten pipeline.
*/
struct ApplicationOptions {
GeneratorOptions generator;
PipelineOptions pipeline;
};
// ============================================================================
// Function Declarations
// ============================================================================
std::optional<ApplicationOptions> ParseArguments(
const int argc, char** argv, std::shared_ptr<ILogger> logger = nullptr);
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_MODELS_H_

View File

@@ -1,12 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_PIPELINE_MODELS_H_
#define BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_PIPELINE_MODELS_H_
/**
* @file data_model/pipeline_models.h
* @brief Convenience include for pipeline-specific data models.
*/
#include "data_model/enriched_city.h"
#include "data_model/generated_brewery.h"
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_PIPELINE_MODELS_H_

View File

@@ -1,22 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_USER_RESULT_H_
#define BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_USER_RESULT_H_
/**
* @file data_model/user_result.h
* @brief Generated user profile payload.
*/
#include <string>
/**
* @brief Generated user profile payload.
*/
struct UserResult {
/// @brief Username handle.
std::string username{};
/// @brief Short user biography.
std::string bio{};
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_USER_RESULT_H_

View File

@@ -1,22 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_
#define BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_
/**
* @file json_handling/json_loader.h
* @brief Loader API for curated location data.
*/
#include <filesystem>
#include <vector>
#include "data_model/location.h"
/// @brief Loads curated world locations from a JSON file into memory.
class JsonLoader {
public:
/// @brief Parses a JSON array file and returns all location records.
static std::vector<Location> LoadLocations(
const std::filesystem::path& filepath);
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_

View File

@@ -0,0 +1,109 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_PRETTY_PRINT_H_
#define BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_PRETTY_PRINT_H_
/**
* @file json_handling/pretty_print.h
* @brief Pretty-printing utilities for JSON values.
*
* Provides formatting capability for boost::json::value with indentation and
* readable output. Adapted from Boost JSON library examples.
*/
#include <boost/json.hpp>
#include <ostream>
#include <string>
/**
* @brief Pretty-prints a JSON value to an output stream with indentation.
*
* Recursively formats JSON objects and arrays with consistent 4-space
* indentation. Adapted from:
* https://raw.githubusercontent.com/boostorg/json/refs/heads/develop/example/pretty.cpp
*
* @param outstream Output stream to write formatted JSON.
* @param json_val JSON value to format.
* @param indent Optional indentation string (managed internally on first call).
*/
inline void PrettyPrint(std::ostream& outstream,
boost::json::value const& json_val,
std::string* indent = nullptr) {
std::string str;
if (indent == nullptr) {
indent = &str;
}
switch (json_val.kind()) {
case boost::json::kind::object: {
outstream << "{\n";
indent->append(4, ' ');
auto const& obj = json_val.get_object();
if (!obj.empty()) {
const auto* iter = obj.begin();
for (;;) {
outstream << *indent << boost::json::serialize(iter->key()) << " : ";
PrettyPrint(outstream, iter->value(), indent);
iter = std::next(iter);
if (iter == obj.end()) {
break;
}
outstream << ",\n";
}
}
outstream << "\n";
indent->resize(indent->size() - 4);
outstream << *indent << "}";
break;
}
case boost::json::kind::array: {
outstream << "[\n";
indent->append(4, ' ');
auto const& arr = json_val.get_array();
if (!arr.empty()) {
const auto* iter = arr.begin();
for (;;) {
outstream << *indent;
PrettyPrint(outstream, *iter, indent);
iter = std::next(iter);
if (iter == arr.end()) {
break;
}
outstream << ",\n";
}
}
outstream << "\n";
indent->resize(indent->size() - 4);
outstream << *indent << "]";
break;
}
case boost::json::kind::string: {
outstream << serialize(json_val.get_string());
break;
}
case boost::json::kind::uint64:
case boost::json::kind::int64:
case boost::json::kind::double_:
outstream << json_val;
break;
case boost::json::kind::bool_:
if (json_val.get_bool()) {
outstream << "true";
} else {
outstream << "false";
}
break;
case boost::json::kind::null:
outstream << "null";
break;
}
if (indent->empty()) {
outstream << "\n";
}
}
#endif // BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_PRETTY_PRINT_H_

View File

@@ -16,16 +16,10 @@
*/ */
class LlamaBackendState { class LlamaBackendState {
public: public:
/// @brief Initializes global llama backend state.
LlamaBackendState() { llama_backend_init(); } LlamaBackendState() { llama_backend_init(); }
/// @brief Cleans up global llama backend state.
~LlamaBackendState() { llama_backend_free(); } ~LlamaBackendState() { llama_backend_free(); }
/// @brief Non-copyable type.
LlamaBackendState(const LlamaBackendState&) = delete; LlamaBackendState(const LlamaBackendState&) = delete;
/// @brief Non-copyable type.
LlamaBackendState& operator=(const LlamaBackendState&) = delete; LlamaBackendState& operator=(const LlamaBackendState&) = delete;
}; };

View File

@@ -0,0 +1,50 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_CURATED_DATA_SERVICE_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_CURATED_DATA_SERVICE_H_
/**
* @file services/curated_data/curated_data_service.h
* @brief Abstraction for loading curated location, persona, and name data.
*/
#include <filesystem>
#include <unordered_map>
#include <unordered_set>
#include <vector>
#include "data_model/models.h"
using ForenameList = std::vector<ForenameEntry>;
using SurnameList = std::vector<std::string>;
using CityList = std::vector<City>;
using PersonasList = std::vector<UserPersona>;
using ForenamesByCountryMap = std::unordered_map<std::string, ForenameList>;
using SurnamesByCountryMap = std::unordered_map<std::string, SurnameList>;
/**
* @brief Interface for services that load curated data used to seed
* brewery/user generation.
*/
class ICuratedDataService {
public:
ICuratedDataService() = default;
virtual ~ICuratedDataService() = default;
ICuratedDataService(const ICuratedDataService&) = delete;
ICuratedDataService& operator=(const ICuratedDataService&) = delete;
ICuratedDataService(ICuratedDataService&&) = delete;
ICuratedDataService& operator=(ICuratedDataService&&) = delete;
/**
* @brief Loads all curated location records.
*/
virtual const CityList& LoadCities() = 0;
/**
* @brief Loads all curated persona records.
*/
virtual const PersonasList& LoadPersonas() = 0;
virtual const ForenamesByCountryMap& LoadForenamesByCountry() = 0;
virtual const SurnamesByCountryMap& LoadSurnamesByCountry() = 0;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_CURATED_DATA_SERVICE_H_

View File

@@ -0,0 +1,63 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_
#define BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_
/**
* @file json_handling/json_loader.h
* @brief JSON-backed implementation of ICuratedDataService.
*/
#include <filesystem>
#include "data_model/models.h"
#include "services/curated_data/curated_data_service.h"
/**
* @brief File locations for the curated JSON fixtures consumed by
* CuratedJsonDataService.
*/
struct CuratedDataFilePaths {
std::filesystem::path cities_path;
std::filesystem::path personas_path;
std::filesystem::path forenames_path;
std::filesystem::path surnames_path;
};
/**
* @brief Loads curated location, persona, and name data from JSON files.
*/
class CuratedJsonDataService final : public ICuratedDataService {
struct cache {
CityList locations;
PersonasList personas;
ForenamesByCountryMap forenames_by_country;
SurnamesByCountryMap surnames_by_country;
cache() = default;
~cache() = default;
};
CuratedDataFilePaths filepaths_;
cache cache_;
public:
explicit CuratedJsonDataService(CuratedDataFilePaths filepaths);
/**
* @brief Parses a JSON array file and returns all city records.
*/
const CityList& LoadCities() override;
/**
* @brief Parses a JSON array file and returns all persona records.
*/
const PersonasList& LoadPersonas() override;
const ForenamesByCountryMap& LoadForenamesByCountry() override;
/**
* @brief Parses a JSON file and returns all the forenames per country.
*/
const SurnamesByCountryMap& LoadSurnamesByCountry() override;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_

View File

@@ -0,0 +1,38 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_MOCK_CURATED_DATA_SERVICE_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_MOCK_CURATED_DATA_SERVICE_H_
/**
* @file services/curated_data/mock_curated_data_service.h
* @brief In-memory ICuratedDataService backed by a small fixed dataset, used
* when file-backed curated data is disabled (mock mode).
*/
#include <filesystem>
#include "data_model/models.h"
#include "services/curated_data/curated_data_service.h"
/**
* @brief Curated data service returning a small fixed in-memory dataset in
* place of the JSON fixture files used by JsonLoader.
*/
class MockCuratedDataService final : public ICuratedDataService {
public:
MockCuratedDataService();
const CityList& LoadCities() override;
const PersonasList& LoadPersonas() override;
const ForenamesByCountryMap& LoadForenamesByCountry() override;
const SurnamesByCountryMap& LoadSurnamesByCountry() override;
private:
CityList locations_;
PersonasList personas_;
ForenamesByCountryMap forenames_by_country_;
SurnamesByCountryMap surnames_by_country_;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_CURATED_DATA_MOCK_CURATED_DATA_SERVICE_H_

View File

@@ -1,5 +1,5 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_EXPORT_SERVICE_H_ #ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_EXPORT_SERVICE_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_EXPORT_SERVICE_H_ #define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_EXPORT_SERVICE_H_
/** /**
* @file services/export_service.h * @file services/export_service.h
@@ -8,7 +8,7 @@
#include <cstdint> #include <cstdint>
#include "data_model/generated_brewery.h" #include "data_model/generated_models.h"
/** /**
* @brief Interface for services that persist generated brewery records. * @brief Interface for services that persist generated brewery records.
@@ -16,8 +16,6 @@
class IExportService { class IExportService {
public: public:
IExportService() = default; IExportService() = default;
/// @brief Virtual destructor for polymorphic cleanup.
virtual ~IExportService() = default; virtual ~IExportService() = default;
IExportService(const IExportService&) = delete; IExportService(const IExportService&) = delete;
@@ -25,7 +23,9 @@ class IExportService {
IExportService(IExportService&&) = delete; IExportService(IExportService&&) = delete;
IExportService& operator=(IExportService&&) = delete; IExportService& operator=(IExportService&&) = delete;
/// @brief Prepares the export destination for a new run. /**
* @brief Prepares the export destination for a new run.
*/
virtual void Initialize() = 0; virtual void Initialize() = 0;
/** /**
@@ -33,10 +33,19 @@ class IExportService {
* *
* @param brewery Generated brewery payload to store. * @param brewery Generated brewery payload to store.
*/ */
virtual uint64_t ProcessRecord(const GeneratedBrewery& brewery) = 0; virtual uint64_t ProcessRecord(const BreweryRecord& brewery) = 0;
/// @brief Finalizes the export destination. /**
* @brief Persists one generated user record.
*
* @param user Generated user payload to store.
*/
virtual uint64_t ProcessRecord(const UserRecord& user) = 0;
/**
* @brief Finalizes the export destination.
*/
virtual void Finalize() = 0; virtual void Finalize() = 0;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_EXPORT_SERVICE_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_EXPORT_SERVICE_H_

View File

@@ -1,5 +1,5 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_SQLITE_CONNECTION_HELPERS_H_ #ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_SQLITE_CONNECTION_HELPERS_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_SQLITE_CONNECTION_HELPERS_H_ #define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_SQLITE_CONNECTION_HELPERS_H_
/** /**
* @file services/sqlite_connection_helpers.h * @file services/sqlite_connection_helpers.h
@@ -12,7 +12,7 @@
#include <string> #include <string>
#include <string_view> #include <string_view>
#include "services/sqlite_handle_types.h" #include "services/database/sqlite_handle_types.h"
namespace sqlite_export_service_internal { namespace sqlite_export_service_internal {
@@ -27,4 +27,4 @@ void RollbackTransactionNoThrow(const SqliteDatabaseHandle& db_handle) noexcept;
} // namespace sqlite_export_service_internal } // namespace sqlite_export_service_internal
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_SQLITE_CONNECTION_HELPERS_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_SQLITE_CONNECTION_HELPERS_H_

View File

@@ -1,5 +1,5 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_SQLITE_EXPORT_SERVICE_H_ #ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_SQLITE_EXPORT_SERVICE_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_SQLITE_EXPORT_SERVICE_H_ #define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_SQLITE_EXPORT_SERVICE_H_
/** /**
* @file services/sqlite_export_service.h * @file services/sqlite_export_service.h
@@ -11,10 +11,10 @@
#include <string> #include <string>
#include <unordered_map> #include <unordered_map>
#include "data_model/application_options.h" #include "data_model/models.h"
#include "services/date_time_provider.h" #include "services/database/export_service.h"
#include "services/export_service.h" #include "services/database/sqlite_export_service_helpers.h"
#include "services/sqlite_export_service_helpers.h" #include "services/datetime/date_time_provider.h"
/** /**
* @brief Persists generated brewery records into a fresh SQLite database. * @brief Persists generated brewery records into a fresh SQLite database.
@@ -30,7 +30,8 @@ class SqliteExportService final : public IExportService {
SqliteExportService& operator=(SqliteExportService&&) = delete; SqliteExportService& operator=(SqliteExportService&&) = delete;
void Initialize() override; void Initialize() override;
uint64_t ProcessRecord(const GeneratedBrewery& brewery) override; uint64_t ProcessRecord(const BreweryRecord& brewery) override;
uint64_t ProcessRecord(const UserRecord& user) override;
void Finalize() override; void Finalize() override;
private: private:
@@ -44,17 +45,29 @@ class SqliteExportService final : public IExportService {
void RollbackAndCloseNoThrow() noexcept; void RollbackAndCloseNoThrow() noexcept;
[[nodiscard]] std::filesystem::path BuildDatabasePath() const; [[nodiscard]] std::filesystem::path BuildDatabasePath() const;
[[nodiscard]] static std::string BuildLocationKey(const Location& location); [[nodiscard]] static std::string BuildCityKey(const City& location);
/**
* @brief Returns the row id for @p location, inserting it first if it has
* not already been seen during this run.
*
* Shared by both ProcessRecord() overloads so breweries and users
* referencing the same location resolve to the same row.
*/
[[nodiscard]] sqlite3_int64 ResolveCityId(const City& location);
std::unique_ptr<IDateTimeProvider> date_time_provider_; std::unique_ptr<IDateTimeProvider> date_time_provider_;
std::filesystem::path output_path_; std::filesystem::path output_path_;
std::string run_timestamp_utc_; std::string run_timestamp_utc_;
std::filesystem::path database_path_; std::filesystem::path database_path_;
SqliteDatabaseHandle db_handle_; SqliteDatabaseHandle db_handle_;
SqliteStatementHandle insert_location_stmt_; SqliteStatementHandle insert_city_stmt_;
SqliteStatementHandle insert_brewery_stmt_; SqliteStatementHandle insert_brewery_stmt_;
SqliteStatementHandle insert_brewery_address_stmt_;
SqliteStatementHandle insert_user_stmt_;
SqliteStatementHandle insert_user_address_stmt_;
bool transaction_open_ = false; bool transaction_open_ = false;
std::unordered_map<std::string, sqlite3_int64> location_cache_; std::unordered_map<std::string, sqlite3_int64> city_cache_;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_SQLITE_EXPORT_SERVICE_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_SQLITE_EXPORT_SERVICE_H_

View File

@@ -0,0 +1,10 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_SQLITE_EXPORT_SERVICE_HELPERS_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_SQLITE_EXPORT_SERVICE_HELPERS_H_
/* Umbrella header for backward compatibility. */
#include "services/database/sqlite_connection_helpers.h"
#include "services/database/sqlite_handle_types.h"
#include "services/database/sqlite_statement_helpers.h"
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_SQLITE_EXPORT_SERVICE_HELPERS_H_

View File

@@ -1,5 +1,5 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_SQLITE_HANDLE_TYPES_H_ #ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_SQLITE_HANDLE_TYPES_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_SQLITE_HANDLE_TYPES_H_ #define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_SQLITE_HANDLE_TYPES_H_
/** /**
* Shared handle and parameter type declarations used by SQLite helper units. * Shared handle and parameter type declarations used by SQLite helper units.
@@ -24,8 +24,10 @@ using SqliteDatabaseHandle = std::unique_ptr<sqlite3, SqliteDatabaseDeleter>;
using SqliteStatementHandle = using SqliteStatementHandle =
std::unique_ptr<sqlite3_stmt, SqliteStatementDeleter>; std::unique_ptr<sqlite3_stmt, SqliteStatementDeleter>;
// Represents a parameter that is bound to a prepared SQLite statement.
// N.B. indices are 1 based.
template <typename T> template <typename T>
struct BindParam { struct BoundParam {
int index; int index;
T value; T value;
std::string_view action; std::string_view action;
@@ -33,4 +35,4 @@ struct BindParam {
} // namespace sqlite_export_service_internal } // namespace sqlite_export_service_internal
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_SQLITE_HANDLE_TYPES_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_SQLITE_HANDLE_TYPES_H_

View File

@@ -0,0 +1,217 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_SQLITE_STATEMENT_HELPERS_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_SQLITE_STATEMENT_HELPERS_H_
/**
* @file services/sqlite_statement_helpers.h
* @brief Declarations for statement-level SQLite helper functions and
* constants.
*/
#include <sqlite3.h>
#include <string>
#include <string_view>
#include <vector>
#include "services/database/sqlite_handle_types.h"
namespace sqlite_export_service_internal {
inline constexpr std::string_view kCreateCitiesTableSql = R"sql(
CREATE TABLE IF NOT EXISTS cities (
id INTEGER PRIMARY KEY AUTOINCREMENT,
city TEXT NOT NULL,
state_province TEXT NOT NULL,
iso3166_2 TEXT NOT NULL,
country TEXT NOT NULL,
iso3166_1 TEXT NOT NULL,
local_languages_json TEXT NOT NULL,
UNIQUE(city, state_province, iso3166_2, country)
);
)sql";
inline constexpr std::string_view kCreateBreweriesTableSql = R"sql(
CREATE TABLE IF NOT EXISTS breweries (
id INTEGER PRIMARY KEY AUTOINCREMENT,
city_id INTEGER NOT NULL,
name_en TEXT NOT NULL,
description_en TEXT NOT NULL,
name_local TEXT NOT NULL,
description_local TEXT NOT NULL,
FOREIGN KEY(city_id) REFERENCES cities(id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_breweries_city_id ON breweries(city_id);
)sql";
inline constexpr std::string_view kCreateUsersTableSql = R"sql(
CREATE TABLE IF NOT EXISTS users (
id INTEGER PRIMARY KEY AUTOINCREMENT,
first_name TEXT NOT NULL,
last_name TEXT NOT NULL,
gender TEXT NOT NULL,
username TEXT NOT NULL,
bio TEXT NOT NULL,
activity_weight REAL NOT NULL,
email TEXT NOT NULL UNIQUE,
date_of_birth TEXT NOT NULL
);
)sql";
inline constexpr std::string_view kCreateBreweryAddressTableSql = R"sql(
CREATE TABLE IF NOT EXISTS brewery_addresses (
id INTEGER PRIMARY KEY AUTOINCREMENT,
brewery_id INTEGER NOT NULL,
address_line_1 TEXT,
address_line_2 TEXT,
postal_code TEXT NOT NULL,
city_id INTEGER NOT NULL,
FOREIGN KEY(brewery_id) REFERENCES breweries(id) ON DELETE CASCADE,
FOREIGN KEY(city_id) REFERENCES cities(id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_brewery_addresses_brewery_id
ON brewery_addresses(brewery_id);
CREATE INDEX IF NOT EXISTS idx_brewery_addresses_city_id
ON brewery_addresses(city_id);
)sql";
inline constexpr std::string_view kCreateUserAddressTableSql = R"sql(
CREATE TABLE IF NOT EXISTS user_addresses (
id INTEGER PRIMARY KEY AUTOINCREMENT,
user_id INTEGER NOT NULL,
address_line_1 TEXT,
address_line_2 TEXT,
postal_code TEXT,
city_id INTEGER NOT NULL,
FOREIGN KEY(user_id) REFERENCES users(id) ON DELETE CASCADE,
FOREIGN KEY(city_id) REFERENCES cities(id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_user_addresses_user_id
ON user_addresses(user_id);
CREATE INDEX IF NOT EXISTS idx_user_addresses_city_id
ON user_addresses(city_id);
)sql";
inline constexpr std::string_view kInsertCitySql = R"sql(
INSERT INTO cities (
city,
state_province,
iso3166_2,
country,
iso3166_1,
local_languages_json
) VALUES (?, ?, ?, ?, ?, ?);
)sql";
inline constexpr std::string_view kInsertBrewerySql = R"sql(
INSERT INTO breweries (
city_id,
name_en,
description_en,
name_local,
description_local
) VALUES (?, ?, ?, ?, ?);
)sql";
inline constexpr std::string_view kInsertUserSql = R"sql(
INSERT INTO users (
first_name,
last_name,
gender,
username,
bio,
activity_weight,
email,
date_of_birth
) VALUES (?, ?, ?, ?, ?, ?, ?, ?);
)sql";
inline constexpr std::string_view kInsertBreweryAddressSql = R"sql(
INSERT INTO brewery_addresses (
brewery_id,
postal_code,
city_id
) VALUES (?, ?, ?);
)sql";
inline constexpr std::string_view kInsertUserAddressSql = R"sql(
INSERT INTO user_addresses (
user_id,
city_id
) VALUES (?, ?);
)sql";
// sqlite3_bind_*() parameter indices are 1-based, matching the "?"
// placeholder order in the SQL above.
enum CityBindIndex {
kCityNameBindIndex = 1,
kCityStateProvinceBindIndex,
kCityIso31662BindIndex,
kCityCountryBindIndex,
kCityIso31661BindIndex,
kCityLanguagesBindIndex,
};
enum BreweryBindIndex {
kBreweryCityIdBindIndex = 1,
kBreweryEnglishNameBindIndex,
kBreweryEnglishDescriptionBindIndex,
kBreweryLocalNameBindIndex,
kBreweryLocalDescriptionBindIndex,
};
enum UserBindIndex {
kUserFirstNameBindIndex = 1,
kUserLastNameBindIndex,
kUserGenderBindIndex,
kUserUsernameBindIndex,
kUserBioBindIndex,
kUserActivityWeightBindIndex,
kUserEmailBindIndex,
kUserDateOfBirthBindIndex,
};
enum BreweryAddressBindIndex {
kBreweryAddressBreweryIdBindIndex = 1,
kBreweryAddressPostalCodeBindIndex,
kBreweryAddressCityIdBindIndex,
};
enum UserAddressBindIndex {
kUserAddressUserIdBindIndex = 1,
kUserAddressCityIdBindIndex,
};
SqliteStatementHandle PrepareStatement(const SqliteDatabaseHandle& db_handle,
std::string_view sql,
const char* action);
void ResetStatement(SqliteStatementHandle& statement);
void Bind(const SqliteStatementHandle& statement,
const BoundParam<std::string_view>& param);
void Bind(const SqliteStatementHandle& statement,
const BoundParam<double>& param);
void Bind(const SqliteStatementHandle& statement,
const BoundParam<sqlite3_int64>& param);
void StepStatement(const SqliteDatabaseHandle& db_handle,
const SqliteStatementHandle& statement,
std::string_view action);
sqlite3_int64 LastInsertRowId(const SqliteDatabaseHandle& db_handle);
std::string SerializeVector(const std::vector<std::string>& str_vec);
} // namespace sqlite_export_service_internal
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATABASE_SQLITE_STATEMENT_HELPERS_H_

View File

@@ -1,5 +1,5 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATE_TIME_PROVIDER_H_ #ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATETIME_DATE_TIME_PROVIDER_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATE_TIME_PROVIDER_H_ #define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATETIME_DATE_TIME_PROVIDER_H_
/** /**
* @file services/date_time_provider.h * @file services/date_time_provider.h
@@ -18,7 +18,6 @@
*/ */
class IDateTimeProvider { class IDateTimeProvider {
public: public:
/// @brief Virtual destructor for polymorphic cleanup.
virtual ~IDateTimeProvider() = default; virtual ~IDateTimeProvider() = default;
IDateTimeProvider() = default; IDateTimeProvider() = default;
@@ -63,4 +62,4 @@ class SystemDateTimeProvider final : public IDateTimeProvider {
} }
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATE_TIME_PROVIDER_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATETIME_DATE_TIME_PROVIDER_H_

View File

@@ -1,5 +1,5 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_TIMER_H_ #ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATETIME_TIMER_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_TIMER_H_ #define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATETIME_TIMER_H_
#include <chrono> #include <chrono>
@@ -32,4 +32,4 @@ class Timer {
} }
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_TIMER_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_DATETIME_TIMER_H_

View File

@@ -1,5 +1,5 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_SERVICE_H_ #ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_ENRICHMENT_SERVICE_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_SERVICE_H_ #define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_ENRICHMENT_SERVICE_H_
/** /**
* @file services/enrichment_service.h * @file services/enrichment_service.h
@@ -8,14 +8,13 @@
#include <string> #include <string>
#include "data_model/location.h" #include "data_model/models.h"
/** /**
* @brief Interface for services that can enrich a location with context. * @brief Interface for services that can enrich a location with context.
*/ */
class IEnrichmentService { class IEnrichmentService {
public: public:
/// @brief Virtual destructor for polymorphic cleanup.
virtual ~IEnrichmentService() = default; virtual ~IEnrichmentService() = default;
/** /**
@@ -24,7 +23,7 @@ class IEnrichmentService {
* @param loc Location to enrich. * @param loc Location to enrich.
* @return Context text, or an empty string if unavailable. * @return Context text, or an empty string if unavailable.
*/ */
virtual std::string GetLocationContext(const Location& loc) = 0; virtual std::string GetLocationContext(const City& loc) = 0;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_SERVICE_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_ENRICHMENT_SERVICE_H_

View File

@@ -0,0 +1,22 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_MOCK_ENRICHMENT_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_MOCK_ENRICHMENT_H_
/**
* @file services/enrichment/mock_enrichment.h
* @brief No-op IEnrichmentService used when network enrichment is disabled.
*/
#include <string>
#include "services/enrichment/enrichment_service.h"
/**
* @brief Enrichment service that returns no context for any location.
*/
class MockEnrichmentService final : public IEnrichmentService {
public:
std::string GetLocationContext(const City& /*loc*/) override {
return {};
}
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_MOCK_ENRICHMENT_H_

View File

@@ -0,0 +1,48 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_WIKIPEDIA_SERVICE_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_WIKIPEDIA_SERVICE_H_
/**
* @file services/wikipedia_service.h
* @brief Wikipedia summary retrieval service with in-memory caching.
*/
#include <memory>
#include <string>
#include <string_view>
#include <unordered_map>
#include "services/enrichment/enrichment_service.h"
#include "services/logging/logger.h"
#include "web_client/web_client.h"
/**
* @brief Provides Wikipedia summary lookups backed by cached raw extracts.
*/
class WikipediaEnrichmentService final : public IEnrichmentService {
public:
/**
* @brief Creates a new Wikipedia service with the provided web client.
*/
explicit WikipediaEnrichmentService(std::unique_ptr<WebClient> client,
std::shared_ptr<ILogger> logger);
/**
* @brief Returns the Wikipedia-derived context for a location.
*/
[[nodiscard]] std::string GetLocationContext(const City& loc) override;
private:
std::string FetchExtract(std::string_view query);
std::unique_ptr<WebClient> client_;
std::shared_ptr<ILogger> logger_;
/**
* @brief Cache for raw Wikipedia query extracts, keyed by query string.
*
* GetLocationContext() always queries "brewing" and reuses "beer in
* {country}" for every city in that country, so caching avoids refetching
* the same extract across locations in a run.
*/
std::unordered_map<std::string, std::string> extract_cache_;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_ENRICHMENT_WIKIPEDIA_SERVICE_H_

View File

@@ -0,0 +1,53 @@
/**
* @file services/logging/log_dispatcher.h
* @brief Dedicated log dispatcher for asynchronous pipeline logging.
*
* The dispatcher drains LogEntry values from a bounded channel and forwards
* them to spdlog on a dedicated thread.
*/
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_DISPATCHER_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_DISPATCHER_H_
#include <spdlog/spdlog.h>
#include "concurrency/bounded_channel.h"
#include "services/logging/log_entry.h"
/**
* @class LogDispatcher
* @brief Consumes log entries from a channel and forwards them to spdlog.
*
* Non-copyable and non-movable. Intended to run on its own dedicated thread
* and exit once the channel has been closed and drained.
*/
class LogDispatcher {
public:
/**
* @brief Construct a log dispatcher.
*
* @param channel Reference to the bounded channel used for log retrieval.
*/
explicit LogDispatcher(BoundedChannel<LogEntry>& channel);
LogDispatcher(const LogDispatcher&) = delete;
LogDispatcher& operator=(const LogDispatcher&) = delete;
LogDispatcher(LogDispatcher&&) = delete;
LogDispatcher& operator=(LogDispatcher&&) = delete;
~LogDispatcher() = default;
/**
* @brief Drain the channel and forward entries to spdlog.
*
* Intended to be called once on a dedicated thread. The loop returns after
* the channel has been closed and all queued entries have been processed.
*/
void Run();
private:
BoundedChannel<LogEntry>& channel_;
static spdlog::level::level_enum ToSpdlogLevel(LogLevel level);
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_DISPATCHER_H_

View File

@@ -0,0 +1,136 @@
/**
* @file services/logging/log_entry.h
* @brief Structured log record shared by the pipeline logging infra.
*
* LogEntry is a lightweight value type that can be passed safely between the
* logging producer and dispatcher through BoundedChannel<LogEntry>.
*/
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_ENTRY_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_ENTRY_H_
#include <chrono>
#include <source_location>
#include <string>
#include <thread>
#include <vector>
/**
* @enum LogLevel
* @brief Severity levels supported by the logging infra.
*/
enum class LogLevel {
/**
* @brief Development/debugging information.
*/
Debug,
/**
* @brief General informational messages.
*/
Info,
/**
* @brief Warning conditions.
*/
Warn,
/**
* @brief Error conditions.
*/
Error,
};
/**
* @enum PipelinePhase
* @brief Pipeline execution phases used to tag log records.
*
* The phase tag makes it easier to correlate log output with the part of the
* pipeline that emitted it.
*/
enum class PipelinePhase {
/**
* @brief Initialization and validation.
*/
Startup,
/**
* @brief Location/context enrichment (e.g. Wikipedia lookups).
*/
Enrichment,
/**
* @brief User profile generation.
*/
UserGeneration,
/**
* @brief Brewery and beer data generation.
*/
BreweryAndBeerGeneration,
/**
* @brief Checkin (visit) record generation.
*/
CheckinGeneration,
/**
* @brief Rating and review generation.
*/
RatingGeneration,
/**
* @brief Follow relationship generation.
*/
FollowGeneration,
/**
* @brief Finalization and cleanup.
*/
Teardown,
};
/**
* @struct LogDTO
* @brief User-provided subset of log fields. Used to capture call-site info
* transparently.
*/
struct LogDTO {
LogLevel level;
PipelinePhase phase;
std::string message;
};
/**
* @struct LogEntry
* @brief Single structured log event.
*
* All fields are value types, which keeps transfer across the bounded channel
* simple and avoids shared ownership.
*
* NOTE: timestamp, thread_id, and origin must be populated by ILogger::Log()
* before the entry is dispatched.
*/
struct LogEntry {
/**
* @brief Timestamp when the entry was created.
*/
std::chrono::system_clock::time_point timestamp{};
/**
* @brief Source location where the log call was made.
*/
std::source_location origin{};
/**
* @brief Thread responsible for emitting the log.
*/
std::thread::id thread_id{};
/**
* @brief Severity level of this entry.
*/
LogLevel level;
/**
* @brief Pipeline phase associated with the entry.
*/
PipelinePhase phase;
/**
* @brief Log message text.
*/
std::string message;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_ENTRY_H_

View File

@@ -0,0 +1,53 @@
/**
* @file services/logging/log_producer.h
* @brief Channel-backed log producer for asynchronous pipeline logging.
*
* The producer captures log records from application code and forwards them to
* a bounded channel for later processing by the dispatcher.
*/
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_PRODUCER_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_PRODUCER_H_
#include <string_view>
#include "concurrency/bounded_channel.h"
#include "services/logging/log_entry.h"
#include "services/logging/logger.h"
/**
* @class LogProducer
* @brief ILogger implementation that forwards entries to a bounded channel.
*
* Non-copyable and non-movable. The channel reference is non-owning and must
* remain valid for the lifetime of the producer.
*/
class LogProducer final : public ILogger {
public:
/**
* @brief Construct a channel-backed producer.
*
* @param channel Reference to the bounded channel used for log transfer.
*/
explicit LogProducer(BoundedChannel<LogEntry>& channel);
LogProducer(const LogProducer&) = delete;
LogProducer& operator=(const LogProducer&) = delete;
LogProducer(LogProducer&&) = delete;
LogProducer& operator=(LogProducer&&) = delete;
~LogProducer() override = default;
/**
* @brief Queue a log message for asynchronous processing.
*
* Blocks while the channel applies backpressure. This blocking behavior
* under heavy load is an accepted trade-off for simplicity.
*/
void DoLog(LogEntry log_entry) override;
private:
BoundedChannel<LogEntry>& channel_;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOG_PRODUCER_H_

View File

@@ -0,0 +1,65 @@
/**
* @file services/logging/logger.h
* @brief Abstract logging interface used by pipeline components.
*
* The interface keeps application code independent from the concrete logging
* transport, buffering, and formatting implementation.
*/
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOGGER_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOGGER_H_
#include <source_location>
#include <string>
#include <utility>
#include "services/logging/log_entry.h"
/**
* @class ILogger
* @brief Minimal interface for submitting structured log messages.
*
* Implementations are non-copyable and non-movable. They are typically owned
* by the composition root and injected into services that emit diagnostics.
*/
class ILogger {
public:
ILogger() = default;
ILogger(const ILogger&) = delete;
ILogger& operator=(const ILogger&) = delete;
ILogger(ILogger&&) = delete;
ILogger& operator=(ILogger&&) = delete;
virtual ~ILogger() = default;
/**
* @brief Submit a log message to the logging subsystem.
*
* @param payload User-provided log data (level, phase, message).
* @param origin Auto-captured source location of the call site.
*/
void Log(LogDTO payload,
std::source_location origin = std::source_location::current(),
std::chrono::system_clock::time_point timestamp =
std::chrono::system_clock::now(),
std::thread::id thread_id = std::this_thread::get_id()) {
LogEntry entry;
entry.timestamp = timestamp;
entry.thread_id = thread_id;
entry.level = payload.level;
entry.phase = payload.phase;
entry.message = std::move(payload.message);
entry.origin = origin;
DoLog(std::move(entry));
}
protected:
/**
* @brief Underlying implementation to transport the log entry.
*
* Implementations must be thread-safe as DoLog can be called concurrently
* from multiple worker threads.
*/
virtual void DoLog(LogEntry log_entry) = 0;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_LOGGING_LOGGER_H_

View File

@@ -0,0 +1,32 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_POSTAL_CODE_MOCK_POSTAL_CODE_SERVICE_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_POSTAL_CODE_MOCK_POSTAL_CODE_SERVICE_H_
/**
* @file services/postal_code/mock_postal_code_service.h
* @brief Placeholder IPostalCodeService used until xeger-based generation is
* implemented.
*/
#include <stdexcept>
#include <string>
#include "data_model/models.h"
#include "services/postal_code/postal_code_service.h"
/**
* @brief Postal code service that returns the location's first known
* example postal code rather than generating one from its regex.
*/
class MockPostalCodeService final : public IPostalCodeService {
public:
std::string GeneratePostalCode(const City& location) override {
if (location.postal_code_examples.empty()) {
throw std::runtime_error(
"MockPostalCodeService: location has no postal_code_examples: " +
location.city);
}
return location.postal_code_examples.front();
}
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_POSTAL_CODE_MOCK_POSTAL_CODE_SERVICE_H_

View File

@@ -0,0 +1,30 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_POSTAL_CODE_POSTAL_CODE_SERVICE_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_POSTAL_CODE_POSTAL_CODE_SERVICE_H_
/**
* @file services/postal_code/postal_code_service.h
* @brief Abstraction for generating a postal code for a location.
*/
#include <string>
#include "data_model/models.h"
/**
* @brief Interface for services that generate a postal code string matching
* a location's postal code format (`City::postal_regex`).
*/
class IPostalCodeService {
public:
virtual ~IPostalCodeService() = default;
/**
* @brief Generates a postal code for the given location.
*
* @param location Location whose postal code format to generate for.
* @return A postal code string matching `location.postal_regex`.
*/
virtual std::string GeneratePostalCode(const City& location) = 0;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_POSTAL_CODE_POSTAL_CODE_SERVICE_H_

View File

@@ -1,5 +1,5 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_PROMPT_DIRECTORY_H_ #ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_PROMPTING_PROMPT_DIRECTORY_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_PROMPT_DIRECTORY_H_ #define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_PROMPTING_PROMPT_DIRECTORY_H_
/** /**
* @file services/prompt_directory.h * @file services/prompt_directory.h
@@ -12,11 +12,14 @@
*/ */
#include <filesystem> #include <filesystem>
#include <memory>
#include <stdexcept> #include <stdexcept>
#include <string> #include <string>
#include <string_view> #include <string_view>
#include <unordered_map> #include <unordered_map>
#include "services/logging/logger.h"
/** /**
* @brief Interface for loading named prompt files. * @brief Interface for loading named prompt files.
*/ */
@@ -56,6 +59,8 @@ class PromptDirectory final : public IPromptDirectory {
* directory. * directory.
*/ */
explicit PromptDirectory(const std::filesystem::path& prompt_dir); explicit PromptDirectory(const std::filesystem::path& prompt_dir);
PromptDirectory(const std::filesystem::path& prompt_dir,
std::shared_ptr<ILogger> logger);
/** /**
* @brief Loads the prompt for @p key, caching the result. * @brief Loads the prompt for @p key, caching the result.
@@ -70,7 +75,8 @@ class PromptDirectory final : public IPromptDirectory {
private: private:
std::filesystem::path prompt_dir_; std::filesystem::path prompt_dir_;
std::shared_ptr<ILogger> logger_;
std::unordered_map<std::string, std::string> cache_; std::unordered_map<std::string, std::string> cache_;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_PROMPT_DIRECTORY_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_PROMPTING_PROMPT_DIRECTORY_H_

View File

@@ -1,10 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_SQLITE_EXPORT_SERVICE_HELPERS_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_SQLITE_EXPORT_SERVICE_HELPERS_H_
/* Umbrella header for backward compatibility. */
#include "services/sqlite_connection_helpers.h"
#include "services/sqlite_handle_types.h"
#include "services/sqlite_statement_helpers.h"
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_SQLITE_EXPORT_SERVICE_HELPERS_H_

View File

@@ -1,116 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_SQLITE_STATEMENT_HELPERS_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_SQLITE_STATEMENT_HELPERS_H_
/**
* @file services/sqlite_statement_helpers.h
* @brief Declarations for statement-level SQLite helper functions and
* constants.
*/
#include <sqlite3.h>
#include <string>
#include <string_view>
#include <vector>
#include "services/sqlite_handle_types.h"
namespace sqlite_export_service_internal {
inline constexpr std::string_view kCreateLocationsTableSql = R"sql(
CREATE TABLE IF NOT EXISTS locations (
id INTEGER PRIMARY KEY AUTOINCREMENT,
city TEXT NOT NULL,
state_province TEXT NOT NULL,
iso3166_2 TEXT NOT NULL,
country TEXT NOT NULL,
iso3166_1 TEXT NOT NULL,
local_languages_json TEXT NOT NULL,
latitude REAL NOT NULL,
longitude REAL NOT NULL,
UNIQUE(city, state_province, iso3166_2, country, latitude, longitude)
);
)sql";
inline constexpr std::string_view kCreateBreweriesTableSql = R"sql(
CREATE TABLE IF NOT EXISTS breweries (
id INTEGER PRIMARY KEY AUTOINCREMENT,
location_id INTEGER NOT NULL,
name_en TEXT NOT NULL,
description_en TEXT NOT NULL,
name_local TEXT NOT NULL,
description_local TEXT NOT NULL,
FOREIGN KEY(location_id) REFERENCES locations(id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_breweries_location_id ON breweries(location_id);
)sql";
inline constexpr std::string_view kInsertLocationSql = R"sql(
INSERT INTO locations (
city,
state_province,
iso3166_2,
country,
iso3166_1,
local_languages_json,
latitude,
longitude
) VALUES (?, ?, ?, ?, ?, ?, ?, ?);
)sql";
inline constexpr std::string_view kInsertBrewerySql = R"sql(
INSERT INTO breweries (
location_id,
name_en,
description_en,
name_local,
description_local
) VALUES (?, ?, ?, ?, ?);
)sql";
inline constexpr int kLocationCityBindIndex = 1;
inline constexpr int kLocationStateProvinceBindIndex = 2;
inline constexpr int kLocationIso31662BindIndex = 3;
inline constexpr int kLocationCountryBindIndex = 4;
inline constexpr int kLocationIso31661BindIndex = 5;
inline constexpr int kLocationLanguagesBindIndex = 6;
inline constexpr int kLocationLatitudeBindIndex = 7;
inline constexpr int kLocationLongitudeBindIndex = 8;
inline constexpr int kBreweryLocationIdBindIndex = 1;
inline constexpr int kBreweryEnglishNameBindIndex = 2;
inline constexpr int kBreweryEnglishDescriptionBindIndex = 3;
inline constexpr int kBreweryLocalNameBindIndex = 4;
inline constexpr int kBreweryLocalDescriptionBindIndex = 5;
SqliteStatementHandle PrepareStatement(const SqliteDatabaseHandle& db_handle,
std::string_view sql,
const char* action);
void ResetStatement(SqliteStatementHandle& statement);
void Bind(const SqliteStatementHandle& statement,
const BindParam<std::string_view>& param);
void Bind(const SqliteStatementHandle& statement,
const BindParam<double>& param);
void Bind(const SqliteStatementHandle& statement,
const BindParam<sqlite3_int64>& param);
void StepStatement(const SqliteDatabaseHandle& db_handle,
const SqliteStatementHandle& statement,
std::string_view action);
sqlite3_int64 LastInsertRowId(const SqliteDatabaseHandle& db_handle);
std::string SerializeVector(const std::vector<std::string>& str_vec);
} // namespace sqlite_export_service_internal
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_SQLITE_STATEMENT_HELPERS_H_

View File

@@ -1,33 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_SERVICES_WIKIPEDIA_SERVICE_H_
#define BIERGARTEN_PIPELINE_INCLUDES_SERVICES_WIKIPEDIA_SERVICE_H_
/**
* @file services/wikipedia_service.h
* @brief Wikipedia summary retrieval service with in-memory caching.
*/
#include <memory>
#include <string>
#include <string_view>
#include <unordered_map>
#include "services/enrichment_service.h"
#include "web_client/web_client.h"
/// @brief Provides Wikipedia summary lookups backed by cached raw extracts.
class WikipediaService final : public IEnrichmentService {
public:
/// @brief Creates a new Wikipedia service with the provided web client.
explicit WikipediaService(std::unique_ptr<WebClient> client);
/// @brief Returns the Wikipedia-derived context for a location.
[[nodiscard]] std::string GetLocationContext(const Location& loc) override;
private:
std::string FetchExtract(std::string_view query);
std::unique_ptr<WebClient> client_;
/// @brief Canonical cache for raw Wikipedia query extracts.
std::unordered_map<std::string, std::string> extract_cache_;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_SERVICES_WIKIPEDIA_SERVICE_H_

View File

@@ -1,54 +0,0 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_WEB_CLIENT_CURL_WEB_CLIENT_H_
#define BIERGARTEN_PIPELINE_INCLUDES_WEB_CLIENT_CURL_WEB_CLIENT_H_
/**
* @file web_client/curl_web_client.h
* @brief libcurl-based WebClient implementation.
*/
#include "web_client/web_client.h"
/**
* @brief RAII wrapper for curl_global_init and curl_global_cleanup.
*
* Create one instance in application startup before using libcurl and keep it
* alive for application lifetime.
*/
class CurlGlobalState {
public:
/// @brief Initializes global libcurl state.
CurlGlobalState();
/// @brief Cleans up global libcurl state.
~CurlGlobalState();
/// @brief Non-copyable type.
CurlGlobalState(const CurlGlobalState&) = delete;
/// @brief Non-copyable type.
CurlGlobalState& operator=(const CurlGlobalState&) = delete;
};
/**
* @brief WebClient implementation backed by libcurl.
*/
class CURLWebClient : public WebClient {
public:
/**
* @brief Executes an HTTP GET request.
*
* @param url Request URL.
* @return Response body.
*/
std::string Get(const std::string& url) override;
/**
* @brief URL-encodes a string value.
*
* @param value Raw value.
* @return URL-encoded string.
*/
std::string UrlEncode(const std::string& value) override;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_WEB_CLIENT_CURL_WEB_CLIENT_H_

View File

@@ -1,36 +1,40 @@
/** /**
* @file web_client/http_web_client.h * @file web_client/http_web_client.h
* @brief cpp-httplib implementation of the WebClient interface. * @brief cpp-httplib implementation of the WebClient interface.
*/ */
#ifndef BIERGARTEN_PIPELINE_INCLUDES_HTTP_WEB_CLIENT_CURL_WEB_CLIENT_H_ #ifndef BIERGARTEN_PIPELINE_INCLUDES_WEB_CLIENT_HTTP_WEB_CLIENT_H_
#define BIERGARTEN_PIPELINE_INCLUDES_HTTP_WEB_CLIENT_CURL_WEB_CLIENT_H_ #define BIERGARTEN_PIPELINE_INCLUDES_WEB_CLIENT_HTTP_WEB_CLIENT_H_
#include "web_client/web_client.h"
#include <memory>
#include <string> #include <string>
#include <utility>
#include "services/logging/logger.h"
#include "web_client/web_client.h"
/** /**
* @brief WebClient implementation backed by cpp-httplib. * @brief WebClient implementation backed by cpp-httplib.
* *
* Supports HTTP and HTTPS (requires * Supports HTTP and HTTPS (requires OpenSSL; see HTTPLIB_REQUIRE_OPENSSL
* OpenSSL; see HTTPLIB_USE_OPENSSL_IF_AVAILABLE in CMakeLists.txt). * in CMakeLists.txt).
* *
* URL parsing splits a full URL into scheme + host and path + query so that * URL parsing splits a full URL into origin (scheme://host[:port]) and
* httplib::Client can be constructed correctly. A new client instance is * path + query so that httplib::Client can be constructed correctly.
* created per request; th is is intentional given the low call volume in the * A new client instance is created per request because the client is
* pipeline (Wikipedia enrichment, near-100 % cache hits). * bound to a single origin at construction time.
*/ */
class HttpWebClient final : public WebClient { class HttpWebClient final : public WebClient {
public: public:
HttpWebClient() = default; explicit HttpWebClient(std::shared_ptr<ILogger> logger)
: logger_(std::move(logger)) {}
~HttpWebClient() override = default; ~HttpWebClient() override = default;
/** /**
* @brief Executes a blocking HTTP/HTTPS GET request against a full URL. * @brief Executes a blocking HTTP/HTTPS GET request against a full URL.
* *
* @param url Fully-qualified URL, e.g. "https://en.wikipedia.org/api/rest_v1/page/summary/Berlin" * @param url Fully-qualified URL, e.g.
* "https://en.wikipedia.org/api/rest_v1/page/summary/Berlin"
* @return Response body on HTTP 2xx; throws std::runtime_error otherwise. * @return Response body on HTTP 2xx; throws std::runtime_error otherwise.
*/ */
std::string Get(const std::string& url) override; std::string Get(const std::string& url) override;
@@ -42,8 +46,10 @@ public:
* @param value Raw string to encode. * @param value Raw string to encode.
* @return Percent-encoded string safe for use in a URL. * @return Percent-encoded string safe for use in a URL.
*/ */
std::string UrlEncode(const std::string& value) override; std::string EncodeURL(const std::string& value) override;
private:
std::shared_ptr<ILogger> logger_;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_WEB_CLIENT_HTTP_WEB_CLIENT_H_
#endif

View File

@@ -13,7 +13,6 @@
*/ */
class WebClient { class WebClient {
public: public:
/// @brief Virtual destructor for polymorphic cleanup.
virtual ~WebClient() = default; virtual ~WebClient() = default;
/** /**
@@ -30,7 +29,7 @@ class WebClient {
* @param value Raw string value. * @param value Raw string value.
* @return Encoded value safe for URL usage. * @return Encoded value safe for URL usage.
*/ */
virtual std::string UrlEncode(const std::string& value) = 0; virtual std::string EncodeURL(const std::string& value) = 0;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_WEB_CLIENT_WEB_CLIENT_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_WEB_CLIENT_WEB_CLIENT_H_

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,42 @@
[
{
"name": "Hophead Explorer",
"description": "Chases the newest hop varieties and double-digit IBUs, tracking hazy releases and limited tap takeovers across town.",
"style_affinities": ["American IPA", "Double IPA", "New England IPA", "Black IPA"]
},
{
"name": "Malt & Tradition Loyalist",
"description": "Prefers clean, balanced lagers brewed the way their region has brewed them for generations, and is quick to defend a well-made Helles.",
"style_affinities": ["Munich Helles", "Vienna Lager", "Märzen", "Festbier"]
},
{
"name": "Dark & Roasty Devotee",
"description": "Orders the darkest beer on the board year-round, drawn to roasted malt, coffee, and chocolate notes over carbonated lightness.",
"style_affinities": ["Imperial Stout", "Robust Porter", "Baltic Porter", "Oatmeal Stout"]
},
{
"name": "Sour & Wild Forager",
"description": "Seeks out barrel-aged, spontaneously fermented, and funky beers, and will travel for a fresh bottling of something wild.",
"style_affinities": ["Lambic", "Gueuze", "Flanders Red Ale", "Wild Ale"]
},
{
"name": "Session Sipper",
"description": "Values an easy-drinking pint that holds up over a long afternoon at the taproom more than raw intensity or novelty.",
"style_affinities": ["Session IPA", "Ordinary Bitter", "Cream Ale", "Blonde Ale"]
},
{
"name": "Belgian Abbey Pilgrim",
"description": "Collects abbey and trappist releases, drawn to the dried-fruit esters and warming strength of classic Belgian ales.",
"style_affinities": ["Belgian Tripel", "Belgian Dubbel", "Belgian Quadrupel", "Trappist Single"]
},
{
"name": "Wheat & Citrus Casual",
"description": "Reaches for something light, cloudy, and refreshing, usually with a citrus or coriander note, especially on a patio in summer.",
"style_affinities": ["Hefeweizen", "Witbier", "American Wheat Beer", "Berliner Weisse"]
},
{
"name": "Big & Boozy Connoisseur",
"description": "Sips slowly from a snifter, seeking out high-ABV releases meant for aging and savoring rather than quick refreshment.",
"style_affinities": ["English Barleywine", "Wee Heavy", "Doppelbock", "Old Ale"]
}
]

View File

@@ -0,0 +1,122 @@
# FULL SYSTEM PROMPT
You are a craft beer community profile writer. You write short, first-person
biographies for fictional users of a brewery-discovery app, grounded in the
exact name, locale, and persona provided. You do not invent the user's name --
it is given to you and must not be altered, translated, or abbreviated beyond
forming a username.
You will receive the inputs like this:
## NAME:
[First name] [Last name]
## GENDER:
[Gender associated with the given first name]
## CITY:
[City Name]
## COUNTRY:
[Country Name]
## PERSONA:
[Persona archetype name]
## PERSONA DESCRIPTION:
[What this persona cares about and how they talk about beer]
## STYLE AFFINITIES:
[Comma-separated beer styles this persona favors]
## CRITICAL OUTPUT FORMAT (READ CAREFULLY):
ABSOLUTELY NO MARKDOWN FORMATTING. Do NOT wrap your response in json or ```
blocks.
Do not add markdown, code fences, or postscript around the final JSON object.
Do not say "Here is the JSON" or "Enjoy!".
The JSON must contain exactly three keys ("username", "bio",
"activity_weight") in that order. Do not rename or add any other keys.
ESCAPE ALL QUOTES inside the bio field using \", or use single quotes (' ')
instead.
DO NOT use actual line breaks (\n) inside any string. Keep the bio as one
continuous string.
The bio must be between 25 and 60 words, written in first person, and must
read as something this specific person would write about themselves -- not a
generic profile blurb.
`activity_weight` is a number between 0 and 1 (inclusive) representing how
often this persona checks in at breweries relative to other users. Persona
archetypes implying frequent, habitual brewery visits should skew higher;
casual or occasional-visitor personas should skew lower. Do not default to
0.5 -- vary it meaningfully based on the persona description.
Expected JSON format:
```json
{
"username": "a handle derived from the given name",
"bio": "The first-person bio goes here.",
"activity_weight": 0.62
}
```
## CONTENT RULES AND CONSTRAINTS:
### THE USERNAME:
Derive the username from the given first and/or last name (e.g. lowercase,
abbreviated, or combined with a short word or number tied to the persona's
style affinities). Do not invent an unrelated handle. Do not include spaces.
### THE BIO:
Ground the bio in the supplied persona description and style affinities --
mention at least one specific beer style from the style affinities list, in a
way that sounds like a personal preference rather than a list. Reference the
city or country naturally if it fits, but do not force it.
### VOICE & PERSPECTIVE:
Write in the first person ("I"/"my"). The tone should match the persona:
an enthusiastic explorer sounds different from a relaxed, easy-drinking
regular. Do not write in second or third person.
### THE BLOCKLIST (FORBIDDEN CONCEPTS):
You absolutely cannot use the following words and phrases. Make sure your
final output doesn't have any of these:
- "hidden gem"
- "passion"
- "authentic"
- "craft beer enthusiast"
- "beer connoisseur"
- "journey"
#### FORBIDDEN WRITING PATTERNS
The following patterns are common AI writing pitfalls and must not appear in
the bio:
- Negative parallelism constructions: "It's not X, it's Y"
- Inflated significance phrases: "stands as a testament," "plays a vital
role," "deeply rooted"
- Superficial trailing analyses: sentences ending in -ing words that add
opinion without content
- Promotional travel-copy tone: "breathtaking," "must-visit," "vibrant"
- Overused conjunctive transitions used as sentence openers: "Moreover,"
"Furthermore," "In addition"
- Rule of three: do not consistently organise ideas or examples in triplets

8
tooling/pipeline/run.sh Normal file
View File

@@ -0,0 +1,8 @@
./biergarten-pipeline \
--model ../models/google_gemma-4-E4B-it-Q6_K.gguf \
--temperature 1.0 \
--top-p 0.95 \
--top-k 64 \
--n-ctx 8192 \
--location-count 15 \
--prompt-dir ../prompts

View File

@@ -0,0 +1,9 @@
# Ignore model files!
*.gguf
*.bin
models/
weights/
# Ignore local build folders
build/
.git/

View File

@@ -0,0 +1,72 @@
# --- Stage 1: Build Environment (The "Heavy" Stage) ---
FROM nvidia/cuda:12.6.3-devel-ubuntu24.04 AS builder
ENV DEBIAN_FRONTEND=noninteractive \
CMAKE_GENERATOR=Ninja
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential ca-certificates curl git libboost-json-dev \
libboost-program-options-dev libssl-dev ninja-build pkg-config zlib1g-dev \
&& rm -rf /var/lib/apt/lists/*
# Install modern CMake
RUN curl -L https://github.com/Kitware/CMake/releases/download/v3.31.0/cmake-3.31.0-linux-x86_64.sh -o cmake.sh && \
sh cmake.sh --skip-license --prefix=/usr/local && rm cmake.sh
# Get headers for C++ build
RUN curl -L https://github.com/ggml-org/llama.cpp/archive/refs/tags/b9012.tar.gz -o /tmp/llama-src.tar.gz && \
tar -xzf /tmp/llama-src.tar.gz -C /tmp && \
cp -r /tmp/llama.cpp-b9012/include/* /usr/local/include/ && \
cp -r /tmp/llama.cpp-b9012/ggml/include/* /usr/local/include/
# Pull llama.cpp binaries to use during build if needed
COPY --from=ghcr.io/ggml-org/llama.cpp:full-cuda /app/lib*.so* /usr/local/lib/
WORKDIR /app
COPY . .
# Build the C++ pipeline
RUN cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release && \
cmake --build build -j$(nproc)
# --- Stage 2: Runtime Environment (The "Slim" Stage) ---
FROM nvidia/cuda:12.6.3-runtime-ubuntu24.04 AS runtime
# Install only necessary runtime shared libraries
RUN apt-get update && apt-get install -y --no-install-recommends \
curl \
ca-certificates \
libboost-json1.83.0 \
libboost-program-options1.83.0 \
libgomp1 \
libssl3 \
zlib1g \
&& rm -rf /var/lib/apt/lists/*
ENV APP_ROOT=/app \
LD_LIBRARY_PATH="/usr/local/lib:${LD_LIBRARY_PATH}"
WORKDIR /app/build
# Copy only the compiled binaries from the builder
COPY --from=builder /app/build/biergarten-pipeline ./
# Copy required config files
COPY cities.json /app/build/
COPY beer-styles.json /app/build/
# Copy prompt templates
COPY prompts /app/prompts
# Copy only the necessary shared libraries from builder/llama-bin
COPY --from=ghcr.io/ggml-org/llama.cpp:full-cuda /app/lib*.so* /usr/local/lib/
# Co-locate plugins
RUN cp /usr/local/lib/libggml-cuda.so . 2>/dev/null || true && \
cp /usr/local/lib/libggml-cpu*.so . 2>/dev/null || true
# Setup Start Script
COPY ./runpod/start.sh /usr/local/bin/biergarten-start
RUN chmod +x /usr/local/bin/biergarten-start
ENTRYPOINT ["/usr/local/bin/biergarten-start"]

View File

@@ -0,0 +1,8 @@
```bash
touch runpod/start.sh
docker build \
--progress=plain \
-t biergarten-pipeline:latest \
-f runpod/Dockerfile \
. 2>&1 | tee build.log
```

View File

@@ -0,0 +1,21 @@
name: biergarten-pipeline-live
imageName: biergarten-pipeline:latest
category: NVIDIA
containerDiskInGb: 50
volumeInGb: 50
volumeMountPath: /workspace
dockerEntrypoint:
- /usr/local/bin/biergarten-start
dockerStartCmd: []
isPublic: false
isServerless: false
env:
BIERGARTEN_MODEL_PATH: /workspace/models/google_gemma-4-E4B-it-Q6_K.gguf
BIERGARTEN_OUTPUT_DIR: /workspace/output
BIERGARTEN_LOG_PATH: /workspace/logs/pipeline.log
BIERGARTEN_GL_LAYERS: "40"
BIERGARTEN_TEMPERATURE: "1.0"
BIERGARTEN_TOP_P: "0.95"
BIERGARTEN_TOP_K: "64"
BIERGARTEN_N_CTX: "8192"
BIERGARTEN_SEED: "-1"

View File

@@ -0,0 +1,58 @@
#!/bin/bash
set -e
MODEL_PATH="${BIERGARTEN_MODEL_PATH:-/workspace/models/google_gemma-4-E4B-it-Q6_K.gguf}"
OUTPUT_DIR="${BIERGARTEN_OUTPUT_DIR:-/workspace/output}"
LOG_PATH="${BIERGARTEN_LOG_PATH:-/workspace/logs/pipeline.log}"
EXECUTABLE="/app/build/biergarten-pipeline"
PROMPT_DIR="/app/prompts"
echo "--- Starting Biergarten Pipeline Environment Check ---"
# Ensure directories exist
mkdir -p "$OUTPUT_DIR"
mkdir -p "$(dirname "$LOG_PATH")"
mkdir -p "$(dirname "$MODEL_PATH")"
# Download model if missing
if [ ! -f "$MODEL_PATH" ]; then
echo "Model not found. Downloading (this may take a while)..."
curl -L -C - \
-o "$MODEL_PATH" \
"https://huggingface.co/bartowski/google_gemma-4-E4B-it-GGUF/resolve/main/google_gemma-4-E4B-it-Q6_K.gguf?download=true"
echo "Download complete."
fi
# Verify model exists
if [ ! -f "$MODEL_PATH" ]; then
echo "ERROR: Model still not found after download attempt."
exit 1
fi
# Default GPU layers
GL_LAYERS="${BIERGARTEN_GL_LAYERS:-40}"
# Build args
ARGS=(
"--model" "$MODEL_PATH"
"--prompt-dir" "$PROMPT_DIR"
"--output" "$OUTPUT_DIR"
"--log-path" "$LOG_PATH"
"--n-gpu-layers" "$GL_LAYERS"
)
# Optional params
[[ -n "$BIERGARTEN_TEMPERATURE" ]] && ARGS+=("--temperature" "$BIERGARTEN_TEMPERATURE")
[[ -n "$BIERGARTEN_TOP_P" ]] && ARGS+=("--top-p" "$BIERGARTEN_TOP_P")
[[ -n "$BIERGARTEN_TOP_K" ]] && ARGS+=("--top-k" "$BIERGARTEN_TOP_K")
[[ -n "$BIERGARTEN_N_CTX" ]] && ARGS+=("--n-ctx" "$BIERGARTEN_N_CTX")
[[ -n "$BIERGARTEN_SEED" ]] && ARGS+=("--seed" "$BIERGARTEN_SEED")
# Extra args
[[ -n "$BIERGARTEN_EXTRA_ARGS" ]] && ARGS+=($BIERGARTEN_EXTRA_ARGS)
echo "--- Executing: $EXECUTABLE ${ARGS[*]} ---"
exec "$EXECUTABLE" "${ARGS[@]}"

View File

@@ -1,19 +1,20 @@
#include <spdlog/spdlog.h> #include <chrono>
#include <format>
#include <iostream>
#include <optional> #include <optional>
#include <sstream> #include <sstream>
#include <string> #include <string>
#include "data_model/application_options.h" #include "data_model/models.h"
#include "services/logging/logger.h"
std::optional<ApplicationOptions> ParseArguments(const int argc, char** argv) { std::optional<ApplicationOptions> ParseArguments(
const int argc, char** argv, std::shared_ptr<ILogger> logger) {
prog_opts::options_description desc("Pipeline Options"); prog_opts::options_description desc("Pipeline Options");
auto opt = desc.add_options(); auto opt = desc.add_options();
opt("help,h", "Produce help message"); opt("help,h", "Produce help message");
// Defaults sourced from SamplingOptions{} so the CLI and LlamaGenerator
// share a single source of truth — changing the struct updates both.
auto add_sampling_options = [&]() -> void { auto add_sampling_options = [&]() -> void {
const SamplingOptions sampling_defaults{}; const SamplingOptions sampling_defaults{};
opt("temperature", opt("temperature",
@@ -30,6 +31,8 @@ std::optional<ApplicationOptions> ParseArguments(const int argc, char** argv) {
"Context window size in tokens"); "Context window size in tokens");
opt("seed", prog_opts::value<int>()->default_value(sampling_defaults.seed), opt("seed", prog_opts::value<int>()->default_value(sampling_defaults.seed),
"Sampler seed: -1 for random, otherwise non-negative integer"); "Sampler seed: -1 for random, otherwise non-negative integer");
opt("n-gpu-layers", prog_opts::value<int>()->default_value(0),
"Number of layers to offload to GPU");
}; };
// --mocked and --model are mutually exclusive; validation is enforced below // --mocked and --model are mutually exclusive; validation is enforced below
@@ -48,8 +51,10 @@ std::optional<ApplicationOptions> ParseArguments(const int argc, char** argv) {
prog_opts::value<std::string>()->default_value("pipeline.log"), prog_opts::value<std::string>()->default_value("pipeline.log"),
"Path for application logs"); "Path for application logs");
opt("prompt-dir", prog_opts::value<std::string>()->default_value(""), opt("prompt-dir", prog_opts::value<std::string>()->default_value(""),
"Directory containing named prompt files (e.g. BREWERY_GENERATION.md)." "Directory containing named prompt files (e.g. "
"BREWERY_GENERATION.md)."
" Required when not using --mocked."); " Required when not using --mocked.");
opt("location-count", prog_opts::value<uint32_t>()->default_value(10));
}; };
add_sampling_options(); add_sampling_options();
@@ -58,10 +63,20 @@ std::optional<ApplicationOptions> ParseArguments(const int argc, char** argv) {
// No flags provided — treat as a help request rather than an error. // No flags provided — treat as a help request rather than an error.
if (argc == 1) { if (argc == 1) {
spdlog::info("Biergarten Pipeline"); const std::string title = "Biergarten Pipeline";
const std::string usage = ([&] {
std::stringstream usage_stream; std::stringstream usage_stream;
usage_stream << "\nUsage: biergarten-pipeline [options]\n\n" << desc; usage_stream << "\nUsage: biergarten-pipeline [options]\n\n" << desc;
spdlog::info(usage_stream.str()); return usage_stream.str();
})();
if (logger) {
logger->Log(LogDTO{.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = title});
logger->Log(LogDTO{.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = usage});
}
return std::nullopt; return std::nullopt;
} }
@@ -73,7 +88,11 @@ std::optional<ApplicationOptions> ParseArguments(const int argc, char** argv) {
if (var_map.contains("help")) { if (var_map.contains("help")) {
std::stringstream help_stream; std::stringstream help_stream;
help_stream << "\n" << desc; help_stream << "\n" << desc;
spdlog::info(help_stream.str()); if (logger) {
logger->Log(LogDTO{.level = LogLevel::Info,
.phase = PipelinePhase::Startup,
.message = help_stream.str()});
}
return std::nullopt; return std::nullopt;
} }
@@ -82,29 +101,55 @@ std::optional<ApplicationOptions> ParseArguments(const int argc, char** argv) {
options.pipeline.output_path = var_map["output"].as<std::string>(); options.pipeline.output_path = var_map["output"].as<std::string>();
options.pipeline.log_path = var_map["log-path"].as<std::string>(); options.pipeline.log_path = var_map["log-path"].as<std::string>();
options.pipeline.prompt_dir = var_map["prompt-dir"].as<std::string>(); options.pipeline.prompt_dir = var_map["prompt-dir"].as<std::string>();
options.pipeline.location_count = var_map["location-count"].as<uint32_t>();
const bool use_mocked = var_map["mocked"].as<bool>(); const bool use_mocked = var_map["mocked"].as<bool>();
const std::string model_path = var_map["model"].as<std::string>(); const std::string model_path = var_map["model"].as<std::string>();
const int n_gpu_layers = var_map["n-gpu-layers"].as<int>();
// Enforce mutual exclusivity before any further configuration is applied. // Enforce mutual exclusivity before any further configuration is
// applied.
if (use_mocked && !model_path.empty()) { if (use_mocked && !model_path.empty()) {
spdlog::error( const std::string msg =
"Invalid arguments: --mocked and --model are mutually exclusive"); "Invalid arguments: --mocked and --model are mutually "
"exclusive";
if (logger) {
logger->Log(LogDTO{.level = LogLevel::Error,
.phase = PipelinePhase::Startup,
.message = msg});
} else {
std::cerr << msg << std::endl;
}
return std::nullopt; return std::nullopt;
} }
if (!use_mocked && model_path.empty()) { if (!use_mocked && model_path.empty()) {
spdlog::error( const std::string msg =
"Invalid arguments: either --mocked or --model must be specified"); "Invalid arguments: either --mocked or --model must be "
"specified";
if (logger) {
logger->Log(LogDTO{.level = LogLevel::Error,
.phase = PipelinePhase::Startup,
.message = msg});
} else {
std::cerr << msg << std::endl;
}
return std::nullopt; return std::nullopt;
} }
// Prompt directory is only meaningful for live inference — the mock // Prompt directory is only meaningful for live inference — the mock
// generator has no use for it and should not require it to be present. // generator has no use for it and should not require it to be present.
if (!use_mocked && options.pipeline.prompt_dir.empty()) { if (!use_mocked && options.pipeline.prompt_dir.empty()) {
spdlog::error( const std::string msg =
"Invalid arguments: --prompt-dir is required when not using " "Invalid arguments: --prompt-dir is required when not using "
"--mocked"); "--mocked";
if (logger) {
logger->Log({.level = LogLevel::Error,
.phase = PipelinePhase::Startup,
.message = msg});
} else {
std::cerr << msg << std::endl;
}
return std::nullopt; return std::nullopt;
} }
@@ -118,13 +163,21 @@ std::optional<ApplicationOptions> ParseArguments(const int argc, char** argv) {
const bool user_provided_sampling = const bool user_provided_sampling =
!var_map["temperature"].defaulted() || !var_map["top-p"].defaulted() || !var_map["temperature"].defaulted() || !var_map["top-p"].defaulted() ||
!var_map["top-k"].defaulted() || !var_map["n-ctx"].defaulted() || !var_map["top-k"].defaulted() || !var_map["n-ctx"].defaulted() ||
!var_map["seed"].defaulted(); !var_map["seed"].defaulted() || !var_map["n_gpu_layers"].defaulted();
if (user_provided_sampling) { if (user_provided_sampling) {
// Warn but do not fail — the run is still valid, the flags are just // Warn but do not fail — the run is still valid, the flags are just
// silently irrelevant when no model is loaded. // silently irrelevant when no model is loaded.
if (use_mocked) { if (use_mocked) {
spdlog::warn("Sampling parameters are ignored when using --mocked"); const std::string msg =
"Sampling parameters are ignored when using --mocked";
if (logger) {
logger->Log(LogDTO{.level = LogLevel::Warn,
.phase = PipelinePhase::Startup,
.message = msg});
} else {
std::cerr << msg << std::endl;
}
} else { } else {
SamplingOptions sampling; SamplingOptions sampling;
sampling.temperature = var_map["temperature"].as<float>(); sampling.temperature = var_map["temperature"].as<float>();
@@ -132,6 +185,7 @@ std::optional<ApplicationOptions> ParseArguments(const int argc, char** argv) {
sampling.top_k = var_map["top-k"].as<uint32_t>(); sampling.top_k = var_map["top-k"].as<uint32_t>();
sampling.n_ctx = var_map["n-ctx"].as<uint32_t>(); sampling.n_ctx = var_map["n-ctx"].as<uint32_t>();
sampling.seed = var_map["seed"].as<int>(); sampling.seed = var_map["seed"].as<int>();
sampling.n_gpu_layers = var_map["n-gpu-layers"].as<int>();
options.generator.sampling = sampling; options.generator.sampling = sampling;
} }
@@ -140,11 +194,23 @@ std::optional<ApplicationOptions> ParseArguments(const int argc, char** argv) {
return options; return options;
} catch (const std::exception& exception) { } catch (const std::exception& exception) {
spdlog::error("Failed to parse command-line arguments: {}", const std::string msg =
exception.what()); std::string("Failed to parse command-line arguments: ") +
exception.what();
if (logger) {
logger->Log(LogDTO{.level = LogLevel::Error,
.phase = PipelinePhase::Startup,
.message = msg});
}
return std::nullopt; return std::nullopt;
} catch (...) { } catch (...) {
spdlog::error("Failed to parse command-line arguments: unknown error"); const std::string msg =
"Failed to parse command-line arguments: unknown error";
if (logger) {
logger->Log(LogDTO{.level = LogLevel::Error,
.phase = PipelinePhase::Startup,
.message = msg});
}
return std::nullopt; return std::nullopt;
} }
} }

View File

@@ -1,16 +0,0 @@
/**
* @file biergarten_data_generator/biergarten_data_generator.cc
* @brief BiergartenDataGenerator constructor implementation.
*/
#include "biergarten_data_generator.h"
#include <utility>
BiergartenDataGenerator::BiergartenDataGenerator(
std::unique_ptr<IEnrichmentService> context_service,
std::unique_ptr<DataGenerator> generator,
std::unique_ptr<IExportService> exporter)
: context_service_(std::move(context_service)),
generator_(std::move(generator)),
exporter_(std::move(exporter)) {}

Some files were not shown because too many files have changed in this diff Show More