2 Commits

Author SHA1 Message Date
Aaron Po
cbadd3bb59 Persist generated users to SQLite and code cleanup 2026-06-21 12:30:58 -04:00
Aaron Po
e3574d56a8 Add user generation feature 2026-06-21 11:59:55 -04:00
44 changed files with 53977 additions and 170 deletions

View File

@@ -13,6 +13,7 @@ low-resource language failures.
- [Model Bias and Language Quality](#model-bias-and-language-quality) - [Model Bias and Language Quality](#model-bias-and-language-quality)
- [Western and Eurocentric Lens](#western-and-eurocentric-lens) - [Western and Eurocentric Lens](#western-and-eurocentric-lens)
- [Wikipedia Enrichment](#wikipedia-enrichment) - [Wikipedia Enrichment](#wikipedia-enrichment)
- [Names-by-Country Dataset](#names-by-country-dataset)
- [The "Avoid AI Phrases" Prompt Instruction](#the-avoid-ai-phrases-prompt-instruction) - [The "Avoid AI Phrases" Prompt Instruction](#the-avoid-ai-phrases-prompt-instruction)
- [Known Issues](#known-issues) - [Known Issues](#known-issues)
- [Hallucinated Brewing Techniques](#hallucinated-brewing-techniques) - [Hallucinated Brewing Techniques](#hallucinated-brewing-techniques)
@@ -91,6 +92,33 @@ generated descriptions.
--- ---
## Names-by-Country Dataset
`tooling/pipeline/forenames-by-country.json` and `surnames-by-country.json`
(used to sample a `Name` per ISO 3166-1 country code for user generation) are
vendored verbatim, unmodified, from
[sigpwned/popular-names-by-country-dataset](https://github.com/sigpwned/popular-names-by-country-dataset)
(the `common-forenames-by-country.json` / `common-surnames-by-country.json`
release assets), released under **CC0** (public domain). That dataset's own
forename/surname lists are pulled from Wikipedia's "Lists of most common
surnames" and "List of most popular given names" as of the week of
2023-07-08 — see that project's README for full provenance. Names are not
LLM-generated; this is curated fixture data per ROADMAP.md §2. Per-forename
gender from the source data is preserved through to the sampled `Name`
(rather than discarded during loading) so it's available for gender-aware
persona/bio generation later.
The full multinational dataset is kept as-is (106 countries for forenames,
75 for surnames) rather than trimmed to `locations.json`'s current country
list, so it doesn't need re-sourcing if more countries are added later.
`NamesByCountry::SampleName()` returns no result for a country present in
neither file; of the countries in `locations.json`, that's currently `KE`,
`SE`, `SG`, `TH`, `VN`, and `ZA``GenerateUsers` skips cities in those
countries the same way brewery generation skips cities whose enrichment
lookup fails.
---
## The "Avoid AI Phrases" Prompt Instruction ## The "Avoid AI Phrases" Prompt Instruction
The system prompt instructs the model to avoid common AI-generated phrasing The system prompt instructs the model to avoid common AI-generated phrasing

View File

@@ -33,28 +33,60 @@ architecture needs.
- [ ] Add `BeerResult`, `CheckinResult`, `RatingResult` result payloads. - [ ] Add `BeerResult`, `CheckinResult`, `RatingResult` result payloads.
- [ ] Add `GenerationMetadata` (`generation_id`, `generated_time`, - [ ] Add `GenerationMetadata` (`generation_id`, `generated_time`,
`context_provided`, `generated_with`). `context_provided`, `generated_with`).
- [ ] Add `activity_weight` to `UserResult` (currently just `username`, - [x] Add `first_name`, `last_name`, `gender`, and `activity_weight` to
`bio`). `UserResult` (currently just `username`, `bio`). `first_name`/
`last_name`/`gender` are copied from the sampled `Name` (see below),
not LLM-invented.
- [x] Add `Name` (`first_name`, `last_name`, `gender`) — the sampled result
handed to `DataGenerator::GenerateUser`. Add `ForenameEntry` (`name`,
`gender`) and a `NamesByCountry` class (not a bare typedef) that owns
two maps — `std::unordered_map<std::string, std::vector<ForenameEntry>>`
and `std::unordered_map<std::string, std::vector<std::string>>`, both
keyed by ISO 3166-1 code — and exposes `SampleName(iso3166_1, rng)`.
Pairing a forename with a surname happens at sample time so gender
(present per-forename in the source data) is never discarded the way a
pre-flattened `{first_name, last_name}` list would lose it; see
`LoadNamesByCountry()` below.
- [ ] Extend `GeneratedBrewery` with `brewery_id`, `context_completeness`, - [ ] Extend `GeneratedBrewery` with `brewery_id`, `context_completeness`,
`metadata` (currently just `{location, brewery}`). `metadata` (currently just `{location, brewery}`).
- [ ] Add `GeneratedBeer`, `GeneratedUser`, `GeneratedCheckin`, - [ ] Add `GeneratedBeer`, `GeneratedCheckin`, `GeneratedRating`,
`GeneratedRating`, `GeneratedFollow` aggregate structs. `GeneratedFollow` aggregate structs.
- [ ] Add `UserPersona` (`name`, `description`, `style_affinities`). - [x] Add `GeneratedUser` aggregate struct. Carries `email`, `date_of_birth`,
and `password` — programmatically generated by the orchestrator, never
LLM-authored — so a downstream auth-account seeding consumer (outside
this repo tree's docs) can register real accounts from the pipeline's
SQLite export. Skips `user_id`/`metadata` for now, matching
`GeneratedBrewery`'s current (pre-`brewery_id`/`metadata`) shape. Not
yet wired into `IExportService`/`SqliteExportService``GenerateUsers`
currently only logs and holds `generated_users_` in memory; no `users`
table exists in the SQLite export yet (see §5).
- [x] Add `UserPersona` (`name`, `description`, `style_affinities`).
## 2. Data Preloading ## 2. Data Preloading
`tooling/pipeline/includes/json_handling/`, fixture files. `tooling/pipeline/includes/json_handling/`, fixture files.
- [ ] Extract a `DataPreloader` interface; have `JsonLoader` implement it - [ ] Extract a `DataPreloader` interface; have `JsonLoader` implement it.
instead of exposing only a static `LoadLocations()`. `JsonLoader` now also exposes static `LoadPersonas()` and
`LoadNamesByCountry()` (see below), but as additional static methods,
not yet behind a formal interface.
- [ ] Add `LoadBeerStyles()`. `beer-styles.json` already exists in the repo - [ ] Add `LoadBeerStyles()`. `beer-styles.json` already exists in the repo
and is already copied into the Docker image and is already copied into the Docker image
(`runpod/Dockerfile`), but no loader reads it yet, and the native (`runpod/Dockerfile`), but no loader reads it yet, and the native
CMake build doesn't copy it into `build/` at all — `CMakeLists.txt`'s CMake build doesn't copy it into `build/` at all — `CMakeLists.txt`'s
"Runtime Assets" step only copies `locations.json` and `prompts/`. "Runtime Assets" step only copies `locations.json` and `prompts/`.
- [ ] Add `LoadPersonas()` and author `personas.json` (doesn't exist yet). - [x] Add `LoadPersonas()` and author `personas.json` (doesn't exist yet).
- [ ] Add `LoadNamesByCountry()` and author `names-by-country.json` (doesn't - [x] Add `LoadNamesByCountry(forenames_filepath, surnames_filepath)`,
exist yet). parsing `tooling/pipeline/forenames-by-country.json` and
`surnames-by-country.json` into a `NamesByCountry`. Both files are
vendored verbatim (unmodified, full multinational coverage) from
`sigpwned/popular-names-by-country-dataset` (CC0) — see
ETHICS-AND-KNOWN-ISSUES.md's Names-by-Country Dataset section for
provenance. Deliberately not pre-paired or filtered down to just the
countries in `locations.json`: keeping the source shape (including
per-forename gender) intact means the loader can support more
countries later, or gender-aware persona/bio generation, without
re-sourcing data.
## 3. Policy / Strategy Layer ## 3. Policy / Strategy Layer
@@ -83,10 +115,14 @@ Entirely new — no `includes/policy/` (or equivalent) directory exists today.
- [ ] Extend the `DataGenerator` interface with `GenerateBeer`, - [ ] Extend the `DataGenerator` interface with `GenerateBeer`,
`GenerateCheckin`, `GenerateRating` (today: only `GenerateBrewery` and `GenerateCheckin`, `GenerateRating` (today: only `GenerateBrewery` and
`GenerateUser`). `GenerateUser`).
- [ ] Implement `LlamaGenerator::GenerateUser` for real. It currently - [x] Implement `LlamaGenerator::GenerateUser` for real, with a retry loop
returns a hardcoded `{"test_user", "This is a test user profile from mirroring `GenerateBrewery` (GBNF grammar, `ValidateUserJson`, up to 3
{locale}."}` regardless of input — see the `// TODO` at the top of attempts with corrective feedback) and a new
`src/data_generation/llama/generate_user.cc`. `prompts/USER_GENERATION.md`. Changed the `DataGenerator::GenerateUser`
signature to
`(city : const EnrichedCity&, persona : const UserPersona&, name : const Name&) : UserResult`,
matching what the activity diagram actually passes. `MockGenerator::GenerateUser`
updated to match the new signature too.
- [ ] Implement `MockGenerator::GenerateBeer` / `GenerateCheckin` / - [ ] Implement `MockGenerator::GenerateBeer` / `GenerateCheckin` /
`GenerateRating`. `GenerateRating`.
- [ ] Add `IPromptFormatter::ExpectedArchitecture()` and - [ ] Add `IPromptFormatter::ExpectedArchitecture()` and

View File

@@ -43,7 +43,7 @@ fork again
:EnrichmentService::PreWarmLocationCache(sampled_locations); :EnrichmentService::PreWarmLocationCache(sampled_locations);
end fork end fork
fork fork
:JsonLoader::LoadNamesByCountry("names-by-country.json"); :JsonLoader::LoadNamesByCountry(\n "forenames-by-country.json",\n "surnames-by-country.json");
fork again fork again
:JsonLoader::LoadPersonas("personas.json"); :JsonLoader::LoadPersonas("personas.json");
end fork end fork

View File

@@ -67,11 +67,47 @@ package "Domain: Models" {
} }
class UserResult { class UserResult {
+ first_name : std::string
+ last_name : std::string
+ gender : std::string
+ username : std::string + username : std::string
+ bio : std::string + bio : std::string
+ activity_weight : float + activity_weight : float
} }
class Name {
+ first_name : std::string
+ last_name : std::string
+ gender : std::string
}
class ForenameEntry {
+ name : std::string
+ gender : std::string
}
class NamesByCountry {
- forenames_by_country_ : std::unordered_map<std::string, std::vector<ForenameEntry>>
- surnames_by_country_ : std::unordered_map<std::string, std::vector<std::string>>
+ SampleName(iso3166_1 : const std::string&,\n rng : std::mt19937&) : std::optional<Name>
}
note right of NamesByCountry
Backed by two source-shaped maps,
keyed by ISO 3166-1 country code,
sourced from popular-names-by-country-dataset
(CC0) -- see ETHICS-AND-KNOWN-ISSUES.md.
Gender is preserved per forename so it can
be threaded through to persona/bio generation
later; surnames carry no gender in the source
data. Pairing happens at sample time, not at
fixture-authoring time, so no information from
the source dataset is discarded up front.
end note
NamesByCountry *-- "0..*" ForenameEntry
NamesByCountry ..> Name : produces
class CheckinResult { class CheckinResult {
+ checked_in_at : std::string + checked_in_at : std::string
+ note : std::string + note : std::string
@@ -110,9 +146,20 @@ package "Domain: Models" {
+ user_id : uint64_t + user_id : uint64_t
+ location : Location + location : Location
+ user : UserResult + user : UserResult
+ email : std::string
+ date_of_birth : std::string
+ password : std::string
+ metadata : GenerationMetadata + metadata : GenerationMetadata
} }
note right of GeneratedUser
email, date_of_birth, and password are
programmatically generated by the orchestrator
(not LLM-authored) so a downstream auth-account
seeding consumer can register real accounts from
this export. See ROADMAP.md §1.
end note
class GeneratedCheckin { class GeneratedCheckin {
+ checkin_id : uint64_t + checkin_id : uint64_t
+ user_id : uint64_t + user_id : uint64_t
@@ -321,15 +368,15 @@ package "Infrastructure: Data Preloading" {
interface DataPreloader <<interface>> { interface DataPreloader <<interface>> {
+ LoadLocations(filepath : const std::filesystem::path&) : std::vector<Location> + LoadLocations(filepath : const std::filesystem::path&) : std::vector<Location>
+ LoadBeerStyles(filepath : const std::filesystem::path&) : std::vector<BeerStyle> + LoadBeerStyles(filepath : const std::filesystem::path&) : std::vector<BeerStyle>
+ LoadPersonas(filepath : const std::filesystem::path&) : std::vector<Persona> + LoadPersonas(filepath : const std::filesystem::path&) : std::vector<UserPersona>
+ LoadNamesByCountry(filepath : const std::filesystem::path&) : NamesByCountry + LoadNamesByCountry(forenames_filepath : const std::filesystem::path&,\n surnames_filepath : const std::filesystem::path&) : NamesByCountry
} }
class JsonLoader { class JsonLoader {
+ LoadLocations(filepath : const std::filesystem::path&) : std::vector<Location> + LoadLocations(filepath : const std::filesystem::path&) : std::vector<Location>
+ LoadBeerStyles(filepath : const std::filesystem::path&) : std::vector<BeerStyle> + LoadBeerStyles(filepath : const std::filesystem::path&) : std::vector<BeerStyle>
+ LoadPersonas(filepath : const std::filesystem::path&) : std::vector<Persona> + LoadPersonas(filepath : const std::filesystem::path&) : std::vector<UserPersona>
+ LoadNamesByCountry(filepath : const std::filesystem::path&) : NamesByCountry + LoadNamesByCountry(forenames_filepath : const std::filesystem::path&,\n surnames_filepath : const std::filesystem::path&) : NamesByCountry
} }
} }
@@ -380,7 +427,7 @@ package "Infrastructure: Data Generation" {
interface DataGenerator <<interface>> { interface DataGenerator <<interface>> {
+ GenerateBrewery(location : const Location&,\n context : const LocationContext&) : BreweryResult + GenerateBrewery(location : const Location&,\n context : const LocationContext&) : BreweryResult
+ GenerateBeer(brewery_id : uint64_t,\n location : const Location&,\n context : const LocationContext&,\n style : const BeerStyle&) : BeerResult + GenerateBeer(brewery_id : uint64_t,\n location : const Location&,\n context : const LocationContext&,\n style : const BeerStyle&) : BeerResult
+ GenerateUser(location : const Location&) : UserResult + GenerateUser(city : const EnrichedCity&,\n persona : const UserPersona&,\n name : const Name&) : UserResult
+ GenerateCheckin(user : const GeneratedUser&,\n brewery : const GeneratedBrewery&,\n timestamp : const std::string&) : CheckinResult + GenerateCheckin(user : const GeneratedUser&,\n brewery : const GeneratedBrewery&,\n timestamp : const std::string&) : CheckinResult
+ GenerateRating(user : const GeneratedUser&,\n beer : const GeneratedBeer&,\n checkin_id : uint64_t) : RatingResult + GenerateRating(user : const GeneratedUser&,\n beer : const GeneratedBeer&,\n checkin_id : uint64_t) : RatingResult
} }

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 106 KiB

After

Width:  |  Height:  |  Size: 139 KiB

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 184 KiB

After

Width:  |  Height:  |  Size: 234 KiB

View File

@@ -151,6 +151,11 @@ target_sources(${PROJECT_NAME} PRIVATE
src/json_handling/json_loader.cc src/json_handling/json_loader.cc
) )
# --- data_model ---
target_sources(${PROJECT_NAME} PRIVATE
src/data_model/names_by_country.cc
)
# --- application_options --- # --- application_options ---
target_sources(${PROJECT_NAME} PRIVATE target_sources(${PROJECT_NAME} PRIVATE
src/application_options/parse_arguments.cc src/application_options/parse_arguments.cc
@@ -161,6 +166,7 @@ target_sources(${PROJECT_NAME} PRIVATE
src/biergarten_pipeline_orchestrator/log_results.cc src/biergarten_pipeline_orchestrator/log_results.cc
src/biergarten_pipeline_orchestrator/biergarten_pipeline_orchestrator.cc src/biergarten_pipeline_orchestrator/biergarten_pipeline_orchestrator.cc
src/biergarten_pipeline_orchestrator/generate_breweries.cc src/biergarten_pipeline_orchestrator/generate_breweries.cc
src/biergarten_pipeline_orchestrator/generate_users.cc
src/biergarten_pipeline_orchestrator/run.cc src/biergarten_pipeline_orchestrator/run.cc
src/biergarten_pipeline_orchestrator/query_cities_with_countries.cc src/biergarten_pipeline_orchestrator/query_cities_with_countries.cc
) )
@@ -204,6 +210,7 @@ target_sources(${PROJECT_NAME} PRIVATE
# --- services: sqlite --- # --- services: sqlite ---
target_sources(${PROJECT_NAME} PRIVATE target_sources(${PROJECT_NAME} PRIVATE
src/services/sqlite/process_record.cc src/services/sqlite/process_record.cc
src/services/sqlite/process_user_record.cc
src/services/sqlite/sqlite_export_service.cc src/services/sqlite/sqlite_export_service.cc
src/services/sqlite/finalize.cc src/services/sqlite/finalize.cc
src/services/sqlite/initialize.cc src/services/sqlite/initialize.cc
@@ -260,6 +267,21 @@ configure_file(
${CMAKE_BINARY_DIR}/locations.json ${CMAKE_BINARY_DIR}/locations.json
COPYONLY COPYONLY
) )
configure_file(
${CMAKE_SOURCE_DIR}/personas.json
${CMAKE_BINARY_DIR}/personas.json
COPYONLY
)
configure_file(
${CMAKE_SOURCE_DIR}/forenames-by-country.json
${CMAKE_BINARY_DIR}/forenames-by-country.json
COPYONLY
)
configure_file(
${CMAKE_SOURCE_DIR}/surnames-by-country.json
${CMAKE_BINARY_DIR}/surnames-by-country.json
COPYONLY
)
add_custom_command(TARGET ${PROJECT_NAME} POST_BUILD add_custom_command(TARGET ${PROJECT_NAME} POST_BUILD
COMMAND ${CMAKE_COMMAND} -E copy_directory COMMAND ${CMAKE_COMMAND} -E copy_directory
${CMAKE_SOURCE_DIR}/prompts ${CMAKE_SOURCE_DIR}/prompts

File diff suppressed because it is too large Load Diff

View File

@@ -91,6 +91,18 @@ class BiergartenPipelineOrchestrator {
*/ */
void GenerateBreweries(std::span<const EnrichedCity> cities); void GenerateBreweries(std::span<const EnrichedCity> cities);
/**
* @brief Generate users grounded in sampled names and personas for
* enriched cities.
*
* Loads personas.json / forenames-by-country.json / surnames-by-country.json
* itself, mirroring how QueryCitiesWithCountries() owns its own
* locations.json load.
*
* @param cities Span of enriched city data.
*/
void GenerateUsers(std::span<const EnrichedCity> cities);
/** /**
* @brief Log the generated brewery results. * @brief Log the generated brewery results.
*/ */
@@ -98,5 +110,8 @@ class BiergartenPipelineOrchestrator {
/// @brief Stores generated brewery data. /// @brief Stores generated brewery data.
std::vector<GeneratedBrewery> generated_breweries_; std::vector<GeneratedBrewery> generated_breweries_;
/// @brief Stores generated user data.
std::vector<GeneratedUser> generated_users_;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_DATA_GENERATOR_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_BIERGARTEN_DATA_GENERATOR_H_

View File

@@ -28,12 +28,16 @@ class DataGenerator {
const std::string& region_context) = 0; const std::string& region_context) = 0;
/** /**
* @brief Generates a user profile for a locale. * @brief Generates a user profile grounded in a sampled name and persona.
* *
* @param locale Locale hint used by generator. * @param city Enriched city the user is associated with.
* @param persona Persona archetype grounding the generated bio.
* @param name Sampled first/last name (and gender) -- not LLM-invented.
* @return User generation result. * @return User generation result.
*/ */
virtual UserResult GenerateUser(const std::string& locale) = 0; virtual UserResult GenerateUser(const EnrichedCity& city,
const UserPersona& persona,
const Name& name) = 0;
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_DATA_GENERATOR_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_DATA_GENERATOR_H_

View File

@@ -65,12 +65,15 @@ class LlamaGenerator final : public DataGenerator {
const std::string& region_context) override; const std::string& region_context) override;
/** /**
* @brief Generates a user profile for the provided locale. * @brief Generates a user profile grounded in a sampled name and persona.
* *
* @param locale Locale hint. * @param city Enriched city the user is associated with.
* @param persona Persona archetype grounding the generated bio.
* @param name Sampled first/last name -- not LLM-invented.
* @return Generated user profile. * @return Generated user profile.
*/ */
UserResult GenerateUser(const std::string& locale) override; UserResult GenerateUser(const EnrichedCity& city, const UserPersona& persona,
const Name& name) override;
private: private:
static constexpr int32_t kDefaultMaxTokens = 10000; static constexpr int32_t kDefaultMaxTokens = 10000;

View File

@@ -47,4 +47,18 @@ void AppendTokenPiece(const llama_vocab* vocab, llama_token token,
std::optional<std::string> ValidateBreweryJson(const std::string& raw, std::optional<std::string> ValidateBreweryJson(const std::string& raw,
BreweryResult& brewery_out); BreweryResult& brewery_out);
/**
* @brief Validates and parses user JSON output.
*
* Only populates `username`, `bio`, and `activity_weight` -- `first_name`
* and `last_name` are not LLM-authored and are set separately from the
* sampled Name.
*
* @param raw Raw model output.
* @param user_out Parsed user payload.
* @return Validation error message if invalid, or std::nullopt on success.
*/
std::optional<std::string> ValidateUserJson(const std::string& raw,
UserResult& user_out);
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_LLAMA_GENERATOR_HELPERS_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_GENERATION_LLAMA_GENERATOR_HELPERS_H_

View File

@@ -28,12 +28,16 @@ class MockGenerator final : public DataGenerator {
const std::string& region_context) override; const std::string& region_context) override;
/** /**
* @brief Generates deterministic user data for a locale. * @brief Generates deterministic user data grounded in a sampled name and
* persona.
* *
* @param locale Locale hint. * @param city Enriched city the user is associated with.
* @param persona Persona archetype.
* @param name Sampled first/last name, copied directly into the result.
* @return Generated user result. * @return Generated user result.
*/ */
UserResult GenerateUser(const std::string& locale) override; UserResult GenerateUser(const EnrichedCity& city, const UserPersona& persona,
const Name& name) override;
private: private:
/** /**
@@ -44,12 +48,26 @@ class MockGenerator final : public DataGenerator {
*/ */
static size_t DeterministicHash(const Location& location); static size_t DeterministicHash(const Location& location);
/**
* @brief Combines city, persona, and name into a stable hash value.
*
* @param location City and country names.
* @param persona Persona archetype.
* @param name Sampled first/last name.
* @return Deterministic hash value.
*/
static size_t DeterministicHash(const Location& location,
const UserPersona& persona,
const Name& name);
// Hash stride constants for deterministic distribution across fixed-size // Hash stride constants for deterministic distribution across fixed-size
// arrays. These coprime strides spread hash values uniformly without // arrays. These coprime strides spread hash values uniformly without
// clustering, ensuring diverse output across different hash inputs. // clustering, ensuring diverse output across different hash inputs.
static constexpr size_t kNounHashStride = 7; static constexpr size_t kNounHashStride = 7;
static constexpr size_t kDescriptionHashStride = 13; static constexpr size_t kDescriptionHashStride = 13;
static constexpr size_t kBioHashStride = 11; static constexpr size_t kBioHashStride = 11;
static constexpr size_t kActivityWeightHashStride = 17;
static constexpr size_t kActivityWeightHashRange = 1000;
static constexpr std::array<std::string_view, 18> kBreweryAdjectives = { static constexpr std::array<std::string_view, 18> kBreweryAdjectives = {
"Craft", "Heritage", "Local", "Artisan", "Pioneer", "Golden", "Craft", "Heritage", "Local", "Artisan", "Pioneer", "Golden",

View File

@@ -36,11 +36,27 @@ struct BreweryResult {
* @brief Generated user profile payload. * @brief Generated user profile payload.
*/ */
struct UserResult { struct UserResult {
/// @brief First (given) name, copied from the sampled Name -- not
/// LLM-invented.
std::string first_name{};
/// @brief Last (family) name, copied from the sampled Name -- not
/// LLM-invented.
std::string last_name{};
/// @brief Gender associated with the sampled first name, copied from the
/// sampled Name -- not LLM-invented.
std::string gender{};
/// @brief Username handle. /// @brief Username handle.
std::string username{}; std::string username{};
/// @brief Short user biography. /// @brief Short user biography.
std::string bio{}; std::string bio{};
/// @brief Relative check-in/activity weight for this user, used to drive
/// a J-curve activity profile in later pipeline phases.
float activity_weight{};
}; };
// ============================================================================ // ============================================================================
@@ -63,4 +79,19 @@ struct GeneratedBrewery {
BreweryResult brewery; BreweryResult brewery;
}; };
/**
* @brief Helper struct to store generated user data.
*
* `email`, `date_of_birth`, and `password` are programmatically generated by
* the orchestrator (never LLM-authored) so a downstream auth-account seeding
* consumer can register real accounts from the pipeline's SQLite export.
*/
struct GeneratedUser {
Location location;
UserResult user;
std::string email{};
std::string date_of_birth{};
std::string password{};
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_GENERATED_MODELS_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_GENERATED_MODELS_H_

View File

@@ -64,6 +64,53 @@ struct BreweryLocation {
std::string_view country_name; std::string_view country_name;
}; };
// ============================================================================
// Name / Persona Models
// ============================================================================
/**
* @brief A sampled first/last name pair, with the source forename's gender.
*
* Produced by NamesByCountry::SampleName();
*/
struct Name {
/// @brief First (given) name.
std::string first_name{};
/// @brief Last (family) name.
std::string last_name{};
/// @brief Gender associated with the sampled forename (e.g. "M", "F"), as
/// reported by the source dataset.
std::string gender{};
};
/**
* @brief A single forename entry from the names-by-country fixture data.
*/
struct ForenameEntry {
/// @brief Romanized forename.
std::string name{};
/// @brief Gender associated with this forename, as reported by the source
/// dataset (e.g. "M", "F").
std::string gender{};
};
/**
* @brief A persona archetype used to ground LLM-generated user bios.
*/
struct UserPersona {
/// @brief Persona display name (e.g. "Hophead Explorer").
std::string name{};
/// @brief Short description of the persona's interests and voice.
std::string description{};
/// @brief Beer styles this persona gravitates toward.
std::vector<std::string> style_affinities{};
};
// ============================================================================ // ============================================================================
// Configuration Models // Configuration Models
// ============================================================================ // ============================================================================
@@ -72,7 +119,7 @@ struct BreweryLocation {
* @brief LLM sampling parameters. * @brief LLM sampling parameters.
*/ */
struct SamplingOptions { struct SamplingOptions {
/// @brief LLM sampling temperature (0.0 to 1.0, higher = more random). /// @brief LLM sampling temperature (higher = more random).
float temperature = 1.0F; float temperature = 1.0F;
/// @brief LLM nucleus sampling top-p parameter. /// @brief LLM nucleus sampling top-p parameter.

View File

@@ -0,0 +1,52 @@
#ifndef BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_NAMES_BY_COUNTRY_H_
#define BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_NAMES_BY_COUNTRY_H_
/**
* @file data_model/names_by_country.h
* @brief Sampling over the names-by-country fixture data.
*/
#include <optional>
#include <random>
#include <string>
#include <unordered_map>
#include <vector>
#include "data_model/models.h"
/**
* @brief Samples locale-appropriate names from curated forename/surname
* fixture data, keyed by ISO 3166-1 country code.
*
* Forenames and surnames are kept in separate maps (mirroring the source
* dataset's own shape) so per-forename gender is preserved; pairing happens
* at sample time rather than being precomputed, so no information from the
* source dataset is discarded up front.
*/
class NamesByCountry {
public:
NamesByCountry(
std::unordered_map<std::string, std::vector<ForenameEntry>>
forenames_by_country,
std::unordered_map<std::string, std::vector<std::string>>
surnames_by_country);
/**
* @brief Samples a first/last name pair for the given country.
*
* @param iso3166_1 ISO 3166-1 country code to sample for.
* @param rng Random source used for sampling.
* @return A sampled Name, or std::nullopt if the country has no forename
* or no surname entries.
*/
[[nodiscard]] std::optional<Name> SampleName(const std::string& iso3166_1,
std::mt19937& rng) const;
private:
std::unordered_map<std::string, std::vector<ForenameEntry>>
forenames_by_country_;
std::unordered_map<std::string, std::vector<std::string>>
surnames_by_country_;
};
#endif // BIERGARTEN_PIPELINE_INCLUDES_DATA_MODEL_NAMES_BY_COUNTRY_H_

View File

@@ -11,6 +11,7 @@
#include <vector> #include <vector>
#include "data_model/models.h" #include "data_model/models.h"
#include "data_model/names_by_country.h"
#include "services/logging/logger.h" #include "services/logging/logger.h"
/// @brief Loads curated world locations from a JSON file into memory. /// @brief Loads curated world locations from a JSON file into memory.
@@ -20,6 +21,23 @@ class JsonLoader {
static std::vector<Location> LoadLocations( static std::vector<Location> LoadLocations(
const std::filesystem::path& filepath, const std::filesystem::path& filepath,
std::shared_ptr<ILogger> logger = nullptr); std::shared_ptr<ILogger> logger = nullptr);
/// @brief Parses a JSON array file and returns all persona records.
static std::vector<UserPersona> LoadPersonas(
const std::filesystem::path& filepath,
std::shared_ptr<ILogger> logger = nullptr);
/**
* @brief Parses the names-by-country fixture pair into a sampling-capable
* NamesByCountry.
*
* @param forenames_filepath Path to forenames-by-country.json.
* @param surnames_filepath Path to surnames-by-country.json.
*/
static NamesByCountry LoadNamesByCountry(
const std::filesystem::path& forenames_filepath,
const std::filesystem::path& surnames_filepath,
std::shared_ptr<ILogger> logger = nullptr);
}; };
#endif // BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_ #endif // BIERGARTEN_PIPELINE_INCLUDES_JSON_HANDLING_JSON_LOADER_H_

View File

@@ -35,6 +35,13 @@ class IExportService {
*/ */
virtual uint64_t ProcessRecord(const GeneratedBrewery& brewery) = 0; virtual uint64_t ProcessRecord(const GeneratedBrewery& brewery) = 0;
/**
* @brief Persists one generated user record.
*
* @param user Generated user payload to store.
*/
virtual uint64_t ProcessRecord(const GeneratedUser& user) = 0;
/// @brief Finalizes the export destination. /// @brief Finalizes the export destination.
virtual void Finalize() = 0; virtual void Finalize() = 0;
}; };

View File

@@ -31,6 +31,7 @@ class SqliteExportService final : public IExportService {
void Initialize() override; void Initialize() override;
uint64_t ProcessRecord(const GeneratedBrewery& brewery) override; uint64_t ProcessRecord(const GeneratedBrewery& brewery) override;
uint64_t ProcessRecord(const GeneratedUser& user) override;
void Finalize() override; void Finalize() override;
private: private:
@@ -46,6 +47,15 @@ class SqliteExportService final : public IExportService {
[[nodiscard]] std::filesystem::path BuildDatabasePath() const; [[nodiscard]] std::filesystem::path BuildDatabasePath() const;
[[nodiscard]] static std::string BuildLocationKey(const Location& location); [[nodiscard]] static std::string BuildLocationKey(const Location& location);
/**
* @brief Returns the row id for @p location, inserting it first if it has
* not already been seen during this run.
*
* Shared by both ProcessRecord() overloads so breweries and users
* referencing the same location resolve to the same row.
*/
[[nodiscard]] sqlite3_int64 ResolveLocationId(const Location& location);
std::unique_ptr<IDateTimeProvider> date_time_provider_; std::unique_ptr<IDateTimeProvider> date_time_provider_;
std::filesystem::path output_path_; std::filesystem::path output_path_;
std::string run_timestamp_utc_; std::string run_timestamp_utc_;
@@ -53,6 +63,7 @@ class SqliteExportService final : public IExportService {
SqliteDatabaseHandle db_handle_; SqliteDatabaseHandle db_handle_;
SqliteStatementHandle insert_location_stmt_; SqliteStatementHandle insert_location_stmt_;
SqliteStatementHandle insert_brewery_stmt_; SqliteStatementHandle insert_brewery_stmt_;
SqliteStatementHandle insert_user_stmt_;
bool transaction_open_ = false; bool transaction_open_ = false;
std::unordered_map<std::string, sqlite3_int64> location_cache_; std::unordered_map<std::string, sqlite3_int64> location_cache_;
}; };

View File

@@ -50,6 +50,27 @@ CREATE INDEX IF NOT EXISTS idx_breweries_location_id ON breweries(location_id);
)sql"; )sql";
inline constexpr std::string_view kCreateUsersTableSql = R"sql(
CREATE TABLE IF NOT EXISTS users (
id INTEGER PRIMARY KEY AUTOINCREMENT,
location_id INTEGER NOT NULL,
first_name TEXT NOT NULL,
last_name TEXT NOT NULL,
gender TEXT NOT NULL,
username TEXT NOT NULL,
bio TEXT NOT NULL,
activity_weight REAL NOT NULL,
email TEXT NOT NULL UNIQUE,
date_of_birth TEXT NOT NULL,
password TEXT NOT NULL,
FOREIGN KEY(location_id) REFERENCES locations(id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_users_location_id ON users(location_id);
)sql";
inline constexpr std::string_view kInsertLocationSql = R"sql( inline constexpr std::string_view kInsertLocationSql = R"sql(
INSERT INTO locations ( INSERT INTO locations (
city, city,
@@ -73,6 +94,21 @@ INSERT INTO breweries (
) VALUES (?, ?, ?, ?, ?); ) VALUES (?, ?, ?, ?, ?);
)sql"; )sql";
inline constexpr std::string_view kInsertUserSql = R"sql(
INSERT INTO users (
location_id,
first_name,
last_name,
gender,
username,
bio,
activity_weight,
email,
date_of_birth,
password
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?);
)sql";
inline constexpr int kLocationCityBindIndex = 1; inline constexpr int kLocationCityBindIndex = 1;
inline constexpr int kLocationStateProvinceBindIndex = 2; inline constexpr int kLocationStateProvinceBindIndex = 2;
inline constexpr int kLocationIso31662BindIndex = 3; inline constexpr int kLocationIso31662BindIndex = 3;
@@ -88,6 +124,17 @@ inline constexpr int kBreweryEnglishDescriptionBindIndex = 3;
inline constexpr int kBreweryLocalNameBindIndex = 4; inline constexpr int kBreweryLocalNameBindIndex = 4;
inline constexpr int kBreweryLocalDescriptionBindIndex = 5; inline constexpr int kBreweryLocalDescriptionBindIndex = 5;
inline constexpr int kUserLocationIdBindIndex = 1;
inline constexpr int kUserFirstNameBindIndex = 2;
inline constexpr int kUserLastNameBindIndex = 3;
inline constexpr int kUserGenderBindIndex = 4;
inline constexpr int kUserUsernameBindIndex = 5;
inline constexpr int kUserBioBindIndex = 6;
inline constexpr int kUserActivityWeightBindIndex = 7;
inline constexpr int kUserEmailBindIndex = 8;
inline constexpr int kUserDateOfBirthBindIndex = 9;
inline constexpr int kUserPasswordBindIndex = 10;
SqliteStatementHandle PrepareStatement(const SqliteDatabaseHandle& db_handle, SqliteStatementHandle PrepareStatement(const SqliteDatabaseHandle& db_handle,
std::string_view sql, std::string_view sql,
const char* action); const char* action);

View File

@@ -35,6 +35,7 @@ enum class LogLevel {
*/ */
enum class PipelinePhase { enum class PipelinePhase {
Startup, ///< Initialization and validation. Startup, ///< Initialization and validation.
Enrichment, ///< Location/context enrichment (e.g. Wikipedia lookups).
UserGeneration, ///< User profile generation. UserGeneration, ///< User profile generation.
BreweryAndBeerGeneration, ///< Brewery and beer data generation. BreweryAndBeerGeneration, ///< Brewery and beer data generation.
CheckinGeneration, ///< Checkin (visit) record generation. CheckinGeneration, ///< Checkin (visit) record generation.

View File

@@ -0,0 +1,42 @@
[
{
"name": "Hophead Explorer",
"description": "Chases the newest hop varieties and double-digit IBUs, tracking hazy releases and limited tap takeovers across town.",
"style_affinities": ["American IPA", "Double IPA", "New England IPA", "Black IPA"]
},
{
"name": "Malt & Tradition Loyalist",
"description": "Prefers clean, balanced lagers brewed the way their region has brewed them for generations, and is quick to defend a well-made Helles.",
"style_affinities": ["Munich Helles", "Vienna Lager", "Märzen", "Festbier"]
},
{
"name": "Dark & Roasty Devotee",
"description": "Orders the darkest beer on the board year-round, drawn to roasted malt, coffee, and chocolate notes over carbonated lightness.",
"style_affinities": ["Imperial Stout", "Robust Porter", "Baltic Porter", "Oatmeal Stout"]
},
{
"name": "Sour & Wild Forager",
"description": "Seeks out barrel-aged, spontaneously fermented, and funky beers, and will travel for a fresh bottling of something wild.",
"style_affinities": ["Lambic", "Gueuze", "Flanders Red Ale", "Wild Ale"]
},
{
"name": "Session Sipper",
"description": "Values an easy-drinking pint that holds up over a long afternoon at the taproom more than raw intensity or novelty.",
"style_affinities": ["Session IPA", "Ordinary Bitter", "Cream Ale", "Blonde Ale"]
},
{
"name": "Belgian Abbey Pilgrim",
"description": "Collects abbey and trappist releases, drawn to the dried-fruit esters and warming strength of classic Belgian ales.",
"style_affinities": ["Belgian Tripel", "Belgian Dubbel", "Belgian Quadrupel", "Trappist Single"]
},
{
"name": "Wheat & Citrus Casual",
"description": "Reaches for something light, cloudy, and refreshing, usually with a citrus or coriander note, especially on a patio in summer.",
"style_affinities": ["Hefeweizen", "Witbier", "American Wheat Beer", "Berliner Weisse"]
},
{
"name": "Big & Boozy Connoisseur",
"description": "Sips slowly from a snifter, seeking out high-ABV releases meant for aging and savoring rather than quick refreshment.",
"style_affinities": ["English Barleywine", "Wee Heavy", "Doppelbock", "Old Ale"]
}
]

View File

@@ -0,0 +1,122 @@
# FULL SYSTEM PROMPT
You are a craft beer community profile writer. You write short, first-person
biographies for fictional users of a brewery-discovery app, grounded in the
exact name, locale, and persona provided. You do not invent the user's name --
it is given to you and must not be altered, translated, or abbreviated beyond
forming a username.
You will receive the inputs like this:
## NAME:
[First name] [Last name]
## GENDER:
[Gender associated with the given first name]
## CITY:
[City Name]
## COUNTRY:
[Country Name]
## PERSONA:
[Persona archetype name]
## PERSONA DESCRIPTION:
[What this persona cares about and how they talk about beer]
## STYLE AFFINITIES:
[Comma-separated beer styles this persona favors]
## CRITICAL OUTPUT FORMAT (READ CAREFULLY):
ABSOLUTELY NO MARKDOWN FORMATTING. Do NOT wrap your response in json or ```
blocks.
Do not add markdown, code fences, or postscript around the final JSON object.
Do not say "Here is the JSON" or "Enjoy!".
The JSON must contain exactly three keys ("username", "bio",
"activity_weight") in that order. Do not rename or add any other keys.
ESCAPE ALL QUOTES inside the bio field using \", or use single quotes (' ')
instead.
DO NOT use actual line breaks (\n) inside any string. Keep the bio as one
continuous string.
The bio must be between 25 and 60 words, written in first person, and must
read as something this specific person would write about themselves -- not a
generic profile blurb.
`activity_weight` is a number between 0 and 1 (inclusive) representing how
often this persona checks in at breweries relative to other users. Persona
archetypes implying frequent, habitual brewery visits should skew higher;
casual or occasional-visitor personas should skew lower. Do not default to
0.5 -- vary it meaningfully based on the persona description.
Expected JSON format:
```json
{
"username": "a handle derived from the given name",
"bio": "The first-person bio goes here.",
"activity_weight": 0.62
}
```
## CONTENT RULES AND CONSTRAINTS:
### THE USERNAME:
Derive the username from the given first and/or last name (e.g. lowercase,
abbreviated, or combined with a short word or number tied to the persona's
style affinities). Do not invent an unrelated handle. Do not include spaces.
### THE BIO:
Ground the bio in the supplied persona description and style affinities --
mention at least one specific beer style from the style affinities list, in a
way that sounds like a personal preference rather than a list. Reference the
city or country naturally if it fits, but do not force it.
### VOICE & PERSPECTIVE:
Write in the first person ("I"/"my"). The tone should match the persona:
an enthusiastic explorer sounds different from a relaxed, easy-drinking
regular. Do not write in second or third person.
### THE BLOCKLIST (FORBIDDEN CONCEPTS):
You absolutely cannot use the following words and phrases. Make sure your
final output doesn't have any of these:
- "hidden gem"
- "passion"
- "authentic"
- "craft beer enthusiast"
- "beer connoisseur"
- "journey"
#### FORBIDDEN WRITING PATTERNS
The following patterns are common AI writing pitfalls and must not appear in
the bio:
- Negative parallelism constructions: "It's not X, it's Y"
- Inflated significance phrases: "stands as a testament," "plays a vital
role," "deeply rooted"
- Superficial trailing analyses: sentences ending in -ing words that add
opinion without content
- Promotional travel-copy tone: "breathtaking," "must-visit," "vibrant"
- Overused conjunctive transitions used as sentence openers: "Moreover,"
"Furthermore," "In addition"
- Rule of three: do not consistently organise ideas or examples in triplets

View File

@@ -15,8 +15,6 @@ std::optional<ApplicationOptions> ParseArguments(
opt("help,h", "Produce help message"); opt("help,h", "Produce help message");
// Defaults sourced from SamplingOptions{} so the CLI and LlamaGenerator
// share a single source of truth — changing the struct updates both.
auto add_sampling_options = [&]() -> void { auto add_sampling_options = [&]() -> void {
const SamplingOptions sampling_defaults{}; const SamplingOptions sampling_defaults{};
opt("temperature", opt("temperature",
@@ -152,7 +150,6 @@ std::optional<ApplicationOptions> ParseArguments(
options.generator.use_mocked = use_mocked; options.generator.use_mocked = use_mocked;
options.generator.model_path = model_path; options.generator.model_path = model_path;
// options.generator.n_gpu_layers = n_gpu_layers;
// Only populate sampling config when the user explicitly overrides at // Only populate sampling config when the user explicitly overrides at
// least one value. Leaving it as std::nullopt lets LlamaGenerator fall // least one value. Leaving it as std::nullopt lets LlamaGenerator fall

View File

@@ -0,0 +1,186 @@
/**
* @file biergarten_pipeline_orchestrator/generate_users.cc
* @brief BiergartenDataGenerator::GenerateUsers() implementation.
*/
#include <cctype>
#include <chrono>
#include <format>
#include <optional>
#include <random>
#include <stdexcept>
#include <string>
#include <string_view>
#include <unordered_set>
#include "biergarten_pipeline_orchestrator.h"
#include "json_handling/json_loader.h"
#include "services/logging/logger.h"
namespace {
std::string Sanitize(std::string_view value) {
std::string out;
out.reserve(value.size());
for (const char character : value) {
if (std::isalnum(static_cast<unsigned char>(character)) != 0) {
out.push_back(
static_cast<char>(std::tolower(static_cast<unsigned char>(character))));
}
}
return out;
}
std::string BuildEmail(const Name& name,
std::unordered_set<std::string>& used_local_parts) {
const std::string base =
std::format("{}.{}", Sanitize(name.first_name), Sanitize(name.last_name));
std::string local_part = base;
for (int suffix = 1; used_local_parts.contains(local_part); ++suffix) {
local_part = std::format("{}{}", base, suffix);
}
used_local_parts.insert(local_part);
return std::format("{}@thebiergarten.app", local_part);
}
std::string GenerateDateOfBirth(std::mt19937& rng) {
using namespace std::chrono;
constexpr int kMinAge = 19;
constexpr int kMaxAge = 48;
constexpr int kMaxDayOffset = 364;
std::uniform_int_distribution<int> age_dist(kMinAge, kMaxAge);
std::uniform_int_distribution<int> day_offset_dist(0, kMaxDayOffset);
const year_month_day today{floor<days>(system_clock::now())};
const year_month_day birth_year_anchor{today.year() - years{age_dist(rng)},
today.month(), today.day()};
const sys_days birth_date =
sys_days{birth_year_anchor} - days{day_offset_dist(rng)};
const year_month_day birth_ymd{birth_date};
return std::format("{:04}-{:02}-{:02}",
static_cast<int>(birth_ymd.year()),
static_cast<unsigned>(birth_ymd.month()),
static_cast<unsigned>(birth_ymd.day()));
}
std::string GenerateRandomPassword(std::mt19937& rng) {
constexpr size_t kPasswordLength = 32;
constexpr std::string_view kCharset =
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*";
std::uniform_int_distribution<size_t> char_dist(0, kCharset.size() - 1);
std::string password;
password.reserve(kPasswordLength);
for (size_t i = 0; i < kPasswordLength; ++i) {
password.push_back(kCharset[char_dist(rng)]);
}
return password;
}
} // namespace
void BiergartenPipelineOrchestrator::GenerateUsers(
std::span<const EnrichedCity> cities) {
logger_->Log({.level = LogLevel::Info,
.phase = PipelinePhase::UserGeneration,
.message = "=== SAMPLE USER GENERATION ==="});
const std::vector<UserPersona> personas =
JsonLoader::LoadPersonas("personas.json", logger_);
if (personas.empty()) {
throw std::runtime_error(
"No personas available in personas.json for user generation");
}
const NamesByCountry names_by_country = JsonLoader::LoadNamesByCountry(
"forenames-by-country.json", "surnames-by-country.json", logger_);
std::mt19937 rng(std::random_device{}());
std::uniform_int_distribution<size_t> persona_dist(0, personas.size() - 1);
generated_users_.clear();
std::unordered_set<std::string> used_email_local_parts;
size_t skipped_count = 0;
size_t export_failed_count = 0;
for (const auto& city : cities) {
const std::optional<Name> sampled_name =
names_by_country.SampleName(city.location.iso3166_1, rng);
if (!sampled_name.has_value()) {
++skipped_count;
logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration,
.message = std::format(
"[Pipeline] Skipping city '{}' ({}): no names available for "
"country '{}'",
city.location.city, city.location.country,
city.location.iso3166_1)});
continue;
}
const UserPersona& persona = personas[persona_dist(rng)];
try {
const UserResult user =
generator_->GenerateUser(city, persona, *sampled_name);
const GeneratedUser generated_user{
.location = city.location,
.user = user,
.email = BuildEmail(*sampled_name, used_email_local_parts),
.date_of_birth = GenerateDateOfBirth(rng),
.password = GenerateRandomPassword(rng),
};
generated_users_.push_back(generated_user);
try {
exporter_->ProcessRecord(generated_user);
} catch (const std::exception& export_exception) {
++export_failed_count;
logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration,
.message = std::format(
"[Pipeline] Generated user for '{}' ({}) but SQLite export failed: {}",
city.location.city, city.location.country,
export_exception.what())});
}
} catch (const std::exception& e) {
++skipped_count;
logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration,
.message = std::format(
"[Pipeline] Skipping city '{}' ({}): user generation failed: {}",
city.location.city, city.location.country, e.what())});
}
}
if (skipped_count > 0) {
logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration,
.message = std::format(
"[Pipeline] Skipped {} city/cities during user generation",
skipped_count)});
}
if (export_failed_count > 0) {
logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::Teardown,
.message = std::format(
"[Pipeline] Failed to export {} generated user/users to SQLite",
export_failed_count)});
}
}

View File

@@ -10,11 +10,12 @@
#include "../../includes/json_handling/pretty_print.h" #include "../../includes/json_handling/pretty_print.h"
#include "biergarten_pipeline_orchestrator.h" #include "biergarten_pipeline_orchestrator.h"
#include "services/logging/logger.h" #include "services/logging/logger.h"
void BiergartenPipelineOrchestrator::LogResults() const { void BiergartenPipelineOrchestrator::LogResults() const {
boost::json::array output; boost::json::array brewery_output;
for (const auto& [location, brewery] : generated_breweries_) { for (const auto& [location, brewery] : generated_breweries_) {
output.push_back(boost::json::object{ brewery_output.push_back(boost::json::object{
{"name_en", brewery.name_en}, {"name_en", brewery.name_en},
{"description_en", brewery.description_en}, {"description_en", brewery.description_en},
{"name_local", brewery.name_local}, {"name_local", brewery.name_local},
@@ -29,9 +30,31 @@ void BiergartenPipelineOrchestrator::LogResults() const {
}}}); }}});
} }
std::ostringstream oss; std::ostringstream brewery_oss;
PrettyPrint(oss, output); PrettyPrint(brewery_oss, brewery_output);
logger_->Log({.level = LogLevel::Info, logger_->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Teardown, .phase = PipelinePhase::Teardown,
.message = oss.str()}); .message = brewery_oss.str()});
boost::json::array user_output;
for (const auto& generated_user : generated_users_) {
user_output.push_back(boost::json::object{
{"first_name", generated_user.user.first_name},
{"last_name", generated_user.user.last_name},
{"gender", generated_user.user.gender},
{"username", generated_user.user.username},
{"bio", generated_user.user.bio},
{"activity_weight", generated_user.user.activity_weight},
{"email", generated_user.email},
{"date_of_birth", generated_user.date_of_birth},
});
}
std::ostringstream user_oss;
PrettyPrint(user_oss, user_output);
logger_->Log({.level = LogLevel::Info,
.phase = PipelinePhase::Teardown,
.message = user_oss.str()});
} }

View File

@@ -22,9 +22,6 @@ bool BiergartenPipelineOrchestrator::Run() {
for (auto& city : cities) { for (auto& city : cities) {
try { try {
std::string region_context = context_service_->GetLocationContext(city); std::string region_context = context_service_->GetLocationContext(city);
// logger_->Log(LogLevel::Debug, PipelinePhase::UserGeneration,
// "[Pipeline] Context for '" + city.city + "' (" +
// city.iso3166_2 + ") gathered:\n" + region_context);
enriched.push_back( enriched.push_back(
EnrichedCity{.location = std::move(city), EnrichedCity{.location = std::move(city),
@@ -33,7 +30,7 @@ bool BiergartenPipelineOrchestrator::Run() {
++skipped_count; ++skipped_count;
logger_->Log( logger_->Log(
{.level = LogLevel::Warn, {.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::Enrichment,
.message = std::format( .message = std::format(
"[Pipeline] Skipping city '{}' ({}): context lookup failed: {}", "[Pipeline] Skipping city '{}' ({}): context lookup failed: {}",
city.city, city.country, exception.what())}); city.city, city.country, exception.what())});
@@ -42,12 +39,13 @@ bool BiergartenPipelineOrchestrator::Run() {
if (skipped_count > 0) { if (skipped_count > 0) {
logger_->Log({.level = LogLevel::Warn, logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::Enrichment,
.message = std::format( .message = std::format(
"[Pipeline] Skipped {} city/cities due to context lookup errors", "[Pipeline] Skipped {} city/cities due to context lookup errors",
skipped_count)}); skipped_count)});
} }
this->GenerateUsers(enriched);
this->GenerateBreweries(enriched); this->GenerateBreweries(enriched);
exporter_->Finalize(); exporter_->Finalize();
this->LogResults(); this->LogResults();

View File

@@ -91,7 +91,6 @@ BreweryResult LlamaGenerator::GenerateBrewery(
std::string last_error; std::string last_error;
// Token budget: too small risks truncating valid JSON mid-string. // Token budget: too small risks truncating valid JSON mid-string.
// Start conservatively but allow adaptive increases on truncation.
int max_tokens = kBreweryInitialMaxTokens; int max_tokens = kBreweryInitialMaxTokens;
// Limit output length to keep it concise and focused // Limit output length to keep it concise and focused

View File

@@ -1,24 +1,121 @@
/** /**
* @file data_generation/llama/generate_user.cc * @file data_generation/llama/generate_user.cc
* @brief Generates locale-aware user profiles with strict two-line formatting, * @brief Builds persona/name-grounded user prompts, performs retry-based
* retry handling, and output sanitization for downstream parsing. * inference, and validates structured JSON output for user records.
*/ */
#include <format> #include <format>
#include <optional>
#include <stdexcept>
#include <string> #include <string>
#include <string_view>
#include "data_generation/llama_generator.h" #include "data_generation/llama_generator.h"
#include "data_generation/llama_generator_helpers.h" #include "data_generation/llama_generator_helpers.h"
// TODO: Implement locale-aware user profile generation. // GBNF grammar for structured user JSON output.
// Current implementation returns a hardcoded test value and ignores the static constexpr std::string_view kUserJsonGrammar = R"json_user(
// locale parameter. Future implementation should: root ::= thought-block "{" ws "\"username\"" ws ":" ws string ws "," ws "\"bio\"" ws ":" ws string ws "," ws "\"activity_weight\"" ws ":" ws number ws "}" ws
// 1. Load a USER_GENERATION.md prompt template with locale context thought-block ::= [^{]*
// 2. Perform LLM inference with locale-specific username/bio generation ws ::= [ \t\n\r]*
// 3. Parse and validate JSON output with retry handling (similar to brewery) string ::= "\"" char+ "\""
// 4. Return locale-aware username and biography char ::= [^"\\\x7F\x00-\x1F] | [\\] escape
UserResult LlamaGenerator::GenerateUser(const std::string& locale) { escape ::= ["\\/bfnrt] | "u" hex hex hex hex
return {.username = "test_user", hex ::= [0-9a-fA-F]
.bio = std::format("This is a test user profile from {}.", locale)}; number ::= "-"? ("0" | [1-9] [0-9]*) ("." [0-9]+)?
)json_user";
static constexpr int kUserInitialMaxTokens = 1200;
UserResult LlamaGenerator::GenerateUser(const EnrichedCity& city,
const UserPersona& persona,
const Name& name) {
std::string style_affinities;
for (const std::string& style : persona.style_affinities) {
if (!style_affinities.empty()) {
style_affinities += ", ";
}
style_affinities += style;
}
const std::string system_prompt = prompt_directory_->Load("USER_GENERATION");
std::string user_prompt = std::format(
"## NAME:\n{} {}\n\n## GENDER:\n{}\n\n## CITY:\n{}\n\n## COUNTRY:\n{}\n\n"
"## PERSONA:\n{}\n\n## PERSONA DESCRIPTION:\n{}\n\n## STYLE "
"AFFINITIES:\n{}",
name.first_name, name.last_name, name.gender, city.location.city,
city.location.country, persona.name, persona.description,
style_affinities);
const std::string retry_context =
std::format("Name: {} {}\nCity: {}, {}\nPersona: {}", name.first_name,
name.last_name, city.location.city, city.location.country,
persona.name);
constexpr int max_attempts = 3;
std::string raw;
std::string last_error;
int max_tokens = kUserInitialMaxTokens;
for (int attempt = 0; attempt < max_attempts; ++attempt) {
raw = this->Infer(system_prompt, user_prompt, max_tokens,
kUserJsonGrammar);
if (logger_) {
logger_->Log(
{.level = LogLevel::Debug,
.phase = PipelinePhase::UserGeneration,
.message = std::format("LlamaGenerator: raw output (attempt {}): {}",
attempt + 1, raw)});
}
UserResult user;
const std::optional<std::string> validation_error =
ValidateUserJson(raw, user);
if (!validation_error.has_value()) {
if (logger_) {
logger_->Log(
{.level = LogLevel::Info,
.phase = PipelinePhase::UserGeneration,
.message = std::format("LlamaGenerator: successfully generated user data on attempt {}",
attempt + 1)});
}
user.first_name = name.first_name;
user.last_name = name.last_name;
user.gender = name.gender;
return user;
}
last_error = *validation_error;
if (logger_) {
logger_->Log(
{.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration,
.message =
std::format("LlamaGenerator: malformed user JSON (attempt {}): {}",
attempt + 1, *validation_error)});
}
user_prompt = std::format(
"Your previous response was invalid. Error: {}\nReturn the thought "
"process before the JSON if needed, then return ONLY valid JSON with "
"exactly these keys, in this exact order: {{\"username\": \"<handle "
"derived from the given name>\", \"bio\": \"<first-person bio "
"grounded in the persona>\", \"activity_weight\": <number between 0 "
"and 1>}}.\nDo not include markdown, comments, extra keys, or "
"literal placeholder values.\n\n{}",
*validation_error, retry_context);
}
if (logger_) {
logger_->Log(
{.level = LogLevel::Error,
.phase = PipelinePhase::UserGeneration,
.message = std::format(
"LlamaGenerator: malformed user response after {} attempts: {}",
max_attempts, last_error.empty() ? raw : last_error)});
}
throw std::runtime_error("LlamaGenerator: malformed user response");
} }

View File

@@ -9,6 +9,7 @@
#include <boost/json.hpp> #include <boost/json.hpp>
#include <cctype> #include <cctype>
#include <optional> #include <optional>
#include <span>
#include <stdexcept> #include <stdexcept>
#include <string> #include <string>
#include <string_view> #include <string_view>
@@ -76,7 +77,7 @@ bool ReadRequiredTrimmedStringField(const boost::json::object& obj,
return !out.empty(); return !out.empty();
} }
bool HasSchemaPlaceholder(const std::array<std::string*, 4>& values) { bool HasSchemaPlaceholder(std::span<std::string* const> values) {
for (const std::string* value : values) { for (const std::string* value : values) {
std::string lowered = *value; std::string lowered = *value;
std::ranges::transform(lowered, lowered.begin(), std::ranges::transform(lowered, lowered.begin(),
@@ -207,3 +208,58 @@ std::optional<std::string> ValidateBreweryJson(const std::string& raw,
return std::nullopt; return std::nullopt;
} }
std::optional<std::string> ValidateUserJson(const std::string& raw,
UserResult& user_out) {
boost::system::error_code error_code;
const std::string_view raw_view(raw);
const size_t opening_brace = raw_view.find('{');
if (opening_brace == std::string_view::npos) {
return "JSON parse error: missing opening brace '{'";
}
const std::string_view json_payload = raw_view.substr(opening_brace);
boost::json::value json_value = boost::json::parse(json_payload, error_code);
if (error_code) {
return "JSON parse error: " + error_code.message();
}
if (!json_value.is_object()) {
return "JSON root must be an object";
}
const auto& obj = json_value.get_object();
if (obj.size() != 3) {
return "JSON object must contain exactly three keys";
}
std::string validation_error;
if (!ReadRequiredTrimmedStringField(obj, "username", user_out.username,
&validation_error)) {
return validation_error;
}
if (!ReadRequiredTrimmedStringField(obj, "bio", user_out.bio,
&validation_error)) {
return validation_error;
}
const boost::json::value* activity_weight_field =
obj.if_contains("activity_weight");
if (activity_weight_field == nullptr || !activity_weight_field->is_number()) {
return "Missing or invalid numeric field: activity_weight";
}
const double activity_weight = activity_weight_field->to_number<double>();
if (activity_weight < 0.0 || activity_weight > 1.0) {
return "activity_weight must be between 0 and 1";
}
user_out.activity_weight = static_cast<float>(activity_weight);
const std::array schema_placeholders = {&user_out.username, &user_out.bio};
if (HasSchemaPlaceholder(schema_placeholders)) {
return "JSON appears to be a schema placeholder, not content";
}
return std::nullopt;
}

View File

@@ -14,3 +14,13 @@ size_t MockGenerator::DeterministicHash(const Location& location) {
boost::hash_combine(seed, location.country); boost::hash_combine(seed, location.country);
return seed; return seed;
} }
size_t MockGenerator::DeterministicHash(const Location& location,
const UserPersona& persona,
const Name& name) {
size_t seed = DeterministicHash(location);
boost::hash_combine(seed, persona.name);
boost::hash_combine(seed, name.first_name);
boost::hash_combine(seed, name.last_name);
return seed;
}

View File

@@ -1,22 +1,31 @@
/** /**
* @file data_generation/mock/generate_user.cc * @file data_generation/mock/generate_user.cc
* @brief Generates deterministic mock user profiles by hashing locale values * @brief Generates deterministic mock user profiles by hashing city, persona,
* into predefined username and bio collections. * and sampled name into predefined username and bio collections.
*/ */
#include <functional>
#include <string>
#include <string_view> #include <string_view>
#include "data_generation/mock_generator.h" #include "data_generation/mock_generator.h"
UserResult MockGenerator::GenerateUser(const std::string& locale) { UserResult MockGenerator::GenerateUser(const EnrichedCity& city,
const size_t hash = std::hash<std::string>{}(locale); const UserPersona& persona,
const Name& name) {
const size_t hash = DeterministicHash(city.location, persona, name);
UserResult result;
const std::string_view username = kUsernames[hash % kUsernames.size()]; const std::string_view username = kUsernames[hash % kUsernames.size()];
const std::string_view bio = kBios[hash / kBioHashStride % kBios.size()]; const std::string_view bio = kBios[hash / kBioHashStride % kBios.size()];
result.username = username; const float activity_weight =
result.bio = bio; static_cast<float>(hash / kActivityWeightHashStride %
return result; kActivityWeightHashRange) /
static_cast<float>(kActivityWeightHashRange);
return {
.first_name = name.first_name,
.last_name = name.last_name,
.gender = name.gender,
.username = std::string(username),
.bio = std::string(bio),
.activity_weight = activity_weight,
};
} }

View File

@@ -0,0 +1,41 @@
/**
* @file data_model/names_by_country.cc
* @brief NamesByCountry::SampleName() implementation.
*/
#include "data_model/names_by_country.h"
#include <utility>
NamesByCountry::NamesByCountry(
std::unordered_map<std::string, std::vector<ForenameEntry>>
forenames_by_country,
std::unordered_map<std::string, std::vector<std::string>>
surnames_by_country)
: forenames_by_country_(std::move(forenames_by_country)),
surnames_by_country_(std::move(surnames_by_country)) {}
std::optional<Name> NamesByCountry::SampleName(const std::string& iso3166_1,
std::mt19937& rng) const {
const auto forenames_it = forenames_by_country_.find(iso3166_1);
const auto surnames_it = surnames_by_country_.find(iso3166_1);
if (forenames_it == forenames_by_country_.end() ||
surnames_it == surnames_by_country_.end() ||
forenames_it->second.empty() || surnames_it->second.empty()) {
return std::nullopt;
}
const std::vector<ForenameEntry>& forenames = forenames_it->second;
const std::vector<std::string>& surnames = surnames_it->second;
std::uniform_int_distribution<size_t> forename_dist(0,
forenames.size() - 1);
std::uniform_int_distribution<size_t> surname_dist(0, surnames.size() - 1);
const ForenameEntry& forename = forenames[forename_dist(rng)];
return Name{.first_name = forename.name,
.last_name = surnames[surname_dist(rng)],
.gender = forename.gender};
}

View File

@@ -16,6 +16,8 @@
#include <stdexcept> #include <stdexcept>
#include <string> #include <string>
#include <string_view> #include <string_view>
#include <unordered_map>
#include <utility>
static std::string ReadRequiredString(const boost::json::object& object, static std::string ReadRequiredString(const boost::json::object& object,
const char* key) { const char* key) {
@@ -59,24 +61,53 @@ static std::vector<std::string> ReadRequiredStringArray(
return items; return items;
} }
std::vector<Location> JsonLoader::LoadLocations( namespace {
const std::filesystem::path& filepath, std::shared_ptr<ILogger> logger) {
boost::json::value ParseJsonFile(const std::filesystem::path& filepath,
const char* what) {
std::ifstream input(filepath); std::ifstream input(filepath);
if (!input.is_open()) { if (!input.is_open()) {
throw std::runtime_error("Failed to open locations file: " + throw std::runtime_error(std::format("Failed to open {} file: {}", what,
filepath.string()); filepath.string()));
} }
std::stringstream buffer; std::stringstream buffer;
buffer << input.rdbuf(); buffer << input.rdbuf();
const std::string content = buffer.str();
boost::system::error_code error; boost::system::error_code error;
boost::json::value root = boost::json::parse(content, error); boost::json::value root = boost::json::parse(buffer.str(), error);
if (error) { if (error) {
throw std::runtime_error("Failed to parse locations JSON: " + throw std::runtime_error(
error.message()); std::format("Failed to parse {} JSON: {}", what, error.message()));
} }
return root;
}
/// @brief Returns the first element of a string array field, falling back to
/// `fallback_key` if `key` is missing/empty (some source entries only have a
/// "localized" form and no "romanized" form).
std::string ReadFirstOfStringArray(const boost::json::object& object,
const char* key,
const char* fallback_key) {
for (const char* candidate_key : {key, fallback_key}) {
const boost::json::value* value = object.if_contains(candidate_key);
if (value == nullptr || !value->is_array()) {
continue;
}
const auto& array = value->as_array();
if (!array.empty() && array.front().is_string()) {
return std::string(array.front().as_string());
}
}
throw std::runtime_error(
std::format("Missing or invalid string array field: {}", key));
}
} // namespace
std::vector<Location> JsonLoader::LoadLocations(
const std::filesystem::path& filepath, std::shared_ptr<ILogger> logger) {
const boost::json::value root = ParseJsonFile(filepath, "locations");
if (!root.is_array()) { if (!root.is_array()) {
throw std::runtime_error( throw std::runtime_error(
@@ -108,3 +139,100 @@ std::vector<Location> JsonLoader::LoadLocations(
return locations; return locations;
} }
std::vector<UserPersona> JsonLoader::LoadPersonas(
const std::filesystem::path& filepath, std::shared_ptr<ILogger> logger) {
const boost::json::value root = ParseJsonFile(filepath, "personas");
if (!root.is_array()) {
throw std::runtime_error(
"Invalid personas JSON: root element must be an array");
}
std::vector<UserPersona> personas;
const auto& items = root.as_array();
personas.reserve(items.size());
for (const auto& item : items) {
if (!item.is_object()) {
throw std::runtime_error(
"Invalid personas JSON: each entry must be an object");
}
const auto& object = item.as_object();
personas.push_back(UserPersona{
.name = ReadRequiredString(object, "name"),
.description = ReadRequiredString(object, "description"),
.style_affinities =
ReadRequiredStringArray(object, "style_affinities"),
});
}
return personas;
}
NamesByCountry JsonLoader::LoadNamesByCountry(
const std::filesystem::path& forenames_filepath,
const std::filesystem::path& surnames_filepath,
std::shared_ptr<ILogger> logger) {
const boost::json::value forenames_root =
ParseJsonFile(forenames_filepath, "forenames-by-country");
const boost::json::value surnames_root =
ParseJsonFile(surnames_filepath, "surnames-by-country");
if (!forenames_root.is_object() || !surnames_root.is_object()) {
throw std::runtime_error(
"Invalid names-by-country JSON: root element must be an object "
"keyed by ISO 3166-1 country code");
}
std::unordered_map<std::string, std::vector<ForenameEntry>>
forenames_by_country;
for (const auto& [country_code, regions] : forenames_root.as_object()) {
if (!regions.is_array()) {
continue;
}
std::vector<ForenameEntry> entries;
for (const auto& region : regions.as_array()) {
if (!region.is_object()) {
continue;
}
const boost::json::value* names = region.as_object().if_contains("names");
if (names == nullptr || !names->is_array()) {
continue;
}
for (const auto& name_value : names->as_array()) {
if (!name_value.is_object()) {
continue;
}
const auto& name_object = name_value.as_object();
entries.push_back(ForenameEntry{
.name = ReadFirstOfStringArray(name_object, "romanized",
"localized"),
.gender = ReadRequiredString(name_object, "gender"),
});
}
}
forenames_by_country.emplace(country_code, std::move(entries));
}
std::unordered_map<std::string, std::vector<std::string>>
surnames_by_country;
for (const auto& [country_code, name_entries] : surnames_root.as_object()) {
if (!name_entries.is_array()) {
continue;
}
std::vector<std::string> surnames;
for (const auto& name_value : name_entries.as_array()) {
if (!name_value.is_object()) {
continue;
}
surnames.push_back(ReadFirstOfStringArray(name_value.as_object(),
"romanized", "localized"));
}
surnames_by_country.emplace(country_code, std::move(surnames));
}
return NamesByCountry(std::move(forenames_by_country),
std::move(surnames_by_country));
}

View File

@@ -53,6 +53,7 @@ int main(const int argc, char** argv) {
std::make_shared<LogProducer>(log_channel); std::make_shared<LogProducer>(log_channel);
std::thread log_thread([&log_dispatcher] { log_dispatcher->Run(); }); std::thread log_thread([&log_dispatcher] { log_dispatcher->Run(); });
auto shutdown = [&](const int exit_code) { auto shutdown = [&](const int exit_code) {
log_channel.Close(); log_channel.Close();
log_thread.join(); log_thread.join();

View File

@@ -21,7 +21,7 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
cache_it != this->extract_cache_.end()) { cache_it != this->extract_cache_.end()) {
if (logger_) { if (logger_) {
logger_->Log({.level = LogLevel::Debug, logger_->Log({.level = LogLevel::Debug,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::Enrichment,
.message = std::format("Wikipedia: Cache hit for {}!", cache_key)}); .message = std::format("Wikipedia: Cache hit for {}!", cache_key)});
} }
return cache_it->second; return cache_it->second;
@@ -47,7 +47,7 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log(
{.level = LogLevel::Warn, {.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::Enrichment,
.message = std::format("WikipediaService: JSON parse error for '{}': {}", .message = std::format("WikipediaService: JSON parse error for '{}': {}",
std::string(query), ec.message())}); std::string(query), ec.message())});
} }
@@ -60,7 +60,7 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log(
{.level = LogLevel::Warn, {.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::Enrichment,
.message = .message =
std::format("WikipediaService: Expected root object for '{}'", std::format("WikipediaService: Expected root object for '{}'",
std::string(query))}); std::string(query))});
@@ -78,7 +78,7 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log(
{.level = LogLevel::Warn, {.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::Enrichment,
.message = .message =
std::format("WikipediaService: Missing query.pages for '{}'", std::format("WikipediaService: Missing query.pages for '{}'",
std::string(query))}); std::string(query))});
@@ -92,7 +92,7 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log(
{.level = LogLevel::Warn, {.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::Enrichment,
.message = std::format("WikipediaService: No pages returned for '{}'", .message = std::format("WikipediaService: No pages returned for '{}'",
std::string(query))}); std::string(query))});
} }
@@ -108,7 +108,7 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log(
{.level = LogLevel::Warn, {.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::Enrichment,
.message = .message =
std::format("WikipediaService: Unexpected page format for '{}'", std::format("WikipediaService: Unexpected page format for '{}'",
std::string(query))}); std::string(query))});
@@ -122,7 +122,7 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
if (page.contains("missing")) { if (page.contains("missing")) {
if (logger_) { if (logger_) {
logger_->Log({.level = LogLevel::Warn, logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::Enrichment,
.message = std::format("WikipediaService: Page '{}' does not exist", .message = std::format("WikipediaService: Page '{}' does not exist",
std::string(query))}); std::string(query))});
} }
@@ -136,7 +136,7 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log(
{.level = LogLevel::Warn, {.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::Enrichment,
.message = .message =
std::format("WikipediaService: No extract string found for '{}'", std::format("WikipediaService: No extract string found for '{}'",
std::string(query))}); std::string(query))});
@@ -149,7 +149,7 @@ std::string WikipediaEnrichmentService::FetchExtract(std::string_view query) {
std::string extract(extract_ptr->as_string()); std::string extract(extract_ptr->as_string());
if (logger_) { if (logger_) {
logger_->Log({.level = LogLevel::Info, logger_->Log({.level = LogLevel::Info,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::Enrichment,
.message = std::format("WikipediaService: Fetched {} chars for '{}'", .message = std::format("WikipediaService: Fetched {} chars for '{}'",
extract.size(), std::string(query))}); extract.size(), std::string(query))});
} }

View File

@@ -16,7 +16,7 @@ std::string WikipediaEnrichmentService::GetLocationContext(
if (!this->client_) { if (!this->client_) {
if (logger_) { if (logger_) {
logger_->Log({.level = LogLevel::Warn, logger_->Log({.level = LogLevel::Warn,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::Enrichment,
.message = "Wikipedia client is nullptr."}); .message = "Wikipedia client is nullptr."});
} }
return {}; return {};
@@ -24,13 +24,6 @@ std::string WikipediaEnrichmentService::GetLocationContext(
std::string result; std::string result;
// std::string region_query(loc.city);
// if (!loc.country.empty()) {
// region_query += loc.state_province,
// region_query += ", ";
// region_query += loc.country;
// }
constexpr std::string_view brewing_query = "brewing"; constexpr std::string_view brewing_query = "brewing";
const std::string location_query = const std::string location_query =
std::format("{}, {}", loc.city, loc.iso3166_2); std::format("{}, {}", loc.city, loc.iso3166_2);
@@ -51,7 +44,7 @@ std::string WikipediaEnrichmentService::GetLocationContext(
append_extract(FetchExtract(beer_query)); append_extract(FetchExtract(beer_query));
if (logger_) { if (logger_) {
logger_->Log({.level = LogLevel::Info, logger_->Log({.level = LogLevel::Info,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::Enrichment,
.message = std::format("Done fetching for {}. Sleeping for 10 seconds.", .message = std::format("Done fetching for {}. Sleeping for 10 seconds.",
location_query)}); location_query)});
} }
@@ -61,7 +54,7 @@ std::string WikipediaEnrichmentService::GetLocationContext(
if (logger_) { if (logger_) {
logger_->Log( logger_->Log(
{.level = LogLevel::Debug, {.level = LogLevel::Debug,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::Enrichment,
.message = std::format("WikipediaService lookup failed for '{}': {}", .message = std::format("WikipediaService lookup failed for '{}': {}",
location_query, e.what())}); location_query, e.what())});
} }

View File

@@ -19,6 +19,8 @@ namespace {
switch (phase) { switch (phase) {
case PipelinePhase::Startup: case PipelinePhase::Startup:
return "Startup"; return "Startup";
case PipelinePhase::Enrichment:
return "Enrichment";
case PipelinePhase::UserGeneration: case PipelinePhase::UserGeneration:
return "User Generation"; return "User Generation";
case PipelinePhase::BreweryAndBeerGeneration: case PipelinePhase::BreweryAndBeerGeneration:

View File

@@ -14,6 +14,7 @@ void SqliteExportService::Finalize() {
} }
try { try {
insert_user_stmt_.reset();
insert_brewery_stmt_.reset(); insert_brewery_stmt_.reset();
insert_location_stmt_.reset(); insert_location_stmt_.reset();
if (transaction_open_) { if (transaction_open_) {

View File

@@ -33,6 +33,9 @@ void SqliteExportService::InitializeSchema() const {
sqlite_export_service_internal::ExecSql( sqlite_export_service_internal::ExecSql(
db_handle_, sqlite_export_service_internal::kCreateBreweriesTableSql, db_handle_, sqlite_export_service_internal::kCreateBreweriesTableSql,
"Failed to create SQLite breweries table"); "Failed to create SQLite breweries table");
sqlite_export_service_internal::ExecSql(
db_handle_, sqlite_export_service_internal::kCreateUsersTableSql,
"Failed to create SQLite users table");
} }
void SqliteExportService::PrepareStatements() { void SqliteExportService::PrepareStatements() {
@@ -42,6 +45,9 @@ void SqliteExportService::PrepareStatements() {
insert_brewery_stmt_ = sqlite_export_service_internal::PrepareStatement( insert_brewery_stmt_ = sqlite_export_service_internal::PrepareStatement(
db_handle_, sqlite_export_service_internal::kInsertBrewerySql, db_handle_, sqlite_export_service_internal::kInsertBrewerySql,
"Failed to prepare SQLite brewery insert statement"); "Failed to prepare SQLite brewery insert statement");
insert_user_stmt_ = sqlite_export_service_internal::PrepareStatement(
db_handle_, sqlite_export_service_internal::kInsertUserSql,
"Failed to prepare SQLite user insert statement");
} }
void SqliteExportService::RollbackAndCloseNoThrow() noexcept { void SqliteExportService::RollbackAndCloseNoThrow() noexcept {
@@ -54,6 +60,7 @@ void SqliteExportService::RollbackAndCloseNoThrow() noexcept {
transaction_open_ = false; transaction_open_ = false;
} }
insert_user_stmt_.reset();
insert_brewery_stmt_.reset(); insert_brewery_stmt_.reset();
insert_location_stmt_.reset(); insert_location_stmt_.reset();
db_handle_.reset(); db_handle_.reset();

View File

@@ -1,6 +1,7 @@
/** /**
* @file services/sqlite/process_record.cc * @file services/sqlite/process_record.cc
* @brief SqliteExportService::ProcessRecord() implementation. * @brief SqliteExportService::ProcessRecord(GeneratedBrewery) implementation
* and the shared location-resolution helper.
*/ */
#include <iomanip> #include <iomanip>
@@ -29,82 +30,84 @@ std::string SqliteExportService::BuildLocationKey(const Location& location) {
return key_stream.str(); return key_stream.str();
} }
sqlite3_int64 SqliteExportService::ResolveLocationId(const Location& location) {
const std::string location_key = BuildLocationKey(location);
const auto cached_location = location_cache_.find(location_key);
if (cached_location != location_cache_.end()) {
return cached_location->second;
}
const std::string local_languages_json =
sqlite_export_service_internal::SerializeVector(
location.local_languages);
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationCityBindIndex,
.value = location.city,
.action = "Failed to bind SQLite location city"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index =
sqlite_export_service_internal::kLocationStateProvinceBindIndex,
.value = location.state_province,
.action = "Failed to bind SQLite location state/province"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationIso31662BindIndex,
.value = location.iso3166_2,
.action = "Failed to bind SQLite location ISO 3166-2 code"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationCountryBindIndex,
.value = location.country,
.action = "Failed to bind SQLite location country"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationIso31661BindIndex,
.value = location.iso3166_1,
.action = "Failed to bind SQLite location ISO 3166-1 code"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationLanguagesBindIndex,
.value = local_languages_json,
.action = "Failed to bind SQLite location languages"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam{
.index = sqlite_export_service_internal::kLocationLatitudeBindIndex,
.value = location.latitude,
.action = "Failed to bind SQLite location latitude"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam{
.index = sqlite_export_service_internal::kLocationLongitudeBindIndex,
.value = location.longitude,
.action = "Failed to bind SQLite location longitude"});
sqlite_export_service_internal::StepStatement(
db_handle_, insert_location_stmt_, "Failed to insert SQLite location row");
const sqlite3_int64 location_id =
sqlite_export_service_internal::LastInsertRowId(db_handle_);
location_cache_.emplace(location_key, location_id);
sqlite_export_service_internal::ResetStatement(insert_location_stmt_);
return location_id;
}
uint64_t SqliteExportService::ProcessRecord(const GeneratedBrewery& brewery) { uint64_t SqliteExportService::ProcessRecord(const GeneratedBrewery& brewery) {
if (db_handle_ == nullptr || !transaction_open_) { if (db_handle_ == nullptr || !transaction_open_) {
throw std::runtime_error("SQLite export service is not initialized"); throw std::runtime_error("SQLite export service is not initialized");
} }
const std::string location_key = BuildLocationKey(brewery.location); const sqlite3_int64 location_id = ResolveLocationId(brewery.location);
const auto cached_location = location_cache_.find(location_key);
sqlite3_int64 location_id = 0;
if (cached_location != location_cache_.end()) {
location_id = cached_location->second;
} else {
const std::string local_languages_json =
sqlite_export_service_internal::SerializeVector(
brewery.location.local_languages);
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationCityBindIndex,
.value = brewery.location.city,
.action = "Failed to bind SQLite location city"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index =
sqlite_export_service_internal::kLocationStateProvinceBindIndex,
.value = brewery.location.state_province,
.action = "Failed to bind SQLite location state/province"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationIso31662BindIndex,
.value = brewery.location.iso3166_2,
.action = "Failed to bind SQLite location ISO 3166-2 code"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationCountryBindIndex,
.value = brewery.location.country,
.action = "Failed to bind SQLite location country"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kLocationIso31661BindIndex,
.value = brewery.location.iso3166_1,
.action = "Failed to bind SQLite location ISO 3166-1 code"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index =
sqlite_export_service_internal::kLocationLanguagesBindIndex,
.value = local_languages_json,
.action = "Failed to bind SQLite location languages"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam{
.index = sqlite_export_service_internal::kLocationLatitudeBindIndex,
.value = brewery.location.latitude,
.action = "Failed to bind SQLite location latitude"});
sqlite_export_service_internal::Bind(
insert_location_stmt_,
sqlite_export_service_internal::BindParam{
.index =
sqlite_export_service_internal::kLocationLongitudeBindIndex,
.value = brewery.location.longitude,
.action = "Failed to bind SQLite location longitude"});
sqlite_export_service_internal::StepStatement(
db_handle_, insert_location_stmt_,
"Failed to insert SQLite location row");
location_id = sqlite_export_service_internal::LastInsertRowId(db_handle_);
location_cache_.emplace(location_key, location_id);
sqlite_export_service_internal::ResetStatement(insert_location_stmt_);
}
sqlite_export_service_internal::Bind( sqlite_export_service_internal::Bind(
insert_brewery_stmt_, insert_brewery_stmt_,

View File

@@ -0,0 +1,85 @@
/**
* @file services/sqlite/process_user_record.cc
* @brief SqliteExportService::ProcessRecord(GeneratedUser) implementation.
*/
#include <stdexcept>
#include "services/database/sqlite_export_service.h"
#include "services/database/sqlite_export_service_helpers.h"
uint64_t SqliteExportService::ProcessRecord(const GeneratedUser& user) {
if (db_handle_ == nullptr || !transaction_open_) {
throw std::runtime_error("SQLite export service is not initialized");
}
const sqlite3_int64 location_id = ResolveLocationId(user.location);
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BindParam<sqlite3_int64>{
.index = sqlite_export_service_internal::kUserLocationIdBindIndex,
.value = location_id,
.action = "Failed to bind SQLite user location id"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserFirstNameBindIndex,
.value = user.user.first_name,
.action = "Failed to bind SQLite user first name"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserLastNameBindIndex,
.value = user.user.last_name,
.action = "Failed to bind SQLite user last name"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserGenderBindIndex,
.value = user.user.gender,
.action = "Failed to bind SQLite user gender"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserUsernameBindIndex,
.value = user.user.username,
.action = "Failed to bind SQLite user username"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserBioBindIndex,
.value = user.user.bio,
.action = "Failed to bind SQLite user bio"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BindParam<double>{
.index = sqlite_export_service_internal::kUserActivityWeightBindIndex,
.value = static_cast<double>(user.user.activity_weight),
.action = "Failed to bind SQLite user activity weight"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserEmailBindIndex,
.value = user.email,
.action = "Failed to bind SQLite user email"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserDateOfBirthBindIndex,
.value = user.date_of_birth,
.action = "Failed to bind SQLite user date of birth"});
sqlite_export_service_internal::Bind(
insert_user_stmt_,
sqlite_export_service_internal::BindParam<std::string_view>{
.index = sqlite_export_service_internal::kUserPasswordBindIndex,
.value = user.password,
.action = "Failed to bind SQLite user password"});
sqlite_export_service_internal::StepStatement(
db_handle_, insert_user_stmt_, "Failed to insert SQLite user row");
sqlite_export_service_internal::ResetStatement(insert_user_stmt_);
return sqlite_export_service_internal::LastInsertRowId(db_handle_);
}

View File

@@ -57,7 +57,7 @@ std::string HttpWebClient::Get(const std::string& url) {
if (logger_) { if (logger_) {
logger_->Log( logger_->Log(
{.level = LogLevel::Error, {.level = LogLevel::Error,
.phase = PipelinePhase::UserGeneration, .phase = PipelinePhase::Enrichment,
.message = .message =
std::format("[HttpWebClient] Request failed for URL: {}", url)}); std::format("[HttpWebClient] Request failed for URL: {}", url)});
} }

File diff suppressed because it is too large Load Diff