Add local language handling

pipeline/LLAMA_CPP_GEMMA4_GUIDE.md

@@ -1,565 +0,0 @@
-# A Beginner's Guide to llama.cpp and Google Gemma 4
-
-## Table of Contents
-
-1. [Introduction](#introduction)
-2. [What is llama.cpp?](#what-is-llamacpp)
-3. [What is Google Gemma 4?](#what-is-google-gemma-4)
-4. [Why Use llama.cpp with Gemma 4?](#why-use-llamacpp-with-gemma-4)
-5. [Getting Started with llama.cpp](#getting-started-with-llamacpp)
-6. [Understanding Chat Templates](#understanding-chat-templates)
-7. [Gemma 4's Reasoning Engine](#gemma-4s-reasoning-engine)
-8. [Performance Optimization](#performance-optimization)
-9. [Common Pitfalls](#common-pitfalls)
-10. [References and Further Reading](#references-and-further-reading)
-
----
-
-## Introduction
-
-This guide is designed for developers and AI enthusiasts who want to run large language models locally and efficiently. Whether you're building a chatbot, conducting research, or simply exploring AI capabilities, understanding llama.cpp and Gemma 4 will help you make informed decisions about your setup.
-
-**Target Audience:** Developers with basic C/C++ knowledge, DevOps engineers, and AI practitioners.
-
----
-
-## What is llama.cpp?
-
-### Overview
-
-llama.cpp is a plain C/C++ implementation for Large Language Model (LLM) inference designed to enable efficient LLM inference with minimal setup and state-of-the-art performance across diverse hardware configurations—both locally and in the cloud.[^1]
-
-According to the official project description: *"The main goal of `llama.cpp` is to enable LLM inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud."*[^1]
-
-### Key Features
-
-llama.cpp provides comprehensive support for inference acceleration:
-
-- **Plain C/C++ Implementation:** No complex dependencies, making it portable and lightweight[^1]
-- **Multi-Platform Support:** 
-  - Apple Silicon optimization via ARM NEON, Accelerate, and Metal frameworks[^1]
-  - x86 architectures: AVX, AVX2, AVX512, and AMX support[^1]
-  - RISC-V architectures: RVV, ZVFH, ZFH, ZICBOP, and ZIHINTPAUSE support[^1]
-
-- **Quantization Support:** 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory usage[^1]
-
-- **GPU Acceleration:** 
-  - Custom CUDA kernels for NVIDIA GPUs[^1]
-  - AMD GPU support via HIP[^1]
-  - Vulkan and SYCL backend support[^1]
-
-- **Hybrid Inference:** CPU+GPU hybrid mode for models larger than total VRAM capacity[^1]
-
-### Installation
-
-llama.cpp can be installed through multiple methods:[^1]
-
-```bash
-# Package managers
-brew install llama.cpp          # macOS
-nix flake show github:ggml-org/llama.cpp  # NixOS
-winget install LlamaCpp         # Windows
-
-# Docker
-docker pull ghcr.io/ggml-org/llama.cpp:server-latest
-
-# From source
-git clone https://github.com/ggml-org/llama.cpp
-cd llama.cpp
-make
-```
-
-### Basic Usage
-
-Once installed, running llama.cpp is straightforward:[^1]
-
-```bash
-# Run locally with a GGUF model file
-llama-cli -m my_model.gguf
-
-# Download and run directly from Hugging Face
-llama-cli -hf ggml-org/gemma-3-1b-it-GGUF
-
-# Launch OpenAI-compatible API server
-llama-server -hf ggml-org/gemma-3-1b-it-GGUF
-```
-
----
-
-## What is Google Gemma 4?
-
-### Overview
-
-Google's Gemma is a family of open-source lightweight Large Language Models that represent the latest breakthroughs in AI research. Gemma models are built with the same research and technology used to create Gemini, Google's advanced AI model.[^2]
-
-The Gemma family includes various sizes optimized for different use cases:
-
-- **Gemma 2:** Available in 9B and 27B parameter variants[^3]
-- **Gemma 4:** The latest generation with advanced reasoning and instruction-tuning capabilities
-
-### Model Variants
-
-Gemma models are available in multiple configurations, with "-it" suffix indicating instruction-tuned versions optimized for chat and dialogue:
-
-- **Base Models:** Designed for text completion and continuation
-- **Instruction-Tuned Models (-it):** Fine-tuned for conversational interactions and following instructions[^3]
-
-### Architecture and Training
-
-Gemma models are built on proven transformer architecture with modern training techniques including:
-
-- Flash Attention for efficient attention computation[^4]
-- Robust quantization-friendly training
-- Extensive safety and alignment training
-
-*Reference:* "Gemma models are trained for safety and helpfulness, incorporating feedback from our safety team across all stages of development."[^2]
-
----
-
-## Why Use llama.cpp with Gemma 4?
-
-### Performance and Efficiency
-
-llama.cpp is specifically optimized for inference workloads, making it ideal for running Gemma 4 models:
-
-1. **Speed:** Highly optimized C/C++ implementation delivers faster token generation compared to Python frameworks[^1]
-2. **Memory Efficiency:** Support for aggressive quantization (4-bit, 3-bit) reduces model size significantly[^1]
-3. **Portability:** Run the same model on laptops, desktops, cloud instances, and edge devices[^1]
-4. **Resource Flexibility:** CPU-only inference is viable; GPU acceleration available when hardware permits[^1]
-
-### Use Cases
-
-**Development and Experimentation**
-- Rapid prototyping without GPU requirements
-- Local testing and debugging of prompts
-- Quantization experimentation
-
-**Production Deployment**
-- Low-latency API servers via `llama-server`[^1]
-- OpenAI-compatible REST API endpoints
-- Edge deployment on resource-constrained devices
-
-**Research**
-- Analyzing model behavior at scale
-- Benchmark studies with consistent inference runtime
-- Fine-tuning and adapter experiments
-
----
-
-## Getting Started with llama.cpp
-
-### Step 1: Build from Source
-
-```bash
-# Clone the repository
-git clone https://github.com/ggml-org/llama.cpp.git
-cd llama.cpp
-
-# Build with optimizations (CPU + optional GPU)
-make
-
-# Optional: Build with CUDA support
-make LLAMA_CUDA=1
-
-# Optional: Build with Metal (Apple Silicon)
-make LLAMA_METAL=1
-```
-
-### Step 2: Obtain a Model
-
-Gemma 4 models are available on Hugging Face in GGUF format (optimized for llama.cpp):[^5]
-
-```bash
-# Download Gemma 4 model (automatic via llama.cpp)
-llama-cli -hf google/gemma-4-9b-it-GGUF
-
-# Or manually download from:
-# https://huggingface.co/google/gemma-4-9b-it-GGUF
-```
-
-**GGUF Format:** GGUF (GUFF) is a quantized model format designed for efficient inference in llama.cpp. It stores model weights in a compressed binary format with metadata.[^6]
-
-### Step 3: Run Inference
-
-```bash
-# Interactive chat mode
-llama-cli -m gemma-4-9b-it.gguf -p "Hello, how are you?" -n 256
-
-# With explicit chat template (if needed)
-llama-cli -m gemma-4-9b-it.gguf --chat-template gemma -p "You are a helpful assistant."
-
-# Start API server
-llama-server -m gemma-4-9b-it.gguf -c 2048
-```
-
----
-
-## Understanding Chat Templates
-
-### What are Chat Templates?
-
-Chat templates are Jinja2-based formatting specifications that define how multi-turn conversations are structured for model input.[^7] They ensure consistent formatting of user messages, system prompts, and assistant responses.
-
-According to the llama.cpp documentation: *"Chat templates are Jinja templates that transform a list of messages into a formatted prompt suitable for the model's training format."*[^7]
-
-### Built-in Templates
-
-llama.cpp includes templates for popular models. The "gemma" template is a built-in alias:[^7]
-
-```bash
-# Use built-in Gemma template
-llama-server --chat-template gemma
-
-# List available templates
-llama-cli --list-templates
-```
-
-### Gemma Chat Format
-
-The Gemma chat template uses `<start_of_turn>` and `<end_of_turn>` markers:[^7]
-
-```
-<start_of_turn>user
-What is quantum computing?<end_of_turn>
-<start_of_turn>model
-Quantum computing uses quantum bits (qubits)...<end_of_turn>
-<start_of_turn>user
-Tell me more.<end_of_turn>
-<start_of_turn>model
-```
-
-### Custom Templates
-
-You can provide custom chat templates via file:
-
-```bash
-llama-server -m model.gguf --chat-template-file my_template.jinja
-```
-
-A custom template file example:
-
-```jinja
-{%- for message in messages %}
-[{{ message['role'].upper() }}]
-{{ message['content'] }}
-{% endfor -%}
-```
-
----
-
-## Gemma 4's Reasoning Engine
-
-### Introduction to Reasoning Capabilities
-
-Google Gemma 4 includes advanced reasoning capabilities that enable the model to think through problems step-by-step before generating responses.[^8]
-
-### Activating the Reasoning Engine
-
-To enable Gemma 4's thinking/reasoning mode, prepend the `<|think|>` token to your system prompt:[^8]
-
-```markdown
-<|think|>
-You are a helpful assistant that solves problems step-by-step.
-Please reason through the user's request carefully.
-```
-
-### How It Works
-
-When the reasoning token is detected, the model:
-
-1. **Allocates computational resources** for intermediate reasoning
-2. **Generates internal thoughts** before the final response
-3. **Produces more accurate answers** by working through logic explicitly
-
-### Example Usage
-
-**Without reasoning:**
-```
-Q: What is 47 × 8?
-A: 376
-```
-
-**With reasoning enabled:**
-```
-<|think|>You have advanced reasoning capabilities.
-
-Q: A store sells widgets at $3 each. If they sell 150 per week, 
-   what's their revenue per month assuming 4.3 weeks per month?
-
-A: [Model reasons through calculation internally]
-
-47 × 8 = 376. But let me verify: 40 × 8 = 320, 7 × 8 = 56, 
-so 320 + 56 = 376. ✓
-```
-
-### Implementation in Application Code
-
-In C++, activate reasoning by including the token in your system prompt:
-
-```cpp
-std::string system_prompt = 
-    "<|think|>\n"
-    "You are an expert problem solver that reasons step-by-step.\n"
-    "Always explain your reasoning before providing the answer.";
-
-std::string user_prompt = "What is the square root of 144?";
-
-// Pass to llama_chat_apply_template as normal
-std::string formatted = ToChatPrompt(model, system_prompt, user_prompt);
-```
-
----
-
-## Performance Optimization
-
-### Quantization Strategy
-
-Model quantization reduces file size and memory requirements while maintaining quality. Gemma 4 works well with multiple quantization levels:[^1]
-
-| Quantization | Size Reduction | Quality Impact | Best For |
-|--------------|----------------|----------------|----------|
-| Q8_0 (8-bit) | ~1/8           | Minimal        | Highest quality, CPU inference |
-| Q6_K         | ~1/4           | Very small     | Balanced (recommended) |
-| Q5_K         | ~1/5           | Small          | Good balance |
-| Q4_K_M       | ~1/3           | Noticeable     | GPU inference, moderate quality |
-| Q3_K         | ~1/3           | Moderate       | Limited memory, acceptable quality |
-
-**Recommendation for Gemma 4:** Use Q6_K or Q5_K quantization for optimal quality-to-performance ratio.[^1]
-
-### Buffer Management
-
-When processing prompts, llama.cpp dynamically resizes buffers to accommodate model output:[^9]
-
-```cpp
-// Initial buffer allocation
-std::vector<char> buffer(
-    std::max(min_buffer_size,
-             (system_prompt.size() + user_prompt.size()) * 4));
-
-// If needed, resize on second pass
-if (result >= buffer_size) {
-    buffer.resize(result + 1);  // Resize to actual required size
-    result = llama_chat_apply_template(
-        template_str, messages, n_msg, true, 
-        buffer.data(), static_cast<int32_t>(buffer.size())  // Use NEW size
-    );
-}
-```
-
-**Critical Point:** Always update the size parameter on retry to reflect the resized buffer capacity.[^9]
-
-### Context Window Optimization
-
-Larger context windows enable longer conversations but use more memory:
-
-```bash
-# Default context (2048 tokens)
-llama-server -m model.gguf
-
-# Larger context for longer conversations
-llama-server -m model.gguf -c 4096
-
-# Maximum context (may require GPU)
-llama-server -m model.gguf -c 16384 -ngl 35  # GPU layers
-```
-
----
-
-## Common Pitfalls
-
-### 1. Template Metadata Missing from GGUF
-
-**Problem:** Model lacks chat template metadata, causing fallback to raw text.
-
-**Solution:** Use the built-in "gemma" alias when metadata is unavailable:
-
-```cpp
-const char* tmpl = llama_model_chat_template(model, nullptr);
-if (tmpl == nullptr) {
-    tmpl = "gemma";  // Fall back to built-in alias
-}
-```
-
-### 2. Buffer Overflow During Template Application
-
-**Problem:** Initial buffer too small, causing truncated output.
-
-**Solution:** Implement dynamic resizing with correct size update:
-
-```cpp
-int32_t result = llama_chat_apply_template(
-    template_str, messages, msg_count, true,
-    buffer.data(), static_cast<int32_t>(buffer.size()));
-
-if (result >= static_cast<int32_t>(buffer.size())) {
-    buffer.resize(result + 1);
-    // IMPORTANT: Pass new buffer size
-    result = llama_chat_apply_template(
-        template_str, messages, msg_count, true,
-        buffer.data(), static_cast<int32_t>(buffer.size())  // New size!
-    );
-}
-```
-
-### 3. Incorrect System Prompt Format
-
-**Problem:** System prompt not recognized by Gemma template.
-
-**Solution:** Use standard role-based format with `<start_of_turn>`:
-
-```
-✓ Correct:
-<start_of_turn>user
-Your question here<end_of_turn>
-
-✗ Incorrect:
-System: [prompt]
-User: [question]
-```
-
-### 4. Token Limit Exceeded
-
-**Problem:** "Token count exceeds context window" errors.
-
-**Solution:** Check and limit input size before inference:
-
-```cpp
-const size_t max_tokens = context_size - safety_buffer;
-if (tokens.size() > max_tokens) {
-    // Truncate or summarize input
-    tokens.resize(max_tokens);
-}
-```
-
-### 5. GPU Memory Exhaustion
-
-**Problem:** Out of VRAM during inference.
-
-**Solution:** Reduce GPU layers or use CPU+GPU hybrid:
-
-```bash
-# Reduce GPU-accelerated layers
-llama-server -m model.gguf -ngl 20
-
-# Use hybrid inference
-llama-server -m model.gguf -ngl 15  # Only load 15 layers on GPU
-```
-
----
-
-## References and Further Reading
-
-### Official Documentation
-
-[^1]: **llama.cpp GitHub Repository**
-    - URL: https://github.com/ggml-org/llama.cpp
-    - Content: Official README with installation, build, and usage instructions
-    - Accessed: April 16, 2026
-
-[^7]: **llama.cpp Chat Template Documentation**
-    - URL: https://github.com/ggml-org/llama.cpp/wiki/Templates-supported-by-llama_chat_apply_template
-    - Content: Comprehensive guide to chat templates and built-in aliases including "gemma"
-    - Accessed: April 16, 2026
-
-### Google Gemma Resources
-
-[^2]: **Google Gemma Official Page**
-    - URL: https://ai.google.dev/gemma
-    - Content: Overview of Gemma model family, architecture, and training details
-    - Accessed: April 16, 2026
-
-[^3]: **Gemma 2 on Hugging Face**
-    - URL: https://huggingface.co/google/gemma-2-9b-it
-    - Content: Model card with architecture details, downloads: 324,845
-    - Accessed: April 16, 2026
-
-[^4]: **Google AI Blog: Gemma Training Details**
-    - URL: https://ai.google.dev/gemma/docs
-    - Content: Technical details on Flash Attention, quantization training, and safety alignment
-    - Accessed: April 16, 2026
-
-[^8]: **Google Gemma Thinking/Reasoning Documentation**
-    - URL: https://ai.google.dev/gemma/docs/capabilities/thinking
-    - Content: Guide to enabling and using Gemma 4's advanced reasoning engine
-    - Accessed: April 16, 2026
-
-### Technical References
-
-[^5]: **Gemma 4 GGUF Models on Hugging Face**
-    - URL: https://huggingface.co/google/gemma-4-9b-it-GGUF
-    - Content: GGUF quantized models optimized for llama.cpp inference
-    - Accessed: April 16, 2026
-
-[^6]: **GGUF Format Specification**
-    - URL: https://github.com/ggml-org/ggml/blob/master/docs/gguf.md
-    - Content: Technical specification of the GGUF binary format for quantized models
-    - Accessed: April 16, 2026
-
-[^9]: **llama.cpp API Reference: Chat Template Application**
-    - URL: https://github.com/ggml-org/llama.cpp/blob/master/include/llama.h
-    - Content: `llama_chat_apply_template()` function signature and buffer management patterns
-    - Accessed: April 16, 2026
-
-### Additional Resources
-
-- **llama.cpp Build Guide:** https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md
-- **Model Quantization Guide:** https://github.com/ggml-org/llama.cpp/blob/master/docs/quantization.md
-- **Docker Support:** https://github.com/ggml-org/llama.cpp/blob/master/docs/docker.md
-- **Hugging Face Model Hub:** https://huggingface.co/models?search=gemma
-
----
-
-## Quick Reference Card
-
-### Common Commands
-
-```bash
-# Interactive chat
-llama-cli -m model.gguf --chat-template gemma
-
-# Start API server
-llama-server -m model.gguf -c 2048
-
-# With GPU acceleration
-llama-server -m model.gguf -ngl 35 -c 4096
-
-# Download and run from Hugging Face
-llama-cli -hf google/gemma-4-9b-it-GGUF
-```
-
-### System Prompt Template for Gemma 4 with Reasoning
-
-```markdown
-<|think|>
-[Model will allocate reasoning resources here]
-
-You are an expert assistant trained to solve problems carefully.
-Your role is to:
-1. Understand the user's question completely
-2. Think through the solution step-by-step
-3. Provide accurate and helpful responses
-4. Explain your reasoning when helpful
-```
-
-### Recommended Settings
-
-- **Model:** Gemma-4-9B-IT (9B parameter instruction-tuned variant)
-- **Quantization:** Q6_K (best quality-performance balance)
-- **Context:** 4096 tokens (good balance for most use cases)
-- **Temperature:** 0.7 (balanced creativity and consistency)
-- **Top-P:** 0.95 (good diversity without nonsense)
-
----
-
-## Conclusion
-
-llama.cpp and Google Gemma 4 represent a powerful combination for running state-of-the-art language models efficiently on various hardware configurations. By understanding chat templates, reasoning capabilities, and performance optimization techniques, you can build robust AI applications that leverage these technologies effectively.
-
-For the latest updates and community support, join the llama.cpp community discussions at https://github.com/ggml-org/llama.cpp/discussions.
-
----
-
-**Last Updated:** April 16, 2026  
-**Guide Version:** 1.0  
-**Compatible With:** llama.cpp b8742+, Gemma 4 models
-

pipeline/README.md

@@ -4,15 +4,6 @@ Biergarten Pipeline is a C++20 command-line tool that reads a local city list, r
 
 ## Tested Hardware & OS
 
-### x86/64 Linux, NVIDIA RTX 2000
-
-- **Host**: ThinkPad P1 Gen 7 (Fedora 43)
-- **CPU**: Intel Core Ultra 7 155H
-- **GPU**: NVIDIA RTX 2000 Ada Generation
-- **Memory**: 32GB
-- **Model**: Gemma 4 E4B: efficient local reasoning; released Apr 2, 2026.
-- **Inference**: llama.cpp with CUDA 12.x support
-
 ### ARM MacOS, M1 Pro
 
 - **Host**: MacBook Pro 14" (2021)
@@ -22,6 +13,15 @@ Biergarten Pipeline is a C++20 command-line tool that reads a local city list, r
 - **Model**: Gemma 4 E4B: efficient local reasoning; released Apr 2, 2026.
 - **Inference**: llama.cpp with Metal (MPS) support
 
+### x86/64 Linux, NVIDIA RTX 2000
+
+- **Host**: ThinkPad P1 Gen 7 (Fedora 43)
+- **CPU**: Intel Core Ultra 7 155H
+- **GPU**: NVIDIA RTX 2000 Ada Generation
+- **Memory**: 32GB
+- **Model**: Gemma 4 E4B: efficient local reasoning; released Apr 2, 2026.
+- **Inference**: llama.cpp with CUDA 12.x support
+
 ## Pipeline
 
 | Stage    | What happens                                                            |

pipeline/includes/data_generation/llama_generator.h

@@ -36,7 +36,7 @@ class LlamaGenerator final : public DataGenerator {
    */
   LlamaGenerator(const ApplicationOptions& options,
                  const std::string& model_path,
-                 std::shared_ptr<IPromptFormatter> prompt_formatter);
+                 std::unique_ptr<IPromptFormatter> prompt_formatter);
 
   ~LlamaGenerator() override;
 

pipeline/includes/data_model/location.h

@@ -7,6 +7,7 @@
  */
 
 #include <string>
+#include <vector>
 
 /**
  * @brief Canonical location record for city-level generation.
@@ -27,6 +28,9 @@ struct Location {
   /// @brief ISO 3166-1 country code.
   std::string iso3166_1{};
 
+  /// @brief Local language codes in priority order.
+  std::vector<std::string> local_languages{};
+
   /// @brief Latitude in decimal degrees.
   double latitude{};
 

pipeline/locations.json

@@ -6,7 +6,8 @@
     "country": "South Africa",
     "iso3166_1": "ZA",
     "latitude": -33.9249,
-    "longitude": 18.4241
+    "longitude": 18.4241,
+    "local_languages": ["af", "en", "xh"]
   },
   {
     "city": "Johannesburg",
@@ -15,7 +16,8 @@
     "country": "South Africa",
     "iso3166_1": "ZA",
     "latitude": -26.2041,
-    "longitude": 28.0473
+    "longitude": 28.0473,
+    "local_languages": ["en", "zu", "st", "af"]
   },
   {
     "city": "Durban",
@@ -24,7 +26,8 @@
     "country": "South Africa",
     "iso3166_1": "ZA",
     "latitude": -29.8587,
-    "longitude": 31.0218
+    "longitude": 31.0218,
+    "local_languages": ["zu", "en"]
   },
   {
     "city": "Franschhoek",
@@ -33,7 +36,8 @@
     "country": "South Africa",
     "iso3166_1": "ZA",
     "latitude": -33.9146,
-    "longitude": 19.1198
+    "longitude": 19.1198,
+    "local_languages": ["af", "en"]
   },
   {
     "city": "Nairobi",
@@ -42,7 +46,8 @@
     "country": "Kenya",
     "iso3166_1": "KE",
     "latitude": -1.2921,
-    "longitude": 36.8219
+    "longitude": 36.8219,
+    "local_languages": ["sw", "en"]
   },
   {
     "city": "Buenos Aires",
@@ -51,7 +56,8 @@
     "country": "Argentina",
     "iso3166_1": "AR",
     "latitude": -34.6037,
-    "longitude": -58.3816
+    "longitude": -58.3816,
+    "local_languages": ["es-AR"]
   },
   {
     "city": "Bariloche",
@@ -60,7 +66,8 @@
     "country": "Argentina",
     "iso3166_1": "AR",
     "latitude": -41.1335,
-    "longitude": -71.3103
+    "longitude": -71.3103,
+    "local_languages": ["es-AR"]
   },
   {
     "city": "Bogotá",
@@ -69,7 +76,8 @@
     "country": "Colombia",
     "iso3166_1": "CO",
     "latitude": 4.711,
-    "longitude": -74.0721
+    "longitude": -74.0721,
+    "local_languages": ["es-CO"]
   },
   {
     "city": "Medellín",
@@ -78,7 +86,8 @@
     "country": "Colombia",
     "iso3166_1": "CO",
     "latitude": 6.2442,
-    "longitude": -75.5812
+    "longitude": -75.5812,
+    "local_languages": ["es-CO"]
   },
   {
     "city": "São Paulo",
@@ -87,7 +96,8 @@
     "country": "Brazil",
     "iso3166_1": "BR",
     "latitude": -23.5505,
-    "longitude": -46.6333
+    "longitude": -46.6333,
+    "local_languages": ["pt-BR"]
   },
   {
     "city": "Curitiba",
@@ -96,7 +106,8 @@
     "country": "Brazil",
     "iso3166_1": "BR",
     "latitude": -25.4284,
-    "longitude": -49.2733
+    "longitude": -49.2733,
+    "local_languages": ["pt-BR"]
   },
   {
     "city": "Rio de Janeiro",
@@ -105,7 +116,8 @@
     "country": "Brazil",
     "iso3166_1": "BR",
     "latitude": -22.9068,
-    "longitude": -43.1729
+    "longitude": -43.1729,
+    "local_languages": ["pt-BR"]
   },
   {
     "city": "Santiago",
@@ -114,7 +126,8 @@
     "country": "Chile",
     "iso3166_1": "CL",
     "latitude": -33.4489,
-    "longitude": -70.6693
+    "longitude": -70.6693,
+    "local_languages": ["es-CL"]
   },
   {
     "city": "Valdivia",
@@ -123,7 +136,8 @@
     "country": "Chile",
     "iso3166_1": "CL",
     "latitude": -39.8142,
-    "longitude": -73.2459
+    "longitude": -73.2459,
+    "local_languages": ["es-CL"]
   },
   {
     "city": "Lima",
@@ -132,7 +146,8 @@
     "country": "Peru",
     "iso3166_1": "PE",
     "latitude": -12.0464,
-    "longitude": -77.0428
+    "longitude": -77.0428,
+    "local_languages": ["es-PE"]
   },
   {
     "city": "Tokyo",
@@ -141,7 +156,8 @@
     "country": "Japan",
     "iso3166_1": "JP",
     "latitude": 35.6762,
-    "longitude": 139.6503
+    "longitude": 139.6503,
+    "local_languages": ["ja"]
   },
   {
     "city": "Osaka",
@@ -150,7 +166,8 @@
     "country": "Japan",
     "iso3166_1": "JP",
     "latitude": 34.6937,
-    "longitude": 135.5023
+    "longitude": 135.5023,
+    "local_languages": ["ja"]
   },
   {
     "city": "Kyoto",
@@ -159,7 +176,8 @@
     "country": "Japan",
     "iso3166_1": "JP",
     "latitude": 35.0116,
-    "longitude": 135.7681
+    "longitude": 135.7681,
+    "local_languages": ["ja"]
   },
   {
     "city": "Sapporo",
@@ -168,7 +186,8 @@
     "country": "Japan",
     "iso3166_1": "JP",
     "latitude": 43.0618,
-    "longitude": 141.3545
+    "longitude": 141.3545,
+    "local_languages": ["ja"]
   },
   {
     "city": "Seoul",
@@ -177,7 +196,8 @@
     "country": "South Korea",
     "iso3166_1": "KR",
     "latitude": 37.5665,
-    "longitude": 126.978
+    "longitude": 126.978,
+    "local_languages": ["ko"]
   },
   {
     "city": "Busan",
@@ -186,7 +206,8 @@
     "country": "South Korea",
     "iso3166_1": "KR",
     "latitude": 35.1796,
-    "longitude": 129.0756
+    "longitude": 129.0756,
+    "local_languages": ["ko"]
   },
   {
     "city": "Ho Chi Minh City",
@@ -195,7 +216,8 @@
     "country": "Vietnam",
     "iso3166_1": "VN",
     "latitude": 10.8231,
-    "longitude": 106.6297
+    "longitude": 106.6297,
+    "local_languages": ["vi"]
   },
   {
     "city": "Hanoi",
@@ -204,7 +226,8 @@
     "country": "Vietnam",
     "iso3166_1": "VN",
     "latitude": 21.0285,
-    "longitude": 105.8542
+    "longitude": 105.8542,
+    "local_languages": ["vi"]
   },
   {
     "city": "Da Nang",
@@ -213,7 +236,8 @@
     "country": "Vietnam",
     "iso3166_1": "VN",
     "latitude": 16.0544,
-    "longitude": 108.2022
+    "longitude": 108.2022,
+    "local_languages": ["vi"]
   },
   {
     "city": "Bangkok",
@@ -222,7 +246,8 @@
     "country": "Thailand",
     "iso3166_1": "TH",
     "latitude": 13.7563,
-    "longitude": 100.5018
+    "longitude": 100.5018,
+    "local_languages": ["th"]
   },
   {
     "city": "Taipei",
@@ -231,7 +256,8 @@
     "country": "Taiwan",
     "iso3166_1": "TW",
     "latitude": 25.033,
-    "longitude": 121.5654
+    "longitude": 121.5654,
+    "local_languages": ["zh-TW"]
   },
   {
     "city": "Beijing",
@@ -240,7 +266,8 @@
     "country": "China",
     "iso3166_1": "CN",
     "latitude": 39.9042,
-    "longitude": 116.4074
+    "longitude": 116.4074,
+    "local_languages": ["zh-CN"]
   },
   {
     "city": "Shanghai",
@@ -249,7 +276,8 @@
     "country": "China",
     "iso3166_1": "CN",
     "latitude": 31.2304,
-    "longitude": 121.4737
+    "longitude": 121.4737,
+    "local_languages": ["zh-CN"]
   },
   {
     "city": "Bengaluru",
@@ -258,7 +286,8 @@
     "country": "India",
     "iso3166_1": "IN",
     "latitude": 12.9716,
-    "longitude": 77.5946
+    "longitude": 77.5946,
+    "local_languages": ["kn", "en"]
   },
   {
     "city": "Singapore",
@@ -267,7 +296,8 @@
     "country": "Singapore",
     "iso3166_1": "SG",
     "latitude": 1.3521,
-    "longitude": 103.8198
+    "longitude": 103.8198,
+    "local_languages": ["en", "zh", "ms", "ta"]
   },
   {
     "city": "Melbourne",
@@ -276,7 +306,8 @@
     "country": "Australia",
     "iso3166_1": "AU",
     "latitude": -37.8136,
-    "longitude": 144.9631
+    "longitude": 144.9631,
+    "local_languages": ["en-AU"]
   },
   {
     "city": "Sydney",
@@ -285,7 +316,8 @@
     "country": "Australia",
     "iso3166_1": "AU",
     "latitude": -33.8688,
-    "longitude": 151.2093
+    "longitude": 151.2093,
+    "local_languages": ["en-AU"]
   },
   {
     "city": "Brisbane",
@@ -294,7 +326,8 @@
     "country": "Australia",
     "iso3166_1": "AU",
     "latitude": -27.4705,
-    "longitude": 153.026
+    "longitude": 153.026,
+    "local_languages": ["en-AU"]
   },
   {
     "city": "Adelaide",
@@ -303,7 +336,8 @@
     "country": "Australia",
     "iso3166_1": "AU",
     "latitude": -34.9285,
-    "longitude": 138.6007
+    "longitude": 138.6007,
+    "local_languages": ["en-AU"]
   },
   {
     "city": "Perth",
@@ -312,7 +346,8 @@
     "country": "Australia",
     "iso3166_1": "AU",
     "latitude": -31.9505,
-    "longitude": 115.8605
+    "longitude": 115.8605,
+    "local_languages": ["en-AU"]
   },
   {
     "city": "Hobart",
@@ -321,7 +356,8 @@
     "country": "Australia",
     "iso3166_1": "AU",
     "latitude": -42.8821,
-    "longitude": 147.3272
+    "longitude": 147.3272,
+    "local_languages": ["en-AU"]
   },
   {
     "city": "Wellington",
@@ -330,7 +366,8 @@
     "country": "New Zealand",
     "iso3166_1": "NZ",
     "latitude": -41.2865,
-    "longitude": 174.7762
+    "longitude": 174.7762,
+    "local_languages": ["en", "mi"]
   },
   {
     "city": "Auckland",
@@ -339,7 +376,8 @@
     "country": "New Zealand",
     "iso3166_1": "NZ",
     "latitude": -36.8485,
-    "longitude": 174.7633
+    "longitude": 174.7633,
+    "local_languages": ["en", "mi"]
   },
   {
     "city": "Christchurch",
@@ -348,7 +386,8 @@
     "country": "New Zealand",
     "iso3166_1": "NZ",
     "latitude": -43.532,
-    "longitude": 172.6306
+    "longitude": 172.6306,
+    "local_languages": ["en", "mi"]
   },
   {
     "city": "Nelson",
@@ -357,7 +396,8 @@
     "country": "New Zealand",
     "iso3166_1": "NZ",
     "latitude": -41.2706,
-    "longitude": 173.284
+    "longitude": 173.284,
+    "local_languages": ["en", "mi"]
   },
   {
     "city": "Munich",
@@ -366,7 +406,8 @@
     "country": "Germany",
     "iso3166_1": "DE",
     "latitude": 48.1351,
-    "longitude": 11.582
+    "longitude": 11.582,
+    "local_languages": ["de"]
   },
   {
     "city": "Berlin",
@@ -375,7 +416,8 @@
     "country": "Germany",
     "iso3166_1": "DE",
     "latitude": 52.52,
-    "longitude": 13.405
+    "longitude": 13.405,
+    "local_languages": ["de"]
   },
   {
     "city": "Cologne",
@@ -384,7 +426,8 @@
     "country": "Germany",
     "iso3166_1": "DE",
     "latitude": 50.9375,
-    "longitude": 6.9603
+    "longitude": 6.9603,
+    "local_languages": ["de"]
   },
   {
     "city": "Bamberg",
@@ -393,7 +436,8 @@
     "country": "Germany",
     "iso3166_1": "DE",
     "latitude": 49.8916,
-    "longitude": 10.8916
+    "longitude": 10.8916,
+    "local_languages": ["de"]
   },
   {
     "city": "Brussels",
@@ -402,7 +446,8 @@
     "country": "Belgium",
     "iso3166_1": "BE",
     "latitude": 50.8503,
-    "longitude": 4.3517
+    "longitude": 4.3517,
+    "local_languages": ["fr", "nl"]
   },
   {
     "city": "Antwerp",
@@ -411,7 +456,8 @@
     "country": "Belgium",
     "iso3166_1": "BE",
     "latitude": 51.2194,
-    "longitude": 4.4025
+    "longitude": 4.4025,
+    "local_languages": ["nl"]
   },
   {
     "city": "Bruges",
@@ -420,7 +466,8 @@
     "country": "Belgium",
     "iso3166_1": "BE",
     "latitude": 51.2093,
-    "longitude": 3.2247
+    "longitude": 3.2247,
+    "local_languages": ["nl"]
   },
   {
     "city": "London",
@@ -429,7 +476,8 @@
     "country": "United Kingdom",
     "iso3166_1": "GB",
     "latitude": 51.5074,
-    "longitude": -0.1278
+    "longitude": -0.1278,
+    "local_languages": ["en-GB"]
   },
   {
     "city": "Bristol",
@@ -438,7 +486,8 @@
     "country": "United Kingdom",
     "iso3166_1": "GB",
     "latitude": 51.4545,
-    "longitude": -2.5879
+    "longitude": -2.5879,
+    "local_languages": ["en-GB"]
   },
   {
     "city": "Edinburgh",
@@ -447,7 +496,8 @@
     "country": "United Kingdom",
     "iso3166_1": "GB",
     "latitude": 55.9533,
-    "longitude": -3.1883
+    "longitude": -3.1883,
+    "local_languages": ["en-GB", "gd"]
   },
   {
     "city": "Glasgow",
@@ -456,7 +506,8 @@
     "country": "United Kingdom",
     "iso3166_1": "GB",
     "latitude": 55.8642,
-    "longitude": -4.2518
+    "longitude": -4.2518,
+    "local_languages": ["en-GB", "gd"]
   },
   {
     "city": "Prague",
@@ -465,7 +516,8 @@
     "country": "Czechia",
     "iso3166_1": "CZ",
     "latitude": 50.0755,
-    "longitude": 14.4378
+    "longitude": 14.4378,
+    "local_languages": ["cs"]
   },
   {
     "city": "Pilsen",
@@ -474,7 +526,8 @@
     "country": "Czechia",
     "iso3166_1": "CZ",
     "latitude": 49.7384,
-    "longitude": 13.3736
+    "longitude": 13.3736,
+    "local_languages": ["cs"]
   },
   {
     "city": "Amsterdam",
@@ -483,7 +536,8 @@
     "country": "Netherlands",
     "iso3166_1": "NL",
     "latitude": 52.3676,
-    "longitude": 4.9041
+    "longitude": 4.9041,
+    "local_languages": ["nl"]
   },
   {
     "city": "Copenhagen",
@@ -492,7 +546,8 @@
     "country": "Denmark",
     "iso3166_1": "DK",
     "latitude": 55.6761,
-    "longitude": 12.5683
+    "longitude": 12.5683,
+    "local_languages": ["da"]
   },
   {
     "city": "Warsaw",
@@ -501,7 +556,8 @@
     "country": "Poland",
     "iso3166_1": "PL",
     "latitude": 52.2297,
-    "longitude": 21.0122
+    "longitude": 21.0122,
+    "local_languages": ["pl"]
   },
   {
     "city": "Krakow",
@@ -510,7 +566,8 @@
     "country": "Poland",
     "iso3166_1": "PL",
     "latitude": 50.0647,
-    "longitude": 19.945
+    "longitude": 19.945,
+    "local_languages": ["pl"]
   },
   {
     "city": "Rome",
@@ -519,7 +576,8 @@
     "country": "Italy",
     "iso3166_1": "IT",
     "latitude": 41.9028,
-    "longitude": 12.4964
+    "longitude": 12.4964,
+    "local_languages": ["it"]
   },
   {
     "city": "Milan",
@@ -528,7 +586,8 @@
     "country": "Italy",
     "iso3166_1": "IT",
     "latitude": 45.4642,
-    "longitude": 9.19
+    "longitude": 9.19,
+    "local_languages": ["it"]
   },
   {
     "city": "Barcelona",
@@ -537,7 +596,8 @@
     "country": "Spain",
     "iso3166_1": "ES",
     "latitude": 41.3851,
-    "longitude": 2.1734
+    "longitude": 2.1734,
+    "local_languages": ["ca", "es"]
   },
   {
     "city": "Madrid",
@@ -546,7 +606,8 @@
     "country": "Spain",
     "iso3166_1": "ES",
     "latitude": 40.4168,
-    "longitude": -3.7038
+    "longitude": -3.7038,
+    "local_languages": ["es"]
   },
   {
     "city": "Paris",
@@ -555,7 +616,8 @@
     "country": "France",
     "iso3166_1": "FR",
     "latitude": 48.8566,
-    "longitude": 2.3522
+    "longitude": 2.3522,
+    "local_languages": ["fr"]
   },
   {
     "city": "Lyon",
@@ -564,7 +626,8 @@
     "country": "France",
     "iso3166_1": "FR",
     "latitude": 45.764,
-    "longitude": 4.8357
+    "longitude": 4.8357,
+    "local_languages": ["fr"]
   },
   {
     "city": "Stockholm",
@@ -573,7 +636,8 @@
     "country": "Sweden",
     "iso3166_1": "SE",
     "latitude": 59.3293,
-    "longitude": 18.0686
+    "longitude": 18.0686,
+    "local_languages": ["sv"]
   },
   {
     "city": "Gothenburg",
@@ -582,7 +646,8 @@
     "country": "Sweden",
     "iso3166_1": "SE",
     "latitude": 57.7089,
-    "longitude": 11.9746
+    "longitude": 11.9746,
+    "local_languages": ["sv"]
   },
   {
     "city": "Oslo",
@@ -591,7 +656,8 @@
     "country": "Norway",
     "iso3166_1": "NO",
     "latitude": 59.9139,
-    "longitude": 10.7522
+    "longitude": 10.7522,
+    "local_languages": ["no"]
   },
   {
     "city": "Dublin",
@@ -600,7 +666,8 @@
     "country": "Ireland",
     "iso3166_1": "IE",
     "latitude": 53.3498,
-    "longitude": -6.2603
+    "longitude": -6.2603,
+    "local_languages": ["en", "ga"]
   },
   {
     "city": "Vienna",
@@ -609,7 +676,8 @@
     "country": "Austria",
     "iso3166_1": "AT",
     "latitude": 48.2082,
-    "longitude": 16.3738
+    "longitude": 16.3738,
+    "local_languages": ["de-AT"]
   },
   {
     "city": "Zurich",
@@ -618,7 +686,8 @@
     "country": "Switzerland",
     "iso3166_1": "CH",
     "latitude": 47.3769,
-    "longitude": 8.5417
+    "longitude": 8.5417,
+    "local_languages": ["de-CH"]
   },
   {
     "city": "Tallinn",
@@ -627,7 +696,8 @@
     "country": "Estonia",
     "iso3166_1": "EE",
     "latitude": 59.437,
-    "longitude": 24.7536
+    "longitude": 24.7536,
+    "local_languages": ["et"]
   },
   {
     "city": "Denver",
@@ -636,7 +706,8 @@
     "country": "United States",
     "iso3166_1": "US",
     "latitude": 39.7392,
-    "longitude": -104.9903
+    "longitude": -104.9903,
+    "local_languages": ["en-US"]
   },
   {
     "city": "Portland",
@@ -645,7 +716,8 @@
     "country": "United States",
     "iso3166_1": "US",
     "latitude": 45.5152,
-    "longitude": -122.6784
+    "longitude": -122.6784,
+    "local_languages": ["en-US"]
   },
   {
     "city": "San Diego",
@@ -654,7 +726,8 @@
     "country": "United States",
     "iso3166_1": "US",
     "latitude": 32.7157,
-    "longitude": -117.1611
+    "longitude": -117.1611,
+    "local_languages": ["en-US", "es-US"]
   },
   {
     "city": "Asheville",
@@ -663,7 +736,8 @@
     "country": "United States",
     "iso3166_1": "US",
     "latitude": 35.5951,
-    "longitude": -82.5515
+    "longitude": -82.5515,
+    "local_languages": ["en-US"]
   },
   {
     "city": "Grand Rapids",
@@ -672,7 +746,8 @@
     "country": "United States",
     "iso3166_1": "US",
     "latitude": 42.9634,
-    "longitude": -85.6681
+    "longitude": -85.6681,
+    "local_languages": ["en-US"]
   },
   {
     "city": "Chicago",
@@ -681,7 +756,8 @@
     "country": "United States",
     "iso3166_1": "US",
     "latitude": 41.8781,
-    "longitude": -87.6298
+    "longitude": -87.6298,
+    "local_languages": ["en-US", "es-US"]
   },
   {
     "city": "Seattle",
@@ -690,7 +766,8 @@
     "country": "United States",
     "iso3166_1": "US",
     "latitude": 47.6062,
-    "longitude": -122.3321
+    "longitude": -122.3321,
+    "local_languages": ["en-US"]
   },
   {
     "city": "Austin",
@@ -699,7 +776,8 @@
     "country": "United States",
     "iso3166_1": "US",
     "latitude": 30.2672,
-    "longitude": -97.7431
+    "longitude": -97.7431,
+    "local_languages": ["en-US", "es-US"]
   },
   {
     "city": "Boston",
@@ -708,7 +786,8 @@
     "country": "United States",
     "iso3166_1": "US",
     "latitude": 42.3601,
-    "longitude": -71.0589
+    "longitude": -71.0589,
+    "local_languages": ["en-US"]
   },
   {
     "city": "Philadelphia",
@@ -717,7 +796,8 @@
     "country": "United States",
     "iso3166_1": "US",
     "latitude": 39.9526,
-    "longitude": -75.1652
+    "longitude": -75.1652,
+    "local_languages": ["en-US"]
   },
   {
     "city": "Brooklyn",
@@ -726,7 +806,8 @@
     "country": "United States",
     "iso3166_1": "US",
     "latitude": 40.6782,
-    "longitude": -73.9442
+    "longitude": -73.9442,
+    "local_languages": ["en-US", "es-US"]
   },
   {
     "city": "Milwaukee",
@@ -735,7 +816,8 @@
     "country": "United States",
     "iso3166_1": "US",
     "latitude": 43.0389,
-    "longitude": -87.9065
+    "longitude": -87.9065,
+    "local_languages": ["en-US"]
   },
   {
     "city": "Richmond",
@@ -744,7 +826,8 @@
     "country": "United States",
     "iso3166_1": "US",
     "latitude": 37.5407,
-    "longitude": -77.436
+    "longitude": -77.436,
+    "local_languages": ["en-US"]
   },
   {
     "city": "Cincinnati",
@@ -753,7 +836,8 @@
     "country": "United States",
     "iso3166_1": "US",
     "latitude": 39.1031,
-    "longitude": -84.512
+    "longitude": -84.512,
+    "local_languages": ["en-US"]
   },
   {
     "city": "St. Louis",
@@ -762,7 +846,8 @@
     "country": "United States",
     "iso3166_1": "US",
     "latitude": 38.627,
-    "longitude": -90.1994
+    "longitude": -90.1994,
+    "local_languages": ["en-US"]
   },
   {
     "city": "Tampa",
@@ -771,7 +856,8 @@
     "country": "United States",
     "iso3166_1": "US",
     "latitude": 27.9506,
-    "longitude": -82.4572
+    "longitude": -82.4572,
+    "local_languages": ["en-US", "es-US"]
   },
   {
     "city": "Minneapolis",
@@ -780,7 +866,8 @@
     "country": "United States",
     "iso3166_1": "US",
     "latitude": 44.9778,
-    "longitude": -93.265
+    "longitude": -93.265,
+    "local_languages": ["en-US"]
   },
   {
     "city": "Burlington",
@@ -789,7 +876,8 @@
     "country": "United States",
     "iso3166_1": "US",
     "latitude": 44.4759,
-    "longitude": -73.2121
+    "longitude": -73.2121,
+    "local_languages": ["en-US"]
   },
   {
     "city": "Portland",
@@ -798,7 +886,8 @@
     "country": "United States",
     "iso3166_1": "US",
     "latitude": 43.6591,
-    "longitude": -70.2568
+    "longitude": -70.2568,
+    "local_languages": ["en-US"]
   },
   {
     "city": "Atlanta",
@@ -807,7 +896,8 @@
     "country": "United States",
     "iso3166_1": "US",
     "latitude": 33.749,
-    "longitude": -84.388
+    "longitude": -84.388,
+    "local_languages": ["en-US"]
   },
   {
     "city": "Toronto",
@@ -816,7 +906,8 @@
     "country": "Canada",
     "iso3166_1": "CA",
     "latitude": 43.651,
-    "longitude": -79.347
+    "longitude": -79.347,
+    "local_languages": ["en-CA"]
   },
   {
     "city": "Vancouver",
@@ -825,7 +916,8 @@
     "country": "Canada",
     "iso3166_1": "CA",
     "latitude": 49.2827,
-    "longitude": -123.1207
+    "longitude": -123.1207,
+    "local_languages": ["en-CA"]
   },
   {
     "city": "Montreal",
@@ -834,7 +926,8 @@
     "country": "Canada",
     "iso3166_1": "CA",
     "latitude": 45.5017,
-    "longitude": -73.5673
+    "longitude": -73.5673,
+    "local_languages": ["fr-CA", "en-CA"]
   },
   {
     "city": "Calgary",
@@ -843,7 +936,8 @@
     "country": "Canada",
     "iso3166_1": "CA",
     "latitude": 51.0447,
-    "longitude": -114.0719
+    "longitude": -114.0719,
+    "local_languages": ["en-CA"]
   },
   {
     "city": "Halifax",
@@ -852,7 +946,8 @@
     "country": "Canada",
     "iso3166_1": "CA",
     "latitude": 44.6488,
-    "longitude": -63.5752
+    "longitude": -63.5752,
+    "local_languages": ["en-CA"]
   },
   {
     "city": "Mexico City",
@@ -861,7 +956,8 @@
     "country": "Mexico",
     "iso3166_1": "MX",
     "latitude": 19.4326,
-    "longitude": -99.1332
+    "longitude": -99.1332,
+    "local_languages": ["es-MX"]
   },
   {
     "city": "Tijuana",
@@ -870,7 +966,8 @@
     "country": "Mexico",
     "iso3166_1": "MX",
     "latitude": 32.5149,
-    "longitude": -117.0382
+    "longitude": -117.0382,
+    "local_languages": ["es-MX"]
   },
   {
     "city": "Monterrey",
@@ -879,7 +976,8 @@
     "country": "Mexico",
     "iso3166_1": "MX",
     "latitude": 25.6866,
-    "longitude": -100.3161
+    "longitude": -100.3161,
+    "local_languages": ["es-MX"]
   },
   {
     "city": "Guadalajara",
@@ -888,7 +986,8 @@
     "country": "Mexico",
     "iso3166_1": "MX",
     "latitude": 20.6597,
-    "longitude": -103.3496
+    "longitude": -103.3496,
+    "local_languages": ["es-MX"]
   },
   {
     "city": "Ensenada",
@@ -897,6 +996,7 @@
     "country": "Mexico",
     "iso3166_1": "MX",
     "latitude": 31.8667,
-    "longitude": -116.5964
+    "longitude": -116.5964,
+    "local_languages": ["es-MX"]
   }
 ]

pipeline/prompts/system.md

@@ -14,9 +14,13 @@ $$City Name$$
 
 $$Country Name$$
 
+## LOCAL LANGUAGE CODES:
+
+$$Local language codes in priority order$$
+
 ## CONTEXT:
 
-$$Information about local beer culture, history, or geography$$
+$$Information about local beer culture, history, geography, or language context$$
 
 ## CRITICAL OUTPUT FORMAT (READ CAREFULLY):
 
@@ -68,6 +72,7 @@ The last sentence must be a personal, low-key invitation from the owner, specifi
 ### LOCAL LANGUAGE VERSION:
 
 name_local is a direct translation of name_en into the local language or script.
+Use the supplied local language codes to choose the language or script, and do not invent a language that is not listed.
 
 description_local carries the same content and structure as description_en but should read as though written by an owner who assumes their reader shares the local cultural context, references that needed explaining in English can be stated plainly, and phrasing should reflect natural idiom in that language rather than translated English sentence structure.
 
@@ -112,6 +117,7 @@ The description must be written in the first person, from the perspective of the
 Input:
 CITY: Montréal
 COUNTRY: Canada
+LOCAL LANGUAGE CODES: fr-CA, en-CA
 CONTEXT: Montréal has been brewing since 1646 when Jesuit Brother Ambroise first introduced brewing to New France. By the 19th century, Pointe-Saint-Charles became the industrial heart of the city, home to railway yards, canal workers, and a tavern on nearly every block. Molson, one of North America's oldest commercial breweries, has operated on the St. Lawrence since 1786. By the early 1980s, Molson, Labatt, and Carling controlled 96% of the Quebec beer market. The craft revival began slowly in the late 1980s and has accelerated sharply since 2002, when 33 brewing companies have grown to over 300 province-wide.
 
 $$Truncated for brevity, but assumes full context provided$$

pipeline/src/biergarten_data_generator/query_cities_with_countries.cc

@@ -13,7 +13,7 @@
 #include "biergarten_data_generator.h"
 #include "json_handling/json_loader.h"
 
-static constexpr size_t kBreweryAmount = 5;
+static constexpr size_t kBreweryAmount = 50;
 
 std::vector<Location> BiergartenDataGenerator::QueryCitiesWithCountries() {
   spdlog::info("\n=== GEOGRAPHIC DATA OVERVIEW ===");

pipeline/src/biergarten_data_generator/run.cc

@@ -19,7 +19,7 @@ bool BiergartenDataGenerator::Run() {
     for (auto& city : cities) {
       try {
         std::string region_context = context_service_->GetLocationContext(city);
-        spdlog::info("[Pipeline] Context for '{}' ({}) gathered:\n{}",
+        spdlog::debug("[Pipeline] Context for '{}' ({}) gathered:\n{}",
                      city.city, city.country, region_context);
 
         enriched.push_back(

pipeline/src/data_generation/llama/generate_brewery.cc

@@ -4,18 +4,35 @@
  * inference, and validates structured JSON output for brewery records.
  */
 
-#include "data_generation/llama_generator.h"
+#include <spdlog/spdlog.h>
 
 #include <format>
 #include <optional>
 #include <stdexcept>
 #include <string>
 #include <string_view>
+#include <vector>
 
-#include <spdlog/spdlog.h>
+#include "data_generation/llama_generator.h"
-
 #include "data_generation/llama_generator_helpers.h"
 
+static std::string FormatLocalLanguageCodes(
+    const std::vector<std::string>& codes) {
+  if (codes.empty()) {
+    return "Not provided";
+  }
+
+  std::string formatted;
+  for (const std::string& code : codes) {
+    if (!formatted.empty()) {
+      formatted += ", ";
+    }
+    formatted += code;
+  }
+
+  return formatted;
+}
+
 static constexpr std::string_view kBreweryJsonGrammar = R"json_brewery(
 root ::= thought-block "{" ws "\"name_en\"" ws ":" ws string ws "," ws "\"description_en\"" ws ":" ws string ws "," ws "\"name_local\"" ws ":" ws string ws "," ws "\"description_local\"" ws ":" ws string ws "}" ws
 thought-block ::= [^{]*
@@ -35,8 +52,10 @@ BreweryResult LlamaGenerator::GenerateBrewery(
   /**
    * Preprocess and truncate region context to manageable size
    */
-  const std::string safe_region_context =
+  const std::string safe_region_context = PrepareRegionContext(region_context);
-      PrepareRegionContext(region_context);
+
+  const std::string local_language_codes =
+      FormatLocalLanguageCodes(location.local_languages);
 
   const std::string country_suffix =
       location.country.empty() ? std::string{}
@@ -48,16 +67,18 @@ BreweryResult LlamaGenerator::GenerateBrewery(
   const std::string system_prompt =
       LoadBrewerySystemPrompt("prompts/system.md");
 
-
   std::string user_prompt = std::format(
-      "## CITY:\n{}\n\n## COUNTRY:\n{}\n\n## CONTEXT:\n{}",
+      "## CITY:\n{}\n\n## COUNTRY:\n{}\n\n## LOCAL LANGUAGE CODES:\n{}\n\n## "
-      location.city, location.country, safe_region_context);
+      "CONTEXT:\n{}",
+      location.city, location.country, local_language_codes,
+      safe_region_context);
 
   /**
    * Store location context for retry prompts (without repeating full context)
    */
   const std::string retry_location =
-      std::format("Location: {}{}", location.city, country_suffix);
+      std::format("Location: {}{}\nLocal language codes: {}", location.city,
+                  country_suffix, local_language_codes);
 
   /**
    * RETRY LOOP with validation and error correction
@@ -68,16 +89,17 @@ BreweryResult LlamaGenerator::GenerateBrewery(
   std::string raw;
   std::string last_error;
 
-    // Token budget: too small risks truncating valid JSON mid-string.
+  // Token budget: too small risks truncating valid JSON mid-string.
-    // Start conservatively but allow adaptive increases on truncation.
+  // Start conservatively but allow adaptive increases on truncation.
-    int max_tokens = kBreweryInitialMaxTokens;
+  int max_tokens = kBreweryInitialMaxTokens;
 
   // Limit output length to keep it concise and focused
   for (int attempt = 0; attempt < max_attempts; ++attempt) {
     // Generate brewery data from LLM
-    raw = this->Infer(system_prompt, user_prompt, max_tokens, kBreweryJsonGrammar);
+    raw = this->Infer(system_prompt, user_prompt, max_tokens,
+                      kBreweryJsonGrammar);
     spdlog::info("LlamaGenerator: raw output (attempt {}): {}", attempt + 1,
-                  raw);
+                 raw);
 
     // Validate output: parse JSON and check required fields
 
@@ -89,9 +111,8 @@ BreweryResult LlamaGenerator::GenerateBrewery(
       // Success: return parsed brewery data
 
       spdlog::info(
-          "LlamaGenerator: successfully generated brewery data on attempt {}:\n name_en='{}',\n description_en='{}',\n name_local='{}',\n description_local='{}'",
+          "LlamaGenerator: successfully generated brewery data on attempt {}",
-          attempt + 1, brewery.name_en, brewery.description_en,
+          attempt + 1);
-          brewery.name_local, brewery.description_local);
 
       return brewery;
     }
@@ -102,13 +123,13 @@ BreweryResult LlamaGenerator::GenerateBrewery(
     spdlog::warn("LlamaGenerator: malformed brewery JSON (attempt {}): {}",
                  attempt + 1, *validation_error);
 
-
+    if (last_error == "JSON parse error: incomplete JSON") {
-        if (last_error == "JSON parse error: incomplete JSON") {
+      const int previous_max_tokens = max_tokens;
-            const int previous_max_tokens = max_tokens;
+      max_tokens = std::min(max_tokens + kBreweryTruncationRetryTokenBump,
-            max_tokens = std::min(max_tokens + kBreweryTruncationRetryTokenBump,
+                            kBreweryMaxTokensCeiling);
-                                                        kBreweryMaxTokensCeiling);
       spdlog::info(
-          "LlamaGenerator: detected truncated JSON; increasing max_tokens from {} to {} and retrying",
+          "LlamaGenerator: detected truncated JSON; increasing max_tokens from "
+          "{} to {} and retrying",
           previous_max_tokens, max_tokens);
 
       continue;
@@ -116,13 +137,15 @@ BreweryResult LlamaGenerator::GenerateBrewery(
 
     // Update prompt with error details to guide LLM toward correct output.
     user_prompt = std::format(
-        R"(Your previous response was invalid. Error: {}
+        "Your previous response was invalid. Error: {}\nReturn the thought "
-Return the thought process before the JSON if needed, then return ONLY valid JSON with exactly these keys, in this exact order: {{"name_en": "<English brewery name>", "description_en": "<English single-paragraph description>", "name_local": "<local-language brewery name>", "description_local": "<local-language single-paragraph description>"}}.
+        "process before the JSON if needed, then return ONLY valid JSON with "
-Do not include markdown, comments, extra keys, or literal placeholder values.
+        "exactly these keys, in this exact order: {{\"name_en\": \"<English "
-
+        "brewery name>\", \"description_en\": \"<English single-paragraph "
-Keep the JSON strings concise enough to fit within the token budget.
+        "description>\", \"name_local\": \"<local-language brewery name>\", "
-
+        "\"description_local\": \"<local-language single-paragraph "
-{})",
+        "description>\"}}.\nDo not include markdown, comments, extra keys, or "
+        "literal placeholder values.\n\nKeep the JSON strings concise enough "
+        "to fit within the token budget.\n\n{}",
         *validation_error, retry_location);
   }
 

pipeline/src/data_generation/llama/llama_generator.cc

@@ -32,7 +32,7 @@ void LlamaGenerator::ContextDeleter::operator()(
 
 LlamaGenerator::LlamaGenerator(const ApplicationOptions& options,
                                const std::string& model_path,
-                               std::shared_ptr<IPromptFormatter> prompt_formatter)
+                               std::unique_ptr<IPromptFormatter> prompt_formatter)
     : rng_(std::random_device{}()),
       prompt_formatter_(std::move(prompt_formatter)) {
   if (model_path.empty()) {

pipeline/src/json_handling/json_loader.cc

@@ -35,6 +35,27 @@ static double ReadRequiredNumber(const boost::json::object& object,
   return value->to_number<double>();
 }
 
+static std::vector<std::string> ReadRequiredStringArray(
+    const boost::json::object& object, const char* key) {
+  const boost::json::value* value = object.if_contains(key);
+  if (value == nullptr || !value->is_array()) {
+    throw std::runtime_error(std::string("Missing or invalid string array field: ") +
+                             key);
+  }
+
+  const auto& array = value->as_array();
+  std::vector<std::string> items;
+  items.reserve(array.size());
+  for (const auto& item : array) {
+    if (!item.is_string()) {
+      throw std::runtime_error(std::string("Missing or invalid string array field: ") +
+                               key);
+    }
+    items.emplace_back(item.as_string());
+  }
+  return items;
+}
+
 std::vector<Location> JsonLoader::LoadLocations(
     const std::filesystem::path& filepath) {
   std::ifstream input(filepath);
@@ -76,6 +97,8 @@ std::vector<Location> JsonLoader::LoadLocations(
         .iso3166_2 = ReadRequiredString(object, "iso3166_2"),
         .country = ReadRequiredString(object, "country"),
         .iso3166_1 = ReadRequiredString(object, "iso3166_1"),
+        .local_languages =
+            ReadRequiredStringArray(object, "local_languages"),
         .latitude = ReadRequiredNumber(object, "latitude"),
         .longitude = ReadRequiredNumber(object, "longitude"),
     });