LLM Engineering

LLM Application Stack `` User Input ↓ Input Processing (validation, PII scrubbing, token counting) ↓ Retrieval (optional RAG: vector search, reranking) ↓ Prompt Construction (system prompt + context + user message) ↓ LLM API Call (model, temperature, max_tokens) ↓ Output Processing (parsing, validation, formatting) ↓ Response → User ↓ Logging & Evaluation (latency, token cost, quality scores) ``

Choosing the Right Model | Dimension | Smaller (GPT-4o-mini) | Larger (GPT-4o) | |---|---|---| | Speed | Faster | Slower | | Cost | 10-20x cheaper | Expensive | | Quality | Good | Excellent | | Reasoning | Basic | Complex | | Use for | Classification, extraction, simple QA | Analysis, synthesis, complex reasoning |

Multi-Model Strategy ``python def route_request(query: str) -> str: complexity = assess_complexity(query) if complexity == "simple": return call_model("gpt-4o-mini", query) elif complexity == "medium": return call_model("claude-3-haiku", query) else: return call_model("gpt-4o", query) ``

Token Budget Management ```python import tiktoken

def count_tokens(text: str, model: str = "gpt-4o") -> int: enc = tiktoken.encoding_for_model(model) return len(enc.encode(text))

def build_prompt_within_budget( system: str, history: list[dict], user_message: str, max_tokens: int = 4096 ) -> list[dict]: system_tokens = count_tokens(system) user_tokens = count_tokens(user_message) budget = max_tokens - system_tokens - user_tokens - 200 # reserve for response

messages = [{"role": "system", "content": system}]

# Add history newest-first until budget exhausted for msg in reversed(history): t = count_tokens(msg["content"]) if budget - t < 0: break messages.insert(1, msg) budget -= t

messages.append({"role": "user", "content": user_message}) return messages ```

Conversation Memory Strategies 1. Full history: include all messages (hits context limit fast) 2. Sliding window: last N messages only 3. Summary buffer: summarize old messages, keep recent ones in full 4. Vector memory: embed and retrieve relevant past messages 5. Entity memory: track named entities across conversation

LLM Evaluation Metrics ```python from ragas import evaluate from ragas.metrics import faithfulness, answer_relevancy, context_precision

# RAG evaluation result = evaluate( dataset=test_dataset, metrics=[faithfulness, answer_relevancy, context_precision] ) # faithfulness: Does answer match retrieved context? (hallucination) # answer_relevancy: Is answer relevant to the question? # context_precision: Are retrieved chunks actually useful? ```

Tracing & Monitoring ```python from langsmith import traceable import langsmith

@traceable(name="RAG Pipeline") def answer_question(question: str) -> str: docs = retrieve(question) context = format_context(docs) return generate_answer(question, context) # LangSmith captures: latency, token usage, prompt, response, errors ```

A/B Testing Prompts ```python import random

PROMPT_VERSIONS = { "v1": "You are a helpful assistant.", "v2": "You are an expert assistant. Be concise and accurate.", "v3": "Answer clearly in 2-3 sentences. Cite your reasoning." }

def get_prompt(user_id: str) -> str: # Deterministic split by user_id bucket = hash(user_id) % 3 version = ["v1", "v2", "v3"][bucket] log_prompt_version(user_id, version) return PROMPT_VERSIONS[version] ```

When to Fine-Tune vs Prompt | Approach | Best For | Cost | |---|---|---| | Prompt engineering | Behavior adjustment, format control | Free | | Few-shot prompting | Consistent output format, style | Token cost | | RAG | Up-to-date knowledge, domain facts | Infra cost | | Fine-tuning | Tone/style, specialized knowledge, speed | Training cost |

LoRA Fine-Tuning ```python from peft import get_peft_model, LoraConfig

lora_config = LoraConfig( r=16, # rank: higher = more parameters trained lora_alpha=32, # scaling factor target_modules=["q_proj", "v_proj"], # which layers to tune lora_dropout=0.1, task_type="CAUSAL_LM" )

model = get_peft_model(base_model, lora_config) model.print_trainable_parameters() # trainable: 8M of 8B params (0.1%) fast and cheap ```

RLHF (Reinforcement Learning from Human Feedback) 1. SFT (Supervised Fine-Tuning): train on human demonstrations 2. Reward Model: train on human preference pairs (A vs B) 3. PPO: optimize policy model to maximize reward model score

OpenAI Fine-Tuning API ```python # Prepare training data {"messages": [ {"role": "system", "content": "You are a customer support agent."}, {"role": "user", "content": "How do I reset my password?"}, {"role": "assistant", "content": "To reset your password: 1) Go to login page..."} ]}

# Upload and fine-tune file = client.files.create(file=open("train.jsonl", "rb"), purpose="fine-tune") job = client.fine_tuning.jobs.create(training_file=file.id, model="gpt-4o-mini") ```