Krystian Safjan's Blog

The Real Cost of Model Migration - What Swapping LLMs Actually Requires

2026-04-09T00:00:00+02:00

So OpenAI deprecated gpt-4o-mini. Or some other model you've built your whole system around just got a sunset date. The email lands and your first thought is: how hard can a model swap be?

I've been through this a few times now. The API call swap? Easy. Twenty minutes, tops. But the swap has a way of revealing every shortcut and assumption your system has been quietly depending on. That's the part people don't usually mention until you're already in it.

What migration actually is
The model options in 2026
Phase 0 - Know what you're migrating from
Phase 1 - Build your evaluation harness first
Phase 2 - The prompt portability problem
Phase 3 - Automated prompt optimization with DSPy
Phase 4 - Reasoning models: where they belong
Phase 5 - Handling missing parameters
Phase 6 - Risk assessment
Phase 7 - Progressive rollout
Phase 8 - Post-migration monitoring
The systems audit you should run regardless

What migration actually is

Here's the (uncomfortable) truth: a model migration is really a systems audit. It just happens to come with a deadline someone else set for you.

When you swap the model under a RAG pipeline, you're removing the environment your system's behavior was calibrated in. And you get to find out how much of that behavior was intentional versus... just kind of happened over time.

Three things consistently surface during migration that were invisible before:

Quality often wasn't measured. A lot of production LLM systems have never been formally evaluated. No golden dataset, no faithfulness score, no format compliance check. "Quality" is whatever the team last looked at and didn't complain about. You can't claim "no quality loss" if you never measured quality to begin with.

Your prompt is coupled to the old model. That system prompt you spent weeks on? It's not a specification. It's a negotiation artifact, the residue of back-and-forth between your intentions and one specific model's quirks. Swap the model and you haven't ported a prompt. You've orphaned it. (This one hurts. More in Phase 2.)

Model behavior often isn't versioned. Pinning a model name isn't the same as pinning model behavior. OpenAI updates weights behind dated aliases without telling you. If you're using gpt-4o-mini as a floating pointer, you may have already had a silent behavioral change in production. If your observability didn't catch it... well, that tells you something about your observability.

The teams who migrate cleanly aren't the ones with the best migration plans. They're the ones who treated their LLM system like a real production system long before a deadline showed up.

The model options in 2026

Note: While this article uses OpenAI models as examples, the migration patterns, evaluation strategies, and architectural decisions apply to any LLM provider. The same principles work whether you're migrating between Anthropic's Claude models, open-source models, or any other provider.

Before you plan anything, you need to know what you're migrating to. OpenAI's current model family has two very different architectures, and picking the wrong one can create more problems than the deprecation itself.

Standard instruction models

GPT-5.4, GPT-5.4-mini, and GPT-5.4-nano are the current flagship models. They support variable reasoning_effort (none/low/medium/high/xhigh), plus all the standard parameters: temperature, system prompts, JSON mode, function calling, streaming. For most RAG answer generation workloads, one of these is where you should land.

Recommended default for gpt-4o-mini replacement: gpt-5.4-nano -comparable cost tier ($0.20 per million input tokens vs $0.10 for gpt-4o-mini), significantly more capable, fully API-compatible. If you need the extra capability and can handle the cost, gpt-5.4-mini ($0.75/MTok input) is a strong middle option.

Pure reasoning models

The o-series models (o3, o4-mini) are pure reasoning models without the hybrid flexibility of GPT-5.x. They only do reasoning, don't support temperature or standard sampling parameters, and use reasoning_effort as the sole control. These are specialists.

Here's what most people get wrong: for typical RAG pipelines, pure reasoning models aren't better answer generators. They're decision infrastructure. I'll get into exactly where they earn their cost in Phase 4.

Model selection decision tree:

Is the answer synthesis step genuinely multi-hop?
(A implies B, B contradicts C, therefore...)
  └── No  → Standard model (gpt-5.4-nano or gpt-5.4-mini)
  └── Yes → Is it high-stakes with real consequences?
              └── No  → Standard model with reasoning_effort: low/medium
              └── Yes → Consider pure reasoning model (o3, o4-mini)
                        OR standard model with reasoning_effort: high
                        for verification layer over synthesis

Phase 0 -Know what you're migrating from

I know, I know. You want to start swapping things. But before you write a line of migration code, document what your system actually does right now. Otherwise you'll have no way to tell if the migration worked or just seemed like it did.

Inventory your integration surface

Pull every place in your codebase where the model name appears. This includes:

Direct API calls with model= parameter
Configuration files and environment variables
Any SDK initialization that sets a default model
Evaluation scripts that may be pinned to the old model for judging

For each integration point, record: the model name, the prompt template used, the parameters passed (temperature, max_tokens, etc.), and the expected output format.

Document implicit behavioral assumptions

This part's harder because these things rarely get written down. You need to look for anywhere your code processes model outputs and makes assumptions about what they look like:

JSON parsing of model responses (field names, nesting depth)
Regex or string matching on output format
Length-based truncation or display logic
Citation extraction that assumes a specific citation format
Any code that branches on output content

Each of these is a behavioral assumption about your current model. And each assumption is a place where things can quietly break.

Get a pre-migration quality snapshot

Run your current system against a sample of real production queries and save the outputs. This is your before-state. Even if you don't have a formal eval harness yet, just having the raw outputs lets you compare later. Future-you will be grateful.

Phase 1 - Build your evaluation harness first

I can't stress this enough. Everything else you do, prompt changes, model selection, rollout strategy, is going to be guided by what your evals tell you. Skip this and you'll discover regressions in production. I've watched teams do this. The cost of fixing things at that point is genuinely 10x higher.

Build a golden dataset

Sample 200–500 real queries from production logs. For each, store:

The original query
The retrieved context chunks
The current model's answer (this becomes your reference)
For as many as you can afford: a human-verified "ideal" answer

Notice: with this approach you are preparing for testing answer generation part of the RAG not the retriever.

Don't sample uniformly. Stratify on purpose. Include:

Stratum	Why
Easy factual queries	Regression canary, should never fail
Multi-chunk synthesis	Where model capability actually matters
Conflicting context	Tests faithfulness under pressure
Out-of-scope queries	Refusal behavior regression
Edge cases your team knows about	The ones that broke things before

Aim for at least 50 human-verified examples. 200 is significantly better.

What to evaluate

For a RAG system, here's what you actually need to measure:

Faithfulness - does the answer only claim things supported by the retrieved context? This is the big one. A model that hallucinates confidently is scarier than one that refuses to answer. Use the Ragas faithfulness metric.

Answer relevance - does it answer what was actually asked? (Ragas answer relevance)

Format compliance - does the output match your schema? JSON structure, citation format, length constraints. You'll likely need a custom LLM-as-judge metric here because format requirements vary widely.

Refusal accuracy - when the context doesn't contain the answer, does the model say "I don't know" instead of making something up?

Groundedness - can you trace specific claims back to specific chunks? Similar to faithfulness but more granular.

Evaluation tooling

Ragas automates faithfulness, answer relevance, context precision, and context recall scoring using an LLM-as-judge approach. Point it at your golden dataset and run both old and new model outputs through it to get comparable scores.

PromptFoo works well for regression testing during prompt iteration. Define test cases with expected outputs or assertions and run them against multiple models simultaneously, which is exactly the side-by-side comparison you need during migration.

LangSmith or Braintrust if you want persistent experiment tracking. They store eval runs with scores, let you diff outputs visually, and can alert on regressions. Worth setting up if this migration will take more than a week.

MLflow for teams already in the MLflow ecosystem. It has native LLM tracking and integrates directly with DSPy (covered in Phase 3).

Define pass/fail gate criteria

Do this before running any evals. Seriously. If you define criteria after seeing results, you'll unconsciously anchor them to whatever the new model happens to achieve. Human brains are terrible at this.

Example gate criteria: - Faithfulness score ≥ 0.92 - Format compliance ≥ 0.98 - Answer relevance ≥ 0.90 - P95 latency within 20% of baseline - Refusal accuracy ≥ 0.95 on out-of-scope stratum

flowchart TD
    A[Sample 200-500 production queries] --> B[Stratify by difficulty and type]
    B --> C[Run current model - store as reference]
    C --> D[Human verify 50-200 examples]
    D --> E[Define metric weights per dimension]
    E --> F[Set gate criteria before migration begins]
    F --> G[Eval harness ready]

    style A fill:#1e3a5f,color:#b8d4f0
    style G fill:#1a3d2e,color:#8fd4b0

Phase 2 - The prompt portability problem

Okay, this is the part that causes the most pain, and people don't usually talk about it honestly.

Written prompts vs tuned prompts

There's a difference between a prompt that specifies behavior and a prompt that was tweaked until outputs stopped looking weird. It's a bigger difference than you'd think.

A written prompt starts from a behavioral spec. You know what the system must do, what it must not do, what the output format looks like. The constraints are verifiable regardless of which model you're running:

- Answer using only information present in the provided context chunks.
- If the context does not contain sufficient information, respond with 
  exactly: "I cannot answer this from the available information."
- Format citations as [source_id] inline, immediately after the claim 
  they support.

These survive a model swap. You can read the prompt and tell whether any output satisfies them, regardless of which model made it.

A tuned prompt is what most of us actually have. It starts from a vague goal and grows through patches. You wrote a first draft. Outputs were mostly fine but the model kept adding chatty preambles, so you added "be concise." Then it started truncating, so you added "be thorough but concise." Citations were inconsistent, so you added "always cite sources." Then it started over-citing, so you added "cite only when directly referencing a specific fact."

Sound familiar? Six months later your prompt is 800 words and full of stuff like:

"Do not add unnecessary preambles" - patch for a greeting behavior specific to an old model weight
"Avoid repeating the question in your answer" - patch for a retriggering behavior
"Use natural language, not bullet points unless the question explicitly asks for a list" - patch for a formatting regression after a silent weight update

None of these describe what your system is supposed to do. They're band-aids for specific past failures of a model that no longer exists.

Why tuned prompts are so common

If you're feeling called out right now, don't. The vast majority of production RAG prompts are more tuned than written.

Patching is faster than specifying. When you're iterating on a RAG system, you see problems and fix them. The fastest fix is usually adding an instruction. Writing a proper spec requires knowing all failure modes before you've seen them, which is... impossible when failures emerge from interaction with real data.

Nothing forces you to notice until migration. A tuned prompt works. It produces acceptable outputs on the current model. The coupling only becomes visible when you remove the thing it's coupled to.

Many teams never wrote a spec to begin with. Writing a specification for "correct behavior" before building the system requires a level of foresight that's genuinely hard to have. So the prompt became the spec over time, simultaneously the behavioral specification and the accumulated technical debt. Good luck telling them apart from the inside.

What to do about it

Start with prompt archaeology. Before you touch anything, go through every instruction in your current prompt and label it as either: - SPEC - this describes intended behavior, survives model changes - PATCH - this suppresses a specific failure, may not be relevant to new model

In my experience, most 500+ word prompts end up being about 40% spec and 60% patch. The patches are candidates for removal or replacement after you test compatibility with the new model.

Run a compatibility baseline before touching anything. Take your existing prompt verbatim, swap only the model name, run your golden dataset through it, and score with Ragas. This tells you how much of the prompt is still needed versus how much is suppressing behaviors the new model doesn't even have.

Reconstruct minimally. Where scores dropped, investigate the failure patterns before changing the prompt. Don't just start adding instructions (that's how you got here). Common adaptation points:

Issue	Diagnosis	Fix
Markdown in plain-text outputs	New model has higher markdown affinity	Explicit format instruction
Longer responses than expected	Different default length calibration	Token budget instruction + few-shot
Citation format drift	Model interprets citation spec differently	Few-shot examples, not more words
Refusal behavior change	Different threshold for "insufficient context"	Explicit refusal instruction with example
JSON key naming change	Model paraphrases key names	Strict schema in system prompt or JSON mode

Few-shot examples are the most reliable format anchor. More reliable than additional instructions. They act as a behavioral anchor that survives prompt wording differences across model versions. For format-critical RAG outputs, 2-3 few-shot examples will do more work than three paragraphs of format instructions.

Phase 3 - Automated prompt optimization with DSPy

So manual prompt iteration is what got you into the tuned-prompt mess. DSPy offers a way out: it learns the optimal prompt for your specific data and target model automatically, guided by whatever metric you care about.

What DSPy actually does

DSPy separates the interface (what should the model do) from the implementation (how do you tell it). You define Signatures (declarative input/output specs) and DSPy optimizers figure out the instructions and examples that best satisfy your metric on your data.

Why this matters for migration: the prompt that worked on gpt-4o-mini isn't guaranteed to be optimal for gpt-5.4-nano. DSPy relearns the prompt for the new model automatically. You get model-specific optimization without the manual iteration that created the problem in the first place.

Translating your pipeline into DSPy

Your answer generation step becomes a Signature:

import dspy

class RAGAnswer(dspy.Signature):
    """Answer the question using only the provided context chunks.
    Cite sources using [chunk_id] notation."""

    context: str = dspy.InputField(desc="retrieved context chunks with IDs")
    question: str = dspy.InputField()
    answer: str = dspy.OutputField(desc="faithful, cited answer")

class RAGPipeline(dspy.Module):
    def __init__(self):
        self.respond = dspy.ChainOfThought(RAGAnswer)

    def forward(self, context, question):
        return self.respond(context=context, question=question)

Switching the target model is one line. Your golden dataset and metric stay identical:

# Before: gpt-4o-mini
dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"))

# After: gpt-5.4-nano - golden dataset unchanged, metric unchanged
dspy.configure(lm=dspy.LM("openai/gpt-5.4-nano"))

Choosing an optimizer

flowchart TD
    A[Start: need to optimize prompt for new model] --> B{How many pipeline stages?}

    B -->|Single stage| C{Budget and time?}
    B -->|Multi-stage| D[MIPROv2 or GEPA]

    C -->|Quick and cheap| E[BootstrapFewShot
~$0.10, ~5 min]
    C -->|Thorough| F[MIPROv2
~$1.50-5, ~30 min]

    D --> G{Complex failures
need diagnosis?}
    G -->|No| F
    G -->|Yes| H[GEPA
Expensive but deepest optimization]

    E --> I[Save compiled program as artifact]
    F --> I
    H --> I

    style E fill:#1a3d2e,color:#8fd4b0
    style F fill:#1e3a5f,color:#b8d4f0
    style H fill:#3d1a2e,color:#d4b0c0
    style I fill:#3d2e1a,color:#d4c0a0

BootstrapFewShot - the baseline. Generates complete demonstrations for each stage of your program, keeping only those that pass your metric. Use this first. Cheap, fast, often sufficient for single-stage answer generation migration.

MIPROv2 - the workhorse. Runs three stages: bootstrapping to collect high-scoring traces, grounded proposal to draft potential instructions, then discrete search to evaluate instruction-example combinations. Costs ~$1.50-5 at medium auto setting, takes 20-40 minutes. Worth running before any production rollout.

from dspy.teleprompt import MIPROv2

optimizer = MIPROv2(
    metric=your_faithfulness_metric,
    auto="medium",           # light / medium / heavy
    num_threads=8
)

compiled_rag = optimizer.compile(
    RAGPipeline(),
    trainset=train_examples,  # 20% of golden dataset
    valset=val_examples        # 80% of golden dataset
)

# Save as versioned artifact
compiled_rag.save("rag_pipeline_gpt54nano_v1.json")

GEPA - the newest. Rather than optimizing only the globally best candidate (which leads to local optima), GEPA maintains a Pareto frontier: candidates that achieve the highest score on at least one evaluation instance. It uses a Teacher model to analyze failures and propose targeted fixes. Use for multi-stage pipelines where interaction effects between stages matter. Requires a strong reflection model (gpt-5.4 or higher recommended).

The data split that matters

This one's counterintuitive: 20% training, 80% validation. Yes, reversed from what you're used to. Prompt-based optimizers overfit to small training sets way more aggressively than neural networks. A prompt optimized on 60 examples and validated on 240 will generalize far better than the other way around.

You can get real value from as few as 30 training examples. The validation set is where quality is actually measured.

DSPy in migration: tool vs framework

Teams adopt DSPy as a permanent architectural layer when the problem only needed a one-time tool.

Use DSPy as a migration tool: run MIPROv2 against your golden dataset on the new model, inspect the optimized prompt, extract it as a string, and deploy it without DSPy in your runtime path. Optimization benefit, no framework dependency.

Adopt DSPy as a framework only if you're going to keep experimenting: swapping models regularly, adding pipeline stages, running continuous optimization against production feedback. That's where the framework investment pays off.

DSPy limitations to know before committing

A few things to be aware of:

Your metric quality determines everything. DSPy optimizes whatever metric you give it. Weak metric? You'll get a confidently wrong optimized prompt. Make sure faithfulness is in there, not just answer relevance.
The optimized prompt can look weird. MIPROv2 generates instructions that work but can be verbose and repetitive in ways a human wouldn't write. Debugging means re-running evals, not reading the prompt.
Reasoning models need special handling. DSPy doesn't have first-class support for reasoning_effort yet. When targeting pure reasoning models (o4-mini, o3) or GPT-5.x with specific reasoning effort levels, wrap the model call in a custom dspy.LM class that sets reasoning_effort at initialization.
Watch the adapter layer. DSPy's adapters wrap your instructions in scaffolding. With reasoning models where prompt verbosity interferes with internal reasoning chains, this can produce unexpected behavior. Test it.

Phase 4 - Reasoning models: where they belong

When people hear "more capable model family," their instinct is to slot the reasoning model right where the standard model was: answer generation. This is usually wrong.

Think of it like hiring. You wouldn't pay a principal engineer to write CRUD endpoints. You hire them to make decisions when things are ambiguous. Same logic here.

The RAG pipeline with reasoning model placement

flowchart TD
    Q[User query] --> QD

    subgraph REASONING ["Reasoning model layer (low-medium effort)"]
        QD["Query decomposition
+ intent routing"]
        VER["Faithfulness
verification"]
    end

    subgraph STANDARD ["Standard model layer"]
        RET["Vector/hybrid
retrieval"]
        RNK["Reranking +
conflict detection"]
        SYN["Answer
synthesis"]
    end

    QD --> RET
    RET --> RNK
    RNK --> SYN
    SYN --> VER
    VER --> ANS[Response]

    style REASONING fill:#1e2a3f,color:#a0c4f0,stroke:#3a5a8f
    style STANDARD fill:#1a2e1f,color:#90c4a0,stroke:#2a5a3a

Slot 1: Query decomposition - strong fit, highest ROI

Most enterprise RAG queries are compound, ambiguous, or require unpacking implicit assumptions before retrieval. A reasoning model at this stage:

Decomposes "what's our exposure if we exit the APAC contracts before Q3 given the renegotiation clauses?" into 3–4 targeted retrieval sub-queries
Classifies intent (factual / comparative / policy / out-of-scope) to route appropriately
Identifies which sub-questions are answerable from the knowledge base vs. need human input

The reasoning happens before retrieval, so latency cost doesn't compound. Use reasoning_effort: low because query decomposition rarely needs deep thinking. You pay for one reasoning call and get better chunks in return for every subsequent step.

Slot 2: Context reranking and conflict detection - strong fit

Standard cross-encoder rerankers like BGE or Cohere Rerank score chunk-query relevance mechanically. A reasoning model can do something they can't: spot when retrieved chunks contradict each other. That's a signal to retry retrieval, not attempt synthesis over conflicting information.

Especially useful in legal, compliance, and financial RAG where "relevance" means logical applicability, not just semantic similarity.

Slot 3: Answer synthesis on complex documents - maybe

Use a reasoning model for synthesis only when the answer requires multi-hop inference across chunks (A implies B, B contradicts C, therefore...) or the domain is high-stakes with real consequences for wrong answers.

Don't bother when the answer is directly stated in one or two chunks, or when query volume is high and latency matters. If you're generating templated output from straightforward lookups, a standard model is fine.

Slot 4: Agentic orchestration - good fit, underused

If you have multi-source RAG (vector DB + SQL + APIs + document store), you need something that decides which retrieval path to take, in what order, and when to stop. That's a planning problem, not a synthesis problem. Reasoning models are trained through RL to reason about when and how to use tools, not just call them when told.

Use the reasoning model as the orchestration brain. Let cheaper instruction models handle the actual synthesis once the right context is assembled.

Slot 5: Faithfulness verification - high value, underused

A second pass after generation: "Given only the following context, does this answer make claims not supported by the context? Flag the specific unsupported sentences."

reasoning_effort: low. You're verifying, not generating. One cheap call that acts as your hallucination guardrail. The economics really work here for high-stakes outputs.

Phase 5 - Handling parameter differences

If you're migrating between standard GPT-5.x models (like gpt-4o-mini to gpt-5.4-nano), this section mostly doesn't apply. The standard parameters work identically.

The main difference: GPT-5.x models add reasoning_effort (none/low/medium/high/xhigh) as an optional parameter. Set it to none for standard instruction-following behavior, or use low/medium/high when you need reasoning.

If you're migrating to or using pure reasoning models (o-series), here's the complete parameter gap:

Parameter	GPT-5.x models	Pure reasoning (o-series)	Migration path
`temperature`	✅	❌	Use `reasoning_effort` instead, or prompt template variants
`top_p`	✅	❌	Not needed, reasoning stabilizes output
`max_tokens`	✅	❌	Use `max_completion_tokens` (covers thinking + output)
`presence_penalty`	✅	❌	Not typically used in RAG anyway
`frequency_penalty`	✅	❌	Not typically used in RAG anyway
`reasoning_effort`	✅ (none/low/medium/high/xhigh)	✅ (low/medium/high only)	Built-in for GPT-5.x, only option for o-series
`system` prompt	✅	Treated as developer message	Don't use both system and developer message
`streaming`	✅	Limited (o3 with access)	Use progress indicators, not streaming text

Temperature and reasoning_effort

For RAG, you were almost certainly running temperature=0 or close to it. You wanted determinism, not creativity.

With GPT-5.x models: You can still use temperature=0 for deterministic output. If you want reasoning on specific queries, add reasoning_effort: low or medium. The two parameters work together.

With pure reasoning models (o-series): No temperature control. The reasoning process itself stabilizes output, giving you consistency by default. If you needed temperature for diversity (generating multiple answer variants), replace it with explicit prompt variants or use a GPT-5.x model with temperature + reasoning_effort: none.

The `max_completion_tokens` trap

Set this generously. Here's why: thinking tokens don't appear in your output but they're billed and count against this budget. At high effort, a single complex query can burn 10,000-50,000 thinking tokens. If max_completion_tokens is too low, the model's reasoning gets truncated mid-thought. You get degraded output and no obvious error signal. Start at 16,000 for medium effort, 32,000+ for high effort, and keep an eye on actual consumption.

Prompt verbosity inversion with reasoning

This one's weird if you've internalized standard prompt engineering advice. When using reasoning (either pure o-series models or GPT-5.x with reasoning_effort > none), being explicit and repetitive hurts. Over-specified prompts interfere with the internal reasoning chain. The model follows your constraints mechanically instead of actually reasoning toward the goal.

Write shorter, higher-trust prompts when reasoning is active. Define the goal and constraints but leave the method to the model. I know this feels uncomfortable. It's the opposite of everything we learned, but it works.

For GPT-5.x with reasoning_effort: none, standard verbose prompts work fine.

Phase 6 - Risk assessment

Before any real traffic touches the new model, take a step back and run a structured risk assessment against your eval results.

flowchart TD
    EVAL[Run eval suite on new model] --> F1 & F2 & F3 & F4

    F1{Format compliance
≥ threshold?}
    F2{Faithfulness
≥ threshold?}
    F3{Latency P95
within range?}
    F4{Edge case
behavior OK?}

    F1 -->|Fail| R1[Debug: markdown bleed,
JSON key drift,
length calibration]
    F2 -->|Fail| R2[Debug: world knowledge
bleeding through,
chunk attribution loss]
    F3 -->|Fail| R3[Evaluate cost/latency
tradeoff, consider
model tier change]
    F4 -->|Fail| R4[Expand golden set
for edge cases,
refusal tuning]

    F1 & F2 & F3 & F4 -->|All pass| PROCEED[Proceed to rollout]

    R1 & R2 & R3 & R4 --> FIX[Fix and re-evaluate]
    FIX --> EVAL

    style PROCEED fill:#1a3d2e,color:#8fd4b0
    style FIX fill:#3d1a1a,color:#d4a0a0

Risk catalog

Format regression - one of the most common things to break. Newer models may wrap answers in markdown when you didn't ask, change capitalization, add disclaimers, or subtly alter JSON key naming. Fixable, but only if you're measuring it.

Faithfulness change - this one can go either direction, which is what makes it tricky. More capable models are generally more faithful to retrieved context, but they also have stronger world-knowledge priors that can bleed through. Watch for answers that are factually correct but not actually grounded in the retrieved context. That's a subtle and dangerous failure mode.

Latency change - gpt-5.4-nano is generally comparable to gpt-4o-mini for standard workloads. gpt-5.4-mini and gpt-5.4 (full) are slower. When you enable reasoning (reasoning_effort: medium or higher), latency increases significantly. At high or xhigh effort, a single query can take 30-90+ seconds. Pure reasoning models (o-series) operate in this higher-latency range. Measure P50, P95, P99 before committing.

Cost change - run a cost projection on your golden set query lengths x new model pricing before migration. The golden set gives you a realistic token distribution. Don't estimate from toy examples.

Streaming behavior - if your UI depends on streaming tokens, verify the new model's streaming token patterns don't break downstream parsing. Partial JSON parsing, in particular, can fail on different tokenization patterns.

Silent behavioral drift post-migration - even after successful migration, model providers update weights silently. This is the argument for pinning to a specific version string (gpt-5.4-nano-2026-03-15 or similar dated versions) rather than a floating alias like gpt-5.4-nano. Floating aliases trade reproducibility for automatic access to improvements.

Phase 7 - Progressive rollout

I recommend don't big-bang this. The cost of a bad rollout isn't the rollout itself. It's the user impact during the time it takes you to notice and roll back.

flowchart LR
    A[Shadow mode
48-72h
Log only] -->|Metrics stable| B[5% traffic
Gate check]
    B -->|Pass| C[20% traffic
Gate check]
    C -->|Pass| D[50% traffic
Gate check]
    D -->|Pass| E[100%
Full cutover]

    B -->|Fail| R[Rollback + investigate]
    C -->|Fail| R
    D -->|Fail| R

    style A fill:#2a2a1e,color:#c4c490
    style E fill:#1a3d2e,color:#8fd4b0
    style R fill:#3d1a1a,color:#d4a0a0

Shadow mode first

Route a percentage of production traffic to the new model, log both responses, but serve only the old model's response to users. Run for 48-72 hours to collect real distribution data. This is the only way to validate behavior on production traffic without user exposure.

Yeah, shadow mode requires infrastructure. You need to fire two model calls per request and log both. Worth it though. You'll be surprised how many behavioral differences show up on real traffic that your golden dataset missed entirely.

Gate criteria at each step

Define your numeric pass/fail criteria before the rollout begins. At each step, automated checks verify:

gate_criteria:
  faithfulness_score: ">= 0.92"
  format_compliance: ">= 0.98"
  answer_relevance: ">= 0.90"
  p95_latency_ms: "<= 3000"
  refusal_accuracy: ">= 0.95"
  error_rate: "<= 0.001"

If any gate fails, the rollout stops. Not pauses for discussion. Stops. The investigation happens before traffic increases, not during.

Traffic segmentation for canary rollout

If you can segment production traffic by query type or user segment, use it. Start with lower-stakes queries. Factual lookups before complex synthesis. Internal users before customers.

Keep rollback ready for a week

Don't retire the old model integration the day you hit 100%. Keep the model name in a config variable so rollback is a single config change, not a code deployment. Wait at least a week before cleaning up the old path.

Phase 8 - Post-migration monitoring

You're not done when you hit 100% traffic. You've just established a new baseline that will itself drift.

Pin your model version string

This is the one I see skipped most often.

Use versioned model strings like gpt-5.4-nano-2026-03-15, not floating aliases like gpt-5.4-nano. OpenAI updates weights behind floating aliases without telling you. When they do, your system's behavior changes and you did nothing. You won't get a deprecation notice. You'll get unexplained metric shifts in your dashboards... if you have dashboards.

If you want automatic access to model improvements, fine. But make that a deliberate choice, not something you fell into because you didn't think about it.

Ongoing eval cadence

Run your eval suite on a random sample of production queries weekly. Use an LLM-as-judge approach so you don't need human annotation at scale. A weekly automated run catches behavioral drift from silent weight updates, shifts in what users are asking, or changes to your retrieval corpus that interact with generation in unexpected ways.

Set up alerting on eval score degradation. If faithfulness drops more than 3 points week-over-week, investigate. Don't wait for users to complain.

What your observability actually needs to cover

At minimum:

Token consumption per request (input, output, and thinking tokens if applicable)
The actual model version string used (not the alias, the resolved version)
Total latency per request, not just generation time
Output format validation pass/fail
Downstream parsing failures from unexpected output format

LangSmith, Arize Phoenix, and Helicone all handle these. One thing worth mentioning: generic APM monitoring won't give you what you need for an LLM system. The failure modes are different. The metrics that matter are different. It's a different kind of system.

The systems audit you should run regardless

Model deprecation is a forcing function. The problems it exposes have been there for months or years. The teams in the best position during migration aren't the ones who saw this specific deprecation coming. They're the ones who built systems that were observable and measurable from the start.

Two questions worth sitting with, migration or not:

Can you actually measure output quality? Not "does the team think it looks fine." Can you produce a number? If not, you're flying blind. A model change, a corpus change, a prompt tweak, a silent weight update... any of these will shift behavior and you won't know.

Is your prompt a specification or a pile of patches? Do the archaeology exercise. If the patch ratio is high (and it probably is), that's a liability that compounds with every future model change.

The migration is the deadline. But the real investment is in system discipline that makes the next migration boring instead of terrifying.

The illusion of control in AI-assisted engineering

2026-03-03T00:00:00+01:00

The illusion of control in AI-assisted engineering

Dashboards are green. Reviews complete on time. Audits pass. And the organization is slowly losing track of what its own systems actually do.

I keep running into this in teams that use AI-assisted engineering a lot. The controls are all still there. They're just not controlling what anyone thinks they're controlling.

The distinction nobody separates

When teams adopt AI-assisted development, the first question management asks is: Is a human still reviewing the output? Yes. Code review gates are there. Architecture sign-offs happen. Compliance checklists get filled in.

But the word "oversight" covers two different things. One is operational control: can you approve, reject, or modify the output? The other is epistemic control: do you actually understand what was built, why it works this way, and what it assumes?

AI-assisted workflows keep the first and slowly destroy the second. Every governance framework I've seen was designed when these two things were inseparable. The engineer who wrote a module was the same person who reviewed it and signed off on it. They understood it because they built it. That coupling is broken now, and most orgs haven't noticed.

What governance theater actually looks like

I watched this happen on a project last year. A team was building an integration layer between an enterprise platform and a third-party data provider. AI tooling generated the service contracts, transformation logic, and error-handling paths. Engineers reviewed everything. And here's what surprised me: the code review was thorough. More thorough than most reviews I've seen. Line-by-line, comments on the PR, long discussions about edge cases. Architecture board signed off.

Six months later, a cascading failure traced back to a silent retry pattern in the error-handling logic that misbehaved under load. The postmortem asked who understood the design intent behind that module. Nobody. The AI generated it. The engineer reviewed it carefully, caught formatting issues and a null-check bug, but never asked why the retry logic was structured that way because the review was focused on "is this correct," not "what is this assuming."

The review wasn't sloppy. That's what made it disturbing. The process did exactly what it was designed to do. It just wasn't designed for code that arrived without anyone having thought through the design.

The compliance version is worse. Imagine software company under financial regulation gets asked during an external audit to explain the reasoning behind an algorithmic approach touching customer data classification. The honest answer? It was AI-generated and the team validated the outputs without tracing the reasoning. No compliance framework accepts that. You either make up a rationale after the fact or admit a gap the auditors weren't looking for.

How responsibility diffuses

In a normal engineering workflow, you can point at someone and say: you designed this, you understood it, you signed off. Architecture review boards exist partly to create that paper trail. When something breaks, you know who to ask.

AI-assisted workflows put something in the authorship chain that can't be asked, can't explain itself, and can't take responsibility when a regulator shows up. Responsibility doesn't move to the AI. It just gets spread thin across everyone who touched the output. Everyone reviewed it. Nobody wrote it. When an incident review or an auditor asks "who understood this design," nobody answers.

You don't notice this in any one sprint. But after a year of it, an organization ends up with a growing pile of production systems where nobody recorded the design intent, nobody captured the assumptions, and the engineers maintaining the code can't explain why a service boundary is where it is or why the retry logic works the way it does. The code passes tests. The architecture diagrams exist. The understanding doesn't.

The slower risks

Leadership conversations about AI in engineering tend to focus on the obvious stuff: hallucinated code, security vulnerabilities, IP leakage. Those are real. The risks that worry me more take longer to show up.

One is architectural drift. When dozens of teams across an organization generate code through AI without shared context, each team's output embeds slightly different assumptions about data models, error handling, and service contracts. Service A assumes retries are idempotent. Service B doesn't. Both pass their own tests. The inconsistency only surfaces when they're under load together, and by then the damage spreads fast.

Another: AI-generated systems are unusually hard to change later. Normal technical debt is annoying but manageable because somebody on the team once understood the code. When a system was generated and reviewed but not really authored, the next team inherits a black box. They can't refactor confidently because nobody can tell them what the original constraints were. So they wrap it, route around it, or just don't touch it. The codebase hardens in a way that's different from normal decay.

Regulators are catching up, too. In financial services and telecom especially, the questions about AI-assisted code generation are getting specific. The organizations I'd worry about aren't the ones using AI the most. They're the ones that adopted it without changing their traceability and governance to match.

Reframing the question

The question isn't whether humans are in the loop. They are. The question is whether the human hitting "approve" on a PR could explain the design to an auditor, rebuild the module if the original was lost, or predict how it fails under conditions the tests don't cover.

Reviewing code for correctness and understanding a system's architecture are different activities. Review processes were designed when they were the same activity. AI-assisted workflows have split them apart, and no one updated the process to compensate.

Some of this is fixable. Not every AI-generated module needs deep human comprehension. Boilerplate, scaffolding, config, sure. But integration contracts, error-handling strategies, anything that encodes assumptions about how other systems behave? Someone needs to understand those, and "reviewed the PR" isn't the same as understanding them. Architectural reviews need to start asking "why is this designed this way" instead of just "does this look right." Traceability for AI-generated code needs to capture the assumptions, not just the output.

If leadership teams keep the review ceremonies running while the understanding underneath thins out, they're not managing risk. They're performing risk management. And that works until someone actually tests it, which in regulated industries tends to happen during an incident or an audit, when the consequences are already serious.

Questions worth sitting with

Can your organization tell the difference between systems your teams approved and systems they actually understand? Does anyone track that? I've never seen it tracked.

If a regulator or an acquirer asked your team to walk through the design intent behind a critical integration, who would take the call? Would they be explaining what they know, or reverse-engineering it on the spot?

Have your sign-off processes changed at all since AI adoption started? Every code review approval and architecture sign-off carries an implicit claim: "I understood this." If that claim has quietly become "I checked this," the paperwork is still flowing but the assurance behind it isn't there anymore.

I don't have good answers to any of this. I'm working through it myself.

Version Your Vectors - Index Versioning as the Missing Layer in RAG Observability and Compliance

2026-02-26T00:00:00+01:00

1. The invisible problem: why RAG systems silently drift

Most teams building Retrieval-Augmented Generation systems obsess over the LLM, its version, its temperature, its prompt. The component that actually changes most often and most quietly is the index: the corpus of documents, their chunked representations, and the embeddings that map them into vector space.

Think about what happens when:

A policy document is updated, but the old version lingers in the index alongside the new one.
An embedding model is upgraded, shifting similarity rankings for every query.
A reindexing job silently drops a document partition due to a pipeline bug.
A new batch of documents is ingested, and one source starts dominating 80% of retrievals on a topic.

In each case, the RAG system's answers change, but the LLM didn't. The model is the same. The prompt is the same. The retrieval configuration is the same. Yet the user gets a different answer today than they got last week.

Without index versioning, you cannot answer the most basic audit question:

"Why did the system give this answer on January 10th?"

This is like running a production database without transaction logs or backups. You can serve queries, but you cannot explain, debug, or defend any historical result.

The index is the part of a RAG pipeline that changes the most, and almost nobody versions it.

flowchart LR
    Q[Query] --> M[Same Model]
    M --> P[Same Prompt]
    P --> I{Changed Index}
    I -->|v1| A1[Answer A]
    I -->|v2| A2[Different Answer]

    style I fill:#f96,stroke:#333,color:#000
    style A2 fill:#f96,stroke:#333,color:#000

2. Use cases that justify index versioning

Index versioning is not a theoretical exercise. Here are concrete scenarios where missing versioning causes real operational, legal, or quality failures.

2.1 Debugging and root cause analysis

Answer quality suddenly degrades. Users report that the system "stopped knowing" about a topic, or started contradicting itself. The engineering team investigates:

Was it a model change? No, same model, same temperature.
Was it a prompt change? No, same template.
Was it an index change? Unknown.

Without index versioning, you cannot isolate retrieval drift from generation drift. You're debugging blind. With version tags on every retrieval request, you can immediately correlate quality drops with specific index changes: "Hallucination rate jumped 15% after index_v23 was deployed, which coincided with a re-chunking of the legal policy corpus."

2.2 Data drift detection

The knowledge base is not static. Documents are added, updated, and removed. Embedding models are upgraded. Chunking strategies evolve.

Index versioning lets you track:

Corpus drift: How many documents were added or removed between versions?
Embedding drift: Did switching from text-embedding-ada-002 to text-embedding-3-large change retrieval behavior?
Source dominance shifts: Is one internal document now responsible for 70% of answers on a topic, when it used to be 30%?

These are not hypothetical concerns. In enterprise deployments, source dominance drift is one of the most common causes of biased or narrow answers, and it is invisible without versioned tracking.

2.3 Regulatory compliance and audit

This is where index versioning goes from "nice to have" to "legally required."

Under the EU AI Act (Regulation 2024/1689), high-risk AI systems must maintain records sufficient to reconstruct past decisions. If a RAG system supports employment decisions, credit assessments, or healthcare recommendations, a regulator may ask:

"Reconstruct the knowledge base state and the retrieval trace for the answer provided to user X on date Y."

Without index versioning, this is impossible.

In financial services, auditors need to verify what knowledge base was active when an AI-assisted recommendation was generated. In healthcare, regulators need to confirm which clinical guidelines were indexed at the time of an AI-supported diagnosis.

2.4 Reproducibility for evaluation

RAG evaluation is meaningless without controlled variables. If you're comparing two retrieval strategies, two embedding models, or two chunking approaches, you need to hold the index constant, or at least know exactly how it differed.

Index versioning gives you:

A/B testing of embedding models with rollback to the control version.
Regression testing: "Did the new index version degrade performance on our golden test set?"
Longitudinal evaluation: tracking quality metrics across index versions over months.

2.5 Incident response and rollback

A corrupted ingestion pipeline indexes the wrong documents. A hallucination spike is traced to a specific batch of poorly formatted PDFs. A critical policy document was accidentally excluded during re-indexing.

In each case, the response must be fast:

Identify the problematic index version.
Rollback to the last known-good version.
Serve from the rolled-back index while the issue is resolved.

Without versioning, step 1 is a forensic investigation, step 2 is impossible, and step 3 requires rebuilding the entire index from scratch.

2.6 Counterfactual replay

For bias investigations and fairness audits, you need to ask:

"What would the system have answered with last month's index?"

This requires the ability to replay a query against a historical index state. Counterfactual replay is the best method we have for demonstrating that a system change improved (or degraded) fairness, accuracy, or safety.

3. What exactly needs versioning?

People often equate "index versioning" with "snapshotting the vector store." That's necessary but not enough. A RAG index is a composite artifact made of multiple independently changing components:

Component	What changes	Why it matters
Document corpus	Documents added, removed, or updated	The content of the knowledge base
Chunk boundaries	Chunking strategy or parameters change	Same document produces different retrieval units
Embedding model	Model version upgraded	Same text maps to different vectors
Vector index structure	HNSW/IVF parameters, rebuild	Approximate search behavior changes
Metadata and filters	ACLs, tags, timestamps, source labels	What is retrievable changes
Prompt templates	System prompt evolves	How retrieved context is used changes

A "version" is the combination of all these, not just the vector snapshot. If you version the vectors but not the chunking strategy, the same document will produce different answers and you won't know why.

In practice, your version identifier should encode or reference all of these components. A simple approach is a composite version string or a version manifest:

{
  "index_version": "v47",
  "corpus_hash": "a3f8c1...",
  "chunk_strategy": "semantic_v2",
  "embedding_model": "text-embedding-3-large",
  "embedding_model_version": "2025-09",
  "vector_index_type": "HNSW",
  "document_count": 14832,
  "last_updated": "2026-02-15T08:30:00Z",
  "prompt_template_version": "v12"
}

4. Regulatory context: when versioning becomes mandatory

4.1 EU AI Act risk classification for RAG

The EU AI Act does not regulate "RAG systems" as a category. It regulates AI systems based on their application domain and impact.

A RAG system becomes high-risk when deployed in:

Employment and worker management (screening, recruitment, task allocation)
Access to essential services (credit scoring, insurance, social benefits)
Education and vocational training (admissions, assessment)
Law enforcement (evidence evaluation, risk assessment)
Healthcare (clinical decision support, triage)

A RAG system used as an internal productivity tool (e.g., searching company wikis, drafting emails) is generally limited-risk or minimal-risk. Transparency obligations apply (users must know they're interacting with AI), but the heavy documentation and logging requirements do not.

For deployers integrating third-party LLMs into RAG pipelines: even if the base model is compliant, your retrieval layer, index, and data governance are your responsibility. The model provider's compliance does not cover your index.

4.2 What the regulation actually requires

For high-risk systems, the EU AI Act mandates:

Article 12 (Record-Keeping): Automatic logging of events sufficient to trace system behavior throughout its lifecycle. For RAG, this means logging retrieval traces, index versions, and context assembly decisions, not just input/output.
Article 11 (Technical Documentation): Description of the system architecture, data sources, training data (or in RAG's case, indexed data), and their versions.
Article 14 (Human Oversight): The system must allow effective human supervision, including the ability to review retrieval results and override decisions.
Article 9 (Risk Management): Ongoing monitoring, testing, and mitigation, which for RAG includes detecting index drift, evaluating retrieval quality, and maintaining rollback capability.

The regulation does not require opening neural weights or explaining attention patterns. It requires system-level traceability and governance. Index versioning is the mechanism that makes this possible for the retrieval layer.

4.3 Proportional versioning: match effort to risk

The principle here is proportionality. Not every RAG system needs a full replay architecture. But every RAG system needs some level of version tracking.

Risk level	What to track	How to implement
Low risk (internal assistant)	Embedding model version, document count, last update timestamp	Metadata tags on telemetry spans
Medium risk (decision support)	Above + document hash sets, retrieval traces, index snapshot references	Structured audit events with blob storage references
High risk (regulatory decisions)	Above + full replay capability, immutable audit trail, point-in-time index reconstruction	Time-travel capable vector DB + archived prompts + evaluation metrics store

Don't over-engineer for low-risk systems. Don't under-engineer for high-risk systems. The cost of getting this wrong runs in both directions: wasted engineering effort on one end, regulatory exposure on the other.

5. Solution architectures: from lightweight to full replay

graph BT
    L0["Level 0: Metadata Tagging
~zero cost 

 basic drift detection"]
    L1["Level 1: Content Hashing
low cost 

 change detection + audit trail"]
    L2["Level 2: Index Snapshots
medium cost 

 point-in-time recovery + A/B testing"]
    L3["Level 3: Full Replay
high cost 

 counterfactual analysis + regulatory demo"]
    L4["Level 4: Self-Healing Index
highest cost 

 automated drift detection + remediation"]

    L0 --> L1 --> L2 --> L3 --> L4

    style L0 fill:#c8e6c9,stroke:#333,color:#000
    style L1 fill:#a5d6a7,stroke:#333,color:#000
    style L2 fill:#fff9c4,stroke:#333,color:#000
    style L3 fill:#ffcc80,stroke:#333,color:#000
    style L4 fill:#ef9a9a,stroke:#333,color:#000

5.1 Level 0: metadata tagging (minimum viable versioning)

The cheapest possible starting point. Tag every retrieval request with basic version metadata:

index_version, a monotonically increasing identifier
embedding_model_version, which model generated the vectors
document_count, how many documents are in the index
last_updated_timestamp, when the index was last modified

Store these as OpenTelemetry span attributes on every rag.retrieval span.

Cost: Near zero. Value: Basic drift detection, version correlation with quality metrics, and historical analysis.

This is the absolute minimum. If you are not doing this today, start here.

5.2 Level 1: content hashing and change detection

At ingestion time, compute a hash for each document and each chunk. Store a document_hash_set_id that uniquely identifies the set of documents (and their versions) in the index.

This gets you:

Change detection without diff storage: know that the index changed between v46 and v47, and which documents changed.
Selective re-embedding: only re-embed documents whose content hash changed, saving API costs.
Audit trail: prove exactly which document versions were indexed at any point.

A practical schema extension for pgvector or similar:

ALTER TABLE embeddings ADD COLUMN content_hash TEXT;
ALTER TABLE embeddings ADD COLUMN model_version TEXT;
ALTER TABLE embeddings ADD COLUMN indexed_at TIMESTAMP DEFAULT NOW();

This pattern is well-documented in production pgvector deployments where content hash and text diff ratios determine whether re-embedding is necessary.

5.3 Level 2: index snapshots with point-in-time recovery

This is where you gain the ability to reconstruct past index states.

Several approaches, depending on your vector store:

Database-native time travel: - LanceDB provides built-in versioning with zero-cost snapshots. Every mutation creates a new version. You can check out any historical version and query against it. - Qdrant supports collection snapshots that can be stored and restored. - Milvus offers point-in-time recovery through its storage layer.

Alias-based zero-downtime deployment: - In Elasticsearch/OpenSearch, create timestamped indexes (rag_index_2026_02_15) and use aliases (rag_index_current) to swap atomically. Old indexes remain queryable for audit.

Storage-based snapshots: - For FAISS or ChromaDB, periodically snapshot the index files to blob storage (S3, Azure Blob). Tag with the version manifest from Section 3.

Blue-green deployment for model upgrades: - When upgrading the embedding model, build the new index in parallel. Route a percentage of traffic to the new index. Compare quality metrics. If acceptable, switch the alias. If not, rollback is instantaneous.

5.4 Level 3: full replay architecture

Store enough state to deterministically replay any historical request:

Index snapshot reference, which version was active
Retrieved document IDs and scores, the actual retrieval results
Exact prompt sent to LLM, including the assembled context
Model version and parameters: temperature, top_p, model name
Evaluation metrics: groundedness score, hallucination risk, confidence

With all five, you can answer:

"What would this query have returned with the index from three months ago?"

This is what makes counterfactual analysis, bias investigations, and regulatory demonstrations possible.

The Drift-Adapter approach offers an interesting optimization here: instead of maintaining full parallel indexes for every embedding model version, use a lightweight learned transformation layer to map new query embeddings into the legacy embedding space. This achieves 95-99% of retrieval performance recovery at a fraction of the storage cost.

5.5 Level 4: self-healing index with drift detection

The most mature pattern: the index monitors its own health and triggers corrective actions.

Continuous monitoring: - Track retrieval quality metrics (precision@k, nDCG, confidence score distributions) per index version. - Monitor content freshness: flag embeddings older than a configurable threshold. - Detect semantic drift: compare embedding distributions between index versions.

Automated drift detection signals: - Content hash changes without corresponding re-embedding - Retrieval score distribution shift beyond a threshold - Embedding age exceeding freshness policy - Source dominance index crossing a configured limit

Automated remediation: - Trigger selective re-embedding for stale content - Alert on anomalous quality metric changes - Initiate alias-based index rebuild if drift exceeds tolerance

This pattern has been demonstrated in production using Elasticsearch with alias-based rebuilds, where the system continuously monitors query quality and initiates zero-downtime reindexing when drift is detected.

6. Implementing observability around index versions

6.1 OpenTelemetry span design

Model every RAG request as a span hierarchy. The index.version attribute must be present on the root span and the retrieval span:

rag.request (root span)
  ├── index.version: "v47"
  ├── embedding.model: "text-embedding-3-large"
  ├── rag.retrieval
  │     ├── top_k: 5
  │     ├── retrieved_doc_ids: ["doc_123", "doc_456", ...]
  │     ├── retrieval_scores: [0.92, 0.87, ...]
  │     └── filter_applied: true
  ├── rag.context_build
  │     └── context_token_count: 2400
  ├── rag.generation
  │     ├── model: "gpt-4o"
  │     └── temperature: 0.1
  └── rag.evaluation
        ├── groundedness_score: 0.91
        └── hallucination_risk: 0.08

This structure is natively understood by Azure Application Insights, Jaeger, SigNoz, and other OpenTelemetry-compatible backends.

6.2 Metrics to track

Metric	What it reveals	Alert threshold (example)
Index freshness	Time since last index update	> 7 days for active knowledge bases
Document churn rate	Additions/removals per period	> 20% churn in 24 hours
Embedding model version distribution	Mixed model versions in index	Any version mismatch
Retrieval score distribution	Shift in confidence over time	Mean score drops > 10% week-over-week
Source dominance index	Single source answering disproportionately	Any source > 60% of retrievals for a topic
Unsupported claim rate	Claims not grounded in retrieved context	> 5% of responses

Configure alerts for:

Hallucination rate spike within 24 hours of a reindexing event
Sudden drop in average retrieval confidence scores
Unexpected change in document count (indicating accidental deletion or duplication)
Retrieval latency spike (possibly caused by index corruption or misconfiguration)

The important thing is correlating quality metrics with index version transitions. Without the version tag, an alert tells you something is wrong. With it, the alert tells you what caused it.

6.4 Governance dashboards

For teams operating RAG in regulated environments:

Index version timeline with quality metrics overlaid, so you can see exactly when quality changed and which index version was responsible.
Document lifecycle view: when was each document added, modified, or removed from the index?
Cross-version comparison: side-by-side retrieval results for the same query across two index versions.
Regulatory audit view: for a given date range, show all index versions that were active and their associated quality metrics.

7. Practical decision framework

Use this decision tree to choose your versioning level:

flowchart TD
    Start([What versioning level do you need?]) --> Q1{Regulated domain?
finance, healthcare,
employment, legal}
    Q1 -->|Yes| R1[Level 2+ with
immutable audit trail]
    Q1 -->|No| Q2{Index changes
more than weekly?}
    Q2 -->|Yes| R2[Level 1+
change detection]
    Q2 -->|No| Q3{Need to explain
past answers?}
    Q3 -->|Yes| R3[Level 3
replay capability]
    Q3 -->|No| Q4{High-risk under
EU AI Act?}
    Q4 -->|Yes| R4[Level 3-4
immutable logs +
point-in-time reconstruction]
    Q4 -->|No| Q5{Experimenting with
embeddings or chunking?}
    Q5 -->|Yes| R5[Level 2+
blue-green + A/B]
    Q5 -->|No| R6[Level 0
metadata tagging
is sufficient]

    style R1 fill:#ef9a9a,stroke:#333,color:#000
    style R4 fill:#ef9a9a,stroke:#333,color:#000
    style R3 fill:#ffcc80,stroke:#333,color:#000
    style R2 fill:#fff9c4,stroke:#333,color:#000
    style R5 fill:#fff9c4,stroke:#333,color:#000
    style R6 fill:#c8e6c9,stroke:#333,color:#000

8. Common pitfalls

You version the vectors but forget the chunking strategy. The same document, chunked differently, retrieves differently and answers differently. Change your chunking approach without bumping the index version and your version history lies to you.

Nobody tracks prompt template changes. Index is the same. Model is the same. But someone tweaked the system prompt last Tuesday, and now identical retrieved context produces different answers. If prompt versions aren't in your audit trail, you have a blind spot.

You snapshot the whole index when you don't need to. A million documents at 1536 dimensions is several gigabytes per snapshot. Often, storing the document hash set, the embedding model version, and the index config is enough to rebuild the index on demand. You don't always need the raw vectors sitting in cold storage.

No retention policy. Keeping every version forever gets expensive and may run afoul of GDPR data minimization rules. Set retention windows based on actual regulatory requirements: maybe six months for an internal tool, five to ten years for healthcare or finance.

You treat versioning as a one-time setup. Pipelines change. New data sources show up. Embedding models get upgraded. Chunking strategies evolve. If your versioning system can't keep up, it becomes stale metadata that nobody trusts. Build it as infrastructure you maintain, not a script you ran once.

9. Index versioning as a first-class concern

The retrieval index is the part of your RAG system that changes the most and gets governed the least. It shifts more often than the model, more quietly than the prompt, and its effect on answer quality is larger than either.

Versioning your index is not optional for any team that cares about debugging, compliance, or evaluation. Without it, you can't explain past answers, you can't detect drift, and you can't roll back when something breaks.

How much versioning you need depends on what's at stake:

Level 0: tag every request with index_version and embedding_model_version. It costs nothing. Do this today.
Level 1-2: add these when your index changes often or when someone asks why the system gave a particular answer last month.
Level 3-4: invest here when regulators require it or when your system's outputs carry real consequences for real people.

The question most RAG systems can't answer today is straightforward:

"What did the system know, and how did it use that knowledge, on a specific date?"

If you can answer that, you're ahead of almost everyone.

Publishing a Python CLI Tool to Homebrew

2026-02-05T00:00:00+01:00

Overview

Homebrew distribution lets users install your Python CLI with brew install your-tool. The process: create a GitHub "tap" repository, generate a Ruby formula file, test it, and publish.

Prerequisites

Your package must be on PyPI with an sdist (source distribution), not just wheels. All dependencies need sdist too. Define entry points in pyproject.toml:

[project.scripts]
your-cli-tool = "your_package.cli:main"

Step-by-Step Process

1. Create a tap repository. A Homebrew tap is just a GitHub repository with a specific naming convention. Create a repo named homebrew-{something} under your account. When someone runs brew tap yourusername/something, Homebrew looks for github.com/yourusername/homebrew-something.

# Users will install your tool like this:
brew tap yourusername/tools
brew install your-cli-tool

# Or in one command using the full path:
brew install yourusername/tools/your-cli-tool

Inside your tap repository, create a Formula directory for your Ruby .rb formula files. See How to Create and Maintain a Tap.

2. Generate the formula using homebrew-pypi-poet in a fresh venv:

cd /tmp && python -m venv venv && source venv/bin/activate
pip install your-cli-tool homebrew-pypi-poet
poet -f your-cli-tool > your-cli-tool.rb

3. Test locally before pushing:

HOMEBREW_NO_INSTALL_FROM_API=1 brew install --build-from-source --verbose --debug ./your-cli-tool.rb

4. Push to GitHub. Users install via brew install yourusername/tapname/your-cli-tool.

flowchart LR
    A[Create tap repo] --> B[Generate formula with poet]
    B --> C[Test locally]
    C --> D[Push to GitHub]
    D --> E[Users can brew install]

Common Pitfalls

Missing sdist - Package or dependency lacks source distribution on PyPI
Contaminated poet env - Always use a fresh venv for formula generation
No license - Required for homebrew-core submission

Going Further

For homebrew-core submission (pre-compiled bottles, brew search discoverability), see homebrew-core CONTRIBUTING. For automated updates, see GitHub Actions automation.

Sources and Further Reading

Packaging a Python CLI tool for Homebrew - Simon Willison's comprehensive walkthrough
Python for Formula Authors - Official Homebrew Python documentation
Formula Cookbook - Writing and testing formulas
homebrew-pypi-poet - Formula generation tool

The limiting factor at work isn't writing code anymore

2026-01-28T00:00:00+01:00

In a Hacker News discussion on the article Management as AI Superpower by Ethan Mollick, I came across a comment that resonated strongly with a realization I’ve had recently while coding with a factory.ai droid:

The limiting factor at work isn’t writing code anymore. It’s deciding what to build and catching when things go sideways.

(link by augusteo)

This observation forces a shift in priorities.

Deciding what to build

Creative minds tend to have no shortage of ideas. With agentic coding, it’s incredibly tempting to start immediately: the goal feels only “a few hours of AI agent work” away, something you can even run in the background.

Starting to build something immediately, however, incurs real costs: - mental engagement Once you commit, you have to keep making decisions: should you continue, pivot, or drop the project? Dropping it comes with a psychological cost—unfinished work, a sense of abandonment, or the feeling of being unable to finish. These are easy mental traps to fall into. You end up with one more thing occupying your cognitive space.

The unfinished projects creating cognitive and emotional debt - “psychological technical debt”. AI accelerates its accumulation by lowering the barrier to starting but not to finishing.

expected returns on investment You still need to ask “why” you are building something—ideally several times, as in the Five Whys method. Even in the era of AI agents, resources remain limited. Your time is certainly finite. I’ll set aside, for now, considerations about token costs—both in dollars and in the broader sense of shared planetary resources.

Decision latency can be a competitive advantage. This could be expanded into a framework for “pre-coding decision filters” in an agentic world: how to deliberately slow down before starting, when everything pushes you to start immediately.

Catching when things go sideways

In many of my projects, there hasn’t been enough time to add strong guardrails for quickly detecting when things start to go wrong. Test suites need to be designed with clear principles and intent.

Today, it’s easy to get close to 100% test coverage with AI agents. The problem lies deeper, in how models are trained: they are optimized to produce an answer rather than to challenge assumptions. I’ve experienced that when a test suite is AI-generated from existing code, it can be fundamentally flawed. The tests often mirror the current implementation of the business logic—even if that logic is incorrect.

Without solid specifications or clearly stated expectations, generated tests tend to confirm what already exists rather than falsify it. They won’t catch flaws unless there is a reliable external reference, such as a well-defined specification. “AI tests can be bad,” is known but more: they systematically reflect existing assumptions unless anchored to external specifications. This confirmatory nature of AI-generated tests is a blind spot for teams that want to quickly close testing gaps and reduce technical debt.

From a Python-specific perspective, extensive usage of typing mechanisms available in Python can help catching bugs and inconsistencies in implementation.

Other thoughts from the discussion

The skills that matter are the same ones that make someone a good manager of people.

(link by augusteo)

The current situation creates pressure for a fast track into managerial competence. Developers supervising and mentoring AI agents get many hours of practice in planning, delegating, and communicating work in a clear, actionable, and precise way. This becomes a kind of manager’s dojo: a training ground entered almost automatically when stepping into AI-assisted coding. Agent supervision exercises the same muscles as people management: planning, delegation, expectation-setting, and review. This reframes career progression and challenges the idea that management skills only come from managing people. I expect in a near future an abundance of technically inclined people with strong managerial skills, even if they have never formally led a human team.

Another comment adds an important nuance:

The apparent speedup only holds if we ignore the cost of comprehension and review; once those are included, the comparison becomes less about raw code throughput and more about where and how understanding is generated in the process.

(link by ithkuil)

This reframes productivity not as code output, but as the efficiency of understanding.

Using MLflow-RAGAS Integration Without Tracing

2026-01-11T00:00:00+01:00

Option 1: Static Data Approach (Recommended)
Key Points
Option 2: Simple predict_fn (No Tracing Decorator)
Comparison: When to Use Each Approach
Complete Working Example
Recommendation

You can use MLflow's RAGAS integration to calculate evaluation scores and log them to experiments without implementing MLflow traces in your RAG pipeline. This is useful when:

You have an existing RAG pipeline you don't want to modify
You're doing batch evaluation of pre-computed results
You want to quickly test RAGAS metrics without full instrumentation

Option 1: Static Data Approach (Recommended)

Pre-compute your RAG outputs, then pass everything to mlflow.genai.evaluate() with the outputs field already populated.

import mlflow
from mlflow.genai.scorers import Faithfulness, FactualCorrectness

# Your existing RAG pipeline (no MLflow code needed)
def my_rag_pipeline(question: str) -> tuple[str, list[str]]:
    """Your RAG implementation - completely unchanged."""
    contexts = my_retriever.retrieve(question)
    answer = my_llm.generate(question, contexts)
    return answer, contexts

# Golden dataset
golden_dataset = [
    {
        "question": "What is MLflow?",
        "ground_truth": "MLflow is an open-source platform for ML lifecycle management.",
        "contexts": ["MLflow is an open-source platform..."]  # Expected contexts
    },
    # ... more samples
]

# Step 1: Run your RAG pipeline and collect outputs
eval_data = []
for item in golden_dataset:
    answer, retrieved_contexts = my_rag_pipeline(item["question"])
    eval_data.append({
        "inputs": {
            "question": item["question"]
        },
        "outputs": {
            "response": answer,
            "retrieved_contexts": retrieved_contexts,  # Required for Faithfulness
        },
        "expectations": {
            "expected_answer": item["ground_truth"],
            "ground_truth": item.get("contexts", []),  # For ContextRecall
        },
    })

# Step 2: Run evaluation with static data (no predict_fn)
mlflow.set_experiment("rag-evaluation")

with mlflow.start_run(run_name="static-data-evaluation"):
    mlflow.log_param("evaluation_mode", "static_data")
    mlflow.log_param("num_samples", len(eval_data))
    results = mlflow.genai.evaluate(
        data=eval_data,  # Data already includes outputs
        scorers=[
            Faithfulness(model="openai:/gpt-4o-mini"),
            FactualCorrectness(model="openai:/gpt-4o-mini"),
        ],
    )
    print(f"Results logged to run: {mlflow.active_run().info.run_id}")

Key Points

No predict_fn needed - outputs are pre-computed
retrieved_contexts in outputs - Required for Faithfulness metric to work
Your RAG code stays unchanged - No decorators or MLflow imports needed in your pipeline

Option 2: Simple predict_fn (No Tracing Decorator)

If you prefer the predict_fn approach but don't want tracing:

def simple_rag_predict(question: str) -> dict:
    """Wrapper around your RAG - no @mlflow.trace decorator."""
    contexts = my_retriever.retrieve(question)
    answer = my_llm.generate(question, contexts)
    return {
        "response": answer,
        "retrieved_contexts": contexts,
    }

# Evaluation data without outputs (they'll be generated)
eval_data = [
    {
        "inputs": {"question": item["question"]},
        "expectations": {
            "expected_answer": item["ground_truth"],
            "ground_truth": item.get("contexts", []),
        },
    }
    for item in golden_dataset
]

with mlflow.start_run():
    results = mlflow.genai.evaluate(
        predict_fn=simple_rag_predict,
        data=eval_data,
        scorers=scorers,
    )

Comparison: When to Use Each Approach

Approach	Tracing	Faithfulness	Best For
Static data (Option 1)	Not needed	Works (needs `retrieved_contexts` in outputs)	Existing pipelines, batch evaluation, CI/CD
predict_fn with @mlflow.trace	Required	Works (reads from RETRIEVER spans)	New pipelines, debugging, detailed traces
predict_fn without tracing (Option 2)	Not needed	Works (needs `retrieved_contexts` in return)	Quick integration, simple pipelines

Complete Working Example

"""
Complete example: Evaluate any RAG pipeline with RAGAS metrics in MLflow.
No tracing required in your RAG implementation.
"""

import mlflow
from mlflow.genai.scorers import Faithfulness, FactualCorrectness

# Configure MLflow
mlflow.set_tracking_uri("http://localhost:5000")  # Or your tracking server
mlflow.set_experiment("my-rag-evaluation")

# Your RAG pipeline (example - replace with your implementation)
class MyRAGPipeline:
    def __init__(self, retriever, llm):
        self.retriever = retriever
        self.llm = llm

    def query(self, question: str) -> tuple[str, list[str]]:
        contexts = self.retriever.search(question, k=3)
        answer = self.llm.generate(
            f"Context: {contexts}\n\nQuestion: {question}\n\nAnswer:"
        )
        return answer, contexts

# Initialize your pipeline
rag = MyRAGPipeline(retriever=my_retriever, llm=my_llm)

# Your evaluation dataset
test_questions = [
    {"question": "What is X?", "ground_truth": "X is..."},
    {"question": "How does Y work?", "ground_truth": "Y works by..."},
]

# Build evaluation data with pre-computed outputs
eval_data = []
for item in test_questions:
    answer, contexts = rag.query(item["question"])
    eval_data.append({
        "inputs": {"question": item["question"]},
        "outputs": {
            "response": answer,
            "retrieved_contexts": contexts,
        },
        "expectations": {
            "expected_answer": item["ground_truth"],
        },
    })

# Run evaluation
with mlflow.start_run(run_name="rag-eval-no-tracing"):
    # Log your pipeline configuration
    mlflow.log_params({
        "retriever_k": 3,
        "llm_model": "gpt-4o-mini",
        "num_samples": len(eval_data),
    })
    # Evaluate with RAGAS scorers
    results = mlflow.genai.evaluate(
        data=eval_data,
        scorers=[
            Faithfulness(model="openai:/gpt-4o-mini"),
            FactualCorrectness(model="openai:/gpt-4o-mini"),
        ],
    )
    # Results are automatically logged to MLflow
    print("Evaluation complete!")
    print(f"View results: mlflow ui --port 5000")
    print(f"Run ID: {mlflow.active_run().info.run_id}")

# Access results programmatically
print(results.metrics)
print(results.tables["eval_results_table"])

Recommendation

For existing RAG pipelines, use Option 1 (Static Data):

Minimal changes - Your RAG code stays completely unchanged
Batch-friendly - Run your pipeline once, evaluate multiple times with different scorers
CI/CD compatible - Easy to integrate into automated testing pipelines
Full metric support - All RAGAS metrics work when you include retrieved_contexts in outputs

The only requirement is that your RAG pipeline returns both the answer and the retrieved contexts, which most RAG implementations already do.

RAG Evaluation with RAGAS and MLflow - A Practical Guide

2026-01-08T00:00:00+01:00

RAG Evaluation with RAGAS and MLflow

Table of contents

Why This Tutorial?
What You'll Learn
Prerequisites
Setup and Configuration
- LLM Provider Configuration
Sample Knowledge Base
Minimal RAG Pipeline
- Enable MLflow Tracing
Load Golden Dataset
- Generate RAG Responses for Evaluation
RAGAS Evaluation with MLflow
- Prepare Evaluation Data
MLflow Results Analysis
- Interpreting RAGAS Scores
Common Pitfalls and Solutions
Extras: Comparing RAG Variants with MLflow
- Example: Comparing Chunk Sizes
- Comparing Results in MLflow UI
How to inspect results in MLflow UI:
More from MLflow
References, further reading

Why This Tutorial?

Evaluating RAG pipelines is surprisingly difficult. You can build a working retrieval system in an afternoon, but answering "Is it actually good?" requires systematic measurement.

The challenge: Manual evaluation doesn't scale. Eyeballing a few responses tells you almost nothing about overall quality. You need metrics that capture different aspects of RAG performance:

Is the retriever finding relevant documents?
Is the LLM staying faithful to the retrieved context (not hallucinating)?
Are the answers factually correct?

The solution: RAGAS provides standardized metrics for RAG evaluation. MLflow provides experiment tracking. Together, they enable systematic, reproducible evaluation that you can run on every pipeline change.

What made this tricky: The MLflow-RAGAS integration looks simple in the docs, but getting it to work with a real LangChain pipeline required navigating several non-obvious requirements:

Specific model URI formats for different providers
Function signatures that match MLflow's expectations
Proper tracing spans for context-aware metrics

This tutorial documents what actually works, including the gotchas I encountered along the way.

This tutorial demonstrates how to evaluate Retrieval-Augmented Generation (RAG) systems using RAGAS (Retrieval Augmented Generation Assessment) metrics through MLflow integration.

NOTE: RAGAS is a third-party evaluation library. For more details, visit the RAGAS GitHub repository. At the time of writing (January 2026), apart from RAGAS, MLFlow supports another third-party scorer/evaluation library: DeepEval.

What You'll Learn

Build a minimal RAG pipeline using LangChain and FAISS
Create a golden evaluation dataset with expected answers
Evaluate RAG quality using RAGAS metrics (Faithfulness, Context Precision, Context Recall, Factual Correctness)
Track results in MLflow for systematic comparison
Support multiple LLM providers: OpenAI, Azure OpenAI, and Ollama

Prerequisites

Python 3.10+
API key for your chosen LLM provider
Basic understanding of RAG concepts

Setup and Configuration

import os
import warnings
from enum import Enum

import pandas as pd

warnings.filterwarnings("ignore")

LLM Provider Configuration

This tutorial supports three LLM providers. Choose your provider and configure the appropriate environment variables:

Provider	Required Environment Variables
OpenAI	`OPENAI_API_KEY`
Azure OpenAI	`AZURE_OPENAI_ENDPOINT`, `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_DEPLOYMENT_NAME`
Ollama	None (runs locally on `http://localhost:11434`)

class LLMProvider(Enum):
    OPENAI = "openai"
    AZURE_OPENAI = "azure_openai"
    OLLAMA = "ollama"


# === CONFIGURE YOUR PROVIDER HERE ===
PROVIDER = LLMProvider.AZURE_OPENAI

# Model names per provider
MODEL_CONFIG = {
    LLMProvider.OPENAI: {
        "chat_model": "gpt-4o-mini",
        "embedding_model": "text-embedding-3-small",
    },
    LLMProvider.AZURE_OPENAI: {
        "chat_model": os.getenv("AZURE_OPENAI_DEPLOYMENT_NAME", "gpt-4o-mini"),
        "embedding_model": os.getenv("AZURE_OPENAI_EMBEDDING_DEPLOYMENT", "text-embedding-3-small"),
    },
    LLMProvider.OLLAMA: {
        "chat_model": "llama3.2:3b",
        "embedding_model": "nomic-embed-text",
    },
}

print(f"Using provider: {PROVIDER.value}")
print(f"Chat model: {MODEL_CONFIG[PROVIDER]['chat_model']}")
print(f"Embedding model: {MODEL_CONFIG[PROVIDER]['embedding_model']}")

Using provider: azure_openai
Chat model: gpt-4o-mini
Embedding model: text-embedding-ada-002-v2

def validate_environment(provider: LLMProvider) -> None:
    """Validate required environment variables for the selected provider."""
    required_vars = {
        LLMProvider.OPENAI: ["OPENAI_API_KEY"],
        LLMProvider.AZURE_OPENAI: [
            "AZURE_OPENAI_ENDPOINT",
            "AZURE_OPENAI_API_KEY",
        ],
        LLMProvider.OLLAMA: [],
    }

    missing = [var for var in required_vars[provider] if not os.getenv(var)]
    if missing:
        raise EnvironmentError(
            f"Missing environment variables for {provider.value}: {missing}\n"
            f"Please set them before continuing."
        )
    
    # Set provider-specific env vars for litellm (used by RAGAS scorers)
    if provider == LLMProvider.OLLAMA:
        os.environ.setdefault("OLLAMA_API_BASE", "http://localhost:11434")
        print(f"OLLAMA_API_BASE set to: {os.environ['OLLAMA_API_BASE']}")
    elif provider == LLMProvider.AZURE_OPENAI:
        # Set litellm Azure env vars from Azure OpenAI vars
        if os.getenv("AZURE_OPENAI_API_KEY") and not os.getenv("AZURE_API_KEY"):
            os.environ["AZURE_API_KEY"] = os.environ["AZURE_OPENAI_API_KEY"]
        if os.getenv("AZURE_OPENAI_ENDPOINT") and not os.getenv("AZURE_API_BASE"):
            os.environ["AZURE_API_BASE"] = os.environ["AZURE_OPENAI_ENDPOINT"]
        os.environ.setdefault("AZURE_API_VERSION", os.getenv("AZURE_OPENAI_API_VERSION", "2024-02-01"))
        print(f"Azure litellm env vars configured")
    
    print(f"Environment validated for {provider.value}")


validate_environment(PROVIDER)

Azure litellm env vars configured
Environment validated for azure_openai

from langchain_openai import ChatOpenAI, OpenAIEmbeddings, AzureChatOpenAI, AzureOpenAIEmbeddings
from langchain_community.embeddings import OllamaEmbeddings
from langchain_community.chat_models import ChatOllama


def get_llm(provider: LLMProvider):
    """Factory function to create LLM instance based on provider."""
    config = MODEL_CONFIG[provider]

    if provider == LLMProvider.OPENAI:
        return ChatOpenAI(
            model=config["chat_model"],
            temperature=0,
        )
    elif provider == LLMProvider.AZURE_OPENAI:
        return AzureChatOpenAI(
            azure_deployment=config["chat_model"],
            api_version=os.getenv("AZURE_OPENAI_API_VERSION", "2024-02-01"),
            temperature=0,
        )
    elif provider == LLMProvider.OLLAMA:
        return ChatOllama(
            model=config["chat_model"],
            temperature=0,
            base_url="http://localhost:11434",
        )


def get_embeddings(provider: LLMProvider):
    """Factory function to create embeddings instance based on provider."""
    config = MODEL_CONFIG[provider]

    if provider == LLMProvider.OPENAI:
        return OpenAIEmbeddings(model=config["embedding_model"])
    elif provider == LLMProvider.AZURE_OPENAI:
        return AzureOpenAIEmbeddings(
            azure_deployment=config["embedding_model"],
            api_version=os.getenv("AZURE_OPENAI_API_VERSION", "2024-02-01"),
        )
    elif provider == LLMProvider.OLLAMA:
        return OllamaEmbeddings(
            model=config["embedding_model"],
            base_url="http://localhost:11434",
        )


def get_mlflow_model_uri(provider: LLMProvider) -> str:
    """Get MLflow model URI for RAGAS scorers (uses litellm format)."""
    config = MODEL_CONFIG[provider]

    if provider == LLMProvider.OPENAI:
        return f"openai:/{config['chat_model']}"
    elif provider == LLMProvider.AZURE_OPENAI:
        # Azure format: azure/<deployment_name>
        return f"azure:/{config['chat_model']}"
    elif provider == LLMProvider.OLLAMA:
        # Ollama format for litellm: ollama/<model_name>
        # Note: ollama_chat format has issues with litellm, use ollama/ prefix
        return f"ollama:/{config['chat_model']}"


llm = get_llm(PROVIDER)
embeddings = get_embeddings(PROVIDER)
mlflow_model_uri = get_mlflow_model_uri(PROVIDER)

print(f"LLM initialized: {type(llm).__name__}")
print(f"Embeddings initialized: {type(embeddings).__name__}")
print(f"MLflow model URI: {mlflow_model_uri}")

LLM initialized: AzureChatOpenAI
Embeddings initialized: AzureOpenAIEmbeddings
MLflow model URI: azure:/gpt-4o-mini

Sample Knowledge Base

We'll create a small knowledge base about MLflow - fitting for a tutorial that uses MLflow for evaluation! This dataset contains key concepts that our RAG system will retrieve from.

import json

# Load knowledge base from external file
with open("data/knowledge_base.json") as f:
    KNOWLEDGE_BASE = json.load(f)

print(f"Knowledge base contains {len(KNOWLEDGE_BASE)} documents")
for i, doc in enumerate(KNOWLEDGE_BASE, 1):
    preview = doc[:80].replace('\n', ' ')
    print(f"  {i}. {preview}...")

Knowledge base contains 20 documents
  1. MLflow Tracking is an API and UI for logging parameters, code versions, metrics,...
  2. The MLflow Model Registry is a centralized model store that provides model linea...
  3. MLflow GenAI provides specialized tools for developing and evaluating generative...
  4. RAGAS (Retrieval Augmented Generation Assessment) is an evaluation framework int...
  5. MLflow Projects package code in a reusable, reproducible form. A project is simp...
  6. MLflow's autolog feature automatically logs metrics, parameters, and models duri...
  7. The MLflow Model format is a standard format for packaging machine learning mode...
  8. Evaluation in MLflow can be performed using mlflow.evaluate() for traditional ML...
  9. MLflow Model Serving enables deploying models as REST API endpoints. You can ser...
  10. MLflow Recipes (formerly MLflow Pipelines) provide predefined templates for comm...
  11. The MLflow CLI provides commands for running projects, serving models, and manag...
  12. MLflow's REST API allows programmatic access to the tracking server. Endpoints i...
  13. MLflow experiments organize runs into logical groups. Each experiment has a uniq...
  14. MLflow provides run comparison capabilities through the UI and API. The Compare ...
  15. MLflow artifacts are files associated with runs, such as models, data files, and...
  16. Model signatures in MLflow define the expected input and output schema for model...
  17. MLflow on Databricks provides managed MLflow tracking, model registry, and model...
  18. MLflow supports multiple environment managers for reproducibility. Projects can ...
  19. MLflow Prompt Engineering tools help develop and version prompts for LLM applica...
  20. MLflow integrates with LangChain through mlflow.langchain module. The integratio...

from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import FAISS
from langchain_core.documents import Document

documents = [Document(page_content=doc.strip()) for doc in KNOWLEDGE_BASE]

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
)
splits = text_splitter.split_documents(documents)

print(f"Split into {len(splits)} chunks")

Split into 20 chunks

vectorstore = FAISS.from_documents(splits, embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

print(f"Vector store created with {vectorstore.index.ntotal} vectors")

Vector store created with 20 vectors

test_query = "What metrics does RAGAS provide?"
test_results = retriever.invoke(test_query)

print(f"Test query: '{test_query}'")
print(f"Retrieved {len(test_results)} documents:")
for i, doc in enumerate(test_results, 1):
    print(f"\n--- Document {i} ---")
    print(doc.page_content[:200] + "...")

Test query: 'What metrics does RAGAS provide?'
Retrieved 3 documents:

--- Document 1 ---
RAGAS (Retrieval Augmented Generation Assessment) is an evaluation framework integrated with MLflow for assessing RAG pipelines. Key metrics include: Faithfulness (measures if the answer is grounded i...

--- Document 2 ---
MLflow GenAI provides specialized tools for developing and evaluating generative AI applications. It includes mlflow.genai.evaluate() for systematic assessment of LLM outputs using configurable scorer...

--- Document 3 ---
Evaluation in MLflow can be performed using mlflow.evaluate() for traditional ML models or mlflow.genai.evaluate() for generative AI applications. For GenAI, evaluation uses Scorer objects that can be...

Minimal RAG Pipeline

We'll build a simple RAG chain using LangChain's LCEL (LangChain Expression Language) that:

Retrieves relevant context from our FAISS vector store
Formats a prompt with the context and question
Generates an answer using the LLM

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

RAG_PROMPT = ChatPromptTemplate.from_template("""
You are a helpful assistant answering questions about MLflow.
Use ONLY the following context to answer the question.
If the context doesn't contain the answer, say "I don't have enough information to answer this question."

Context:
{context}

Question: {question}

Answer:
""")


def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)


rag_chain = (
    {
        "context": retriever | format_docs,
        "question": RunnablePassthrough(),
    }
    | RAG_PROMPT
    | llm
    | StrOutputParser()
)

print("RAG chain created successfully")

RAG chain created successfully

test_answer = rag_chain.invoke("What is MLflow Tracking?")
print("Test Question: What is MLflow Tracking?")
print(f"\nAnswer: {test_answer}")

Test Question: What is MLflow Tracking?

Answer: MLflow Tracking is an API and UI for logging parameters, code versions, metrics, and artifacts when running your machine learning code. It allows you to log and query experiments using Python, REST, R API, and Java API. The MLflow Tracking component lets you log source code, models, and visualizations. Each run records: code version, start and end time, source, parameters, metrics, and artifacts.

Enable MLflow Tracing

MLflow's LangChain integration can automatically capture traces of our RAG pipeline invocations. This is essential for evaluation - RAGAS scorers analyze these traces to compute metrics.

import mlflow

mlflow.set_experiment("RAG-Evaluation-Tutorial")

mlflow.langchain.autolog(log_traces=True)

print(f"MLflow experiment: {mlflow.get_experiment_by_name('RAG-Evaluation-Tutorial').name}")
print("LangChain autologging enabled with tracing")

2026/01/11 09:22:25 INFO mlflow.store.db.utils: Creating initial MLflow database tables...
2026/01/11 09:22:25 INFO mlflow.store.db.utils: Updating database tables
2026/01/11 09:22:25 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2026/01/11 09:22:25 INFO alembic.runtime.migration: Will assume non-transactional DDL.
2026/01/11 09:22:25 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2026/01/11 09:22:25 INFO alembic.runtime.migration: Will assume non-transactional DDL.

MLflow experiment: RAG-Evaluation-Tutorial
LangChain autologging enabled with tracing

NOTE: If you don't have tracing enabled in your RAG there is still possibility to pass the evaluation data to MLflow to ease analysis. If this is your case please refer to the text on my blog explaining how to do it: RAG Evaluation with RAGAS and MLflow - without tracing

Load Golden Dataset

A golden dataset (also called ground truth or evaluation dataset) contains:

Questions: User queries we want to evaluate
Expected Answers: The correct/ideal responses
Expected Contexts (optional): Which documents should be retrieved

This dataset allows us to systematically measure our RAG system's quality.

# Load golden dataset from external file
with open("data/golden_dataset.json") as f:
    GOLDEN_DATASET = json.load(f)

eval_df = pd.DataFrame(GOLDEN_DATASET)
print(f"Golden dataset contains {len(GOLDEN_DATASET)} evaluation samples")
eval_df.head(2)

Golden dataset contains 20 evaluation samples

	question	ground_truth	contexts
0	What is MLflow Tracking used for?	MLflow Tracking is used for logging parameters...	[MLflow Tracking is an API and UI for logging ...
1	What features does the MLflow Model Registry p...	The MLflow Model Registry provides model linea...	[The MLflow Model Registry is a centralized mo...

Generate RAG Responses for Evaluation

We'll run our RAG pipeline on each question and collect the responses along with the retrieved contexts. This data will be used by RAGAS scorers.

# Define traced RAG function for evaluation
# IMPORTANT: Function parameter names must match keys in data['inputs']
# Since inputs={'question': ...}, the function must accept 'question' parameter

@mlflow.trace(span_type="CHAIN")
def traced_rag_predict(question: str) -> dict:
    """Traced RAG prediction function for mlflow.genai.evaluate().
    
    Args:
        question: The question to answer (matches inputs['question'] key)
    
    Returns:
        dict with 'response' and 'retrieved_contexts' for RAGAS scorers
    """
    # Retrieval step - creates RETRIEVER span
    with mlflow.start_span(name="retriever", span_type="RETRIEVER") as span:
        retrieved_docs = retriever.invoke(question)
        contexts = [doc.page_content for doc in retrieved_docs]
        span.set_inputs({"question": question})
        span.set_outputs({"retrieved_contexts": contexts})
    
    # Generation step - creates LLM span
    with mlflow.start_span(name="generator", span_type="LLM") as span:
        answer = rag_chain.invoke(question)
        span.set_inputs({"question": question, "contexts": contexts})
        span.set_outputs({"response": answer})
    
    return {
        "response": answer,
        "retrieved_contexts": contexts,
    }

# Preview a sample from the golden dataset
# Note: With predict_fn approach, answers are generated during evaluation
sample_idx = 2
print(f"Sample evaluation record #{sample_idx + 1}:")
print(f"\nQuestion: {eval_df.iloc[sample_idx]['question']}")
print(f"\nExpected Answer: {eval_df.iloc[sample_idx]['ground_truth']}")

# Show what the traced function would produce for this question
print(f"\n--- Testing RAG response for this question ---")
test_output = traced_rag_predict(question=eval_df.iloc[sample_idx]['question'])
print(f"\nRAG Answer: {test_output['response']}")
print(f"\nRetrieved Contexts ({len(test_output['retrieved_contexts'])}):\n")
for j, ctx in enumerate(test_output['retrieved_contexts'], 1):
    print(f"  Context {j}: {ctx[:100]}...\n")

Sample evaluation record #3:

Question: What metrics does RAGAS provide for RAG evaluation?

Expected Answer: RAGAS provides four key metrics: Faithfulness (measures if the answer is grounded in context), Context Precision (evaluates if relevant documents are ranked higher), Context Recall (checks if context contains all needed information), and Factual Correctness (compares output against expected answers).

--- Testing RAG response for this question ---

RAG Answer: RAGAS provides the following key metrics for RAG evaluation: Faithfulness, Context Precision, Context Recall, and Factual Correctness.

Retrieved Contexts (3):

  Context 1: RAGAS (Retrieval Augmented Generation Assessment) is an evaluation framework integrated with MLflow ...

  Context 2: Evaluation in MLflow can be performed using mlflow.evaluate() for traditional ML models or mlflow.ge...

  Context 3: MLflow GenAI provides specialized tools for developing and evaluating generative AI applications. It...

RAGAS Evaluation with MLflow

Now we'll use MLflow's RAGAS integration to evaluate our RAG pipeline. The key metrics we'll compute:

Metric	What it measures	Required Data	Common Failure
Faithfulness	Is the answer grounded in retrieved context?	answer, contexts	Missing RETRIEVER spans
Context Precision	Are relevant docs ranked higher?	question, contexts, ground_truth	No ground_truth provided
Context Recall	Does context contain needed info?	contexts, ground_truth	No ground_truth provided
Factual Correctness	Does answer match expected?	answer, ground_truth	Semantic mismatch (strict)

These metrics provide an initial quantitative assessment of RAG quality across multiple dimensions. There are more RAGAS tool metrics available through the MLflow integration.

Note on LLM Judge: RAGAS metrics use an LLM as a judge. For best results, use OpenAI (gpt-4o-mini) as the judge model even if you're using Ollama for RAG generation. Ollama/local models may have issues with litellm's structured output parsing. Set JUDGE_PROVIDER = LLMProvider.OPENAI below if you encounter scoring errors with Ollama.

# Test litellm connectivity (optional - helps debug scoring issues)
import litellm

def test_litellm_connection(model_uri: str) -> bool:
    """Test if litellm can connect to the model."""
    try:
        response = litellm.completion(
            model=model_uri,
            messages=[{"role": "user", "content": "Say 'test' and nothing else."}],
            max_tokens=10,
        )
        print(f"✓ litellm connection successful: {model_uri}")
        print(f"  Response: {response.choices[0].message.content[:50]}...")
        return True
    except Exception as e:
        print(f"✗ litellm connection failed: {model_uri}")
        print(f"  Error: {type(e).__name__}: {str(e)[:100]}")
        return False

# Test the judge model connection
judge_model_uri = get_mlflow_model_uri(PROVIDER)
print(f"Testing judge model: {judge_model_uri}\n")
litellm_ok = test_litellm_connection(judge_model_uri)

if not litellm_ok:
    print("\n⚠️  Consider using OpenAI as judge model for reliable scoring.")
    print("   Set JUDGE_PROVIDER = LLMProvider.OPENAI in the next cell.")

Testing judge model: azure:/gpt-4o-mini


Provider List: https://docs.litellm.ai/docs/providers


Provider List: https://docs.litellm.ai/docs/providers

✗ litellm connection failed: azure:/gpt-4o-mini
  Error: BadRequestError: litellm.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call.

⚠️  Consider using OpenAI as judge model for reliable scoring.
   Set JUDGE_PROVIDER = LLMProvider.OPENAI in the next cell.

from mlflow.genai.scorers.ragas import (
    Faithfulness,
    ContextPrecision,
    ContextRecall,
    FactualCorrectness,
)

# Configure the judge model for RAGAS evaluation
# For reliable scoring, use OpenAI even when using Ollama for RAG generation
JUDGE_PROVIDER = PROVIDER  # Change to LLMProvider.OPENAI for better results
judge_model_uri = get_mlflow_model_uri(JUDGE_PROVIDER)

print(f"Judge model: {judge_model_uri}")
print(f"(Change JUDGE_PROVIDER to LLMProvider.OPENAI if scoring fails with Ollama)\n")

# Note: ContextPrecision and ContextRecall require traces with RETRIEVER spans
# For evaluation without traces, use Faithfulness and FactualCorrectness
scorers = [
    Faithfulness(model=judge_model_uri),
    FactualCorrectness(model=judge_model_uri),
    # These require traces with retriever spans - may show errors without proper tracing:
    ContextPrecision(model=judge_model_uri),
    ContextRecall(model=judge_model_uri),
]

print(f"Configured {len(scorers)} RAGAS scorers:")
for scorer in scorers:
    print(f"  - {type(scorer).__name__}")

Judge model: azure:/gpt-4o-mini
(Change JUDGE_PROVIDER to LLMProvider.OPENAI if scoring fails with Ollama)

Configured 4 RAGAS scorers:
  - Faithfulness
  - FactualCorrectness
  - ContextPrecision
  - ContextRecall

Prepare Evaluation Data

MLflow's genai.evaluate() expects data in a specific format. We need to map our data to the expected schema.

# Prepare evaluation data for predict_fn approach
# With predict_fn, we pass inputs and expectations - outputs come from the traced function
eval_data = []
for _, row in eval_df.iterrows():
    eval_data.append({
        "inputs": {"question": row["question"]},
        "expectations": {
            "ground_truth": row["ground_truth"],
            "contexts": row.get("contexts", []),  # For ContextRecall
        },
    })

print(f"Prepared {len(eval_data)} samples for evaluation")
print(f"\nSample format:")
print(f"  inputs: {list(eval_data[0]['inputs'].keys())}")
print(f"  expectations: {list(eval_data[0]['expectations'].keys())}")
if eval_data[0]['expectations'].get('ground_truth'):
    print(f"  ground_truth contexts: {len(eval_data[0]['expectations']['ground_truth'])} items")
print(f"\nNote: outputs will be generated by traced_rag_predict() during evaluation")
print(f"      ground_truth enables ContextPrecision and ContextRecall metrics")

Prepared 20 samples for evaluation

Sample format:
  inputs: ['question']
  expectations: ['ground_truth', 'contexts']
  ground_truth contexts: 205 items

Note: outputs will be generated by traced_rag_predict() during evaluation
      ground_truth enables ContextPrecision and ContextRecall metrics

print("Running RAGAS evaluation with traced predict_fn...")
print("This generates traces with RETRIEVER spans for Faithfulness metric.\n")

with mlflow.start_run(run_name="ragas-evaluation-traced") as run:
    mlflow.log_param("provider", PROVIDER.value)
    mlflow.log_param("model", MODEL_CONFIG[PROVIDER]["chat_model"])
    mlflow.log_param("num_samples", len(eval_data))
    mlflow.log_param("retriever_k", 3)
    mlflow.log_param("evaluation_mode", "predict_fn")

    # Use predict_fn to generate traces with RETRIEVER spans
    # This allows Faithfulness scorer to access retrieved_contexts
    eval_results = mlflow.genai.evaluate(
        predict_fn=traced_rag_predict,
        data=eval_data,
        scorers=scorers,
    )

    run_id = run.info.run_id

print(f"\nEvaluation complete! Run ID: {run_id}")

2026/01/11 09:22:33 INFO mlflow.models.evaluation.utils.trace: Auto tracing is temporarily enabled during the model evaluation for computing some metrics and debugging. To disable tracing, call `mlflow.autolog(disable=True)`.
2026/01/11 09:22:33 INFO mlflow.genai.utils.data_validation: Testing model prediction with the first sample in the dataset. To disable this check, set the MLFLOW_GENAI_EVAL_SKIP_TRACE_VALIDATION environment variable to True.
2026/01/11 09:22:33 WARNING mlflow.tracing.fluent: Failed to start span VectorStoreRetriever: 'NonRecordingSpan' object has no attribute 'context'. For full traceback, set logging level to debug.

Running RAGAS evaluation with traced predict_fn...
This generates traces with RETRIEVER spans for Faithfulness metric.

2026/01/11 09:22:34 WARNING mlflow.tracing.fluent: Failed to start span RunnableSequence: 'NonRecordingSpan' object has no attribute 'context'. For full traceback, set logging level to debug.

Evaluating:   0%|          | 0/20 [Elapsed: 00:00, Remaining: ?]

✨ Evaluation completed.

Metrics and evaluation results are logged to the MLflow run:
  Run name: ragas-evaluation-traced
  Run ID: 35029b87d0e542128dedd53531ba0710

To view the detailed evaluation results with sample-wise scores,
open the Traces tab in the Run page in the MLflow UI.


Evaluation complete! Run ID: 35029b87d0e542128dedd53531ba0710

MLflow Results Analysis

Let's examine the evaluation results both programmatically and understand how to view them in the MLflow UI.

print("=" * 60)
print("RAGAS EVALUATION RESULTS")
print("=" * 60)

results_df = eval_results.tables["eval_results"]

# Find RAGAS scorer columns (Faithfulness, FactualCorrectness, Context*)
import pandas as pd
ragas_metrics = ['Faithfulness', 'FactualCorrectness', 'ContextPrecision', 'ContextRecall']
value_columns = [col for col in results_df.columns 
                 if col.endswith('/value') and any(m in col for m in ragas_metrics)]
error_columns = [col for col in results_df.columns 
                 if col.endswith('/error') and any(m in col for m in ragas_metrics)]

print("\nRAGAS Metrics:")
print("-" * 40)
successful_metrics = 0
failed_metrics = 0

for col in value_columns:
    # Convert to numeric, coercing errors to NaN
    numeric_col = pd.to_numeric(results_df[col], errors='coerce')
    non_null = numeric_col.dropna()
    total = len(results_df)
    success_count = len(non_null)

    if success_count > 0:
        mean_val = non_null.mean()
        std_val = non_null.std() if len(non_null) > 1 else 0
        print(f"  ✓ {col}: {mean_val:.3f} (±{std_val:.3f}) [{success_count}/{total} samples]")
        successful_metrics += 1
    else:
        print(f"  ✗ {col}: NO SCORES (0/{total} samples succeeded)")
        failed_metrics += 1

print(f"\nSummary: {successful_metrics} metrics succeeded, {failed_metrics} metrics failed")
print(f"   Total samples: {len(results_df)}")

# Error diagnostics (if any metrics failed)
if failed_metrics > 0:
    print("\n" + "=" * 60)
    print("🔍 DIAGNOSTIC: Error Details for Failed Metrics")
    print("=" * 60)
    
    for col in error_columns:
        metric_name = col.replace('/error', '')
        errors = results_df[col].dropna()
        
        if len(errors) > 0:
            print(f"\n❌ {metric_name}:")
            # Get first unique error message
            unique_errors = errors.unique()
            for err in unique_errors[:2]:  # Show max 2 unique errors
                # Truncate long error messages
                err_str = str(err)[:300]
                if len(str(err)) > 300:
                    err_str += "..."
                print(f"   {err_str}")
    
    print("\n" + "-" * 60)
    print("Common fixes:")
    print("   1. Use OpenAI as judge: JUDGE_PROVIDER = LLMProvider.OPENAI")
    print("   2. For Ollama: ensure model is running and OLLAMA_API_BASE is set")
    print("   3. ContextPrecision/ContextRecall require traces with RETRIEVER spans")
else:
    print("\n✅ All metrics computed successfully!")

============================================================
RAGAS EVALUATION RESULTS
============================================================

RAGAS Metrics:
----------------------------------------
  ✓ ContextRecall/value: 0.853 (±0.196) [20/20 samples]
  ✓ ContextPrecision/value: 0.967 (±0.116) [20/20 samples]
  ✓ Faithfulness/value: 0.984 (±0.047) [20/20 samples]
  ✓ FactualCorrectness/value: 0.613 (±0.250) [20/20 samples]

Summary: 4 metrics succeeded, 0 metrics failed
   Total samples: 20

✅ All metrics computed successfully!

# Helper function to extract question from request column
def extract_question(request_data):
    """Extract question from MLflow request column."""
    if isinstance(request_data, dict):
        return str(request_data.get("question", "N/A"))[:60]
    elif isinstance(request_data, str):
        return request_data[:60]
    return "N/A"

# Display results summary with metric columns
available_cols = [col for col in value_columns if col in results_df.columns]
results_summary = results_df[available_cols].copy()

# Add question column from request data
if "request" in results_df.columns:
    results_summary.insert(0, "question", results_df["request"].apply(extract_question))

results_summary


print("\nIdentifying Low-Scoring Samples:")
print("-" * 40)

for col in value_columns:
    if col in results_df.columns:
        numeric_col = pd.to_numeric(results_df[col], errors='coerce')
        low_mask = numeric_col < 0.5
        low_scores = results_df[low_mask]
        if len(low_scores) > 0:
            print(f"\n⚠️  {col} < 0.5: {len(low_scores)} samples")
            for idx, row in low_scores.iterrows():
                question = extract_question(row.get("request", {}))
                score = numeric_col.loc[idx]
                if pd.notna(score):
                    print(f"    - [{score:.2f}] {question}...")

Identifying Low-Scoring Samples:
----------------------------------------

⚠️  ContextRecall/value < 0.5: 1 samples
    - [0.33] What is MLflow GenAI used for?...

⚠️  ContextPrecision/value < 0.5: 1 samples
    - [0.50] How can you run MLflow Projects?...

⚠️  FactualCorrectness/value < 0.5: 6 samples
    - [0.46] What is MLflow Tracking used for?...
    - [0.30] What is MLflow GenAI used for?...
    - [0.47] How can you run MLflow Projects?...
    - [0.12] What is Faithfulness in RAGAS?...
    - [0.31] What frameworks support MLflow autolog?...
    - [0.40] How can you access MLflow programmatically via REST API?...

Interpreting RAGAS Scores

All RAGAS metrics return scores between 0.0 and 1.0. Here's rough guidance:

Score Range	Interpretation
0.9 - 1.0	Excellent - production ready
0.7 - 0.9	Good - minor improvements needed
0.5 - 0.7	Fair - significant room for improvement
< 0.5	Poor - investigate specific failures

Important caveats:

These thresholds are guidelines, not absolutes
Different applications have different quality requirements
Low FactualCorrectness often reflects semantic similarity issues, not actual incorrectness
Focus on relative improvements when comparing variants, not absolute scores

To view detailed results in the MLflow UI:

Start MLflow UI (if not running): $ mlflow ui --port 5000
Open http://localhost:5000 in your browser
Navigate to:
- Experiment: 'RAG-Evaluation-Tutorial'
- Run: 'ragas-evaluation'
In the run details, you'll find:
- Parameters: model configuration
- Metrics: aggregate RAGAS scores
- Artifacts: detailed evaluation tables
- Traces: individual RAG invocations

print(f"Run ID: {run_id}")

Run ID: 35029b87d0e542128dedd53531ba0710

print("\n" + "=" * 60)
print("🎉 Tutorial Complete!")
print("=" * 60)
print(f"""
Summary:
  - Provider: {PROVIDER.value}
  - Model: {MODEL_CONFIG[PROVIDER]['chat_model']}
  - Samples evaluated: {len(eval_data)}
  - MLflow Run ID: {run_id}

View results: mlflow ui --port 5000
""")

============================================================
🎉 Tutorial Complete!
============================================================

Summary:
  - Provider: azure_openai
  - Model: gpt-4o-mini
  - Samples evaluated: 20
  - MLflow Run ID: 35029b87d0e542128dedd53531ba0710

View results: mlflow ui --port 5000

Common Pitfalls and Solutions

During development of this tutorial, several non-obvious issues emerged:

Model URI Format

MLflow uses litellm under the hood. The URI format matters:

OpenAI: openai:/gpt-4o-mini (note the colon-slash)
Azure: azure:/deployment-name
Ollama: ollama:/llama3.2:3b

Using azure/ instead of azure:/ will fail silently or produce cryptic errors.

Function Signature Must Match Input Keys

When using predict_fn, the function parameter names must exactly match the keys in your inputs dictionary:

# If your data has: {"inputs": {"question": "..."}}
# Your function MUST be: def predict(question: str)  # NOT def predict(query: str)

RETRIEVER Spans for Context Metrics

Faithfulness, ContextPrecision, and ContextRecall require traces with RETRIEVER-type spans. Without them, these metrics return errors or incorrect values. The traced_rag_predict function in this tutorial creates these spans explicitly.

Judge Model Limitations

RAGAS metrics use an LLM as a judge. Local models (Ollama) may struggle with the structured output parsing that RAGAS requires. For reliable scoring, consider using OpenAI/Azure as the judge even when your RAG uses a different provider.

Extras: Comparing RAG Variants with MLflow

One of MLflow's key strengths is enabling systematic A/B comparisons between different RAG configurations. Here's how to structure experiments comparing variants like chunk sizes, models, or retrieval strategies.

Example: Comparing Chunk Sizes

# Comparing RAG Variants: Different Chunk Sizes
# This demonstrates how to evaluate the same RAG pipeline with different configurations

print("Running chunk size comparison experiments...")
print("=" * 60)

CHUNK_SIZES = [50, 150]
experiment_run_ids = []

for chunk_size in CHUNK_SIZES:
    print(f"\nTesting chunk_size={chunk_size}")
    
    # Rebuild the vector store with new chunk size
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=chunk_size // 10
    )
    chunks = text_splitter.split_documents(documents)
    
    # Update the global vectorstore and retriever used by traced_rag_predict
    global vectorstore, retriever
    vectorstore = FAISS.from_documents(chunks, embeddings)
    retriever = vectorstore.as_retriever(search_kwargs={"k": 3})
    
    print(f"   Created {len(chunks)} chunks")
    
    # Run evaluation
    with mlflow.start_run(run_name=f"chunk-size-{chunk_size}") as run:
        mlflow.log_param("chunk_size", chunk_size)
        mlflow.log_param("chunk_overlap", chunk_size // 10)
        mlflow.log_param("num_chunks", len(chunks))
        
        eval_results = mlflow.genai.evaluate(
            predict_fn=traced_rag_predict,
            data=eval_data,
            scorers=scorers,
        )
        experiment_run_ids.append(run.info.run_id)
        print(f"   ✓ Run ID: {run.info.run_id}")

print("\n" + "=" * 60)
print(f"Completed {len(CHUNK_SIZES)} experiments. Run IDs saved for comparison.")

Running chunk size comparison experiments...
============================================================

Testing chunk_size=50

2026/01/11 09:23:10 INFO mlflow.genai.utils.data_validation: Testing model prediction with the first sample in the dataset. To disable this check, set the MLFLOW_GENAI_EVAL_SKIP_TRACE_VALIDATION environment variable to True.
2026/01/11 09:23:10 WARNING mlflow.tracing.fluent: Failed to start span VectorStoreRetriever: 'NonRecordingSpan' object has no attribute 'context'. For full traceback, set logging level to debug.
2026/01/11 09:23:10 WARNING mlflow.tracing.fluent: Failed to start span RunnableSequence: 'NonRecordingSpan' object has no attribute 'context'. For full traceback, set logging level to debug.

   Created 175 chunks

Evaluating:   0%|          | 0/20 [Elapsed: 00:00, Remaining: ?]

✨ Evaluation completed.

Metrics and evaluation results are logged to the MLflow run:
  Run name: chunk-size-50
  Run ID: 34464d4c3bb34a0a8ce4d43760149f1e

To view the detailed evaluation results with sample-wise scores,
open the Traces tab in the Run page in the MLflow UI.

   ✓ Run ID: 34464d4c3bb34a0a8ce4d43760149f1e

Testing chunk_size=150

2026/01/11 09:23:33 INFO mlflow.genai.utils.data_validation: Testing model prediction with the first sample in the dataset. To disable this check, set the MLFLOW_GENAI_EVAL_SKIP_TRACE_VALIDATION environment variable to True.
2026/01/11 09:23:33 WARNING mlflow.tracing.fluent: Failed to start span VectorStoreRetriever: 'NonRecordingSpan' object has no attribute 'context'. For full traceback, set logging level to debug.
2026/01/11 09:23:33 WARNING mlflow.tracing.fluent: Failed to start span RunnableSequence: 'NonRecordingSpan' object has no attribute 'context'. For full traceback, set logging level to debug.

   Created 62 chunks

Evaluating:   0%|          | 0/20 [Elapsed: 00:00, Remaining: ?]

✨ Evaluation completed.

Metrics and evaluation results are logged to the MLflow run:
  Run name: chunk-size-150
  Run ID: c77546d4d930426b98ebeb1bf1a09a3f

To view the detailed evaluation results with sample-wise scores,
open the Traces tab in the Run page in the MLflow UI.

   ✓ Run ID: c77546d4d930426b98ebeb1bf1a09a3f

============================================================
Completed 2 experiments. Run IDs saved for comparison.

Comparing Results in MLflow UI

After running multiple variants:

Open MLflow UI: mlflow ui --port 5000
Navigate to your experiment
Select runs to compare using checkboxes
Click Compare to see side-by-side metrics
Use Chart view to visualize metric differences

You can also compare programmatically:

# Compare Results Programmatically
# Query MLflow for runs and display a formatted comparison table

experiment_name = "RAG-Evaluation-Tutorial"

# Get runs with chunk_size parameter (our comparison experiments)
runs_df = mlflow.search_runs(
    experiment_names=[experiment_name],
    filter_string="params.chunk_size != ''",
    order_by=["params.chunk_size ASC"]
)

if len(runs_df) == 0:
    print("No chunk size comparison runs found. Run the comparison cell above first.")
else:
    # Debug: show available metric columns
    metric_cols = [c for c in runs_df.columns if c.startswith("metrics.")]
    print(f"Available metric columns ({len(metric_cols)} total):")
    for col in metric_cols[:8]:  # Show first 8
        print(f"  - {col}")
    
    # Define metrics we want (will search for partial matches)
    metric_names = ["Faithfulness", "FactualCorrectness", "ContextPrecision", "ContextRecall"]
    
    # Find actual column names (may have backticks or different format)
    def find_metric_col(df, metric_name):
        """Find column containing metric_name in its name."""
        for col in df.columns:
            if metric_name in col and "mean" in col:
                return col
        return None
    
    comparison_data = []
    for _, run in runs_df.iterrows():
        row = {
            "Run Name": run.get("tags.mlflow.runName", "N/A"),
            "Chunk Size": run.get("params.chunk_size", "N/A"),
            "Num Chunks": run.get("params.num_chunks", "N/A"),
        }
        for metric_name in metric_names:
            col = find_metric_col(runs_df, metric_name)
            if col:
                value = run.get(col)
                row[metric_name] = f"{value:.3f}" if pd.notna(value) else "N/A"
            else:
                row[metric_name] = "N/A"
        comparison_data.append(row)
    
    comparison_df = pd.DataFrame(comparison_data)
    print("\nChunk Size Comparison Results")
    print("=" * 80)
    print(comparison_df.to_string(index=False))
    
    # Find best configuration
    if "FactualCorrectness" in comparison_df.columns:
        best_idx = comparison_df["FactualCorrectness"].apply(
            lambda x: float(x) if x != "N/A" else 0
        ).idxmax()
        print(f"\n✨ Best configuration: {comparison_df.iloc[best_idx]['Run Name']}")

Available metric columns (4 total):
  - metrics.Faithfulness/mean
  - metrics.FactualCorrectness/mean
  - metrics.ContextRecall/mean
  - metrics.ContextPrecision/mean

Chunk Size Comparison Results
================================================================================
      Run Name Chunk Size Num Chunks Faithfulness FactualCorrectness ContextPrecision ContextRecall
chunk-size-150        150         62        0.911              0.768            0.992         0.890
chunk-size-150        150         62        0.915              0.781            0.983         0.834
 chunk-size-50         50        175        0.942              0.743            0.983         0.840
 chunk-size-50         50        175        0.944              0.758            0.983         0.844

✨ Best configuration: chunk-size-150

How to inspect results in MLflow UI:

Select the experiment to inspect

Figure 1: Select the experiment you want to analyze.

Experiment type should be automatically recognized as "GenAI Evaluation" - when opening the experiment for the first time - you need to confirm this. Perhaps there is a way to pass this parameter when creating the experiment via code, but I have not found it yet.

Configure comparison of the runs

Figure 2: Runs overview - high level overview of the results achieved for the various configurations/variant of your RAG under evaluation.

You can edit experiment name and description here. Informative names help when comparing multiple experiments. Description can provide additional context about the experiment's purpose.

NOTE: Perhaps there is a way to pass description when creating the experiment via code, but I have not found it yet.

In this view you can see all the runs (evaluate RAG variants) that belongs to this experiment. Each run corresponds to a different RAG configuration (e.g., different chunk sizes, models, etc.). You can see parameters (e.g., model name, chunk size), aggregated metrics (e.g., mean Faithfulness, mean Context Precision). The displayed columns with parameters, metrics can be customized using the "Columns" button on the top right, so you can focus on the most relevant information.
If you want to do the comparison of two runs, select the runs you want to compare by checking the checkboxes next to each run. Note that, the second run you select will be treated as the "baseline" run in the comparison. The score changes of the first selected run will be calculated against the second selected run.
You can select multiple runs to compare their metrics side-by-side. This is useful for evaluating different RAG configurations.
You can also select columns to display in the comparison table.

View Comparison Results

Figure 3: Comparison of individual question results for selected runs. In this view, you can see detailed comparison of individual question results for the selected runs. The zoom of the single panel is presented in the Figure 4** below. This helps to analyze how each RAG configuration performed on specific queries. You can inspects full RAG output and details like retrieved contexts for each question.

Figure 4: Zoom at the detailed results for the single variant

Figure 5: Comparison of individual question results for selected runs - detailed view showing metrics for each question.

More from MLflow

This tutorial focused on RAG evaluation using RAGAS metrics. MLflow offers many more features for RAG or GenAI model management, including:

built-in metrics MLFlow predefined metrics for GenAI models
Guidelines-based LLM Scorers Guidelines-based LLM Scorers

from MLflow Documentation:

Guidelines is a powerful scorer class designed to let you quickly and easily customize evaluation by defining natural language criteria that are framed as pass/fail conditions. It is ideal for checking compliance with rules, style guides, or information inclusion/exclusion.

Guidelines have the distinct advantage of being easy to explain to business stakeholders ("we are evaluating if the app delivers upon this set of rules") and, as such, can often be directly written by domain experts.

MCP server See the documentation on how to add MLflow MCP server in poular IDEs and Agentic conding tools: MLflow MCP Server

...and many more features. Explore the MLflow GenAI documentation for more details.

References, further reading

the code for this tutorial is available on my GitHub: 2026-01-08-ragas-in-mlfow-rag-eval-demo
MLflow Documentation
Introduction to RAG with MLflow and LangChain - MLflow documentation - exemplary implementation of RAG with LangChain and MLflow (without RAGAS evaluation).
GitHub - rag_evaluation_and_tracking - This project houses a Retrieval Augmented Generation (RAG) LLM application built for robust and context-aware text generation. It leverages the combined power of LangChain for orchestration, MLflow for tracking and experimentation, DVC for version control, and RAGAS for evaluation.

Working Faster with Git Worktrees and AI-Based Multi-Workflow Development

2025-11-28T00:00:00+01:00

A practical step-by-step tutorial using Git worktrees, small and large refactorings, and GitHub Copilot.

Prerequisites

Before starting, ensure you have:

Git 2.5+ (check with git --version)
Python 3.8+ and pip
VS Code (optional but recommended)
pytest (pip install pytest)
GitHub Copilot or another AI coding assistant
Basic familiarity with Git branching and command-line operations

Introduction

There is a moment when a project grows from a single-track workflow into something more complex. The codebase stays small, but you want to try a few big ideas at once. Or you want GitHub Copilot (or any AI coding agent) to attempt two different solutions while you continue working normally. One branch is not enough. Stashes are too messy. Full clones eat time and attention.

Git worktrees solve this elegantly. They let one repository produce several working directories, each on its own branch, all sharing the same underlying store. One directory becomes the human-edited version. Another becomes the AI sandbox. A third holds an alternative AI proposal. You compare the results side by side without the mental reset of switching branches.

This tutorial shows you how to build a multi-workflow setup using a simple Python mini-project. You will run two AI-assisted refactoring tasks: one small and one heavy. You will then evaluate them, compare results, validate correctness, and merge only the parts worth keeping.

Starting Point: A Tiny Codebase

Initial Project Setup

First, create a fresh repository:

mkdir project
cd project
git init
git checkout -b develop

Create a file called stats.py with some simple statistics functions:

# stats.py

def average(values):
    return sum(values) / len(values)

def median(values):
    sorted_values = sorted(values)
    mid = len(sorted_values) // 2
    if len(sorted_values) % 2 == 0:
        return (sorted_values[mid - 1] + sorted_values[mid]) / 2
    return sorted_values[mid]

def variance(values):
    # Wrong: variance uses mean, not sum of squared values directly
    return sum(v * v for v in values) / len(values)

Commit this baseline:

git add stats.py
git commit -m "Initial commit with stats module"

There is an obvious bug in variance. The right definition uses squared deviations from the mean.

# Correct version of variance (population variance)
def variance(values):
    m = average(values)
    return sum((v - m) ** 2 for v in values) / len(values)

Note: This calculates population variance (dividing by N). Sample variance would use N-1.

This is our baseline. A candidate for small refactoring is the variance fix. A candidate for heavy refactoring is reorganising everything into a class-based structure or splitting the module into multiple files.

Step 1: Create a Clean Main Worktree

You start in the main repository folder, checked out on develop. This is your everyday environment.

cd project
git status

Everything looks normal.

Step 2: Prepare Worktrees for AI Workflows

Now create worktrees with new branches for experimentation. The git worktree add command creates both the branch and the working directory in one step:

git worktree add -b ai/refactor-small ../refactor-small
git worktree add -b ai/refactor-heavy ../refactor-heavy

Your directory structure now looks like:

parent-directory/
├── project/           # main worktree (develop branch)
│   └── stats.py
├── refactor-small/    # worktree (ai/refactor-small branch)
│   └── stats.py
└── refactor-heavy/    # worktree (ai/refactor-heavy branch)
    └── stats.py

You now have three directories:

project/ for your normal coding
refactor-small/ for the AI to attempt precise, localised edits
refactor-heavy/ for a broad exploration of reorganising the project

You can verify existing workflows with:

git worktree list

Each is a full working directory with its own branch and commit history, but the .git data is shared. Switching happens simply by changing directories, not by resetting state.

Step 3: Run the First Workflow – A Small AI Refactor

Go into the refactor-small directory and open your IDE.

cd ../refactor-small
code .

Open stats.py and use GitHub Copilot Chat to fix the variance implementation. Exemplary prompt:

Clean up this file. Fix the variance bug, add type hints, and make error handling more robust. Keep changes minimal.

Review the AI's suggestions carefully. Accept changes that look correct. The result should be something like:

# stats.py after small refactor

from typing import Iterable

def average(values: Iterable[float]) -> float:
    values = list(values)
    if not values:
        raise ValueError("Cannot compute average of empty list")
    return sum(values) / len(values)

def median(values: Iterable[float]) -> float:
    values = sorted(values)
    if not values:
        raise ValueError("Cannot compute median of empty list")
    mid = len(values) // 2
    if len(values) % 2 == 0:
        return (values[mid - 1] + values[mid]) / 2
    return values[mid]

def variance(values: Iterable[float]) -> float:
    values = list(values)
    if not values:
        raise ValueError("Cannot compute variance of empty list")
    m = average(values)
    return sum((v - m) ** 2 for v in values) / len(values)

Commit the result.

git add stats.py
git commit -m "Small refactor: fix variance, add type hints, add basic validation"

Nothing touches the project/ folder. No stashes. No switching.

Step 4: Run the Heavy Workflow – A Large-Scale AI Refactor

Open the second worktree in a separate VS Code window.

cd ../refactor-heavy
code .

This time, ask your AI assistant:

Rewrite this module using an object-oriented structure. Split statistics operations into separate classes if needed. Add validation. Keep the public API clear and ergonomic. Keep the code in the same file do not split into multiple files.

Expect a much bigger rewrite. Verify changes and commit again:

git add stats.py
git commit -m "Major rewrite: object-oriented Stats class with reorganized structure"

Now you have two valid, complete refactor attempts captured in versioned form, side by side, without any switching friction.

Step 5: Validate and Compare the Results

Here is where multi-workflows shine. You can run tests, linters, or performance checks in each directory independently and in parallel.

Back in the main project directory, create a test script:

# test_stats.py (in project/ and refactor-small/)

from stats import average, median, variance

def test_all():
    values = [1, 2, 3, 4, 5]
    assert average(values) == 3
    assert median(values) == 3
    assert round(variance(values), 5) == 2.0

if __name__ == "__main__":
    test_all()
    print("✓ All tests passed")

Copy this file to both worktrees. For the heavy version, create a different test:

# test_stats.py (in refactor-heavy/)

from stats import Stats

def test_stats_class():
    s = Stats([1, 2, 3, 4, 5])
    assert s.average() == 3
    assert s.median() == 3
    assert round(s.variance(), 5) == 2.0

if __name__ == "__main__":
    test_stats_class()
    print("All tests passed")

Run tests separately in each worktree:

cd project
python test_stats.py
# Output: All tests passed

cd ../refactor-small
python test_stats.py
# Output: All tests passed

cd ../refactor-heavy
python test_stats.py
# Output: All tests passed

You get independent validation. If the heavy refactor has bugs, you know immediately without polluting the main line of work.

Step 6: Decide What to Merge

You now have three evolving branches:

develop with your original code
ai/refactor-small with a careful, stable refactor
ai/refactor-heavy with a more ambitious rewrite

Decision time usually involves:

Code clarity
Compatibility with existing imports
Test coverage
Future extensibility
Performance

For many teams, the small refactor becomes a straightforward merge:

cd ../project  # Return to main worktree
git switch develop
git merge ai/refactor-small
# Output: Fast-forward or Merge made by the 'recursive' strategy.

After merging, run tests again to ensure everything works:

python test_stats.py

The heavy refactor might need additional time, iteration, or might stay as a long-running branch until more confidence is built. You can keep that worktree around for continued experimentation.

Step 7: Cleaning Up

After merging what you want, remove worktrees you no longer need.

git worktree remove ../refactor-small

For branches you want to keep but not actively work on:

# Archive the heavy refactor branch for future reference
git branch --move ai/refactor-heavy archive/refactor-heavy-2025-12
git worktree remove ../refactor-heavy

Or delete the branch entirely if you don't need it:

git worktree remove ../refactor-heavy
git branch -D ai/refactor-heavy

If Git complains the directory still exists, delete it manually and run:

rm -rf ../refactor-heavy
git worktree prune

List remaining worktrees:

git worktree list

Why This Workflow Works So Well

This approach removes the friction of experimental coding. AI tools prefer entire files or folders as context, and they sometimes rewrite aggressively rather than patch small areas. Worktrees give them dedicated sandboxes, and they give you mental clarity: each branch corresponds to a purpose. You can compare outputs by simply opening two folders side by side instead of juggling switching commands that reset your IDE state.

Worktrees also encourage discipline. You evaluate AI-generated changes deliberately, not impulsively. You treat each branch as a self-contained proposal. This makes working with powerful automated tools safer, faster, and easier to reason about.

The biggest advantage is velocity without chaos: several workflows move forward independently, but none interfere with your main development branch. You keep the codebase clean, and you can run experiments all day without ever fearing that an agent will overwrite something you care about.

If you use AI in your everyday coding, this structure should become second nature. It transforms the way you evaluate ideas, especially when dealing with both small fixes and large rewrites.

Troubleshooting Common Issues

Problem: "fatal: 'ai/refactor-small' is already checked out at..."
Solution: You can't check out the same branch in multiple worktrees. Use different branch names.

Problem: Changes appear in the wrong worktree
Solution: Check which directory you're in with pwd. Verify the branch with git branch --show-current.

Problem: VS Code commits to wrong branch
Solution: Always open the worktree directory as the workspace root, not a parent folder. Check the branch indicator in the bottom-left corner.

Problem: Tests fail after AI refactor
Solution: Review AI changes line-by-line. AI tools sometimes introduce subtle bugs. Use git diff to see exactly what changed.

Problem: Can't delete worktree
Solution: Close all editors and terminals in that worktree, then use git worktree remove --force if needed.

Extras

### Using VS Code with Multiple Worktrees Without Losing Your Mind

Now comes the tricky part. Developers get confused in multi-worktree setups not because Git is hard, but because the editor happily opens whatever folder you point at and then people forget which workflow they’re editing. One tiny commit into the wrong branch and your careful separation collapses.

The easiest way to stay sane is to open each worktree as a separate VS Code window.

If you open the root project folder and start navigating into `.worktrees/...`, you’re setting yourself up for mistakes. Instead, open the worktree directory itself:

```bash
code ../project
code ../refactor-small

This gives each workflow its own VS Code instance, with its own branch badge in the bottom-left status bar. You never want both instances to point at the same folder tree.

The moment you enter a file in the wrong window, you’ll see the branch indicator complaining. Git worktree integration in VS Code is surprisingly stable as long as each window corresponds to just one worktree.

Using Git Worktrees as Clean Rooms for AI-Assisted Coding

2025-11-28T00:00:00+01:00

The Mental Model

Git worktrees let a single repository sprout multiple working directories, each tied to a different branch but sharing the same object store. It feels almost like having parallel realities: one directory on develop, another on a feature branch, yet another on a temporary test branch, all living side by side without fighting for checkout state. There is no deep magic behind this; Git simply keeps the .git metadata in one place and mounts lightweight directory views that track their own branch heads.

A tiny example explains the idea better than abstraction ever could:

# Add a new worktree checked out to branch feature/refactor
git worktree add ../refactor-sandbox feature/refactor
````

Your filesystem now holds two siblings:

__OBSIDIAN_CODEBLOCK_1__

The first stays on your main line of work. The second is a clean, isolated world where an AI agent can generate, rewrite, or destructively refactor code without trampling over your ongoing tasks.

Here is a simplified illustration of how worktrees relate to the underlying object store:

__OBSIDIAN_CODEBLOCK_2__

This is the core arrangement: many worktrees, one shared repository brain.

## What You Can Do

The real advantage of worktrees is not in the basic command but in the freedom of running multiple contexts at the same time. This becomes especially useful when coding with agents that expect a clean directory and stable branch state.

You can place an AI agent in a dedicated worktree and let it operate as if it were the only thing happening in the project. Meanwhile, you continue normal development somewhere else, without stashing, switching, or cleaning up.

__OBSIDIAN_CODEBLOCK_3__

The agent gets its own branch and folder, untouched by half-written ideas or files you forgot to commit. When it produces large structural changes, you can evaluate them in isolation and merge only when convinced the proposal makes sense.

You can go further and set up multiple competing proposals:

__OBSIDIAN_CODEBLOCK_4__

Agents operate freely. You compare their results with human eyes.

## Practical Scenarios

This workflow shines when you are working with tools that tend to rewrite code rather than patch it. Many agents do this. A worktree acts like a disposable environment.

You might use them when engaging in heavy refactors, upgrades across the whole codebase, or exploratory changes generated by an agent, where the risk of polluting your current branch is high. You can let the agent run tests or static checks inside that environment as well, because nothing touches your main working directory.

There are situations where a worktree is not a good fit. If you are working in a simple repository with a single branch and you rely heavily on ephemeral changes, then switching branches or using commit-based snapshots might be simpler. Also, if an agent expects to operate on a fully isolated clone rather than a shared store, then a real clone is safer. Some poorly implemented tools still assume that `.git` lives inside the working directory; those break in a worktree layout.

A real-world example: you have a large Python codebase. You want an agent to attempt a dependency upgrade across many modules. You create a fresh worktree and branch called `upgrade/fastapi-0.115`. The agent runs its rewrite steps there. You stay focused on a bug fix on `develop`, undisturbed. Later, you review the agent’s output, run tests, and only then merge. The separation reduces mistakes.

## The Fine Print

There are a few subtle behaviors worth knowing before using worktrees as part of your AI-assisted coding workflow. The first is that each worktree tracks its own branch head. If you forget which directory you are in, it is easy to commit to the wrong branch. The fix is simple: always inspect `git status` before committing, especially when moving between folders.

Another nuance is the deletion cycle. Removing a worktree requires both removing the folder and cleaning up Git’s internal registry. If an AI tool aggressively deletes directories, Git can get confused until you run `git worktree prune`.

Performance is usually excellent, since objects are shared, but some agent tools expect the `.git` directory to be physically inside the folder. Worktrees create a `.git` file that points elsewhere. When such tools fail, the workaround is to copy the `.git` directory from the main repo into the worktree. This is safe only for debugging; the correct fix is to use tools that support worktrees properly.

Here is an example of what not to do:

```bash
# Wrong: manually replacing the pointer file
rm agent-run/.git
cp -r project/.git agent-run/.git   # Bad

This produces a corrupted setup. The correct approach is to use tools that understand Git’s layout or run them in the primary worktree.

Another detail: worktrees are lightweight but they do not isolate installed dependencies, temporary files, or build artefacts. If your agent modifies environment-level resources, you still need separate virtual environments. Using per-worktree environments avoids cross-contamination:

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Finally, merging large agent-generated changes becomes easier because diffs are clean and self-contained. The trade-off is that you must resist the temptation to accept everything wholesale. A worktree encourages discipline: treat the agent’s output as a draft, not a command.

# Bad: blindly accepting generated rewrite
def load_user(id):
    return db.get_user(id)  # Lost validation

# Good: incorporate intention, keep safeguards
def load_user(id):
    user = db.get_user(id)
    if user is None:
        raise ValueError("User not found")
    return user

Here is the decision flow in a format Mermaid will accept:

flowchart TD
    A[Start AI task]
    B{Create worktree?}
    C[Isolated branch and folder]
    D[Agent operates safely]
    E[Human review]
    F{Accept changes?}
    G[Merge into main]
    H[Delete worktree]
    I[Risk mixing changes]

    A --> B
    B --> C
    B --> I
    C --> D
    D --> E
    E --> F
    F --> G
    F --> H

Avoiding Homebrew Upgrades That Require Sudo on macOS

2025-10-31T00:00:00+01:00

Understanding the Problem

Homebrew is designed to install software without root privileges. Most formulae live peacefully under /usr/local or /opt/homebrew, updating silently. Yet sometimes you hit an upgrade that stops mid-flow asking for your password. This happens when the formula is actually a cask that installs via a macOS .pkg file. Those installers, like dotnet-sdk, Java, or Docker, need admin rights to write under /Library.

Running brew update && brew upgrade in automation, or even during a morning maintenance routine, becomes painful if one package demands manual confirmation.

==> Upgrading dotnet-sdk
==> Uninstalling packages with `sudo` (which may request your password)...
Password:
````

The challenge is: how to let Homebrew upgrade everything else _except_ those sudo-needy packages?

## How Homebrew Handles Upgrades

The key to understanding this lies in the distinction between **formulae** and **casks**.

- **Formulae** are built from source or precompiled binaries and install locally.

- **Casks** wrap macOS installers (`.pkg` or `.dmg`), often using `sudo` under the hood.


__OBSIDIAN_CODEBLOCK_1__

By splitting upgrades this way, you can run the formula upgrades safely and delay the cask upgrades for when you’re ready to enter a password.


## Useful Patterns for Controlling Upgrades

There are a few practical patterns that help you avoid password prompts without breaking your update flow.

### 1. Pin the problem packages

Pinning prevents a package from being upgraded or cleaned up. It’s ideal when you want to hold off on upgrades that are known to require admin privileges.

__OBSIDIAN_CODEBLOCK_2__

You can unpin later with:

__OBSIDIAN_CODEBLOCK_3__

Pinned packages are skipped automatically by both `brew upgrade` and `brew cleanup`.

### 2. Skip casks entirely

If your upgrade script doesn’t need GUI tools or frameworks, this is the simplest fix:

__OBSIDIAN_CODEBLOCK_4__

No sudo prompts, no surprises.

### 3. Filter out known offenders dynamically

If you want a more flexible approach, you can automate selective upgrading with a simple shell loop:

__OBSIDIAN_CODEBLOCK_5__

This simulates each upgrade first. Anything that would invoke `sudo` is skipped.


## When to Use These Approaches

### When It Makes Sense

- You’re automating Homebrew upgrades in a CI job, scheduled task, or startup script.
- You’re on a work laptop where you don’t have admin rights.
- You want to keep your local developer tools up to date but prefer to manually review framework-level changes like .NET or Java.

### When Not to Use

- You manage system-wide development environments and actually _need_ the latest frameworks for compatibility.
- You rely on the full cask ecosystem (e.g., GUI apps) and are okay with interactive upgrades.

## Things to Watch Out For

Skipping casks means you may fall behind on security updates for those apps. If you pin something, remember to unpin periodically and check whether a new version resolves past issues.

Also, not every `.pkg` installer actually uses `sudo` every time—it depends on how the vendor packaged it. Sometimes a version update switches from a simple binary to a `.pkg`, suddenly introducing password prompts. You’ll notice it quickly because your upgrade automation will hang.

Here’s a mental model to decide:

__OBSIDIAN_CODEBLOCK_6__

The trick is to control the boundary between convenience and responsibility. Homebrew gives you the tools—`--formula`, `--cask`, and `pin`—but expects you to use them deliberately.

## The Fine Print

There’s no hidden flag like `--ignore-sudo` in Homebrew’s command set. You have to rely on composition: limiting the upgrade scope and pinning selectively. The nice part is that all of this integrates cleanly with your daily maintenance habits.

A typical non-interactive safe sequence on macOS might look like this:

```bash
brew update
brew upgrade --formula
brew cleanup

Then, once a week or month, handle casks manually:

brew upgrade --cask

This rhythm keeps your CLI tools fresh while letting you decide when to deal with packages that need elevated rights. It’s a small adjustment that turns Homebrew into a far smoother experience on macOS.

Understanding Python Protocols - Structural Subtyping in Practice

2025-10-29T00:00:00+01:00

The Basic Idea

Introduced in Python 3.8 via PEP 544, Protocols extend Python’s static typing system by allowing structural subtyping — also called static duck typing.
It means that instead of checking what a class inherits from, Python type checkers can verify what methods and attributes it provides.

Let’s unpack this idea with a simple example.

```python from typing import Protocol

Define a protocol describing the "shape" of a writer

class Writer(Protocol): def write(self, text: str) -> None: ...

Any class implementing 'write' will satisfy this protocol

class FileWriter: def write(self, text: str) -> None: print(f"Writing to file: {text}")

class ConsoleLogger: def write(self, text: str) -> None: print(f"LOG: {text}")

def log_message(writer: Writer, msg: str) -> None: writer.write(msg)

log_message(ConsoleLogger(), "Hello") # OK log_message(FileWriter(), "Hello") # OK ````

Neither FileWriter nor ConsoleLogger inherit from Writer.
But both structurally match it — they have a write method with the same signature.
This is the essence of protocols: “If it quacks like a duck, type checkers treat it as a duck.”

Useful Patterns

Protocols aren’t just about pretending to be an interface. They unlock new expressive power for static type hints and help keep your code decoupled. Let’s explore some common variations.

Read-only Attributes

You can describe expected attributes directly in a protocol.

OBSIDIAN_CODEBLOCK_1

Here, any object with a name attribute of type str will pass the type check.

Optional and Partial Protocols

Sometimes you don’t need to specify all possible methods — only those relevant to your function.

OBSIDIAN_CODEBLOCK_2

This works beautifully with many built-in classes (io.TextIOWrapper, sockets, etc.).

Runtime Checkable Protocols

By default, isinstance() and issubclass() don’t work with protocols.
You can enable that explicitly using @runtime_checkable.

OBSIDIAN_CODEBLOCK_3

Be cautious, though — runtime checking only verifies that a class explicitly implements the methods (not dynamically added ones).

Where It Shines

Protocols shine in situations where you want flexibility without inheritance — particularly in large systems or when using third-party libraries.

When to Use

When you want to specify expected behavior without forcing inheritance.
When integrating code from multiple libraries that weren’t designed to work together.
When designing APIs that rely on capabilities (“things that can write”, “things that can close”, etc.) rather than hierarchies.

When Not to Use

When you actually need shared implementation (then a base class is better).
When your codebase doesn’t use static type checking tools like mypy, pyright, or pylance — the benefits won’t materialize at runtime.
When the protocol would describe a highly dynamic or runtime-altered interface.

Example: Logging Abstraction

Let’s look at a practical scenario. Suppose you have a service that writes logs, but you want to support multiple backends.

OBSIDIAN_CODEBLOCK_4

You’ve decoupled your code from concrete classes — that’s clean design.

The Fine Print

Protocols are deceptively simple but have a few subtleties worth knowing.

Structural vs Nominal Subtyping

Traditional inheritance in Python is nominal: types are related by declared lineage.
Protocols enable structural subtyping: types are compatible based on structure.

The diagram below shows the conceptual difference.

OBSIDIAN_CODEBLOCK_5

Generic Protocols

You can make protocols generic to express relationships between types.

OBSIDIAN_CODEBLOCK_6

Common Mistakes

OBSIDIAN_CODEBLOCK_7

Always match both method name and signature exactly — including annotations.
Otherwise, the type checker will reject the class.

Performance and Design Notes

Protocols exist only for type checking; at runtime, they add virtually no overhead.
They don’t enforce behavior — they just describe it.
That’s a strength, but also a risk: they can give a false sense of safety if your codebase doesn’t use static analysis.

HINT: In real projects, protocols often replace ad-hoc duck typing checks like hasattr(obj, "write") with explicit, type-checked contracts — a major readability and safety win. You can check your code base whether you have intensive usage of hasattr

Closing Thought

Protocols bring static typing and duck typing together in a surprisingly elegant way.
They don’t replace inheritance or abstract base classes — they complement them.
Where ABCs define what must be implemented, protocols describe what is already provided.

References

- longer, bottom-up introduction to Protocols: Python Protocols: Leveraging Structural Subtyping – Real Python

Evolution of Type Hints in Python — From Comments to Inline Typing and Beyond

2025-10-24T00:00:00+02:00

The Mental Model

Python typing wasn’t born overnight. It crept into the language slowly, first as a loose suggestion and later as a core part of modern codebases.
Originally, you could only hint types using comments (# type: int), but as Python matured, its typing syntax grew more expressive, more compact, and more powerful.

The journey started with PEP 484 in Python 3.5, introducing the typing module and annotations as first-class citizens. Since then, nearly every minor version brought a refinement or simplification, allowing developers to express richer constraints without resorting to verbose generics.

# Python 3.5 (PEP 484)
from typing import List

def greet_all(names: List[str]) -> None:
    for name in names:
        print(f"Hello, {name}!")
````

This basic form of static typing opened the door for tools like **mypy**, **pyright**, **pylance** and most recently **ty**, **pyrefly** to provide editor-level correctness checks.

This basic form of static typing opened the door for tools like [**mypy**](https://mypy-lang.org/), [**pyright**](https://github.com/microsoft/pyright), [**pylance**](https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-pylance), and most recently [**ty**](https://docs.astral.sh/ty/), [**pyrefly**](https://pyrefly.org/) to provide editor-level correctness checks.

## Key Features Through Versions

### Python 3.5 (PEP 484) — The Birth of Typing

Introduced the `typing` module:

- `List`, `Dict`, `Tuple`, `Optional`, `Union`
- `Callable`, `Any`, `TypeVar`, `Generic`
- Function annotations became official.

### Python 3.6 — Variable Annotations

You could finally type variables directly without using comments:

__OBSIDIAN_CODEBLOCK_1__

Also introduced `typing.NamedTuple` and `typing.NewType`.

### Python 3.7 — Postponed Evaluation (PEP 563)

Type hints were treated as **strings** by default (via `from __future__ import annotations`), delaying their evaluation until runtime.  
This solved circular import issues and reduced overhead in function definitions.

### Python 3.8 — TypedDict and Literal Types

Added:

- `TypedDict` for describing dicts with specific key/value types.
- `Final` and `Literal` for immutability and constant values.
- `Protocol` (PEP 544) for structural subtyping.

__OBSIDIAN_CODEBLOCK_2__

### Python 3.9 — Built-in Generics (PEP 585)

This was a huge quality-of-life upgrade:  
`list[int]` replaced `List[int]`, `dict[str, int]` replaced `Dict[str, int]`.

__OBSIDIAN_CODEBLOCK_3__

### Python 3.10 — Union Operator (PEP 604)

Simplified `Union` syntax using the `|` operator.

__OBSIDIAN_CODEBLOCK_4__

Also improved type narrowing in `match` statements (structural pattern matching).

### Python 3.11 — Self Type, Variadic Generics, TypedDict Enhancements

- `Self` for methods returning their own type (PEP 673).
- `TypeVarTuple` and `Unpack` for variadic generics (PEP 646).
- `NotRequired` and `Required` for `TypedDict` fields.

__OBSIDIAN_CODEBLOCK_5__

### Python 3.12 — The `typing` Cleanup

Deprecated old-style generics (`List`, `Dict`, etc.).  
Introduced `TypeAliasType` (PEP 695) and made `type` annotations simpler:

__OBSIDIAN_CODEBLOCK_6__

### Python 3.13 — No More `from __future__ import annotations`

Postponed evaluation of annotations is now **default**, removing one of the last confusing steps in typing setup.

## Useful Patterns

### Simplified Union Types

Readable and concise unions are now idiomatic:

```python
def load_data(source: str | Path) -> str: ...

Structural Subtyping with Protocols

Protocol-based typing (instead of inheritance) allows flexible contracts:

from typing import Protocol

class SupportsClose(Protocol):
    def close(self) -> None: ...

def cleanup(obj: SupportsClose) -> None:
    obj.close()

TypedDict for JSON-like Data

Great for static checking of structured but dict-based data:

class Config(TypedDict, total=False):
    host: str
    port: int

Self Type for Fluent Interfaces

Commonly used in builder-style classes:

class Query:
    def where(self, condition: str) -> Self:
        ...
    def execute(self) -> None:
        ...

A Quick Summary of Typing Evolution

graph LR
A[3.5: typing module (PEP 484)] --> B[3.6: variable annotations]
B --> C[3.7: postponed evaluation]
C --> D[3.8: TypedDict, Literal, Protocol]
D --> E[3.9: built-in generics]
E --> F[3.10: union | operator]
F --> G[3.11: Self, variadic generics]
G --> H[3.12: TypeAliasType, PEP 695]
H --> I[3.13: annotations postponed by default]

Typing has moved from verbose and experimental to elegant and essential.
Modern Python encourages typing as part of its design philosophy — readable, predictable, and expressive.

Keeping performance results in a separate Git branch using `git checkout --orphan`

2025-10-24T00:00:00+02:00

Understanding orphan branches in Git

There’s a lesser-known Git feature that allows you to start a branch with no history at all:
git checkout --orphan <branch>.
It creates a new branch disconnected from any commits or files that exist in the current branch.

Think of it as a fresh repository living inside your repo — same .git database, different history.
Everything from your current branch remains in your working directory, but the new branch starts with no commits.
You can then decide which files to keep, stage them, and make an initial commit that becomes the root of this branch.

# Create a new orphan branch named 'gh-pages'
git checkout --orphan gh-pages

# Clean out files from the working directory
git rm -rf .

# Add whatever you want to keep in this branch (e.g. built docs, reports)
echo "Performance results" > index.html
git add index.html
git commit -m "Initial gh-pages commit"
````

At this point, you’ve got a brand new branch — no link to your project’s main history.  
It’s perfect for things you don’t want tangled with your codebase: generated reports, documentation, benchmarks, or built assets.

__OBSIDIAN_CODEBLOCK_1__

Unlike a normal branch, an orphan branch doesn’t have a parent commit, so merges or rebases to it don’t make sense — it’s meant to live independently.

## Using it with GitHub Actions

Let’s say you run automated performance tests in your CI pipeline and want to **publish their results** to a separate branch (`gh-pages`) — so you can host them via GitHub Pages or just keep them visible.

Here’s the minimal GitHub Actions workflow to make that happen:

__OBSIDIAN_CODEBLOCK_2__

The important detail:  
`git checkout --orphan gh-pages` creates a clean branch inside the same repo,  
and `git push --force origin gh-pages` overwrites the branch each time, keeping only the latest reports.

You could later enable **GitHub Pages** to serve that branch, and your performance dashboard would live at  
`https://<username>.github.io/<repo>/`.


## When this pattern makes sense

This approach shines when you want to:

- Publish generated files (reports, docs, dashboards) without polluting your main branch.

- Keep automation output under version control but separated from your code.

- Host static files directly from the repository with GitHub Pages.


It’s less suited when:

- You need historical diffs between runs — the `--force` push wipes history.

- Reports are huge — pushing megabytes of HTML or JSON repeatedly is slow.

- The repo is private and GitHub Pages isn’t enabled — you might prefer an artifact store or S3.


For long-term tracking, you might instead append results to a dedicated folder in the main branch or store them in a database.

## Subtle details and common pitfalls

A few things can bite you when using orphan branches from automation:

1. **Forgetting to clean up.**  
    After `git checkout --orphan`, your working directory still has files from the previous branch.  
    If you forget `git rm -rf .`, those files will get re-committed.

2. **Forcing overwrites.**  
    Each run should do a clean commit and a `--force` push. Without that, you may get “non-fast-forward” errors because the branch histories are unrelated.

3. **Detached history.**  
    You can’t merge or diff easily between `gh-pages` and `main`. This is by design, but it means you can’t `git diff main...gh-pages`.

4. **Keeping GitHub Pages clean.**  
    If you only need artifacts for internal review, consider naming the branch `reports` or `artifacts` instead of using `gh-pages`.


Here’s what _not_ to do:

```bash
# Wrong: this will mix commits from main
git checkout -b gh-pages  # Bad, keeps full history

# Correct
git checkout --orphan gh-pages

Orphan branches are like disposable notebooks: use them for output, not for code - and they’ll serve you well.

Understanding the Language Server Protocol through a Minimal Working Example

2025-10-16T00:00:00+02:00

The Mental Model: What Is the Language Server Protocol

The Language Server Protocol (LSP) is one of those invisible technologies that quietly revolutionized the developer experience. It defines a standard way for code editors (clients) to communicate with language-specific analysis engines (servers).

Before LSP, every editor (VS Code, Vim, Sublime, Atom, etc.) had to build their own support for each programming language. That meant dozens of editors and dozens of languages — an explosion of duplicated work.

LSP fixed that by saying: “Let’s agree on how editors talk to language tools.”

Now, editors can speak a common protocol over JSON-RPC (a lightweight request-response format over stdin/stdout). A single server can work with many editors.

You can think of it like this:

graph TD
    A[VS Code Editor] -->|JSON-RPC: initialize, textDocument/didOpen| B[Language Server]
    B -->|diagnostics, hover info, completions| A

In short:

The editor (client) sends notifications and requests: “File opened”, “User typed this”, “What symbols are here?”
The server responds with diagnostics, completions, definitions, etc.

Once you get this model, the magic of modern editor intelligence starts to feel less mysterious.

Using CSS Variables for Dynamic and Reusable Styling

2025-10-14T00:00:00+02:00

The Core Idea

CSS variables (custom properties) let you store reusable values directly in CSS and update them at runtime.
They use the --name syntax and are accessed with var():

:root {
  --bg: #222;
  --text: #eee;
}

body {
  background: var(--bg);
  color: var(--text);
}
````

Variables defined in `:root` are global but can be overridden in specific scopes or components.

## Practical Usage

### Where It Shines

- **Dynamic theming:** dark/light mode, high-contrast themes, seasonal designs.   
- **Design consistency:** same color or spacing reused across multiple components.
- **Runtime manipulation:** interactive UIs where JS can tweak design parameters instantly.
- **Design systems:** variables act as a shared language between designers and developers.

### When Not to Use It

- **Heavy preprocessing needs:** if you rely on loops, conditionals, or functions like `darken()`—Sass still wins there.

- **Older browser support:** IE11 doesn’t support CSS variables (and never will).

### Example: color picker that changes page color live

An example of a simple color picker that changes page color live:
```js
<input type="color" id="picker">
<script>
  const picker = document.getElementById('picker');
  picker.addEventListener('input', e => {
    document.documentElement.style.setProperty('--main-bg', e.target.value);
  });
</script>

Example: Theming

:root {
  --bg: #fff;
  --text: #000;
}
.dark {
  --bg: #121212;
  --text: #fff;
}

Switch themes easily:

document.body.classList.toggle('dark');

Fallbacks and Responsiveness

h1 {
  color: var(--title-color, #444); /* fallback if missing */
}

@media (min-width: 800px) {
  :root {
    --gap: 2rem;
  }
}

Things to Watch Out For

Scoped variables don’t leak outside their selector
IE11 doesn’t support them
Changing variables with JS triggers repaints, so avoid in animations

CSS variables behave like live, inheritable properties—perfect for theming and dynamic UI tweaks without recompiling styles.

Bare Asterisk in Python Function Signatures - Keyword Only Arguments

2025-10-10T00:00:00+02:00

Core Principle

The * by itself in a function signature forces everything after it to be keyword-only arguments. It's a syntax barrier - arguments before the asterisk can be positional, arguments after it must be named.

def foo(a, b, *, c, d):
    pass

foo(1, 2, c=3, d=4)  # works
foo(1, 2, 3, 4)      # breaks

History: Introduced in Python 3.0 via PEP 3102. This was part of the Python 3 overhaul, so it's never been available in Python 2.

Useful Extensions

You can mix this with other parameter types in ways that make sense for your API:

With default values:

def __init__(self, required_arg, *, optional=None, debug=False):
    pass

*_Combined with _args:__

def process(*items, separator=", ", prefix=""):
    # items catches unlimited positional args
    # separator and prefix must be keyword-only
    pass

All keyword-only (rare but valid):

def configure(*, host, port, timeout=30):
    # Everything must be named, nothing positional
    pass

Specific Use Cases

Library APIs where clarity matters - When you have optional parameters that could be confused if passed positionally. The httpx example I saw does this: def __init__(self, message: str, request: httpx.Request, *, body: object | None) - the body parameter is important enough to deserve an explicit name.

Future-proofing - If you might add more required positional parameters later, putting optional ones after * means you won't break existing calls. Old code keeps working even as your signature evolves.

Preventing argument order bugs - When you have multiple parameters of the same type, forcing keywords prevents foo(timeout, retries) vs foo(retries, timeout) mixups.

Nuances

The bare asterisk doesn't capture anything - it's purely a delimiter that says "keyword-only from here on." This is different from *args, which actually captures excess positional arguments into a tuple.

# Bare asterisk - just marks the boundary
def func(a, *, b):
    pass

func(1, b=2)        # works
func(1, 2)          # TypeError: too many positional arguments
func(1, 2, b=3)     # TypeError: too many positional arguments

The function above only accepts one positional argument (a). The * doesn't "consume" anything - it just blocks additional positional arguments from being accepted.

*_When you use _args with keyword-only parameters__ - you put a name on the asterisk, and now it captures all the excess positional arguments, but you can still have keyword-only parameters after it:

def func(a, *args, b):
    print(f"a={a}, args={args}, b={b}")

func(1, b=2)              # a=1, args=(), b=2
func(1, 2, 3, b=4)        # a=1, args=(2, 3), b=4
func(1, 2, 3, 4)          # TypeError: missing keyword-only argument 'b'

Here *args is greedy - it captures all positional arguments after a. But b still must be passed by keyword because it comes after the *args.

The key distinction:

def func(a, *, b): - accepts exactly one positional arg, b must be keyword
def func(a, *args, b): - accepts unlimited positional args into args, b must be keyword
def func(a, *args) - accepts unlimited positional args, no keyword-only parameters

This matters when designing APIs. Use bare * when you want to restrict positional arguments. Use *args when you want to accept a variable number of them but still have some parameters that must be named.

References

PEP 3102 – Keyword-Only Arguments | peps.python.org

Six Weeks, Real Progress - Exploring Shape Up for Product Work

2025-10-06T00:00:00+02:00

What makes Shape Up different?
When Shape Up works well
When it falls short
So what should you actually do?
Further reading

I keep coming back to the same frustration with how we build software. Not the code, the process around it. Sprint planning that eats three hours. Standups that turn into debugging sessions. A backlog so long nobody even scrolls to the bottom anymore.

Agile was supposed to fix all this. Scrum, Kanban, SAFe, whatever. And those frameworks helped, I won't deny that. But at some point things drifted. Two-week sprints started fragmenting work into artificial chunks. Context switching never stopped. There was this constant pressure to estimate story points for work we barely understood yet, and I started noticing we were spending more time managing the process than actually building things.

Then I found Shape Up, the approach from Basecamp (now 37signals). It described exactly what was bothering me, better than I could.

What makes Shape Up different?

Shape Up is not Agile with different labels. The structure is different enough that it changes how you think about planning.

Work in six-week cycles. Not two weeks. Six. Long enough to build something real, short enough that you can't hide. "Plus it gives you about eight chances a year to recalibrate and decide what to work on next." (see: 37signals). Between cycles, there's a two-week cooldown for bugs, tech debt, or just breathing.

Before a cycle starts, senior people do what Basecamp calls "shaping." They take raw ideas and turn them into pitches with clear limits. Not detailed specs, but something with clear boundaries and a sense of what you're not going to build. That last part is important.

Then small teams, usually a designer and one or two programmers, take the shaped project and run with it. No daily standups. No one hovering for status updates. The time is fixed at six weeks, but the scope flexes. If something is taking too long, you cut scope, not time.

There's no backlog. Ideas that don't make it into a cycle just disappear. If they matter, they'll resurface. If they don't, you saved yourself from a list of tickets nobody will ever touch.

When Shape Up works well

I've seen it work for product teams that are tired of the sprint treadmill. It fits when you're building features that need real design thinking, the kind of work that breaks apart when you force it into two-week slices.

It works when your team is experienced enough to work without someone checking on them every day. When people can make decisions without approval for every small thing, and when they'll actually say something when they're stuck instead of waiting for a standup.

Six weeks lets you sit with a problem long enough to solve it properly. A couple people I've talked to said they felt less scattered, and one mentioned actually having time to think about edge cases instead of just filing them as follow-up tickets. I've had the same feeling on longer projects.

The no-backlog thing is better than it sounds. You stop maintaining a guilt list of things you'll never get to. You stop feeling bad about the tickets that have been sitting there for a year and a half. Every cycle is a clean slate.

When it falls short

Shape Up is not a universal fix.

If most of your work is small customer requests and bug fixes, the six-week cycle feels heavy. Sometimes you just need to knock out twenty little improvements, and trying to shape each one into a project is forced. The cooldown weeks help some, but if 80% of your work looks like this, you're probably fighting the framework.

Shaping requires people who understand both the business and the technology. If you don't have senior folks who can do this well, you end up with vague projects that either blow up in scope or leave teams stuck. Good shaping is hard. It takes practice.

Teams that need predictability may struggle too. "We'll ship something good in six weeks" is a very different promise than "We'll deliver these fourteen story points by the end of Sprint 23." If your stakeholders need firm dates on specific features, the flexible scope will create problems.

It's also wrong for true discovery mode. Six weeks is too much commitment when you need to pivot every few days based on user feedback. Shape Up assumes you have some idea of what you're building.

And if your organization is deeply invested in Agile tooling, velocity charts, burndown graphs, story points, the transition is a hard sell. Some teams aren't ready to give those up, and pushing Shape Up into that environment just creates a different kind of process overhead.

So what should you actually do?

The useful question isn't "is Shape Up better than Agile?" It's "what's actually broken in how we work right now?"

If your problem is fragmented work and constant context switching, Shape Up is worth trying. If you're drowning in process overhead, read the book and see what makes sense for you. But if you mostly need to ship small changes on a predictable schedule, you probably don't need to change anything.

I don't really care if Shape Up is "the right framework." What I care about is whether teams are honest about why they work the way they do. Most of us inherited our process from a previous team or a blog post from 2015. Maybe it's time to revisit that.

Simpler Parallelism with concurrent.futures

2025-09-21T00:00:00+02:00

The High-Level Approach

Introduced in Python 3.2 via PEP 3148, concurrent.futures gives you a unified interface for running code in parallel. Instead of wrestling with threads and processes directly, you get executors that handle the messy details. You submit tasks, get back futures, and collect results when they're ready.

The module provides two main executors: ThreadPoolExecutor for I/O-bound work and ProcessPoolExecutor for CPU-bound tasks. Both share the same API, which means you can swap them out with minimal code changes.

from concurrent.futures import ThreadPoolExecutor
import requests

def fetch_url(url):
    response = requests.get(url)
    return len(response.content)

urls = [
    'https://example.com',
    'https://python.org',
    'https://github.com'
]

# Context manager handles cleanup automatically
with ThreadPoolExecutor(max_workers=3) as executor:
    results = executor.map(fetch_url, urls)

for url, size in zip(urls, results):
    print(f"{url}: {size} bytes")

The executor manages a pool of workers for you. You don't create threads manually or worry about joining them. The context manager ensures everything gets cleaned up properly, even if exceptions occur.

Working with Futures

The real power shows up when you need more control than map() provides. The submit() method returns a Future object immediately, letting you track individual tasks and handle them independently.

from concurrent.futures import ThreadPoolExecutor, as_completed
import time

def process_item(item):
    time.sleep(item['delay'])
    return item['id'], item['value'] * 2

items = [
    {'id': 1, 'value': 10, 'delay': 0.5},
    {'id': 2, 'value': 20, 'delay': 0.1},
    {'id': 3, 'value': 30, 'delay': 0.3}
]

with ThreadPoolExecutor(max_workers=3) as executor:
    # Submit all tasks, get futures back
    future_to_item = {
        executor.submit(process_item, item): item 
        for item in items
    }

    # Process results as they complete
    for future in as_completed(future_to_item):
        item = future_to_item[future]
        try:
            result_id, result_value = future.result()
            print(f"Item {result_id}: {result_value}")
        except Exception as e:
            print(f"Item {item['id']} failed: {e}")

The as_completed() function is particularly useful because it yields futures as soon as they finish, rather than in submission order. This means you can start processing early results while slower tasks are still running.

You can also wait for specific conditions using wait():

from concurrent.futures import wait, FIRST_COMPLETED, ALL_COMPLETED

# Submit multiple tasks
futures = [executor.submit(slow_function, i) for i in range(10)]

# Wait for the first one to finish
done, pending = wait(futures, return_when=FIRST_COMPLETED)
fastest_result = next(iter(done)).result()

# Cancel the rest if you only needed one result
for future in pending:
    future.cancel()

The Future objects themselves provide several useful methods. You can check if a task is done with .done(), cancel pending tasks with .cancel(), and attach callbacks with .add_done_callback() that fire when the task completes.

When Each Executor Makes Sense

ThreadPoolExecutor works best for I/O-bound operations where your code spends time waiting. Network requests, file I/O, database queries—these are all good candidates. Python's Global Interpreter Lock (GIL) doesn't hurt you here because threads release the GIL during I/O operations.

from concurrent.futures import ThreadPoolExecutor
import sqlite3

def query_database(db_path, query):
    conn = sqlite3.connect(db_path)
    cursor = conn.cursor()
    cursor.execute(query)
    results = cursor.fetchall()
    conn.close()
    return results

databases = ['users.db', 'orders.db', 'inventory.db']
query = "SELECT COUNT(*) FROM main_table"

with ThreadPoolExecutor(max_workers=3) as executor:
    counts = executor.map(
        lambda db: query_database(db, query), 
        databases
    )

ProcessPoolExecutor is your choice for CPU-intensive work like data processing, image manipulation, or mathematical computations. Each process gets its own Python interpreter and memory space, bypassing the GIL completely.

from concurrent.futures import ProcessPoolExecutor
import hashlib

def hash_file(filepath):
    hasher = hashlib.sha256()
    with open(filepath, 'rb') as f:
        for chunk in iter(lambda: f.read(4096), b''):
            hasher.update(chunk)
    return filepath, hasher.hexdigest()

files = ['large_file1.bin', 'large_file2.bin', 'large_file3.bin']

with ProcessPoolExecutor(max_workers=4) as executor:
    for filepath, digest in executor.map(hash_file, files):
        print(f"{filepath}: {digest}")

Don't use ProcessPoolExecutor for quick tasks or when you're passing large amounts of data. Spawning processes and serializing data between them has significant overhead. If your tasks take less than 0.1 seconds, the overhead probably exceeds the benefit.

Avoid threads for pure CPU-bound work. The GIL means only one thread executes Python bytecode at a time, so you won't get parallel execution. You might even see slower performance due to context switching overhead.

The Subtle Bits

The max_workers parameter matters more than you might think. Too few workers and you're not utilizing available resources. Too many and you waste memory while adding context-switching overhead. For I/O-bound work, you can often use more workers than CPU cores. For CPU-bound work, using more processes than cores typically doesn't help.

import os
from concurrent.futures import ProcessPoolExecutor

# Good default for CPU-bound work
with ProcessPoolExecutor(max_workers=os.cpu_count()) as executor:
    results = executor.map(cpu_intensive_function, data)

# For I/O-bound work, you might go higher
with ThreadPoolExecutor(max_workers=50) as executor:
    results = executor.map(fetch_url, urls)

When using ProcessPoolExecutor, remember that arguments and return values must be picklable. This means you can't pass lambdas, local functions, or objects with unpicklable attributes. If you need to share configuration, consider using functools.partial():

from concurrent.futures import ProcessPoolExecutor
from functools import partial

def process_with_config(item, config):
    # Use config dict to guide processing
    return item * config['multiplier']

config = {'multiplier': 3}
data = [1, 2, 3, 4, 5]

# Wrong - lambdas aren't picklable
# with ProcessPoolExecutor() as executor:
#     results = executor.map(lambda x: process_with_config(x, config), data)

# Right - use partial to bind the config argument
with ProcessPoolExecutor() as executor:
    process_func = partial(process_with_config, config=config)
    results = executor.map(process_func, data)

Exception handling requires attention because exceptions happen in worker threads or processes, not your main thread. Always wrap .result() calls in try-except blocks. If you use map(), exceptions won't raise until you iterate over the results.

def might_fail(x):
    if x < 0:
        raise ValueError("Negative values not allowed")
    return x * 2

with ThreadPoolExecutor() as executor:
    results = executor.map(might_fail, [1, -2, 3])

    for value in results:
        try:
            print(value)  # Exception raises here, not during map()
        except ValueError as e:
            print(f"Error: {e}")

The executors don't automatically time out. If a task hangs, it'll block forever unless you specify a timeout:

future = executor.submit(potentially_slow_function, arg)

try:
    result = future.result(timeout=5.0)  # Wait max 5 seconds
except TimeoutError:
    print("Function took too long")
    future.cancel()  # Won't stop already-running tasks

One common mistake is thinking .cancel() will stop running tasks. It only prevents pending tasks from starting. Once a task begins execution, cancellation doesn't interrupt it. If you need interruptible tasks, you'll need to implement that logic yourself, typically using threading events or multiprocessing shared values.

The module handles resource cleanup well through context managers, but if you don't use them, call .shutdown(wait=True) explicitly. This ensures all pending tasks complete and resources get released. Forgetting this can leave threads or processes hanging around.

Threading vs Multiprocessing in Python - GIL Implications and Choosing the Right Tool

2025-09-10T00:00:00+02:00

Core Principle

Python has two built-in ways to run code concurrently: threading and multiprocessing. The critical difference comes down to the Global Interpreter Lock (GIL) - a mutex that protects access to Python objects, preventing multiple threads from executing Python bytecode at once. This means:

Threading - multiple threads in one process, but only one thread executes Python code at a time due to the GIL. Good for I/O-bound tasks.
Multiprocessing - separate processes with separate Python interpreters, each with its own GIL. True parallelism for CPU-bound tasks.

import threading
import multiprocessing
import time

def cpu_bound_task(n):
    # Heavy computation
    return sum(i * i for i in range(n))

# Threading - doesn't help with CPU work
start = time.time()
threads = [threading.Thread(target=cpu_bound_task, args=(10_000_000,)) for _ in range(4)]
for t in threads: t.start()
for t in threads: t.join()
print(f"Threading: {time.time() - start:.2f}s")  # ~4 seconds on 4-core machine

# Multiprocessing - achieves true parallelism
start = time.time()
processes = [multiprocessing.Process(target=cpu_bound_task, args=(10_000_000,)) for _ in range(4)]
for p in processes: p.start()
for p in processes: p.join()
print(f"Multiprocessing: {time.time() - start:.2f}s")  # ~1 second on 4-core machine

History: The GIL has been in CPython since the beginning. PEP 703 (approved in 2023) outlines a path to making the GIL optional in Python 3.13+, but it's still fundamental to understand for current Python versions.

Useful Extensions

Threading with ThreadPoolExecutor (simpler interface):

from concurrent.futures import ThreadPoolExecutor
import requests

def fetch_url(url):
    response = requests.get(url)
    return len(response.content)

urls = ["http://example.com"] * 10

# Old way - manual thread management
threads = [threading.Thread(target=fetch_url, args=(url,)) for url in urls]
for t in threads: t.start()
for t in threads: t.join()

# Better way - thread pool handles everything
with ThreadPoolExecutor(max_workers=5) as executor:
    results = executor.map(fetch_url, urls)
    print(list(results))

Multiprocessing with ProcessPoolExecutor:

from concurrent.futures import ProcessPoolExecutor

def expensive_calculation(n):
    return sum(i * i for i in range(n))

numbers = [10_000_000] * 4

with ProcessPoolExecutor() as executor:
    results = executor.map(expensive_calculation, numbers)
    print(list(results))

Sharing data between processes:

from multiprocessing import Process, Queue, Value, Array
import ctypes

# Queue - safe inter-process communication
def worker(queue):
    queue.put("Result from worker")

q = Queue()
p = Process(target=worker, args=(q,))
p.start()
print(q.get())
p.join()

# Shared memory with Value and Array
def increment_counter(counter, arr):
    counter.value += 1
    arr[0] += 10

counter = Value('i', 0)  # shared integer
arr = Array('i', [0, 1, 2])  # shared array

processes = [Process(target=increment_counter, args=(counter, arr)) for _ in range(5)]
for p in processes: p.start()
for p in processes: p.join()

print(f"Counter: {counter.value}")  # 5
print(f"Array: {list(arr)}")  # [50, 1, 2]

Thread-safe operations with Lock:

import threading

counter = 0
lock = threading.Lock()

def increment():
    global counter
    for _ in range(100_000):
        with lock:  # Only one thread can execute this block at a time
            counter += 1

threads = [threading.Thread(target=increment) for _ in range(10)]
for t in threads: t.start()
for t in threads: t.join()

print(counter)  # 1,000,000 (correct with lock, random without)

Specific Use Cases

Use threading for I/O-bound tasks:

Making multiple HTTP requests (web scraping, API calls)
Reading/writing multiple files
Database queries where you're waiting for responses
Network operations (socket communication)
Any operation where you spend time waiting for external resources

The GIL doesn't matter here because threads release it during I/O operations. While one thread waits for network/disk, others can run.

Use multiprocessing for CPU-bound tasks:

Image/video processing
Data analysis and numerical computations
Encryption/decryption
Machine learning model training
Parsing large files
Any computation-heavy work where the CPU is the bottleneck

You need separate processes to get around the GIL and use multiple CPU cores effectively.

Real-world example - web scraper:

from concurrent.futures import ThreadPoolExecutor
import requests

def scrape_page(url):
    response = requests.get(url)
    # Process the page
    return extract_data(response.text)

urls = ["http://example.com/page1", "http://example.com/page2", ...]

# Threading is perfect here - lots of waiting for network responses
with ThreadPoolExecutor(max_workers=10) as executor:
    results = list(executor.map(scrape_page, urls))

Real-world example - image processing:

from concurrent.futures import ProcessPoolExecutor
from PIL import Image

def process_image(filename):
    img = Image.open(filename)
    # CPU-intensive operations
    img = img.resize((800, 600))
    img = img.filter(ImageFilter.SHARPEN)
    img.save(f"processed_{filename}")
    return filename

images = ["img1.jpg", "img2.jpg", "img3.jpg", ...]

# Multiprocessing is needed - CPU-intensive work
with ProcessPoolExecutor() as executor:
    results = list(executor.map(process_image, images))

Nuances

The GIL releases during I/O operations - this is why threading works for I/O-bound tasks. When a thread calls a blocking I/O function (like requests.get() or file.read()), it releases the GIL so other threads can run. The GIL only prevents multiple threads from executing Python bytecode simultaneously.

Multiprocessing has overhead - creating processes is expensive (memory and startup time). Each process needs its own Python interpreter and memory space. For small tasks, this overhead can outweigh the benefits of parallelism:

# Bad - overhead dominates
with ProcessPoolExecutor() as executor:
    results = executor.map(lambda x: x * 2, range(100))

# Good - task is substantial enough to justify processes
with ProcessPoolExecutor() as executor:
    results = executor.map(expensive_function, large_dataset)

Data serialization between processes - when you pass data to a process or get results back, Python uses pickle to serialize it. Large objects or objects that can't be pickled cause problems:

# This won't work - lambda functions can't be pickled
with ProcessPoolExecutor() as executor:
    results = executor.map(lambda x: x * 2, numbers)  # Error

# This works - regular functions can be pickled
def multiply_by_two(x):
    return x * 2

with ProcessPoolExecutor() as executor:
    results = executor.map(multiply_by_two, numbers)  # Works

Threading has less isolation - all threads share the same memory space, which means bugs in one thread (like accessing shared data without locks) can corrupt data across the entire program. Processes are isolated - a crash in one process doesn't affect others.

When neither helps - if your program is both CPU and I/O bound, you might need a hybrid approach: processes for CPU work, each using threads for I/O. Or consider asyncio for I/O operations instead of threads if you're doing lots of concurrent I/O.

Process pool size guidelines - for CPU-bound work, use os.cpu_count() workers (one per CPU core). For I/O-bound work with threads, you can use many more (tens or hundreds) since threads spend most time waiting. Experiment to find the sweet spot.

import os
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

# CPU-bound - match core count
with ProcessPoolExecutor(max_workers=os.cpu_count()) as executor:
    results = executor.map(cpu_intensive_func, data)

# I/O-bound - can use many more
with ThreadPoolExecutor(max_workers=50) as executor:
    results = executor.map(io_intensive_func, data)

The simple decision tree:

Waiting for network/disk/external services? Use threading (or asyncio)
Doing heavy calculations/data processing? Use multiprocessing
Doing simple sequential work? Use neither - regular code is simpler

asyncio Basics - async/await and When to Actually Use Them

2025-09-09T00:00:00+02:00

Core Principle

async/await is Python's way of writing concurrent code that can handle multiple I/O operations without blocking. The key insight: while one task is waiting for something (network response, file read, database query), other tasks can run. This is cooperative multitasking - tasks voluntarily yield control during waits.

import asyncio

async def fetch_data(url):
    # This function can be paused and resumed
    await asyncio.sleep(1)  # Simulating network delay
    return f"Data from {url}"

async def main():
    # Run three fetches concurrently
    results = await asyncio.gather(
        fetch_data("api.example.com/1"),
        fetch_data("api.example.com/2"),
        fetch_data("api.example.com/3")
    )
    print(results)

# Run the async code
asyncio.run(main())

This takes roughly 1 second total, not 3, because the waits happen concurrently.

History: async/await syntax was introduced in Python 3.5 via PEP 492. Earlier async code used @asyncio.coroutine and yield from, but async/await is cleaner and now standard.

Useful Extensions

Multiple ways to run concurrent tasks:

# gather - run multiple tasks, collect all results
results = await asyncio.gather(task1(), task2(), task3())

# create_task - start a task in the background
task = asyncio.create_task(long_running_operation())
# Do other stuff
result = await task  # Wait for it when you need the result

# as_completed - process results as they finish
for coro in asyncio.as_completed([task1(), task2(), task3()]):
    result = await coro
    print(f"Got result: {result}")

Async context managers and iterators:

# Async context manager
async with aiohttp.ClientSession() as session:
    async with session.get(url) as response:
        data = await response.text()

# Async iterator
async for item in async_generator():
    process(item)

Timeouts and cancellation:

try:
    result = await asyncio.wait_for(slow_operation(), timeout=5.0)
except asyncio.TimeoutError:
    print("Operation took too long")

# Cancel a running task
task = asyncio.create_task(operation())
task.cancel()

Specific Use Cases

When async actually helps - I/O-bound operations where you're waiting for external resources:

HTTP requests to APIs (using aiohttp or httpx)
Database queries (using asyncpg, motor for MongoDB)
Reading/writing files (using aiofiles)
WebSocket connections
Microservice communication

When async doesn't help - CPU-bound work like calculations, data processing, or image manipulation. Async doesn't give you parallelism for compute work - for that you need multiprocessing or threads (and threads don't help much due to the GIL). Async is about doing other things while waiting, not doing multiple CPU-intensive things simultaneously.

Real-world example where async shines:

# Without async: 10 API calls taking 1 second each = 10 seconds total
# With async: 10 API calls taking 1 second each = ~1 second total

async def fetch_user_data(user_ids):
    async with aiohttp.ClientSession() as session:
        tasks = [fetch_user(session, uid) for uid in user_ids]
        return await asyncio.gather(*tasks)

async def fetch_user(session, user_id):
    async with session.get(f"api.example.com/users/{user_id}") as resp:
        return await resp.json()

Nuances

You can't mix sync and async freely - once you go async, you need an async ecosystem. You can't await in a regular function, and you can't call regular blocking functions in async code without consequences:

async def bad_example():
    # This blocks the entire event loop!
    time.sleep(5)  # Wrong - use await asyncio.sleep(5)

    # This also blocks everything
    requests.get(url)  # Wrong - use aiohttp or httpx async client

async def good_example():
    await asyncio.sleep(5)  # Non-blocking sleep
    async with aiohttp.ClientSession() as session:
        await session.get(url)  # Non-blocking HTTP

Running blocking code when necessary - sometimes you have to use a blocking library. Use run_in_executor to run it in a thread pool:

import asyncio
from concurrent.futures import ThreadPoolExecutor

async def use_blocking_library():
    loop = asyncio.get_event_loop()
    # Run blocking code in a thread
    result = await loop.run_in_executor(
        None,  # Use default executor
        blocking_function,
        arg1, arg2
    )
    return result

The event loop is the engine - asyncio.run() creates an event loop, runs your main coroutine, and cleans up. In older code you'll see manual loop management with loop.run_until_complete(), but asyncio.run() (added in Python 3.7) is simpler.

Common mistake - forgetting await:

async def fetch_data():
    return "data"

async def main():
    result = fetch_data()  # Wrong! This is a coroutine object, not the result
    print(result)  # Prints: <coroutine object fetch_data at 0x...>

    result = await fetch_data()  # Correct
    print(result)  # Prints: data

When NOT to use async - if you're only doing one I/O operation at a time, regular synchronous code is simpler and just as fast. Async adds complexity (mental overhead, debugging difficulty, library compatibility) that's only worth it when you're doing multiple I/O operations concurrently. A script that makes one API call doesn't benefit from async. A web scraper hitting 100 URLs does.

Replacing Makefile with Invoke for Cross-Platform Python Tasks

2025-08-14T00:00:00+02:00

I’ve always liked Make. It’s quick to type, powerful, and honestly kind of fun once you know the quirks. On macOS and Linux, it just works.

Then my teammate on Windows ran make test-unit. Boom. Red text. Paths broke. Shell flags disappeared. Instead of testing code, we were testing our patience.

After a couple of these moments, I realized I had two choices:

Keep duct-taping Windows support into the Makefile.
Switch to a tool that doesn’t care which OS you’re on.

I chose option 2.

From my old Makefile

It looked like this — perfectly fine for macOS/Linux, but brittle on Windows:

SRC_FILES = src
SRC_AND_TEST_FILES = src tests
R_PYPROJECT = requirements-pyproject.txt

test-unit: ## Run the unit tests with pytest.
    @echo -e "$(COLOR_CYAN)Running unit tests...$(COLOR_RESET)"
    pytest --log-cli-level=INFO -rA tests/unit/

format: ## Running code formatter: black and isort
    @echo "(isort) Ordering imports..."
    @isort $(SRC_AND_TEST_FILES)
    @echo "(black) Formatting codebase..."
    @black --config pyproject.toml $(SRC_AND_TEST_FILES)
    @echo "(ruff) Running fix only..."
    @ruff check $(SRC_AND_TEST_FILES) --fix-only

lint: ## Run the linter (ruff) to check the code style.
    @echo -e "$(COLOR_CYAN)Checking code style with ruff...$(COLOR_RESET)"
    ruff check $(SRC_AND_TEST_FILES)

changelog: ## Generate a changelog using git-cliff
    git-cliff -o CHANGELOG.md

It’s not bad code. It’s just not friendly to every shell.
echo -e works in Bash, not in cmd.exe. Color codes are ignored in PowerShell. Paths and quoting rules differ. Small stuff, but it adds up.

How I rewrote it to work everywhere

I moved the build logic into Python with Invoke. The syntax is simple, it runs anywhere Python runs, and I can call the exact same commands on macOS, Linux, and Windows.

# tasks.py — Invoke version
from invoke import task

SRC_FILES = "src"
SRC_AND_TEST_FILES = "src tests"

@task
def test_unit(c):
    print("Running unit tests...")
    c.run("python -m pytest --log-cli-level=INFO -rA tests/unit/")

@task
def format(c):
    print("(isort) Ordering imports...")
    c.run(f"python -m isort {SRC_AND_TEST_FILES}")
    print("(black) Formatting codebase...")
    c.run(f"python -m black --config pyproject.toml {SRC_AND_TEST_FILES}")
    print("(ruff) Running fix only...")
    c.run(f"python -m ruff check {SRC_AND_TEST_FILES} --fix-only")

@task
def lint(c):
    print("Checking code style with ruff...")
    c.run(f"python -m ruff check {SRC_AND_TEST_FILES}")

@task
def changelog(c):
    c.run("git-cliff -o CHANGELOG.md")

Now my teammate runs:

invoke test-unit
invoke format
invoke lint
invoke changelog

No “works on my machine” syndrome. No branching logic for OS detection. It just works.

Why this change paid off

I write a command once — it runs everywhere.
No one needs to know shell quirks to contribute.
Commands are short and consistent across the team.

Make is still a great tool. But for a Python project with a mixed-OS team, moving the build brain into Python was a quiet productivity win.

Using OpenAI Python SDK with Local Ollama Models (and When to Opt for Alternatives)

2025-07-29T00:00:00+02:00

I've been diving into how to use the official openai Python package to talk to local Ollama models—and when it makes sense to bring in abstraction layers like LiteLLM. Let me walk you through what I learned.

1. Can I use the OpenAI Python package for local Ollama models?

Yes! Since early February 2024, Ollama supports the OpenAI Chat Completions API, exposing compatible endpoints locally. You can simply point the OpenAI client at "http://localhost:11434/v1", pass a dummy API key, and call completions just like you would to OpenAI’s hosted API (see Ollama blog).

from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="unused-key")

response = client.chat.completions.create(
  model="llama2",
  messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What’s the capital of France?"}
  ],
)
print(response.choices[0].message.content)

You can also do embeddings similarly:

resp = client.embeddings.create(model="llama2", input="Hello world!")
print(resp.data[0].embedding)

So for fairly simple local projects, the OpenAI SDK works perfectly with Ollama.

2. When should I use LiteLLM instead?

LiteLLM is a lightweight Python SDK (and optional proxy server) that provides a unified API for over 100 LLM providers—including OpenAI, Anthropic, HuggingFace—and crucially, Ollama/local models ( nice example with minimal Flask app - poem generator KodeKloud Notes).

Here are some benefits of using LiteLLM: - It standardizes completions, embeddings, streaming, retries, and fallback logic
- You can swap providers (e.g. openai/gpt‑4, anthropic/claude, ollama/llama2) with no code changes
- Proxy server mode offers observability, logging, rate limiting, and cost tracking across providers (see LiteLLM documentation: LiteLLM)

3. Example: using LiteLLM Python SDK

First install:

pip install litellm

Then in Python:

import os
from litellm import completion, embeddings

os.environ["OPENAI_API_KEY"] = "dummy"
os.environ["LITELLM_OLLAMA_BASE"] = "http://localhost:11434/v1"

# Completion via Ollama
resp = completion(model="ollama/llama2", messages=[{"role":"user","content":"Hello!"}])
print(resp["choices"][0]["message"]["content"])

# Embeddings via Ollama
emb = embeddings(model="ollama/llama2", input="Hello world")
print(emb["data"][0]["embedding"])

Later you can just switch to openai/gpt-4o or another provider. You keep the same completion(...) call. No branching logic in your app (Langfuse, LiteLLM, LiteLLM).

5. Alternatives to LiteLLM

There are several other frameworks you may consider:

Langchain, Llama‑Index, Guidance, instructor – great for structured output, chaining, tool-use, agents, prompt templating. Read more in these sources:
- 4 Proven Ways to Use Ollama Locally & OpenAI APIs in Python: Fast, Flexible, and Scalable | by Brain Glitch | Jun, 2025 | Towards Dev
- Unlocking the Power of LiteLLM: A Lightweight, Unified Interface for LLMs
- Top 5 LiteLLM alternatives of 2025
- The best library for structured LLM output – Paul Simmering - gives different recommendations for four use cases.
TrueFoundry – a more enterprise‑ready orchestration layer with observability, scaling, and deployment support, but heavier than LiteLLM (truefoundry.com)

Summary

I like using the OpenAI Python SDK with Ollama—it’s quick, reliable, and simple for local use cases. But as soon as I need to add other providers, handle retries/fallbacks, use embeddings, or manage observability and switching logic, LiteLLM becomes more convenient. And if I’m building complex agent pipelines or need structure, then libraries like Langchain or TrueFoundry fit right in.

Building a Multi-Notebook Report with Quarto

2025-07-23T00:00:00+02:00

Over the past few months, I’ve been using Jupyter notebooks to explore and document various data analysis tasks. At some point, I realized I wanted to share the results in a more polished way, so I used quarto to generate report. Over the time analysis notebook has grown to the monster size. I wanted to split it into multiple notebooks, each focusing on a specific aspect of the analysis. But I also wanted to combine them into a single cohesive report. Something that reads like a real report. I didn't want to just dump a bunch of disconnected notebooks on someone.

That’s where Quarto with book project functionality came in. In this post, I’ll show you two practical ways to combine multiple notebooks into a single, unified report using Quarto:

Option 1: Turn your notebooks into a structured book (like GitBook)
Option 2: Embed notebooks into a single .qmd file

Both methods let you keep your notebooks clean and modular while generating a polished report that looks great in HTML, PDF, or even EPUB formats.

Option 1: Create a Book-Like Report with Quarto

This is the approach I went with for a larger project. It gives you a navigation sidebar, automatic chapter numbering, and multiple output formats. Think of it as building a small website or PDF book.

Step 1: Install Quarto

If you haven’t installed Quarto yet:

# On macOS/Linux
brew install quarto

# Or download manually
https://quarto.org/docs/get-started/

Make sure Jupyter is also installed and working.

Step 2: Create a New Book Project

Let’s set up the project:

quarto create project book my-report
cd my-report

You’ll now see a few files and folders:

my-report/
├── _quarto.yml
├── index.qmd
├── intro.qmd
└── references.qmd

This is your basic structure. You can remove intro.qmd or rename it. For now, we’ll leave it.

NOTE: you can use vscode quarto extensions and start project from vscode as described in Quick Start in documentation.

Step 3: Add Your Notebooks

Drop your notebooks into the folder. For example:

my-report/
├── chapter1.ipynb
├── chapter2.ipynb
├── conclusion.ipynb

You can keep working in Jupyter as usual.

NOTE: I encourage you to read about structuring your "book" in the official docs here. There are many other options than chapter: book parts, appendices

Step 4: Configure `_quarto.yml`

This is where the magic happens. You define the structure of your report here:

project:
  type: book

book:
  title: "My Analysis Report"
  author: "Your Name"
  chapters:
    - index.qmd
    - chapter1.ipynb
    - chapter2.ipynb
    - conclusion.ipynb

format:
  html:
    toc: true
  pdf:
    toc: true
    documentclass: book

Make sure the filenames match your actual notebooks. You can mix .qmd and .ipynb files freely.

Each "chapter" notebook can have its own quarto front matter (YAML header) if you want to customize titles, authors, or other metadata.

NOTE: The official documentation describes many options for customizing style, layout, controls, social media sharing options. Check it out here.

Step 5: Preview Your Report

To preview as you work:

quarto preview

This starts a live-reloading web server at http://localhost:4200.

When you’re ready to render:

quarto render

This generates the output in _book/ (for HTML) and optionally .pdf or .epub depending on your formats.

Option 2: One `.qmd` File That Embeds Notebooks

Sometimes you want to pick and choose what parts of your notebooks go into a report. Or maybe you want to glue them together with a lot of custom narrative. That’s where embedding comes in.

Instead of building a whole book, you write one .qmd file and embed specific cells or whole notebooks into it.

Step 1: Create a New Quarto Project

This time we’ll use a regular document project:

quarto create project article embedded-report
cd embedded-report

Your structure will look like:

embedded-report/
├── _quarto.yml
└── report.qmd

You can drop your notebooks here too:

embedded-report/
├── notebook_a.ipynb
├── notebook_b.ipynb

Step 2: Write the Report with Embeds

Edit report.qmd like this:

---
title: "Combined Report"
format: html
execute:
  enabled: false
---

# Introduction

This is a combined report from two notebooks.

# Results from Notebook A

```{=embed}
notebook_a.ipynb

Specific Figure from Notebook B

notebook_b.ipynb#cell-3

Full Notebook B

notebook_b.ipynb

You can embed the whole notebook or reference individual cells by their labels or indices. If you want to embed only specific cells, give them labels in Jupyter using #| label: my-cell.

Step 3: Render It

quarto render

You’ll get a clean report with selected content pulled from your notebooks. You can even combine this approach with more Markdown narrative, custom styling, and conditional content.

Summary

Both methods have their strengths:

The Quarto Book approach is great when you want a navigable multi-chapter report or site. It’s structured, scalable, and looks professional out of the box.
The embedding approach gives you more flexibility when curating content or writing more detailed commentary around selected outputs.

I personally use the book format for larger projects, and embedding for quick curated reports. With both approaches, notebooks stay clean and modular — and reports look great with just a few lines of configuration.

Alternative tools

Additional tools (optional)

Mercury: converts notebooks into interactive web apps, with widgets, hideable code, etc. More suited for dashboards rather than structured multi-notebook books. Quarto+1Reddit+1 Reddit+15Reddit+15Quarto+15 Quarto+6csoneson.github.io+6jumpingrivers.com+6 blog.adyog.com Reddit+2jumpingrivers.com+2Medium+2 GitHub+1Quarto+1 Reddit+4Reddit+4Reddit+4

- Pretty Jupyter: a tool for styling notebook outputs into elegant self‑contained HTML—less structured than Quarto “books”, but quick and visually appealing. Reddit+2Reddit+2Reddit+2

References

Creating a Book – Quarto - official documentation for quarto books
Python for Data Analysis, 3E - exemplary book created with Quarto

Downgrade or Upgrade Your Python Version with uv

2025-06-26T00:00:00+02:00

To downgrade your project’s virtual environment e.g. from Python 3.11 to 3.10 using uv, here’s a step‑by‑step process:

1. Install the desired Python version

Run:

uv python install 3.10

This downloads and manages Python 3.10 in ~/.local/share/uv/python/... (or the equivalent on your OS) (docs.astral.sh, docs.astral.sh).

2. Pin your project to that version

Within your project directory:

uv python pin 3.10

This writes 3.10 to .python-version, ensuring that future commands use that interpreter (docs.astral.sh).

This might not work in case if your pyproject.toml file has a python requirement that prevents upgrade - edit this first, e.g. change:

requires-python = ">=3.11"

requires-python = ">=3.10"

You can also edit .python-version file to have it consistent with the rest of the project.

3. Recreate the virtual environment

The simplest and clean method:

rm -rf .venv
uv venv

This creates a fresh venv using Python 3.10, respecting the pin.

Alternatively, if you're managing dependencies via pyproject.toml + uv.lock:

uv sync

This will recreate the environment from locked specs using the pinned Python version (news.ycombinator.com, docs.astral.sh).

Optional: Verify interpreter version

Run:

. .venv/bin/activate
python --version  # should show Python 3.10.x

Beyond Coverage - Building Truly Complete Test Suites with GitHub Copilot

2025-06-15T00:00:00+02:00

Introduction
The Coverage Trap
Beyond Line Coverage: Behavioral Auditing
The API-First Testing Strategy
Automating the Tedious Parts
The Flaky Test Problem
Test Quality as a First-Class Concern
Integration and End-to-End Validation
Looking Forward

Introduction

Over the past year, I've found myself increasingly dissapointed with the traditional approach to test coverage. Sure, hitting 90% line coverage feels good, but I've watched too many "well-tested" codebases crumble under the weight of production bugs that somehow slipped through. The problem isn't just that we're measuring the wrong things - it's that we're treating testing as a checkbox exercise rather than a comprehensive quality assurance strategy.

That's when I started experimenting with GitHub Copilot's Agent Mode, not just as a code completion tool, but as a systematic approach to building truly complete test suites. What I discovered was a fundamentally different way of thinking about testing - one that goes beyond coverage percentages to focus on behavioral completeness, integration reliability, and long-term maintainability.

The Coverage Trap

Let me start with a confession: I used to be obsessed with coverage numbers. There's something deeply satisfying about seeing that green 95% coverage badge, but recent research finds "disconcerting trends for maintainability" when we rely too heavily on automated tools without proper oversight. The truth is, coverage metrics can lull you into a false sense of security.

I learned this the hard way when our system failed spectacularly in production due to a race condition in our retry logic. The lines were covered, but the behavior wasn't tested. That's when I realized we needed to shift from measuring what code runs to validating what the code actually does.

Beyond Line Coverage: Behavioral Auditing

The first breakthrough came when I started using Copilot to audit our entire codebase for behavioral gaps. Instead of focusing on lines of code, I began asking it to identify untested public functions and methods. This simple prompt changed everything:

"Scan the codebase identify all public functions and methods, then report which of them lack any direct test invocation. Group them by module."

(NOTE: you should include your code base as context in Agent mode with e.g. #codebase or specific dir #file:src )

This might look similar to coverage testing but instead of covered lines you are getting information about functions that are not called directly by any of the tests.

What emerged was startling. We had entire utility functions, error handling routines, and data transformation methods that had never been directly tested. They were covered by higher level tests, but their specific behaviors - especially edge cases remained completely unvalidated.

This behavioral audit approach revealed gaps that traditional coverage tools simply can't detect. When you're validating input spaces rather than code paths, you uncover scenarios like empty inputs, malformed data, and maximum size payloads that can break your system in ways that line coverage never anticipates.

The API-First Testing Strategy

One of the most valuable insights from this journey has been the importance of API surface auditing. Every Flask endpoint, every REST API, every public interface represents a contract with the outside world. Breaking these contracts doesn't just cause bugs - it breaks trust with users and downstream systems.

I started having Copilot systematically inventory all our endpoints and cross-reference them with our integration tests. The results were eye-opening: critical user journeys like password reset and account verification had comprehensive unit tests but no end-to-end validation. Copilot did the work of finding relevant files, extracting the relevant styles and patterns, and applying those forward to the new test suite that it generated, creating coherent integration tests that followed our established patterns.

This approach catches issues that unit tests simply can't see serialization problems, authentication flows, error response formats, and the subtle ways that components interact when they're wired together in a real system.

Automating the Tedious Parts

Once I had a clear picture of what needed testing, the next challenge was actually writing all those tests. This is where GitHub Copilot's ability to generate tests becomes invaluable - you can select the code you want to test, right-click in your IDE and select Copilot -> Generate Tests, or use slash commands to quickly scaffold test suites.

But I discovered that the real power isn't in generating individual tests - it's in systematically working through entire modules. I'd point Copilot at a file like payment_processor.py and ask it to generate pytest tests covering valid payments, negative amounts, and simulated network failures using mocks. The agent would create the test file, inject proper fixtures, write assertions, and even run the tests to check for immediate failures.

More importantly, Copilot excels at converting repetitive test patterns into parameterized tests. Instead of five nearly identical tests for different input values, I could ask it to consolidate them into a single @pytest.mark.parametrize block. This not only reduces maintenance overhead but makes it trivial to add new edge cases as you discover them.

The Flaky Test Problem

Fleaky tests - the tests that pass sometimes and fail other times are the bane of every CI pipeline. They waste developer time, obscure real issues, and erode trust in your test suite.

No discussion of comprehensive testing is complete without addressing the elephant in the room: flaky tests. There are two main types of flaky tests: those that are flaky due to some external conditions, such as network issues, machine crashes, power outages, and those that are flaky due to test design issues.

To spot flaky tests, you need to compare test results from multiple test runs. This analysis would be a time consuming process to perform manually, but fortunately, many CI servers detect flaky tests automatically. The key insight is that Copilot can go beyond just detection to root cause analysis and remediation.

For timing related flakiness, it suggests explicit waits or better synchronization. For external dependency issues, it recommends proper mocking. For shared state problems, it proposes better isolation techniques. The goal isn't to eliminate all flakiness - that's impossible, but to make your test suite reliable enough that failures actually mean something.

Test Quality as a First-Class Concern

As our test suite grew, I realized that test quality itself needed to become a first-class concern. Bad tests are worse than no tests - they give you false confidence while slowing down development. This is where Copilot's analytical capabilities really shine.

I started having it audit our test directory for common anti-patterns: empty test functions, duplicated assertions, magic constants, and tests that rely on implicit ordering. The agent would flag these issues and suggest refactors-converting magic numbers to named constants, extracting common setup into fixtures, and consolidating duplicate logic.

But the most valuable insight was learning to cross-reference coverage reports with module criticality. Not all code is equally important, and not all untested code represents the same level of risk. By having Copilot map coverage data against business-critical modules like payment processing and authentication, I could focus our testing efforts where they would have the most impact.

Integration and End-to-End Validation

Unit tests form the foundation, but they can't catch the subtle ways that components interact in production. This is where integration and end-to-end testing become crucial, and where Copilot's ability to understand entire workflows becomes invaluable.

I've had great success asking Copilot to generate integration tests that exercise entire user journeys - from account creation through data processing to final output. These tests use in-memory databases for speed but validate the complete data flow including serialization, authentication, and error handling.

The key is to focus on critical user paths rather than trying to test every possible integration. A single end-to-end test that uploads a CSV file, triggers data ingestion, and verifies the resulting database entries can catch a surprising number of issues that unit tests miss entirely.

Looking Forward

After a year of experimenting with this approach, I've come to believe that comprehensive testing isn't about reaching some magical coverage percentage - it's about building systems that give you confidence in your code's behavior. Copilot has been instrumental in making this transition from coverage-focused to behavior-focused testing.

The techniques I've described here - behavioral auditing, API surface validation, automated test generation, flaky test management, and continuous quality monitoring - work together to create a testing strategy that's both comprehensive and maintainable. Each element addresses a different aspect of the testing challenge, from initial coverage gaps to long-term sustainability.

What excites me most is that this is just the beginning. As AI tools become more sophisticated, I expect we'll see even more powerful approaches to test analysis and generation. The key is to remember that these tools are amplifiers of human insight, not replacements for it. The goal is to spend less time on mechanical test-writing and more time on the kinds of deep, thoughtful testing that actually prevents bugs.

The future of testing isn't about perfect coverage - it's about perfect understanding of what your code actually does, and having the confidence that comes from knowing you've validated the behaviors that matter most.

Edits:

2026-02-07: Added table of contents with anchor links

Understanding Python's `copy` vs `deepcopy` - When to Use Each

2025-03-20T00:00:00+01:00

When working with Python objects, understanding how to properly copy them is crucial for avoiding unexpected behaviors in your code. Python provides two main approaches for copying objects: copy and deepcopy. Let's explore the differences, use cases, and potential pitfalls of each.

The Basics: Shallow vs. Deep Copying

Python's copy module provides two primary functions:

copy.copy() - Creates a shallow copy
copy.deepcopy() - Creates a deep copy

The difference lies in how they handle nested objects.

Shallow Copy (`copy.copy()`)

A shallow copy creates a new object, but doesn't create copies of nested objects - it just copies references to them. This means changes to nested objects in the copy will affect the original, and vice versa.

Deep Copy (`copy.deepcopy()`)

A deep copy creates a completely independent clone - it recursively copies all nested objects, creating a fully separate hierarchy of objects.

Let's See It In Action

Here's a practical example showing the difference:

import copy

# Let's create a nested list
original = [1, 2, [3, 4]]

# Create a shallow copy
shallow_copied = copy.copy(original)

# Create a deep copy
deep_copied = copy.deepcopy(original)

# Let's modify the nested list in the original
original[2][0] = 'changed!'

print("Original:", original)
print("Shallow copy:", shallow_copied)
print("Deep copy:", deep_copied)

Output:

Original: [1, 2, ['changed!', 4]]
Shallow copy: [1, 2, ['changed!', 4]]
Deep copy: [1, 2, [3, 4]]

Notice how changing the nested list in the original affected the shallow copy but not the deep copy!

Typical Use Cases

When to Use Shallow Copy (`copy`)

Performance Matters: When dealing with large objects where a deep copy would be expensive, and you're confident you won't modify nested objects.
Simple Data Structures: When your object contains only immutable values like numbers, strings, or tuples.

import copy

# Dictionary with simple values
user_settings = {
    "theme": "dark",
    "notifications": True,
    "volume": 75
}

# Safe to use shallow copy here
backup_settings = copy.copy(user_settings)

When to Use Deep Copy (`deepcopy`)

Complex Nested Objects: When working with objects that contain mutable objects like lists, dictionaries, or custom classes.
When Independence is Critical: When you need to ensure modifications to the copy don't affect the original at all.

import copy

# Complex nested structure representing a user profile
user_profile = {
    "name": "Alex",
    "preferences": {
        "theme": "dark",
        "notifications": ["email", "push"]
    },
    "friends": [
        {"name": "Taylor", "status": "online"},
        {"name": "Jordan", "status": "offline"}
    ]
}

# We need a true independent copy to modify
new_profile = copy.deepcopy(user_profile)
# Now we can safely modify nested lists and dictionaries
new_profile["friends"][0]["status"] = "away"
new_profile["preferences"]["notifications"].append("sms")

print("Original friend status:", user_profile["friends"][0]["status"])  # Still "online"
print("Copy friend status:", new_profile["friends"][0]["status"])  # "away"

Common Gotchas and Pitfalls

1. Forgetting the import

Remember to import copy before using these functions!

2. Assignment Is Not Copying

# This is NOT a copy - it's just another reference to the same object
list2 = list1

3. List Slicing Creates Shallow Copies

original = [1, 2, [3, 4]]
sliced_copy = original[:]  # Equivalent to copy.copy()

original[2][0] = "changed!"
print(sliced_copy)  # Will show [1, 2, ['changed!', 4]]

4. Performance Considerations

Deep copying can be resource-intensive for large nested structures:

import copy
import time

# Create a large nested structure
large_structure = [list(range(1000)) for _ in range(1000)]

# Time shallow copy
start = time.time()
shallow = copy.copy(large_structure)
print(f"Shallow copy took: {time.time() - start:.6f} seconds")

# Time deep copy
start = time.time()
deep = copy.deepcopy(large_structure)
print(f"Deep copy took: {time.time() - start:.6f} seconds")

5. Circular References

Be careful with circular references when using deepcopy:

import copy

# Create a circular reference
circular = [1, 2, 3]
circular.append(circular)  # The list contains itself!

# This works fine, handling the circular reference properly
deep_circular = copy.deepcopy(circular)

Custom Copying Behavior

You can customize how your objects are copied by implementing __copy__ and __deepcopy__ methods:

import copy

class Person:
    def __init__(self, name, address):
        self.name = name
        self.address = address

    def __copy__(self):
        # Create a new instance and copy attributes
        return Person(self.name, self.address)

    def __deepcopy__(self, memo):
        # Create a new instance with deeply copied attributes
        return Person(self.name, copy.deepcopy(self.address, memo))

    def __repr__(self):
        return f"Person(name={self.name}, address={self.address})"

# Example usage
person = Person("Alice", {"city": "New York", "zip": "10001"})
person_copy = copy.copy(person)
person_deepcopy = copy.deepcopy(person)

# Modify the address in the original
person.address["city"] = "Boston"

print(person)  # Shows updated city
print(person_copy)  # Also shows updated city (shallow copy)
print(person_deepcopy)  # Still shows "New York" (deep copy)

Immutable vs. Mutable Objects: Understanding this fundamental Python concept helps clarify when copying is necessary.
Object References in Python: Diving deeper into how Python handles object references.
Memory Management: Learning how Python allocates and deallocates memory can help you make better choices about copying.
The pickle Module: For serializing and deserializing Python objects, another approach to object copying.
Performance Optimization: Strategies for efficient copying when working with large data structures.

References

Key takeaways

Understanding the distinction between shallow and deep copying is essential for writing robust Python code that behaves as expected. Choose copy() when you need a quick, lightweight duplication of simple structures, and deepcopy() when you need complete independence between the original and copied objects.

Tracking Down zsh Alias Plugin Sources

2025-02-28T00:00:00+01:00

When hunting for the origin of mysteriously defined by unknown plugin zsh aliases:

zsh -xv 2>&1 | grep "alias_name"

This works by:

Starting zsh with -x (trace) and -v (verbose) flags
Redirecting both stdout and stderr (2>&1) to capture all output
Filtering with grep to find when your alias is defined

For a more targeted approach with less output:

zsh -xv 2>&1 | grep -A 3 "source.*plugin" | grep "alias_name"

This helps identify which plugin file is sourcing your alias.

Guide to Managing VS Code Keyboard Shortcuts

2025-02-11T00:00:00+01:00

I've been using VS Code for years, and keyboard shortcuts can definitely get messy, especially when you have multiple extensions installed. Let me break this down for you.

The interesting thing about VS Code shortcuts is that they can indeed be context-dependent. This means the same keyboard combination might do different things depending on whether you're in a Python file, Jupyter notebook, or even the integrated terminal.

How VS Code Handles Shortcuts

VS Code uses a system of "when clauses" to determine which shortcut should fire in a given context. Think of it like a priority system where the more specific context wins. For example:

A shortcut that works only in Python files (when: "editorLangId == 'python'")
A shortcut that works in any text editor (when: "editorFocus")
A global shortcut that works everywhere

Finding and Resolving Conflicts

Here's how you can investigate and fix shortcut conflicts:

Open the Keyboard Shortcuts editor:
Press Cmd+K Cmd+S (Mac) or Ctrl+K Ctrl+S (Windows/Linux)
Or go to Code > Preferences > Keyboard Shortcuts
Type Cmd+L (or whatever shortcut you're investigating) in the search bar. This will show you all commands using that combination.
Look for overlapping shortcuts. If you see multiple entries, that's your conflict!

Fixing Conflicts

You have several options:

Change the shortcut for the command you use less frequently
Add a more specific "when" clause to make the shortcuts context-dependent
Disable one of the conflicting shortcuts

To modify a shortcut: 1. Find it in the Keyboard Shortcuts editor 2. Click the pencil icon 3. Press your new desired key combination 4. Press Enter to save

Context Matters

Different shortcuts can be active in different contexts like: - editorFocus (when text editor has focus) - terminalFocus (when terminal is focused) - notebookEditorFocus (in Jupyter notebooks) - editorLangId == 'python' (in Python files specifically)

Want to see what context you're in? Try the "Developer: Toggle Keyboard Shortcuts Troubleshooting" command. It'll show you active contexts as you work.

Pro Tips

Back up your custom shortcuts! They're stored in keybindings.json, which you can access through the Keyboard Shortcuts editor.
Use the "Show Conflicts" option in the Keyboard Shortcuts editor to quickly spot problematic bindings.
Extensions can add their own keybindings. Check their documentation or the Keyboard Shortcuts editor to see what they've added.

References

Simple In-Memory Knowledge Graphs for Quick Graph Querying

2025-01-16T00:00:00+01:00

NetworkX - The Python Swiss Army Knife
RDFLib - When You Need Semantic Power
PyGraphviz - Visualization with Query Capabilities
DIY Solution - Custom Graph Structure
Making the Right Choice
Performance Considerations
Beyond Simple Solutions
A Note on Automated Graph Construction
Wrapping Up
Further Reading, References

Working with knowledge graphs doesn't always require Neo4j or other heavyweight solutions. Sometimes you need a lightweight way to represent and query graph data right in memory. Let me share some approachable solutions I've found particularly useful.

NetworkX - The Python Swiss Army Knife

NetworkX has been my reliable companion for simple graph operations. It's incredibly intuitive and perfect for smaller knowledge graphs:

import networkx as nx

# Create a knowledge graph
G = nx.Graph()

# Add some knowledge
G.add_edge("Alice", "knows", "Bob")
G.add_edge("Bob", "works_at", "TechCorp")

# Simple queries
def find_connections(G, person):
    return [(n, G[person][n]) for n in G[person]]

You can test it with this example:

# Test NetworkX
print(find_connections(G, "Alice"))
print(find_connections(G, "Bob"))
print("\n")

The output is:

NetworkX Example:
[('Bob', {'relationship': 'knows'})]
[('Alice', {'relationship': 'knows'}), ('TechCorp', {'relationship': 'works_at'})]

RDFLib - When You Need Semantic Power

If you're dealing with semantic data and need SPARQL-like querying, RDFLib provides a perfect middle ground:

from rdflib import Graph, Literal, RDF, URIRef

# Create an in-memory graph
g = Graph()

# Add triples
g.add((URIRef('Alice'), URIRef('knows'), URIRef('Bob')))

# Query using SPARQL
qres = g.query(
    """SELECT ?s ?o
       WHERE {
          ?s knows ?o .
       }""")

for row in qres:
 print(f"{row.s} -> {row.o}")
 print("\n")

The output is:

RDFLib Example:
Bob -> TechCorp
Alice -> Bob

PyGraphviz - Visualization with Query Capabilities

When you need both visualization and querying use pygraphviz:

import pygraphviz as pgv

G = pgv.AGraph()
G.add_edge("Alice", "Bob", relationship="knows")

def find_relationships(G, node):
    return [e for e in G.edges() if node in e]

NOTE: There might be an problem when installing pygraphviz in Google Colab, you can use matlplotlib + networkx instead

DIY Solution - Custom Graph Structure

Sometimes, a custom solution fits best:

from collections import defaultdict
from typing import List, Dict, Any

class SimpleKG:
    def __init__(self):
        self.graph = defaultdict(dict)

    def add_relation(self, subject: str, predicate: str, object: str):
        if predicate not in self.graph[subject]:
            self.graph[subject][predicate] = []
        self.graph[subject][predicate].append(object)

    def query_by_subject(self, subject: str) -> Dict[str, List[str]]:
        return self.graph.get(subject, {})

    def get_connected_nodes(self, node: str) -> List[str]:
        connected = []
        for predicate, objects in self.graph.get(node, {}).items():
            connected.extend(objects)
        return connected

    def find_paths(self, start: str, end: str, max_depth: int = 4) -> List[List[str]]:
        if start == end:
            return start

        visited = set()
        queue = [(start, [start])]
        paths = []

        while queue:
            (vertex, path) = queue.pop(0)
            connected_nodes = self.get_connected_nodes(vertex)

            for next_node in connected_nodes:
                if next_node == end:
                    paths.append(path + [next_node])
                elif next_node not in visited and len(path) < max_depth:
                    visited.add(next_node)
                    queue.append((next_node, path + [next_node]))

        return paths

    def find_by_predicate(self, predicate: str) -> List[tuple]:
        results = []
        for subject, predicates in self.graph.items():
            if predicate in predicates:
                for obj in predicates[predicate]:
                    results.append((subject, obj))
        return results

    def find_connected_through_predicate(self, node: str, predicate: str) -> List[str]:
        return self.graph.get(node, {}).get(predicate, [])

Here is simple example how you can test it:

# Test the implementation
kg = SimpleKG()
kg.add_relation("Alice", "knows", "Bob")
kg.add_relation("Bob", "works_at", "TechCorp")
kg.add_relation("TechCorp", "located_in", "San Francisco")

print("Query by subject 'Alice':", kg.query_by_subject("Alice"))
print("Find paths from Alice to TechCorp:", kg.find_paths("Alice", "TechCorp"))

The output is:

Query by subject 'Alice': {'knows': ['Bob']}
Find paths from Alice to TechCorp: 'Alice', 'Bob', 'TechCorp'

More advanced example:

# Create sample data
kg = SimpleKG()

# Movies data
movies = [
    "Inception", "The Dark Knight", "Interstellar", "Dunkirk",
    "Memento", "The Prestige", "Tenet", "Fight Club", "Se7en",
    "The Social Network", "Gone Girl", "Panic Room"
]

directors = [
    "Christopher Nolan", "David Fincher", "Martin Scorsese",
    "Quentin Tarantino", "Steven Spielberg"
]

actors = [
    "Leonardo DiCaprio", "Christian Bale", "Matthew McConaughey",
    "Brad Pitt", "Tom Hardy", "Marion Cotillard", "Michael Caine",
    "Anne Hathaway", "Cillian Murphy", "Joseph Gordon-Levitt",
    "Ellen Page", "Jesse Eisenberg", "Ben Affleck", "Rosamund Pike"
]

# Add relationships
# Directors directed movies
movie_director = {
    "Inception": "Christopher Nolan",
    "The Dark Knight": "Christopher Nolan",
    "Interstellar": "Christopher Nolan",
    "Dunkirk": "Christopher Nolan",
    "Memento": "Christopher Nolan",
    "The Prestige": "Christopher Nolan",
    "Tenet": "Christopher Nolan",
    "Fight Club": "David Fincher",
    "Se7en": "David Fincher",
    "The Social Network": "David Fincher",
    "Gone Girl": "David Fincher",
    "Panic Room": "David Fincher"
}

for movie, director in movie_director.items():
    kg.add_relation(movie, "directed_by", director)
    kg.add_relation(director, "directed", movie)

# Add actors to movies (random assignment for demonstration)
movie_actors = {
    "Inception": ["Leonardo DiCaprio", "Tom Hardy", "Marion Cotillard", "Michael Caine", "Ellen Page", "Joseph Gordon-Levitt"],
    "The Dark Knight": ["Christian Bale", "Michael Caine", "Cillian Murphy"],
    "Interstellar": ["Matthew McConaughey", "Anne Hathaway", "Michael Caine"],
    "Fight Club": ["Brad Pitt"],
    "The Social Network": ["Jesse Eisenberg"],
    "Gone Girl": ["Ben Affleck", "Rosamund Pike"]
}

for movie, cast in movie_actors.items():
    for actor in cast:
        kg.add_relation(movie, "stars", actor)
        kg.add_relation(actor, "acted_in", movie)

# Add some awards
awards = ["Oscar", "Golden Globe", "BAFTA"]
for director in directors[:3]:
    for award in random.sample(awards, random.randint(1, len(awards))):
        kg.add_relation(director, "won", award)

for actor in actors[:6]:
    for award in random.sample(awards, random.randint(0, len(awards))):
        kg.add_relation(actor, "won", award)

# Example queries
print("1. Find all movies directed by Christopher Nolan:")
nolan_movies = kg.find_connected_through_predicate("Christopher Nolan", "directed")
print(nolan_movies)
print()

print("2. Find actors who worked with Christopher Nolan (through any movie):")
nolan_actors = set()
for movie in nolan_movies:
    actors = kg.find_connected_through_predicate(movie, "stars")
    nolan_actors.update(actors)
print(list(nolan_actors))
print()

print("3. Find path between Leonardo DiCaprio and Christopher Nolan:")
paths = kg.find_paths("Leonardo DiCaprio", "Christopher Nolan")
print("Found paths:")
for path in paths:
    print(" -> ".join(path))
print()

print("4. Find Oscar winners:")
oscar_winners = kg.find_by_predicate("won")
print([winner[0] for winner in oscar_winners if winner[1] == "Oscar"])
print()

print("5. Find common movies between Michael Caine and Leonardo DiCaprio:")
caine_movies = set(kg.find_connected_through_predicate("Michael Caine", "acted_in"))
dicaprio_movies = set(kg.find_connected_through_predicate("Leonardo DiCaprio", "acted_in"))
print(list(caine_movies & dicaprio_movies))

Output:

1. Find all movies directed by Christopher Nolan: 
['Inception', 'The Dark Knight', 'Interstellar', 'Dunkirk', 'Memento', 'The Prestige', 'Tenet'] 

2. Find actors who worked with Christopher Nolan (through any movie):
['Christian Bale', 'Michael Caine', 'Leonardo DiCaprio', 'Joseph Gordon-Levitt', 'Cillian Murphy', 'Anne Hathaway', 'Matthew McConaughey', 'Marion Cotillard', 'Ellen Page', 'Tom Hardy'] 

3. Find path between Leonardo DiCaprio and Christopher Nolan: 
Found paths: 
Leonardo DiCaprio -> Inception -> Christopher Nolan 
Leonardo DiCaprio -> Inception -> Michael Caine -> The Dark Knight -> Christopher Nolan 
Leonardo DiCaprio -> Inception -> Michael Caine -> Interstellar -> Christopher Nolan 

5. Find Oscar winners: 
['Leonardo DiCaprio', 'Tom Hardy', 'Marion Cotillard', 'Christian Bale', 'Matthew McConaughey', 'Brad Pitt', 'Martin Scorsese'] 

5. Find common movies between Michael Caine and Leonardo DiCaprio: 
['Inception']

Making the Right Choice

The best solution depends on your specific needs:

Use NetworkX for general graph operations and algorithms
Choose RDFLib when working with semantic data and SPARQL
Go with PyGraphviz when visualization is important
Consider a custom solution for specialized query patterns

Performance Considerations

These solutions work well for graphs with thousands of nodes and edges. The key is keeping everything in memory and optimizing your query patterns. For NetworkX and RDFLib, using their built-in query methods is usually faster than writing custom traversal code.

Beyond Simple Solutions

When your knowledge graph grows beyond memory constraints or you need more complex querying capabilities, it might be time to consider solutions like Neo4j or Amazon Neptune. However, for many use cases, these in-memory solutions provide the perfect balance of simplicity and functionality.

A Note on Automated Graph Construction

Building knowledge graphs by hand, as shown in our examples, is straightforward. However, automatically constructing them from documents or unstructured data is a complex challenge worthy of its own article. Here are some key challenges you'll face:

Entity recognition and disambiguation is perhaps the trickiest part - determining whether "Apple" refers to the fruit or the company, or whether two mentions of "John Smith" refer to the same person. You'll need to handle coreference resolution (understanding that "he" refers to "John" mentioned earlier) and deal with variations in how entities are written ("NYC" vs "New York City").

Relationship extraction comes with its own set of problems. Natural language is complex and often implicit - extracting clear, structured relationships from sentences like "After years at Microsoft, Sarah brought her expertise to the startup" requires sophisticated NLP techniques.

Data quality and consistency are also major concerns. Sources might conflict with each other, contain outdated information, or present opinions as facts. You'll need strategies for handling uncertainty and conflicting information in your graph.

If you're interested in automatic graph construction, I'd recommend starting with established NLP libraries and knowledge graph toolkits rather than building everything from scratch. But that's a topic for another deep dive!

Wrapping Up

Don't jump to complex graph databases when simpler solutions might suffice. These in-memory approaches can handle surprisingly complex tasks while keeping your codebase clean and maintainable. Plus, they're perfect for prototyping before committing to a full-scale graph database solution.

Quick Ways to Disable GitHub Actions Workflows Without Deletion

2024-10-03T00:00:00+02:00

GitHub Actions workflows are powerful automation tools, but sometimes you need to temporarily disable them. Here are three simple methods to pause a workflow without deleting its YAML file:

Comment out the file: Add '#' at the start of each line in the workflow file.
Use manual triggers: Replace existing triggers with on: workflow_dispatch.
Add a false condition: Insert if: false under the jobs key.

Example of method 3

name: My Workflow

on:
  push:
    branches: [main]

jobs:
  if: false  # This disables the entire workflow
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      # ... rest of the job steps

Krystian Safjan's Blog

The Real Cost of Model Migration - What Swapping LLMs Actually Requires

Table of contents

What migration actually is

The model options in 2026

Standard instruction models

Pure reasoning models

Phase 0 -Know what you're migrating from

Inventory your integration surface

Document implicit behavioral assumptions

Get a pre-migration quality snapshot

Phase 1 - Build your evaluation harness first

Build a golden dataset

What to evaluate

Evaluation tooling

Define pass/fail gate criteria

Phase 2 - The prompt portability problem

Written prompts vs tuned prompts

Why tuned prompts are so common

What to do about it

Phase 3 - Automated prompt optimization with DSPy

What DSPy actually does

Translating your pipeline into DSPy

Choosing an optimizer

The data split that matters

DSPy in migration: tool vs framework

DSPy limitations to know before committing

Phase 4 - Reasoning models: where they belong

The RAG pipeline with reasoning model placement

Slot 1: Query decomposition - strong fit, highest ROI

Slot 2: Context reranking and conflict detection - strong fit

Slot 3: Answer synthesis on complex documents - maybe

Slot 4: Agentic orchestration - good fit, underused

Slot 5: Faithfulness verification - high value, underused

Phase 5 - Handling parameter differences

Temperature and reasoning_effort

The max_completion_tokens trap

Prompt verbosity inversion with reasoning

Phase 6 - Risk assessment

Risk catalog

Phase 7 - Progressive rollout

Shadow mode first

Gate criteria at each step

Traffic segmentation for canary rollout

Keep rollback ready for a week

Phase 8 - Post-migration monitoring

Pin your model version string

Ongoing eval cadence

What your observability actually needs to cover

The systems audit you should run regardless

Further reading

The illusion of control in AI-assisted engineering

The illusion of control in AI-assisted engineering

The distinction nobody separates

What governance theater actually looks like

How responsibility diffuses

The slower risks

Reframing the question

Questions worth sitting with

Version Your Vectors - Index Versioning as the Missing Layer in RAG Observability and Compliance

1. The invisible problem: why RAG systems silently drift

2. Use cases that justify index versioning

2.1 Debugging and root cause analysis

2.2 Data drift detection

2.3 Regulatory compliance and audit

2.4 Reproducibility for evaluation

2.5 Incident response and rollback

2.6 Counterfactual replay

3. What exactly needs versioning?

4. Regulatory context: when versioning becomes mandatory

4.1 EU AI Act risk classification for RAG

4.2 What the regulation actually requires

4.3 Proportional versioning: match effort to risk

5. Solution architectures: from lightweight to full replay

5.1 Level 0: metadata tagging (minimum viable versioning)

5.2 Level 1: content hashing and change detection

5.3 Level 2: index snapshots with point-in-time recovery

5.4 Level 3: full replay architecture

5.5 Level 4: self-healing index with drift detection

6. Implementing observability around index versions

The `max_completion_tokens` trap