Hands-on architects Blog RSS Feed

Agent Harness vs Work Harness: One Term, Two Systems, Two Owners

Tue, 14 Jul 2026 00:00:00 GMT

💡 Deciding where the AI budget goes rather than writing the prompts? Jump to the TL;DR section for the executive summary.

Ask three engineers what an AI "harness" is and you will get three confident, incompatible answers. The vendor engineer means the orchestration loop and sandbox their product ships with. The framework author means every line of code that is not the model. The developer down the hall means the CLAUDE.md and the custom linters in their repository. All three are right, which is exactly the problem - the term is going through what Martin Fowler long ago called semantic diffusion.

The cleanest anchor comes from LangChain's Vivek Trivedy: Agent = Model + Harness. "If you're not the model, you're the harness" - every piece of code, configuration, and execution logic that isn't the model itself. Birgitta Böckeler adopted that equation and immediately flagged it as "a very wide definition, and therefore worth narrowing down for common categories of agents." Addy Osmani arrived at the same place independently: a coding agent is the model plus everything you build around it.

We agree with the equation, but the narrowing is where the useful insight lives. Most published material obsesses over the inner layer - the layer you mostly buy; the outer layer is the one you build, and it is where your leverage is. In this post we split the harness into those two nested systems with two different owners - the agent harness and the work harness - walk through what lives in each, and then use that vocabulary to revisit the Delivery cluster of our Harness Model (in short words the execution - how work gets done). The punchline: the jump from Stage 3 to Stage 4+ is not a smarter agent. It is a change in which harness you, the human, are talking to.

Two Harnesses, Nested

Peel the buzzword apart and two layers fall out - with two different builders.

The agent harness is the control layer the vendor ships around the model. It contains:

System prompts - the behavioral baseline you never see
Orchestration - subagent spawning, handoffs, model routing
Building-block interfaces - tools, skills, hooks, MCP support, and their descriptions
Bundled infrastructure - filesystem access, sandboxes, memory, browser
The user interface - terminal, IDE plugin, web app

This component list converges across sources that have no reason to agree: LangChain's anatomy, Osmani's restatement (which adds observability), and the architecture taxonomy in a 138-talk practitioner study all describe essentially the same machinery. Anthropic says it plainly: "The Claude Agent SDK is a powerful, general-purpose agent harness adept at coding..."

The work harness is everything your team builds on top of that, for your codebase and your delivery process: AGENTS.md and CLAUDE.md files, skills, MCP servers and CLIs, hooks, the tests and linters wired into the agent loop, and the workflow definitions that string them together. The vendor cannot ship this layer - it does not know your module boundaries, your compliance constraints, or your definition of done. Stripe's minions are the textbook example: "our custom minion harness tightly integrates with that tooling" - the developer-productivity stack Stripe built for its humans - because "if it's good for humans, it's good for LLMs, too."

Terminology, openly. "Work harness" is our coinage - no published source uses it. Böckeler calls this outer layer the user harness (also "outer harness"), and her name for the inner layer is the builder harness; the Zenn writeup that names the split most cleanly calls the two layers Agent Harness and User Harness. We prefer "work harness" because it names the work, not the user - the layer exists to make a specific body of work deliverable, regardless of who triggers it.

One boundary case before the picture: teams that build their own orchestration - with LangGraph or similar - own both layers, and the test is ownership, not vendor. Whatever your team builds and iterates on is your work harness; what you consume as shipped is the agent harness.

The layers nest: the model sits inside the agent harness, and both sit inside your work harness.

Most of what is published today - benchmarks, orchestration tricks, context management - covers the inner layer. Useful, but as we said up front: that is the part you purchase. The work harness is the part you engineer, and the rest of this post is about how.

Inside the Work Harness: Guides and Sensors

Böckeler's decomposition of the outer harness is the canonical one, and we have not found a better axis. Everything in the work harness is either a guide or a sensor:

Guides (feedforward controls) steer the agent before it acts. They raise the probability that the first attempt is right.
Sensors (feedback controls) observe after the agent acts and feed the result back, so the agent can self-correct before a human ever looks.

Each comes in two execution flavors: computational - deterministic, fast, CPU-run - and inferential - LLM-run, slower, more expensive, non-deterministic. That gives a practical 2×2:

	Computational (deterministic, CPU)	Inferential (LLM-run)
Guides - steer before the act	Bootstrap scripts, codemods, language servers	`AGENTS.md` conventions, skills, reference docs
Sensors - observe after the act	Linters, type checks, tests, module-boundary checks	Review agents, LLM-as-judge, semantic duplication checks

You need both directions. As Böckeler puts it: "you get either an agent that keeps repeating the same mistakes (feedback-only) or an agent that encodes rules but never finds out whether they worked (feed-forward-only)." A well-built work harness serves two goals - it increases the probability the agent gets it right the first time, and it self-corrects as many issues as possible before they reach human eyes.

The two controls wrap the agent from both sides - guides steering the input, sensors observing the output - with the human tuning both:

Where do workflow definitions - slash commands, pipelines, triggers - fit? They are a component of the work harness that sequences guides and sensors into a repeatable process, not a third axis. That distinction matters later, because workflow definitions are exactly what migrates as teams mature.

We have already built one of these in public: our harness engineering for E2E tests post is a worked example of a guides-and-sensors stack - scenario corpora as guides, pytest-in-Docker plus reviewer agents as computational and inferential sensors.

The Steering Loop

If guides and sensors are the parts, the steering loop is the operating principle. Böckeler:

"The human's job in this is to steer the agent by iterating on the harness. Whenever an issue happens multiple times, the feedforward and feedback controls should be improved to make the issue less probable to occur in the future, or even prevent it."

This is the same practice Mitchell Hashimoto described as engineering the harness so the agent "never makes that mistake again" - the founding move of the Harness Model. Your job is not to babysit the agent through each task. Your job is to notice the second occurrence of a mistake and turn it into a guide or a sensor. That is the loop the diagram above closes: recurring issues flow back to the human, who hardens the guides and sensors.

Two warnings come with the loop. First, it never finishes - Böckeler observes that building the outer harness "is emerging as an ongoing engineering practice, not a one-time configuration." Second, stopping early is the most common failure. The Zenn article makes the point sharply: if your work harness is CLAUDE.md plus some skills plus a few hooks, you are missing the computational sensors - linting, type checks, verification loops in the agent's path - and you are missing measurement. Can you demonstrate, with data, that your harness actually works?

Guides without sensors is vibes with extra files.

Driving the Agent vs Running the Work Engine

Now the payoff, and a part we want to flag honestly: what follows is our own synthesis. No published source maps the harness layers onto a staged maturity ladder - what the literature supports is the steering loop underneath it. We think the mapping earns its keep.

In the Harness Model, the early stages share one trait across the Delivery cluster: whenever a human works with AI at all, they interact with the agent harness. You open the terminal or the chat window, you prompt, you watch, you correct. Your prompt is the workflow, and it evaporates when the session ends.

At Stage 4+, the human's hands move one layer out. You interact with the work harness: you trigger a versioned workflow, the work harness drives the agent through guides and sensors, and what comes back has already survived the feedback loop. The whole stack - work harness wrapped around the agent - becomes what we call a work engine.

"Work engine" is also our coinage - but not from thin air. LangChain's own framing is that harness engineering exists to turn models into "work engines." We are extending their phrase one layer up: the model becomes a work engine only when your harness, not just the vendor's, wraps it.

The shift in one sentence: at Stage 1-3 your prompt is the workflow; at Stage 4+ the workflow is versioned in the repo and you trigger it. That is not a model upgrade. It is a change of interface - and of what you spend your day improving.

Walking the Delivery Cluster

The two-harness vocabulary makes the three Delivery dimensions of the Harness Model concrete.

If you have not read that post: the model spans five stages, from No AI Process through Chatbot-Assisted and Human-in-the-Loop to Systematic Harness and Agentic Flywheel. Across the Delivery cluster, the early stages mean that whenever a human works with AI at all, their hands are on the agent harness - prompt, watch, correct - while at Stage 4+ they move to the work harness and trigger versioned workflows.

Human-Agent Interaction

Kief Morris's loop progression - outside the loop, in the loop, on the loop, flywheel - restates cleanly: "on the loop" literally means your hands move from the agent harness to the work harness. And the reason you cannot skip ahead is documented. Böckeler's catalogue of coding-agent steering found that even in successful sessions, "intervention, correction and steering happened all the time" - what she calls "supervised agent" mode, exactly the Stage 1-3 territory of our model. Until your guides and sensors absorb that steering, a human has to supply it live, on the agent harness.

Workflow & Process

Maturing here means workflow definitions migrate from ad-hoc prompts into the work harness. Stripe is the Stage 4+ exhibit: "A typical minion run starts in a Slack message and ends in a pull request which passes CI and is ready for human review, with no interaction in between" - over a thousand merged PRs per week, with deterministic lint and test steps interleaved into the agent loop so required checks always run. OpenAI's harness engineering post compresses the same operating model into four words - "Humans steer. Agents execute." - and notes they have "pushed almost all review effort towards being handled agent-to-agent." Review itself became a sensor inside the work harness.

Reliability & Operations

Unattended operation is only trustworthy if the work harness carries state and verification, because the agent harness alone cannot. Anthropic's long-running agent pattern is the machinery: a specialized initializer agent sets up an init.sh, a claude-progress.txt log, and an initial git commit; every later session reads the progress file and git history, verifies the app still runs, and commits incremental work. Their motivating finding - "compaction isn't sufficient" - is the reliability argument in miniature: durable state lives in the filesystem and git, which are work-harness artifacts, not in the agent harness's context tricks. This is the same investment as the runbook and alert-handling axes in the matrix: agents earn operational trust through structure, not optimism.

It's Engineering, Not Model Magic

If this sounds like a lot of unglamorous systems work - it is, and the skeptical evidence says that is precisely the point. A study of 138 practitioner talks on deployed agent systems concludes: "allocate most of your resources to integrating tools and APIs, designing memory and workflow layers, implementing orchestration, and setting up observability and debugging. These engineering tasks dominate development time more than model behavior." (Fair caveat: that is the authors' qualitative takeaway from self-selected talks, not a measured-hours figure.)

The 2025 DORA report lands the organizational version of the same finding. It warns that existing delivery bottlenecks will "neutralize any gains from AI", because "Every line of AI-generated code still needs to go through your review process, your test suite, and your deployment pipeline." The teams positioned to capture the value are the ones that treat internal platforms as real products with dedicated ownership. Pipelines, tests, fast feedback, governance - DORA does not say "work harness," but it is describing one.

Our conclusion from both: the agent harness is necessary, and the vendors are racing each other to build it for you. The work harness is what gates the value, and nobody is going to build it for you.

Start Building Your Work Harness

The vocabulary in one breath: three layers (model, agent harness, work harness), two owners (the vendor builds the inner layers, your team builds the outer), one loop (steer the agent by iterating on guides and sensors). And one marker of maturity: which harness your humans touch all day.

Two things to do this week:

Audit the interface. Watch where your team's hands actually are. Mostly prompting, watching, and correcting in a terminal? You are on the agent harness - pre-Stage-4 territory, and that is fine, as long as it is a deliberate stop and not a permanent address.
Convert one recurring mistake. Pick the agent error your team has corrected at least twice this month and turn it into a guide (a rule, a skill) or a sensor (a linter, a check in the loop). That single move is the steering loop in practice.

And if you are earlier in the journey - no agent in your daily work yet, nothing to audit - start with our getting-started guide. The two moves above will still be here when you are ready.

We are collecting patterns across teams for the Q2 2026 update of the Harness Model - if your team has moved a workflow from prompts into a work harness (or tried and bounced off), we would like to compare notes. Email us at contact@handsonarchitects.com or DM either of us on LinkedIn: Maciej · Tomasz.

TL;DR

"Harness" is one word for two different systems. The agent harness is the control layer that ships with the product - orchestration, tools, sandboxes - and the vendors are racing each other to build it; whatever edge it gives you, your competitors rent at the same price. The work harness is the layer your team builds on top: the conventions agents read before they act, the linters, tests, and review agents that check what comes out, and the versioned workflows that string them together. Nobody will build that layer for you - it encodes your module boundaries, your compliance constraints, your definition of done - and it is the part you own and the part that compounds.

The budget translation: most of the published excitement is about the layer you buy, but the returns are gated by the layer you build. The empirical evidence points the same way - across deployed agent systems, integration, workflow, and observability engineering dominates development time more than model behavior, and DORA warns that existing delivery bottlenecks will "neutralize any gains from AI": every AI-generated line still passes through your review process, your test suite, and your deployment pipeline. Funding the work harness largely means funding those.

A walk-the-floor diagnostic for your next review: watch where your engineers' hands are. If the day is spent prompting, watching, and correcting agents in a terminal, the prompt is the workflow and it evaporates with the session - fine as a stage, expensive as a permanent address. The maturity marker is engineers triggering versioned workflows whose output has already survived automated checks before a human looks at it. That jump is not a model upgrade you can purchase. It is an interface change you have to engineer.

Two questions to ask your leads this month: which harness do your hands touch all day? and which recurring agent mistake did we convert into a guide or a sensor? Then place your team on the Delivery rows of our maturity matrix.

Team Topologies in the AI Era: Where Does the Cognitive Load Go?

Tue, 30 Jun 2026 00:00:00 GMT

💡 Running an engineering org rather than a team? Jump to the TL;DR section for the executive summary.

At the Pragmatic Summit, Martin Fowler asked the question every technology leader reorganizing around AI is circling: "Are we seeing two-pizza teams becoming one-pizza teams because agents don't eat pizza, or do we see two-pizza teams staying and becoming much more effective and capable? My bet is on more effective two-pizza teams." It sounds like a question about headcount (since two-pizza team is 5-8 people, then one-pizza team would be 3-4?). We think it is really a question about where the cognitive load goes.

Team Topologies is usually read as a book about org design. Its engine is something narrower and more useful: cognitive load as a finite, per-team budget. Every team type the book defines is a different way to keep one team's budget from overflowing. An AI harness is a new lever on exactly that budget - so in this post we re-read all four team types through one question: when the harness moves the cheap load, where does the expensive load end up?

An AI harness is a cognitive-load lever - which is what platform teams do

Terms first, because "harness" is a contested word. We use it the way Birgitta Böckeler does in Harness Engineering, and the way our governance post already does: a harness is everything in an AI agent except the model itself.

Agent = Model + Harness

The harness is the context the agent reads, the tools it can call, the guardrails and linters and tests that check its output, and the feedback loops that improve all of those over time.

Now the Team Topologies side. The book defines exactly four team types - stream-aligned, enabling, complicated-subsystem, and platform - and exactly three interaction modes: collaboration, X-as-a-Service, and facilitating. We built on this foundation before, in Services Architecture and Code Ownership. The part that gets lost in re-tellings is why the types exist: a team's cognitive capacity is finite, and when the domain it owns exceeds that capacity, delivery slows, quality drops, and people burn out. The other three types exist to absorb load off the stream-aligned team, so it can stay focused on its stream.

Read the harness's job description against that. It absorbs boilerplate, glue code, codebase search, and first-draft implementation off a stream-aligned team. That is the exact job description of a platform team. This is not just our analogy: the platform engineering community explicitly describes its own discipline as cognitive-load mitigation.

Which raises the tempting move: should the team that builds the org's harness be crowned a new, 4.5th team type? Team Topologies is emphatic that there are four types and no hybrids - and we think it is right. The "what type is it?" question is the wrong one. Ask instead: whose load does it reduce? A team that builds and operates the harness - context templates, agent workflows, eval loops, guardrails - is a platform team whose product happens to be the harness. It is consumed the way Team Topologies says platforms should be consumed: X-as-a-Service, behind a clear interface (a well-designed MCP server is exactly that interface - and, as we return to at the end, perhaps a Conway's Law force of its own), with governance baked into the harness rather than enforced in review - architecture tests that fail the build are the canonical example.

Here is the whole argument in one picture. Watch the return arrows - every load the harness platform takes off the stream team sends something back.

Stream-aligned: the Product Engineer question

The stream-aligned team is Team Topologies' default: five to nine people owning a slice of the product end to end. It is also the team type with the most at stake in the AI era, because it is the team the harness was built for. A wave of practitioner writing has already renamed its members - the Product Engineer, a generalist who, amplified by agents, spans design to code to operations.

So which way does Fowler's question break? Does the team shrink to fewer generalists - or hold its headcount and own more scope? Both forks are live, and we are not going to pretend the data settles it.

What we can say is what the surviving work looks like. Kent Beck, on the same Summit stage, described his experience of pairing - two humans plus a genie - and praised the genie's slowness: "When the AI goes away for three minutes, we can talk about our philosophy of naming, or how we express conditionals, or about what we should be doing next." That three-minute gap is not idle time. It is where the irreducible work happens: deciding what to build, judging what came back. The genie generates; the humans verify and steer.

Beck also supplies the brake on the strongest "smaller teams" reading: "AI is an amplifier." We made the same argument in How to Start using AI in Software Development - an amplifier does not subtract team members; it changes what each one carries. It also resolves a tension with Services Architecture and Code Ownership, where we argued that a stream team should own its services full-lifecycle: a leaner team does not break that argument, it concentrates it. The verification load is the new weight of ownership.

One more data point, from inside a 4,000-engineer AI-forward organization: Jamie Hurst reports that the compression runs upward. "The work that used to need a team now sometimes needs one person with the right tools, and that person tends to be senior because seniority is where the system-level understanding lives." The generalist who absorbs the team is senior - and Hurst is blunt that it is not free: "the role isn't sustainable at this pace." A field report of one, but worth sitting with.

Enabling: consultants who teach calibrated trust, then leave

The Team Topologies authors' 2025 guidance on interaction modes sharpened something the book always implied: collaboration is temporary by design, and facilitating is self-terminating - entry and exit criteria are part of the mode. That maps one-to-one onto a model we expect to see everywhere: a small group of consultants - internal or external - who introduce the harness to a stream team, then leave.

The scope matters. The job is not installing the harness; the job is teaching the team to carry the verification load the harness creates. "Calibrated trust" sounds soft, so make it concrete:

When to override the agent. Anywhere the work touches domain rules no sensor in the harness checks - correctness is outside the harness's remit when nobody specified it.
How to tell the harness itself is wrong. If the guardrails never fire, treat that as a smell, not a triumph: either quality is exceptional or detection is inadequate.
What honest verification looks like. "We merged it because the tests are green" is not verification when the same agent generated the tests.

The exit criterion writes itself: the enabling team leaves when the stream team can answer those three without them in the room.

Hurst's field report shows why this work needs an explicit owner. In his organization, mentoring and 1-2-1s were "the first thing to go" - precisely because they are the load AI does not absorb. Unless someone is accountable for transferring judgment, it silently disappears from everyone's calendar. The cheapest defense we know: write the engagement's exit date, exit criteria, and a named owner into the kickoff doc on day one - that single line keeps the facilitating mode self-terminating and the judgment transfer on someone's accountability list. We argued the upskilling half of this years ago in Just-in-time Knowledge Sharing; what that post stopped short of, and Team Topologies supplies, is the time-box - the exit date.

Complicated-subsystem: the AI-resistant boundary

If you had asked us in 2024 which team type AI would dissolve first, we would have guessed this one: specialists, absorbed by models trained on everything. The early evidence points the other way. A CHI 2026 study found AI assistance is weakest exactly where tacit, specialist knowledge matters - the settings where you cannot easily verify the output because the expertise lives in people's heads. One study, n=60, so treat it as directional rather than definitive. But it matches a nuance our own maturity matrix already states: agents provide technical depth on demand - frameworks, languages, patterns - not domain depth.

Re-read the complicated-subsystem team through the load lens. It exists so stream-aligned teams do not have to carry deep specialist load - the video codec, the pricing engine, the ML internals. The harness relocates generalist load; it cannot relocate tacit-specialist load, because tacit knowledge is precisely the kind you cannot write into a context file. So of the four types, this team's load budget is the least changed by AI. Everyone else's work got cheaper to generate; theirs did not.

That inverts the intuitive prediction: the complicated-subsystem team is not the first boundary to dissolve - it is the most durable one. There is a sharper, forward-looking version of this idea (if AI is weakest at tacit knowledge, is tacit knowledge now a reason to draw a boundary?), but it rests on that same single study, so we hold it for the open questions at the end.

The Stage 3→4 jump: load relocated, not removed

Time to name what we have been planting all along. In the Harness Model, our maturity matrix for AI engineering, the Team (Humans + Agents) dimension moves from Stage 3 - "Leaner delivery-minded team; agents use file/terminal" - to Stage 4 - "Small generalist team per initiative; agents run full stack." The matrix states the shape of the jump. This post is about its mechanism and its price: The harness does not delete cognitive load. It relocates it.

Generation gets cheap; verification, integration, and judgment get expensive. The CHI study gives the receipt: verification load is measurable and mediates fatigue - more checking means more tiredness, even while raw output climbs. The Stage 3→4 jump is that trade executed deliberately: the team can get leaner only because the platformizable load left - and it stays viable only if it can pay for the load that came back. Here is the stream team's ledger across the jump:

To make the ledger less abstract, picture a five-person stream team a quarter into the jump - a composite sketch, not a case study. Nobody on it hand-writes boilerplate, glue code, or first-draft tests anymore; the agents do. The hours that work used to fill now go to reviewing agent output before it merges, making the integration calls the agent cannot make, keeping the context files current enough that next week's drafts stay trustworthy, and learning. Same five people, same forty hours - every line item moved.

The matrix already says this in its own voice: "Teams that try to shrink without paying either cost end up with the same number of humans and less agent trust."

And here is where we push on our own model. The matrix commits to "small generalist team"; Fowler bets on more-effective same-size teams. We now think the next revision should assert less and explain more: hold the headcount open, assert the load relocation. The relocation is what the evidence supports; the headcount is downstream of where the relocated load lands, and that differs by team and by domain. The Q1 matrix is explicitly a snapshot built to be revised - consider this our first input to its next version.

Hurst compresses the whole thesis into one sentence: "The cost of building has collapsed, but the cost of aligning organisationally has not." One practitioner's report - but it is exactly what relocated load looks like from the inside.

The Software House angle: the harness is the IP

A deliberate gear-change here, from org design to economics - and if you are not in the agency business, stay anyway: everything below applies equally to any organization running many product teams on a similar stack. For a Software House, the relocation thesis has a commercial consequence. Relocated cognitive load does not evaporate - when relocation is done well, it gets captured: in context templates, eval loops, guardrails, agent workflows. Captured relocation is durable and reusable. The model is rented; the harness is owned.

The mechanism is well supported: harness design, not the model, drives outcomes. One benchmark cited by Hugging Face moved from 61.5% to 87.2% on the same model through harness design alone. Our own E2E test harness is the worked proof at small scale: sharper specs and a better harness cut commits per test step roughly 5× while CI time stayed flat - with the leverage in the spec, not the model.

The inference we draw from that - and it is our extrapolation, not a finding - is that a Software House accumulating proven harness templates starts every project ahead: faster delivery, more consistent quality, and an asset that survives model churn because a good harness re-targets to the next model. Before anyone turns that into a sales deck, two brakes:

Experience precedes platform. You cannot harness-platformize patterns you have not lived. The harness platform should be extracted from a few real engagements - thinnest-viable, demand-pulled, exactly the rule Team Topologies applies to every platform. Building it speculatively is premature optimization with a roadmap.
ROI scales with stack homogeneity. Harness templates behave like service templates: cheap to reuse across twenty similar Spring Boot services, expensive across twenty different stacks. A heterogeneous portfolio pays more to platformize and gets less back.

The economics are moving this way regardless. Hurst again: a piece of internal platform that previously served 4,000 humans "now potentially serves 4,000 humans plus an unbounded number of agent instances those humans deploy." The industry is starting to call this agent experience, and it is moving platform investment from team-level to board-level priority. A harness platform team is that investment with a name and a team type.

What we can't settle yet

Three questions we are deliberately leaving open - not because they are unanswerable, but because answering them today would be guessing:

The headcount fork. Do stream-aligned teams shrink, or hold headcount and own more scope? Fowler bets on more-capable two-pizza teams; our matrix currently bets on leaner ones.
Conway's Law with agents. If an agent is a communicating participant in building the system, does Conway's Law now describe human↔agent and agent↔agent communication - and is choosing a harness a reverse-Conway move, shaping architecture by shaping the agent topology? The matrix's "agents are first-class team members" principle is the on-ramp; nothing stronger than practitioner speculation exists yet.
Tacit knowledge as a fracture plane. If AI is weakest at tacit knowledge, does tacit knowledge become a place to draw team boundaries, not just defend them? We think so. Nobody has tested it, and the supporting evidence is one n=60 study.

All three are inputs to the next, post-Q1 version of the Harness Model. This post is meant to shape that model, not just apply it.

If you want to do something with this tomorrow: run each of your teams through the one question this post runs on - whose cognitive load does this team reduce, and what load flows back? Then self-assess on the Team dimension of the matrix. And if your answers disagree with ours - especially on the headcount fork - argue with us: email contact@handsonarchitects.com or DM either of us on LinkedIn: Maciej · Tomasz. The best version of this argument is the one we have not heard yet.

TL;DR

AI coding agents do not delete engineering work - they relocate it. Writing code gets cheap; checking it, integrating it, and deciding what to build stay human and become the expensive part. Any plan built on "AI means fewer engineers" without asking where that expensive work lands is budgeting from half the ledger.

On headcount, the honest answer is that nobody has reliable numbers yet - even the best study keeps correcting itself. Martin Fowler bets teams stay the same size and become more capable; our own maturity model bets on leaner teams. Both outcomes are live. What is certain either way: each remaining person carries more reviewing and judgment per head, so budget for that load, not just for seats.

The tooling around the AI - the harness: the context, guardrails, and workflows wrapped around the rented model - is where the results actually come from, not the model itself. Treat the team that builds it as a real product team, not a side project. For a Software House this is the commercial headline: everyone rents the same models at the same price, but the harness is owned, and it compounds - every new project starts ahead. Two cautions before that becomes a sales deck: build it out of a few real projects rather than speculation, and expect the return to scale with how uniform your tech stacks are.

One counter-intuitive staffing signal: your deep specialists - the people whose value is hard-won knowledge that lives in their heads, not in documents - are the least exposed to AI, not the most. AI is weakest exactly where unwritten expertise matters. Plan around keeping them.

The single question worth running every team through at your next org review: whose workload does this team reduce - and what work flows back when it does? Then place yourself on the Team dimension of our maturity matrix.

When the Weakest Dimension Wins: Architectural Governance at Scale

Wed, 10 Jun 2026 00:00:00 GMT

💡 For a quick summary, jump to the TL;DR section.

Thoughtworks puts the task-vs-system gap as cleanly as anyone has: "It's like buying a Ferrari and driving it on muddy roads: the engine is powerful, but your arrival time is determined by road conditions and traffic." That image - borrowed from their work on Structured-Prompt-Driven Development - is what every productivity debate about AI tooling keeps returning to. The engine has changed. The roads haven't.

METR's February 2026 productivity update makes the same point in numbers. Its late-2025 cohort - 57 developers, 800+ tasks - puts both returning and newly recruited developers within a few points of zero, with confidence intervals wide enough to cross it in both directions. The honest read is not that AI helps or doesn't help - it is that task-level metrics are the wrong instrument for system-level questions.

Architectural governance is a system-level question. That is the layer this article is about - and the argument is that when individual teams reach Stage 3 in context engineering and workflow, governance is the dimension that determines what the platform actually delivers.

The productivity measurement trap

Per-task speed and platform throughput are not the same thing. If we make one developer 20% faster at writing a controller, the controller ships sooner. If two teams independently build incompatible contracts around the same bounded context, the integration cost is migration-level work - and it does not appear in any task-level study because it is not a task. It is a structural collision between systems that each shipped on time. Later in this article we walk through one concrete case: two teams both using "external integration server" to mean different things, which turns a documentation gap into a production incident.

This is why the METR caveat matters more than the central estimate. METR is measuring the right thing for the question it asks (does AI speed up individual tasks?), and the answer is "probably, with caveats that are getting harder to control." But that answer does not transfer to platform throughput - it is a different kind of question. Platform throughput is bounded by how fast a multi-team system can converge on correct cross-team decisions - and that is governed by shared vocabulary, shared diagrams, shared decision records, and the discipline of keeping them current. None of which appear in a task-level randomized controlled trial (RCT).

We will use the term harness throughout the rest of this article in the sense Birgitta Böckeler develops in her "Harness Engineering" piece: a harness is everything in an AI agent except the model itself.

Agent = Model + Harness

The harness is the context, the constraints, the verification - the system around the agent that turns raw model output into something a team can trust. Harness engineering is the practice of designing and improving that system, the same way you would design and improve a test suite or a CI pipeline.

The rest of this article is about what happens when the harness is built per-team but the system being governed is cross-team.

The Harness Model and the multi-dimensional nature of maturity

The short version: the Harness Model Maturity Matrix scores ten dimensions of AI-engineering practice, each across five maturity stages, so a team can see where it actually stands rather than where it feels like it stands. Those ten dimensions group into four clusters: Foundation (context engineering, team), Governance (security, architectural governance), Delivery (human-agent interaction, workflow, reliability), and Outcomes & Learning (verification, knowledge, planning). Each dimension has five stages: from "no AI process" (Stage 1) through "chatbot-assisted" (Stage 2), "human-in-the-loop" (Stage 3), and "systematic harness" (Stage 4) to the speculative "agentic flywheel" (Stage 5). The matrix is a diagnostic tool. It is not a certification, and Stage 5 is informed speculation, not a documented endpoint.

The central claim of the model is structural: maturity does not average across dimensions; the weakest dimension dominates the whole. A team at Stage 3 in context engineering paired with Stage 1 in verification does not operate at the mean. The higher stages still ship at their own velocity, but the lowest dimension accumulates structural debt that erodes the value of everything above it. Unlike Liebig's law of the minimum - where plant growth is capped by the scarcest nutrient with a hard physical bound - the weakest dimension here does not impose a zero-yield ceiling. The debt compounds until it dominates the outcome. The mechanism is value erosion over time, not an instantaneous cap. We lean on the Liebig framing because it captures the right intuition (the bottleneck is what governs), but the mechanism is more like compounding interest than a hard wall.

A concrete cross-dimension example: a team at Stage 3 in workflow generates code rapidly with AI assistance. The same team at Stage 1 in architectural governance has no linters, no structural tests, no codified invariants - boundary checks happen in code review or not at all. The Stage 3 velocity is real. Pull requests open and close faster than they used to. What also happens is that agent-generated code violates structural boundaries faster than humans can catch them manually. Six months later, the team owns more code, but the architectural shape has drifted. The Stage 3 speed shipped Stage 1 entropy. The weakest dimension dominated the outcome.

This is the lens we want to apply to a specific dimension and a specific class of platform.

Why AI platforms amplify the governance gap

We use an AI Platform as the running example because the governance tension is sharpest there. The reasons are structural, not particular to any organization. AI standards (Model Context Protocol servers, Skills) are changing constantly - what was a sensible abstraction in Q1 is a legacy compatibility layer by Q3. Business expectations are enormous and under continuous executive scrutiny. Market demand makes shipping speed non-negotiable, which amplifies the pressure to skip cross-team coordination. Multiple teams build simultaneously, each with strong local context and weak cross-team visibility.

Under these conditions, the architectural governance dimension - shared domain language, cross-team contracts, structural invariants, decision records - gets neglected not from negligence but from structural reality. Cross-team coordination requires cross-team ownership, and on a platform without a dedicated architect role, no single team owns it. Each team optimizes its own velocity. The aggregate cost is invisible until contracts collide.

A specific class of collision: domain vocabulary. Imagine a platform that distinguishes between four types of Model Context Protocol server - an Internal Platform Server (platform-owned, internal data access), an External Integration Server (platform-owned, external tool access), a Verified Community Server (third-party, audited), and a Custom Configuration Server (admin-configured, unaudited). Before this vocabulary is documented, two teams use these terms interchangeably. The collision surfaces at the security layer: a permission granted to "an external integration server" means different things depending on which definition was in scope when the rule was written. The fix at the documentation level is a ubiquitous language entry. The fix at the production level is an incident.

A bounded context is the boundary within which a particular vocabulary and model are valid - the unit at which a team owns its own naming. A domain ubiquitous language, in the Domain-Driven Design sense, is the shared vocabulary developed by engineers and domain experts and tested continuously against working software. For LLMs, the vocabulary is the contract. If it is precise and consistent, the model maps intent to implementation reliably. If it is vague or inconsistent, the model guesses - and it guesses faster than the team can catch the guess.

Unmesh Joshi makes this argument cleanly in "What Is Code?": code is a conceptual model of the problem domain, and vocabulary is how that model becomes visible. His framing of the risk: "The code may compile. It may even pass basic tests. But if the team does not understand the conceptual model behind those structures, the codebase has gained vocabulary without shared understanding." He calls the gap "cognitive debt." LLMs amplify it because they generate plausible vocabulary at a rate that outpaces a team's ability to develop shared meaning.

The knowledge base - making the platform model machine-readable

The first move in lifting architectural governance is making the platform-wide model machine-readable. The goal is concrete: when an engineer asks an agent "how does feature X interact with team B's component?", the answer comes back with structural context - bounded contexts named, ownership identified, contracts referenced, prior decisions cited. Not from memory. From the knowledge base.

That knowledge base has four components:

A shared domain model: bounded contexts with their ubiquitous language entries
Logical component-to-team mapping: who owns what
Architectural diagrams that expose cross-team flows and invariants
Architecture Decision Records (ADRs) that make implicit decisions legible

We will say more about the last two because they are the parts most often treated as documentation hygiene rather than first-class infrastructure.

The C4 model (Context, Container, Component, Code) is a framework for visualizing software architecture at four levels of abstraction. The Level 0 view - the System Landscape diagram - is the one that proves its value at platform scale. It puts every team's system on the same canvas. It shows cross-system relationships. Before a System Landscape exists, teams have no shared picture of the full platform to disagree with concretely. Disagreements happen in generalities. After it exists, principals from different teams can point at the same box and say "this is owned by my team, here is the contract" - or dispute the assignment. That is the point at which governance becomes possible: a common artifact that surfaces disagreements before they become migrations.

"Diagrams as code" is the second discipline. The diagram lives in version control, has a review process, and can be validated automatically - the same hygiene we apply to test suites. The diagram does not rot silently because automated checks (linting names, complexity thresholds, freshness timers) run on every change. The diagram is part of the build.

ADRs record the decisions: not just what was chosen, but the context, the alternatives considered, and the rationale for the chosen option. (We have written before about using generative AI as an architect buddy for ADRs.) For human teams, ADRs catch the "why did we do it this way?" question three months later. For LLMs, they do something more important: they make implicit decisions explicit context. A decision made in a Slack thread three months ago is invisible to an agent reasoning about a new feature. An ADR linked from the knowledge base is not.

A wiki has all of these things and decays anyway. The reason the knowledge base described here does not decay is not a semantic distinction. It is mechanical - the ingestion harness in the next section. This is the responsibility Mark Richards frames in Analyzing Architecture: Structural Decay: continually analyzing the architecture to keep it vital and to detect structural decay is part of the architect's job, not optional documentation hygiene.

Building the knowledge base - the ingestion harness

Maintaining cross-team documentation is hard for a known reason: teams fill templates under deadline pressure and produce consistency drift, silent gaps, and naming collisions that nobody flags until a contract breaks. On the platform we work on, we tried the template approach first. The output looked complete. It was not.

What works is a conversation-driven ingestion harness. The harness is an AI session that asks one focused question at a time, offers concrete options with their consequences, and refuses to accept a well-formatted document that has not been challenged. The pattern has three properties worth naming.

Conversation first, materialization second. The session is interactive. Every question comes with an initial proposal to reduce friction and boost engagement. The AI does not generate the documentation as its primary output - it challenges the model until ambiguity is gone. When the session ends, the findings are materialized into a review artifact in a staging area, not written directly into the canonical knowledge base.

Separate review and apply steps. The materialized session is a proposal. An explicit apply step, with per-item confirmation, is required before anything enters the canonical store. The human never has to fix AI-generated documentation inline - they approve or reject structured proposals. This is the "human on the loop" pattern Kief Morris describes in "Humans and Agents": "Rather than personally inspecting what the agents produce, we can make them better at producing it." The review step exists because the harness is not infallible. It is the last line of defense, not a formality.

Gap discovery as the working signal. The most useful feedback we have gotten from this process came from a platform principal engineer after one of the early sessions:

"I answered one question in a tricky way, there was a hidden gap. A while later I was asked exactly the question I had in my mind."

That moment - the harness arriving at the gap the participant already suspected - is the credibility anchor. The AI is not generating good documentation. It is arriving at the gaps the participant already half-knew were there. To stay honest about what this means: the harness surfaces what neither party is blind to - it cannot catch a shared assumption both the contributor and the model hold. The early sessions are a promising initial result with motivated principals, not a claim validated at scale.

The harness has evolved across iterations. The first version was a single advanced prompt - ~200 lines, embedding the full workflow, launched as a chat session. Value was immediate; the workflow was monolithic. The current version is split into small commands (start a session, apply approved findings, reject a session with reason) backed by a separate C4 model skill that validates structural rules: naming linting, complexity analysis, and maturity checks computed from git history. The maturity-policy timers flag a System Landscape diagram not updated in 12 months and a System Context diagram not updated in 6. The freshness signal is enforced automatically, not by convention.

Re-ingestion is triggered not only by those timers but by any platform update, by new approved or developed work (there is a documentation-update loop when new work ships), and by a monthly Principal meeting where direction is discussed and sessions are orchestrated when needed.

The honest note about maintenance: the harness makes ingestion tractable, not free. The ongoing cost is the human review cadence and the willingness to re-run sessions when the model changes. And freshness does not equal correctness - a diagram that is updated incorrectly generates false confidence. The harness reduces the cost of staying current; it does not eliminate it.

What this section describes is harness engineering applied to the documentation pipeline itself. Each iteration improves the system that produces the output, not just the output. The C4 skill is a Guide (a feedforward control on what the diagram should look like). The maturity check is a Sensor (a feedback control on whether it is still current). Both are part of the same discipline.

The decision harness - from knowledge base to governed decisions

The knowledge base is the foundation. The decision harness is where it pays off. With a shared domain model in place, a new feature can be analyzed with awareness of cross-team contracts, complexity, and cost - before implementation is committed to.

A note on confidence: the ingestion harness has been iterated through sessions with platform principals and produced the gap-discovery result above. The decision harness has not. It is at the plan and prototype stage - described from a design that has been worked through, not a production system that has run at scale. Treat the rest of this section as a designed pattern, not a validated result.

The harness uses a canvas adapted from the Structured-Prompt-Driven Development (SPDD) method by Thoughtworks. SPDD treats prompts as first-class delivery artifacts and structures them with the REASONS canvas (Requirements, Entities, Approach, Structure, Operations, Norms, Safeguards). The adaptation here is called REASINS: we substitute the O (Operations) for an I (Integration), which changes both the canvas and its name.

Letter	Section	Purpose
R	Requirements	What the feature must accomplish; outcomes, constraints, success criteria
E	Entities	Domain concepts in play; bounded contexts and ubiquitous language entries that apply
A	Approach	High-level direction; the option chosen among the alternatives considered
S	Structure	Component-level shape; which components are affected, how they relate
I	Integration	API contracts and dependency edges between components (replaces SPDD's Operations)
N	Norms	Standards and conventions the solution must follow (naming, patterns, prior ADRs)
S	Safeguards	Explicit risk mitigations and failure-mode handling

The Norms vs. Safeguards distinction is worth keeping in mind. Norms are what good looks like by default - the conventions that the solution should respect because the platform already runs that way. Safeguards are what the solution must do specifically to mitigate identified risks - the explicit "if this fails, here is the fallback" handling. Conflating them is how a canvas grows a section that looks complete but does not separate "follow the platform pattern" from "this specific risk needs this specific mitigation."

The reason for I (Integration) replacing O (Operations): SPDD's Operations section covers deployment, runtime behavior, and operational concerns. At the cross-team governance level, what we need instead is an explicit record of API contracts and dependency edges between components. The decision harness governs what crosses team boundaries. Operations stays with the teams who own the components.

To make this concrete, run the MCP-server permission feature through the canvas. Suppose a team wants to add a feature: "let platform admins grant a scoped permission to an external integration server." This is exactly the feature that, left ungoverned, produced the collision earlier - "external integration server" meant one thing to the security team and another to the team writing the rule. Here is how REASINS forces the ambiguity into the open before any code is written:

Section	Populated for the permission feature
R Requirements	Admins can grant a scoped, revocable permission to an external integration server; the grant is auditable
E Entities	Pulls the bounded-context vocabulary: External Integration Server (platform-owned, external tool access) - explicitly not Verified Community Server or Custom Configuration Server, which carry different trust assumptions
A Approach	Permission resolved against the server's declared type at grant time, not at call time
S Structure	Admin console component and the permission-resolution component in the security context are affected
I Integration	The contract edge that would otherwise collide: the permission API must take a typed server reference, not a free-text "external integration server" string the two teams read differently
N Norms	Follows the platform's existing permission-naming convention and the prior ADR on server-type taxonomy
S Safeguards	If a server's declared type is missing or ambiguous, deny by default rather than fall back to the broadest scope

The Entities row is where the bounded-context vocabulary stops being documentation and starts being a constraint: the canvas cannot advance with "external integration server" left as prose, because the entry has to resolve to one of the four named server types. The Integration row is where the collision is caught - it names the contract edge (a typed server reference) that the two teams would otherwise have implemented as a string each interpreted its own way. The documentation gap and the production incident are the same gap, surfaced before the code instead of after.

The lifecycle is gated. A canvas advances only when its sections are non-empty, affected components are named, and lifecycle checks pass.

What the harness produces is a structured prompt per affected component, not implementation code. Teams receive the prompt and use whatever technique they prefer to plan implementation - spec-driven workflows, their own templates, plain pairing. Teams retain autonomy over how; the harness governs what crosses the seam. This is the design choice that keeps team independence intact while ensuring cross-team contracts are verified before implementation diverges.

The harness does not resolve domain model conflicts on its own. When two teams hold contradictory views on a shared concept, the harness surfaces the conflict, frames it as a structured question with options and consequences, and escalates. Resolution happens between humans. The harness produces the structured input that makes that conversation possible.

The payoff splits cleanly across roles. For engineers, the value is pivot speed at contract level. With the knowledge base loaded, the harness can answer "if we add multi-context state, which contracts change and in which sequence?" in a session rather than across a week of meetings. The cost of changing direction becomes visible while it is still cheap to change direction. For product managers, the value is informed scope decisions before the sprint hardens. The decision harness gives PMs a structured view of how a feature idea maps to technical complexity and cost, before implementation starts. A PM sees not a green/yellow/red signal but the actual contract changes a feature would require - which is the information needed to choose between "ship the small version now" and "fund the bigger change properly."

Context engineering at the architectural level - the Stage 3 → Stage 4 jump

The maturity matrix describes the architectural governance dimension at Stage 3 as "human-enforced boundaries; constrained solution space." Stage 4 is "custom linters with remediation; taste invariants as code." The jump between them is what this article has been describing, decomposed:

The knowledge base makes the architectural model machine-readable and agent-accessible
The ingestion harness keeps that model current and challenged
The decision harness encodes cross-team governance into a reviewable, versionable artifact
C4 diagrams as code plus maturity validation gives the architectural view a freshness signal

At Stage 3, context engineering happens at the per-engineer or per-team level. Each engineer maintains their own context - AGENTS.md, structured prompts, local conventions. Each team curates its own slice of documentation. At Stage 4, context engineering happens at the platform level. The domain context that crosses team boundaries is a core artifact with the same lifecycle discipline as code: versioned, reviewed, validated, refreshed.

The Harness Model thesis is that each stage unlocks the next. That is true. The corollary is what this article has been building toward: each stage also constrains the next. If architectural governance stays at Stage 1 while everything else advances, the higher stages do not disappear. They accumulate debt. The weakest dimension does not impose a hard ceiling; it erodes the returns from everything above it. Individual velocity continues. The aggregate cost grows invisibly until contracts collide.

15-minute exercise

Pick a team you work with - yours, or one you collaborate with. Sit with these three questions and answer them out loud, not in your head:

Do your teams share a documented domain vocabulary that both humans and LLMs can read - and is that vocabulary tested against working software, not just agreed in a meeting?
Are your architectural invariants enforced by tooling (linters, structural tests, maturity checks), or by code review? When a boundary is violated, how long before anyone notices?
When a new team joins the platform, is there a structured way for them to ingest the existing contract surface - or does orientation happen through word of mouth and Slack threads?

These are diagnostic, not rhetorical. A team that cannot answer the first affirmatively does not yet have the substrate for a Stage 4 knowledge base. A team that cannot answer the second is running architectural governance as tribal knowledge - Stage 3 at best, regardless of how good its individual harness looks. A team that cannot answer the third is treating governance as a per-team property, which it is not: governance that lives in one team's heads is not platform governance.

Spend the remaining minutes writing down one concrete artifact you could ship next week to move from a "no" to a "partial yes" on whichever question stung the most. A System Landscape diagram in version control. A first ADR. A glossary entry for the three terms your teams use differently. The first artifact is the one that costs the least and surfaces the next gap.

Conclusion

Task-level metrics will keep getting noisier. METR has been honest about why: the more AI adoption spreads, the harder it is to construct a control group, and the more the question itself drifts away from what teams actually want to know. Platforms are not built one task at a time. They are built by multiple teams converging on shared decisions about contracts they will all live with.

Architectural governance is the dimension that crosses team boundaries. On an AI platform - where standards shift faster than annual planning cycles and where multiple teams ship in parallel - it is also the dimension most likely to lag while everything else advances. The Stage 3 → Stage 4 jump is not a tooling decision. It is the decision to treat context engineering as an architectural practice, not a per-team one: a shared domain model, an ingestion harness that keeps it current, a decision harness that turns cross-team disagreements into structured prompts before implementation diverges.

The weakest dimension dominates the outcome. The question is whether the architectural governance dimension dominates by design - through a governed system that is owned, maintained, and machine-readable - or by neglect, through the cost of a migration nobody budgeted for. Both options ship code. Only one ships a platform.

TL;DR

On an AI platform, maturity doesn't average across dimensions - the weakest one dominates the outcome. Here are some signs that architecture governance is your weakness: AI standards shift faster than planning cycles, multiple teams ship in parallel, and on a platform without a dedicated architect role, no single team owns the cross-team seam.

The cost stays invisible until contracts collide. Two teams use "external integration server" to mean different things; the collision surfaces at the security layer, and a documentation gap becomes a production incident. For an LLM the vocabulary is the contract - when it is vague, the model guesses, and it guesses faster than the team can catch the guess.

The way out is to stop treating context engineering as a per-team habit and make it an architectural practice. That starts with a machine-readable shared domain model - bounded contexts, C4 diagrams as code, and ADRs - kept current by a conversation-driven ingestion harness that challenges the model until ambiguity is gone and materializes findings into a reviewable artifact rather than writing straight into the canonical store.

A decision harness then turns cross-team contracts into reviewable artifacts before implementation diverges. It runs a feature through a REASINS canvas - Thoughtworks' REASONS adapted to swap Operations for Integration - which forces ambiguous vocabulary to resolve to a named entity and catches the contract collision before any code is written. The harness governs what crosses the seam; teams keep autonomy over how they implement.

That is the Stage 3 → Stage 4 jump: context engineering as a platform artifact with the same lifecycle discipline as code. Both neglect and design ship code; only one ships a platform.

E2E test harness for AI agents: better specs, not smarter models

Tue, 12 May 2026 00:00:00 GMT

Three weeks of green CI. A flow that never executed the way the spec said it should. An agent that, given the choice between the documented OAuth dance and a debug shortcut that happened to return the same HTTP status, picked the shortcut. We fixed the documentation. The next test came out right on the first try.

That bug is the small story. The bigger story is the harness engineering we did around it - over three and a half months, alongside normal feature work, on a Spring Boot microservice - the MCP Servers Registry - that manages server configurations and OAuth flows. Fifteen end-to-end suites. 162 test steps. CI wall-clock flat at three to four minutes while coverage grew five times, and a fivefold drop in commits per test step as the harness matured. A pipeline where four specialized AI agents propose, write, review, and run E2E tests, and where the way we move work between them is itself the artifact we are now optimizing. This post is the engineering tour of that harness - what we built, what we measured, and what we would warn the next team about.

Tests were green. The agent was testing the wrong thing.

Open the CI dashboard. Green for three weeks. A new OAuth test had been merged, the suite added one more dot to the row, and nobody had a reason to look harder. The discovery did not come from staring at the test. It came from the Planner agent - one of four we run on this codebase - flagging a coverage gap while drafting a new scenario. The "real" OAuth flow it was looking at, the one with discovery and Dynamic Client Registration (DCR), was not exercised anywhere.

Tracing the gap led back to the existing test. It had been calling an internal debug endpoint instead of the discovery and registration calls the contract described. The HTTP statuses lined up. The assertions passed. The test was wrong about which flow it was testing - and "wrong" had been silent for three weeks of merges.

That endpoint existed for a good reason. Debugging OAuth against external MCP servers is hard, and a shortcut that hands you a working token end-to-end saves hours. The endpoint is internal to the service, fully admin-gated, and not exposed in production - it exists so engineers can debug OAuth integrations without losing an afternoon. The endpoint was never the problem. The problem was that the API contract did not say "for local debugging only; not part of the OAuth flow." The agent saw an endpoint that produced the right status at the right step, and used it.

We didn't fix the test. We fixed the documentation.

We will come back to the mechanics of that fix later in the post. We start here because this incident is the cleanest statement of the principle the entire harness is built around: test quality is a function of what the agent receives as input. The spec decides. The agent executes. We do iterate the agents and skills too - more accurate planners, more specialized review skills - but that work moves slowly compared to the context the agents read, which changes every sprint. It is a small simplification, but mostly true in our case: the leverage is in the spec, not the model. Once you accept that, the question shifts from "how do we get better agents?" to "what is the system that produces the spec the agent reads, and how do we evolve it without slowing delivery down?" The rest of this post is the answer we have so far.

Why E2E became the place the harness took shape

One of us joined the service while it was under delivery pressure. Documentation was a handful of API examples in Postman JSON format. The unit and integration tests had high coverage and low business value - vibe-coded by AI, they checked HTTP 200s and response shapes, not behaviour. CI was green on every push and told us almost nothing.

We had a choice about where to start. We picked E2E deliberately. Three reasons. First, an E2E suite asserts on business outcomes - "the admin published a server and it now shows up in the list" - which is exactly the layer of behaviour the existing tests were not protecting. Second, an E2E test exercises the integration boundary, which is where a brownfield Spring Boot service hides most of its scars: WireMock stubs for external dependencies, OAuth flows, secrets handling, real database transitions. Third - and this is the part that surprised us - an E2E suite makes the missing documentation visible. You cannot write a useful E2E test without naming the precondition, the endpoint, the expected status, and the external stub. Either the artifacts that name those things exist, or they do not. The suite forces the question.

The work happened over roughly three and a half months, in sprint-sized increments. Ten weeks on the foundation - C4 system and container diagrams (kept in Mermaid alongside the code, generated with AI from a separate architecture spec and refreshed when the spec changes), a real API contract, a CONTRIBUTING.md, a README.md, and the first scenarios in what would become SCENARIOS.md. CI that verified instead of only building. Two weeks of harness work on top - CLAUDE.md as the index the agent reads first, plus the four-agent pipeline (Planner, Implementer, Reviewer, Runner) we still run today. Then the suites started landing. March: three suites and 33 steps. April: nine new suites and 83 new steps. May 1–11: three more suites and 46 steps - eleven days, more than half of April's step count. We will get to the data in detail. As a rough order of magnitude, the foundation phase took about a day per sprint per engineer; the expansion phase, a few hours.

Mapped onto the Harness Model maturity matrix, the journey is the climb from Stage 1 (no AI process, tribal knowledge, ad-hoc tests) into Stage 3 (human-in-the-loop with structured documentation and an agent that generates from it). We did not jump. We climbed. If you are earlier on the curve and still asking where to start with AI-assisted development at all, our companion post on how to start with AI-assisted development today covers the thirty-minute experiment we recommend before any harness work. The same documentation that onboarded a new engineer turned out to be exactly what the agent needed, and that coincidence is the part we keep coming back to. There is no "AI documentation" in this repository. There is just documentation, written well enough that an agent can use it.

Domain aggregates: where the spec actually starts

Before any SCN, before any contract, there is a smaller artifact we under-invested in for the first few weeks and have been correcting for ever since: the domain aggregate - DDD's aggregate, sketched on one page. A single business concept with its states, its lifecycle transitions, who owns each transition, and the invariants that survive across them.

For the MCP Servers Registry, the aggregate looks roughly like this:

Domain aggregate: server configuration (MCP Servers Registry)
States:        UNPUBLISHED -> PUBLISHED
Owners:        admin edits (create, update, publish, unpublish)
               power user reads (selects for runtime, never edits)
Transitions:   create               -> UNPUBLISHED
               update (admin)       -> UNPUBLISHED
               publish (admin)      -> PUBLISHED
               unpublish (admin)    -> UNPUBLISHED
Invariants:    a power user only sees PUBLISHED entries
               server URL and configuration ID are distinct fields

This kind of artifact belongs before SCNs in the spec lineage, not alongside them. Two reasons. First, every SCN that touches "publish" or "select" needs the same vocabulary; without the aggregate, three SCN authors invent three slightly different vocabularies, and the agent learns the inconsistency. Second - and this is the part the OAuth bug pointed at - most agent misbehaviour we have traced back to spec problems came from missing aggregate-level knowledge, not missing SCN steps. A field called "server URL" being silently overloaded to also carry a configuration ID is the kind of confusion that the aggregate catches and an SCN does not.

We use AI in this step too, but as an elicitor rather than a writer. A skill in the repository prompts the engineer through "what are the states, who owns each transition, what are the invariants," and produces a draft aggregate the team edits. The work that takes time is naming the states and the owners - exactly the work the SCN cannot do for itself. The aggregate is short, lives next to the API contract, and is referenced by every SCN that crosses its boundary. The next section is where it gets cashed in.

The SCN format - and where it sits in the spec lineage

Every E2E test starts as a scenario in SCENARIOS.md. We call them SCNs. The format is intentionally small.

SCN-001: Admin configures MCP server

Preconditions:
- Application is healthy (actuator/health -> UP)
- Admin authenticated (identity headers + userType=ADMIN)
- WireMock as stub for the external MCP server

Steps:
1. Admin reads the current server list.
   GET  /mcp-servers/config -> 200
2. Admin tests the connection to a new server; tool discovery succeeds.
   POST /mcp-servers/test   -> 200, discover tools
3. Admin saves the configuration as UNPUBLISHED.
   POST /mcp-servers/config (state=UNPUBLISHED) -> 200
4. Admin publishes the configuration.
   POST /mcp-servers/config (state=PUBLISHED)   -> 200
5. Admin re-reads the list; the new entry is present.
   GET  /mcp-servers/config -> 200, count + 1

Five steps. Three preconditions. Each step has an HTTP verb, an endpoint, and an expected outcome. That is the entire interface between human intent and the agent.

The format is not new. Specification by Example and Gojko Adzic's work on it predate this by more than a decade. BDD tools like Gherkin gave us Given/When/Then. Contract testing tools like Pact push the same idea down to the wire. More recently, Martin Fowler's writing on structured prompt-driven development and projects like GitHub Spec Kit treat structured prompts as a first-class engineering artifact. SCNs sit inside that lineage. The shared idea is older than any of the tooling: there is a layer of artifact above the test, written in human language, that captures intent precisely enough to survive change.

The difference here is who the consumer is. Gherkin is the test, executed by Cucumber. SCN generates the test, written in Python by an agent. That distinction is why we keep the SCN minimal. The agent fills in fixtures, WireMock stubs, assertions, and cleanup; we keep the document at the level a product manager could read. If we pushed the SCN to be more precise, we would be writing the test in slow motion. If we let it drift looser, the agent would invent details.

The other job the SCN does - and we keep underlining this - is to act as a contract. It is the precise statement of what "this flow works" means, written before the test exists, and it stays the contract whether a human drafts it from a feature ticket or the Planner drafts it from a contract delta. The SCN is also a diagnostic: somebody has to sit down and write the steps, preconditions, and success criteria. If neither the human nor the planner can, the flow is not understood yet - and no model will fix that for you.

If you can't write the scenario, the agent can't either.

We treat that sentence as a hard rule. To make the diagnostic concrete, here is an SCN that does not work - and the small set of edits that does:

SCN-XXX (weak): User publishes server
Steps:
1. Save and publish the server configuration -> success
2. Check that the server is listed

The verbs are missing. "Success" is unspecified. There is no precondition about authentication or what the list contained beforehand. An agent reading this writes a test that calls some endpoint with some payload and asserts a 2xx - and the test is green for the wrong reasons.

SCN-XXX (fixed): Admin publishes a new MCP server
Preconditions:
- Admin authenticated (userType=ADMIN)
- Configuration list is empty
Steps:
1. Admin saves a new server configuration as UNPUBLISHED.
   POST /mcp-servers/config (state=UNPUBLISHED) -> 200
2. Admin publishes the saved configuration.
   POST /mcp-servers/config (state=PUBLISHED)   -> 200
3. Admin re-reads the list; the configuration is PUBLISHED.
   GET  /mcp-servers/config                     -> 200, count == 1, entry is PUBLISHED

Same intent. Wholly different prompt. The fix is not in the agent or the model - it is in five lines of plain text that a human writes, or the Planner drafts and a human approves, before regenerating.

The OAuth bug, in mechanics

Back to the OAuth bug, this time with the wiring. We want to walk through it because it is the clearest example we have of "fix the spec, not the test" paying off in a single afternoon.

The correct OAuth Dynamic Client Registration (DCR) flow looks like this:

Correct OAuth DCR flow	What the agent did
`GET /.well-known/oauth-authorization-server`	skipped
`POST /register` (Dynamic Client Registration)	skipped
`GET /authorize`	`GET /authorize`
`POST /token`	`POST /token` (against the debug endpoint)
`GET /resource`	`GET /resource`

The agent skipped discovery and registration and used the debug endpoint to obtain a working token directly. The remaining calls all succeeded. The test asserted that a resource came back. CI was green.

The fix was two lines of work, neither in the test file:

 ## OAuth endpoints
+### POST /oauth/debug-token
+For local debugging of OAuth integrations only. NOT part of any production
+OAuth flow. Tests MUST go through /.well-known and /register first.
 ### GET /.well-known/oauth-authorization-server
 Returns the authorization server metadata used by the DCR flow.

We added the missing line to the API contract describing what the debug endpoint is for. Then we wrote a detailed OAuth request-flow diagram into the architecture docs - a Mermaid sequence diagram listing the correct endpoints in order, AI-generated from our OAuth spec following the same pattern as the C4 diagrams.

On the next Planner iteration, the Planner discovered the new file as part of its context, noticed that the existing test did not match the documented flow, and proposed a regenerated test. We accepted it. The new test made the discovery call, performed the registration, used the registered client to get a token, fetched the resource, and asserted the lot. One shot. Fix the spec, not the test: write the missing artifact, let the harness re-read it, accept the corrected test.

The agent was not confused about OAuth. It knew the protocol perfectly well. What it was missing was the boundary statement - which endpoints in our service are part of the flow and which are not. That is exactly the gap an aggregate-plus-contract pair is supposed to close, and the lesson we took out of the incident was that the domain aggregate for "MCP server OAuth integration" had been implicit in our heads and absent from the repository.

The implication compounds both ways. Every future OAuth scenario the agent generates now starts from the corrected understanding - it cannot accidentally use the debug endpoint. Every other agent, and every human reading the docs for onboarding, gets the same clarification for free. A line of API contract is doing more work than a fix in a single test could.

The artifact loop, plus what's upstream and downstream of it

The artifacts are not isolated documents. They form a loop. The loop is what makes the investment compound.

That diagram is the inner loop - the one we sketch first when someone asks how the harness works. SPEC, GENERATE, RUN, SHARPEN. We have linked it before to Mitchell Hashimoto's harness engineering idea: any time an agent makes a mistake, you engineer a solution so it never makes that mistake again. The artifact loop is one shape that engineering takes. Custom linters and structural tests - like those in our Python architecture-tests post - are another. Both belong in the harness.

Concretely, skills live at .claude/skills/<name>/SKILL.md, agents at .claude/agents/<name>.md, and slash commands at .claude/commands/<name>.md. The orchestration sits in a slash command - /e2e/scenario - that walks the four agents through the pipeline with a human approval gate after the Planner step. Each agent in turn calls a focused skill: the Implementer uses e2e-scaffold for the suite skeleton (templates for the test file, docker-compose, and WireMock layout) and wiremock for stub generation; the Runner uses e2e-run to execute the suite in an isolated container. Conventions live in the skills; control flow lives in the command; the agents are the things that get reviewed.

What the artifact loop hides is the SDLC pipeline that surrounds it. Once we started building specialized agents for domain elicitation, contract drafting, and SCN authoring - not only for test generation - a wider loop became visible. The diagram below is how we draw that loop today.

The six stations on the diagram correspond to the five states a scenario moves through.

Analyzed - the scenario's domain aggregate and SCN exist. The domain skill elicits the states and invariants, the scenario-author skill drafts the SCN, and the team edits.

Proposed - the contract-author skill writes the API contract and the C4 diff against the aggregate.

Approved - the /e2e/scenario slash command drives the Planner / Implementer / Reviewer / Runner agents to produce the pytest file, WireMock stubs, and docker-compose.yml updates. Each agent calls the relevant skill (e2e-scaffold for the suite skeleton, wiremock for stubs, e2e-run for execution), so the project's E2E conventions live in one place.

Implemented - CI runs the suite. The Runner attributes each outcome - pass, fail, flake - to a cause: spec, application code, test code, or infrastructure.

Sharpened - the cause gets fixed at the source. A fuzzy precondition becomes concrete, an ambiguous endpoint gets a paragraph, the aggregate gains a state if it needs one.

One honest caveat. The Sharpen step is not pure automation. The Planner surfaces gaps, the Runner attributes outcomes, a human decides what to write. Outcome-to-spec attribution is a classification problem we can in principle automate further - we are not there yet. More on that gap in the human-in-the-loop section.

The data: 15 suites, 162 steps, and the shape of the curve

Three months after we migrated the first suites to pytest, we have 15 suites and 162 test steps committed. The growth was not even.

March was the pytest foundation: three suites and 33 steps, written by two engineers who were also inventing the conventions as they went. April was the expansion: nine new suites and 83 new steps, with SCENARIOS.md, the e2e-scaffold skill, and shared WireMock stub patterns emerging in parallel. May, through publication on the 12th, added three agent-management suites in eleven days - and those are the ones where spec-first discipline paid its largest dividend.

The clearest signal of compounding is commits per test step: how many commits the team needed to land a working suite, divided by the number of test steps in it. Here is what that looks like in the git log.

Suites 01 through 07 averaged about 0.78 commits per step - the agent's work while the harness was still thin. Scenario format in flux. API contract gaps still being discovered. WireMock conventions not yet shared. Suites 08 through 15 averaged about 0.17 commits per step, a roughly five-times improvement. Suite 06, "Re-auth DCR reuse," is the most striking single point: 20 steps, one commit. The agent reused DCR work it had already verified in earlier suites; the SCN corpus itself was now part of the spec.

There is a quieter, more organizational signal in the same git log. The principal software engineer on the team committed SCENARIOS.md alone first - a commit with no test code, only scenarios - and then delivered three agent suites in a single implementation commit afterwards. Spec-first separation, visible in git. A software engineer on the team writes scenarios and implementation together in one commit rather than ahead of it; growing engagement, not yet full separation. We did not mandate the spec-first pattern in a process document. It showed up in the commits as soon as the harness made it cheaper than the alternative.

The third chart is the one we did not expect.

The e2e-test stage of our Jenkins pipeline went from three suites and 33 steps to fifteen suites and 162 steps - and the wall-clock barely moved. Every CI run still finishes E2E in three to four minutes. Two changes did most of the work. We optimised how we reconfigure the environment between suites - re-prime WireMock and reload stubs, do not restart the application - and we reimplemented the test runner from bash to Python, which dropped per-step execution time substantially. What dominates the three-minute budget today is the application Docker image build and startup, including database schema setup; that is the fixed cost the growing step count gets amortised against. The next section is where we cash in on that property.

A reminder we keep repeating to ourselves: three months and one team is not a controlled experiment. The dataset is small. We were the variable that changed the most. The numbers are visible, but the trajectory is what we were trying to see, not the absolute deltas.

The Docker image and harness build: making prod-like the default

A harness is not only documentation and agents. It is also the infrastructure the tests run against. Three engineering decisions did the most to keep agent-generated tests honest.

One production-shaped image, curated overrides only

The application Docker image used by the E2E suites is the same image that runs in staging and production. No @Profile("e2e") bean tree, no @TestConfiguration that swaps in fakes at boot. The only thing the E2E run does is apply a small, curated set of Spring profile overrides - logging, the WireMock stub URLs, and a couple of JVM options.

# application-e2e.yaml - the entire override surface
logging:
  pattern.console: "%-5level %logger{36} - %msg%n"
  level.root: INFO
mcp:
  external-server.base-url: "http://wiremock:8080"
  oauth.authorization-server-url: "http://wiremock:8080/oauth"
spring:
  datasource.url: "jdbc:postgresql://postgres-e2e:5432/mcp"

The discipline here matters more than the file's contents. If the production image cannot satisfy the test, the test is the place to argue, not the image.

Per-suite docker-compose and the `suite_containers` fixture

Each suite has its own docker-compose.yml next to its tests, declaring the WireMock instances it needs and an optional auth proxy when OAuth is in play. Suites share the application's Docker network so they can reach it, but each suite owns its stubs.

services:
  wiremock:
    image: wiremock/wiremock:3.5.0
    networks: [e2e]
    volumes:
      - ./wiremock:/home/wiremock
    command: --port 8080 --verbose
networks:
  e2e:
    external: true
    name: mcp-config-e2e

A shared conftest.py exposes a session-scoped suite_containers fixture that handles the lifecycle. The skeleton below is illustrative - the project polls health endpoints with a small retry helper rather than the testcontainers built-in:

from testcontainers.compose import DockerCompose  # custom health-check polling below

@pytest.fixture(scope="session")
def suite_containers(suite_dir, app_logs_dir):
    compose = DockerCompose(context=suite_dir, compose_file_name="docker-compose.yml")
    compose.start()
    try:
        wait_until(lambda: wiremock_admin_ready(compose) and proxy_healthy(compose))
        yield compose
    finally:
        save_app_logs(compose, app_logs_dir)
        compose.stop()

Sharing a single WireMock across suites was the obvious starting point and the wrong one - stubs leaked, recordings overlapped, debugging a flake meant guessing which suite poisoned the global state. Session-scoped UUID fixtures (test_run_id, workgroup_id, user_id) handle data isolation the same way, even when suites share an app instance.

JVM DNS cache and HTTP keep-alive

This is the fix that earned the flat CI curve. Between suites, Docker restarts the WireMock containers and they get new IPs. The JVM's default DNS cache (indefinite TTL) and HTTP keep-alive (reused TCP connections) kept the application talking to stale container IPs - which meant restarting the app between every suite, and the wall-clock budget evaporated.

Two files, mounted into the application container as Paketo buildpack layer overrides - a Cloud Native Buildpack mechanism that binds small files into the running image without a rebuild. If your image is built differently, use a Docker bind mount or bake them in.

# e2e/java-security-e2e.properties
networkaddress.cache.ttl=0
networkaddress.cache.negative.ttl=0

# e2e/jvm-e2e-options.append
-Dhttp.keepAlive=false

With these in place, the JVM resolves DNS through Docker's 127.0.0.11 resolver on every call and never reuses a TCP connection across suite boundaries. The app no longer needs to restart between suites. The e2e-test stage stays at three to four minutes - image build and schema setup dominate the budget today, and we watch the curve on every CI run.

A caveat for modern Spring Boot stacks: -Dhttp.keepAlive=false only covers HttpURLConnection-based clients. If your service uses RestClient, WebClient, or Apache HTTP Client 5 for outbound calls, configure keep-alive timeouts on those clients explicitly - the system property does not reach them.

Specialized agents, skills, and commands across the SDLC

Today the test pipeline runs four agents - Planner, Implementer, Reviewer, Runner - with a single human approval gate after the Planner step. Each one has a narrow job. The Planner identifies coverage gaps and turns business intent into SCN drafts. The Implementer generates the Python test, WireMock stubs, and docker-compose.yml updates from the SCN, using the e2e-scaffold skill for project conventions. The Reviewer checks the diff against style rules, the API contract, and the existing test corpus. The Runner executes the suite in an isolated container and reports the outcome.

In the matrix's vocabulary this is human-in-the-loop, Stage 3 on the Human-Agent Interaction dimension. It is where most teams running agent pipelines on production code sit, and for a reason: we do not yet have a metric that lets us safely skip the plan review. Across fifteen scenarios, that is fifteen approval gates total - one per scenario, between the Planner's test plan and the Implementer that turns it into code. Each gate is five to fifteen minutes of human attention in our experience; the team is still calibrating what "good" looks like for the plan the Planner proposes, and that is where the time goes. That is the bottleneck the data already shows; the commits-per-step above 0.10 are usually not the agent doing twenty revision rounds - they are the agent doing the work in one pass and the human approving and adjusting the plan in two or three.

The direction we are building toward is wider than the test pipeline. The SDLC harness loop in the previous diagram already shows where we are headed: domain analysis, scenario authoring, contract drafting, generation, verification, and sharpen - each station served by a specialized agent or skill, with the human moving from per-step approver to system editor. The shorthand for that state is human-on-the-loop.

We will not pretend we are there. Three things we do not have yet:

a quality metric we trust enough to gate merges on (today, commits per step is a lagging proxy);
automated drift detection that ties contract changes to affected SCNs and forces a regeneration;
a flake-rate signal that distinguishes infrastructure noise from real regressions.

This is exactly the matrix's Stage 3 → Stage 4 transition on Verification & Quality and Human-Agent Interaction, and it is the direction the Harness Model post frames in detail. We are doing the work. We are not writing the post yet that says we are finished.

The shape we expect the next two months to take: more specialized skills upstream of test generation (a contract-drift detector, a domain-aggregate refiner), more outcome attribution downstream (a Runner that can tell "the spec is wrong" from "the application code is wrong" automatically), and a quality metric we trust enough to merge on. None of those alone is dramatic. Together, they collapse the number of approval gates the engineer has to walk through, and they do it without giving up the ability to catch a debug-endpoint shortcut before it ships.

Traps when moving the team along the harness maturity stages

Moving from "we have some AI-assisted coding" to "agents generate E2E tests from SCNs and contracts" is not a single jump. It is a sequence of small moves along the dimensions in the maturity matrix. Six traps cost us the most time at the transitions.

Documentation accuracy becomes load-bearing. When humans were the only readers, "mostly right" docs were fine; people noticed contradictions and asked. Agents do not ask. They take the contract at face value. A field marked optional that is actually required produces a wrong test every time. The Stage 2 → Stage 3 transition is when documentation stops being a productivity aid and becomes correctness infrastructure.

Tribal knowledge becomes a correctness ceiling. The OAuth bug came from tribal knowledge - "everyone knows the debug endpoint is for debugging" - that had never reached the contract. While humans wrote the tests, the gap was invisible because the author also knew. The moment agents wrote the tests, the gap surfaced as a wrong test in production CI. Stage 2 → Stage 3 again, on the Context Engineering dimension.

Vibe-coded existing tests sabotage the new ones. The unit and integration tests we inherited were technically high-coverage and business-low-value. When the agent reads tests/ looking for examples of how the codebase asserts things, those tests teach the wrong style. We had to mark a chunk of them as "do not learn from" and write a CLAUDE.md section pointing the agent at the SCN-driven E2E suites as the canonical examples. This is the trap at Stage 3 → Stage 4 on Verification & Quality.

API contract drift between controller and OpenAPI spec. Once the agent reads the OpenAPI spec instead of guessing from controller code, the controller stops being the source of truth - but only if you defend that move. The first time a developer in a hurry adds a query parameter in the controller and forgets the OpenAPI annotation, the agent writes a wrong test against the right code. We caught two of these in April and a pre-commit hook in the contract repository now blocks the drift. Stage 3 → Stage 4 on Context Engineering.

WireMock JSON mappings as first-class contract artifacts. The first instinct, ours included, is to treat WireMock stub JSON as "test fixtures" and let each suite invent its own. After the second OAuth bug that came from a stub that was technically wrong about the external service's response shape, we moved the WireMock mappings into a curated directory, tagged each with the upstream contract version they represent, and started reviewing them on PRs alongside the application code. Stub correctness is contract correctness. Stage 3 → Stage 4 on Verification & Quality.

Skipping sprint-aligned investment. The most expensive trap is the one that does not look like a trap. The temptation, when the harness is taking shape, is to run a "documentation sprint" - three days of writing docs, then back to features. We tried it once. The docs were stale by week three. What works for us is sprint-aligned investment: every sprint, the harness gets one of its dimensions sharpened - a new skill, a contract clean-up, a Runner improvement. The pace is unglamorous; it is also the only pace that produced the curves above. Stage 1 → Stage 2 on Process and Workflow, and the trap that decides whether the whole journey starts at all.

The matrix's brownfield reality check applies directly: stages 1 through 3 work on brownfield systems, but Stage 4 - systematic harness with mechanical verification - assumes a level of architectural cleanliness that legacy code rarely has. Our service was extracted from a larger one, with real organizational scars, and even then we needed ten weeks of foundation work before agent-generated tests were trustworthy. A genuinely legacy system would need that work to land first on a single bounded module before any agent-driven test generation made sense.

The honest framing is this: harness engineering for E2E does not skip the boring work. It rewards teams that did the boring work, and it punishes teams that did not.

For Monday - write one SCN, sketch one aggregate

If you want a concrete next step that costs you nothing in tooling and an hour of attention, do two things.

First, pick one critical business flow in your service that has no E2E test, and write the SCN. A scenario ID. Three preconditions. Three to six steps with an HTTP verb, an endpoint, and an expected status. You do not need an agent. You need a text editor and the discipline to be specific.

Second, sketch the aggregate the SCN references - one paragraph. What are the states of the thing the scenario manipulates? Who owns each transition? What is the invariant the scenario implicitly assumes (the list contains only published entries, the user is authenticated, the configuration ID is distinct from the server URL)? You will discover, in the writing, what you do not yet know - and that is the answer to "can the agent write this test for me."

If the SCN comes out clean and the aggregate is one paragraph you would defend at a review, you have the shape of the spec the harness rewards. If either falls apart, fix the documentation that has to exist for them to make sense. Then come back to the SCN. Then, and only then, ask the agent.

This is the bet we are making, three and a half months and 162 steps in: better specs produce better tests; the spec corpus compounds; the harness is the system that holds the compounding together. The flat CI curve, the five-fold drop in commits per step, the fact that a 20-step OAuth re-auth scenario now lands in a single commit - none of that came from a smarter model. It came from documentation an agent could read, an aggregate small enough to keep accurate, a scenario format that respects the agent's lane, and a discipline that points every fix at the spec.

If you can't write the scenario, the agent can't either. Write one this week. Sketch the aggregate next to it. See what you learn from the writing alone.

The Harness Model - AI Engineering Maturity Matrix, Q1 2026

Thu, 16 Apr 2026 00:00:00 GMT

Most organizations measure AI adoption by which models or tools they use. That tells you almost nothing about whether those tools produce reliable outcomes. The teams winning with AI are not the ones with the best models. They are the ones who have stopped treating AI output as a deliverable and started treating it as an unverified input to a system that catches errors.

The discipline does not disappear - it relocates.

This post introduces The Harness Model, a maturity matrix for AI engineering practices across ten dimensions. It is not a certification framework or a prescriptive roadmap. It is a diagnostic tool - a structured way for engineering teams to assess where they are and decide where to invest next.

The visual format is inspired by the Cloud Native Maturity Matrix by Container Solutions: same dimension-based grid approach, applied to AI engineering practices. This is a Q1 2026 snapshot. The field is moving fast. We expect to revise this quarterly as practices evolve and community feedback arrives.

The Harness Thesis

A convergence is happening across the AI engineering community: when AI writes the code, the craft shifts to designing the system that controls AI. Multiple practitioners have arrived at this insight from different angles. The individual concepts - harness engineering, context engineering, loop progression - have been described separately. Our contribution is the synthesis: a structured assessment model that maps where a team stands across all these dimensions simultaneously.

A harness is the system of context, constraints, and verification that wraps an AI agent. It is what turns raw model output into something you can trust. The harness includes the documentation agents read, the linters that enforce boundaries, the tests that catch regressions, and the feedback loops that improve all of these over time.

Mitchell Hashimoto described the core practice: anytime an agent makes a mistake, you engineer a solution so it never makes that mistake again. He called this harness engineering. OpenAI's team reported a striking demonstration: over five months, three engineers used Codex to build a million-line codebase with zero manually written code - a dogfooding result from the team that built the agent, not yet reproduced by customer teams at that scale. Their harness had three layers - context engineering that made the repository itself the system of record, architectural enforcement through custom linters and structural tests, and garbage collection agents that fought entropy before it compounded.

Birgitta Boeckeler observed that harnesses might become the new service templates - standardized starting points for common application topologies. She also raised the critical brownfield question: which harnessing techniques work for existing applications, and which only work for systems built with a harness in mind?

Chad Fowler placed this in historical context: when XP replaced phase-gate development, rigor relocated from documents to tests. The formula now: probabilistic inside, deterministic at the edges - let AI generate freely within boundaries that are strict, explicit, and mechanically enforced.

Kief Morris mapped the progression of how humans relate to agents. Start outside the loop (vibe coding - agent runs, human hopes). Move to in the loop (human reviews every line). Then on the loop (human builds the harness that controls agents). Finally, the agentic flywheel (human directs agents to improve the harness itself). Each step shifts human effort from execution to system design - from writing code to writing the system that writes code.

Taken together, the five practitioners describe different entry points into the same conclusion: reliable AI outcomes require a designed system around the agent, not a better agent.

The better your harness, the more you can trust the output. The worse your harness, the more time you spend reviewing, fixing, and re-prompting - which is often slower than writing the code yourself. METR's 2025 RCT found experienced open-source developers using AI tools were 19% slower on average, while believing they were faster. The harness thesis is exactly what explains that gap: without the system around the agent, the review-fix-reprompt tax dominates.

One principle runs through every stage of this model: accountability stays with the human. Maturity is not a transfer of responsibility to agents; it is the humans earning the right to delegate more execution because the system around the agents has been engineered to catch what they get wrong. When an agent ships a bug, it is still a human who answers for it - and that framing should shape every decision about where to invest next.

A related principle shapes the matrix itself: agents are first-class team members. If a piece of information - a backlog item, an architectural decision, a runbook, a metric - is available to humans but not to agents, the harness has a hole in it. "It sits in my slide deck" is the anti-pattern. Everything the team works from should be equally reachable by the humans and the agents on that team.

The Maturity Model

The Harness Model defines five stages across ten dimensions. Each stage describes a dominant interaction paradigm between humans and AI agents - not a maturity score to optimize. Most organizations today are at Stage 1 or 2. Stages 4 and 5 are aspirational endpoints, not expected norms. Stage 5 is our informed speculation - no validated production examples exist yet, and we included it because extrapolating where this is heading was too interesting to leave out.

We organize the ten dimensions into four clusters that form a causal story: Foundation (inputs - what agents work with), Governance (constraints - what keeps agents safe), Delivery (execution - how work gets done), and Outcomes & Learning (measurement and evolution - how results are assessed and how the system improves). Reading the matrix top-to-bottom follows this causal flow - inputs feed into constraints, which shape execution, which produces outcomes that feed the next iteration - while reading left-to-right shows progression through the five maturity stages.

A candid note before the table: we typically start greenfield teams at Stage 3 and have reached some Stage 4 characteristics in select dimensions - never all at once. If that sounds modest, it is meant to.

Foundation

Governance

Delivery

Outcomes & Learning

Dimension	No AI Process	Chatbot-Assisted	Human-in-the-Loop	Systematic Harness	Agentic Flywheel
Context Engineering	N/A	Copy-paste snippets into chat	Human maintains AGENTS.md; manual context loading	Repository as system of record; progressive disclosure	Human designs context structure; agents maintain and evolve it
Team (Humans + Agents)	Large specialist team; humans execute; standard tools	Same specialist team; humans execute with AI suggestions	Leaner delivery-minded team; agents use file/terminal	Small generalist team per initiative; agents run full stack	Same generalists span multiple initiatives; agents self-provision
Security & Trust	No AI-specific security concerns	Human reviews AI output for obvious issues	AI output runs through existing SAST/DAST tooling	Scoped, auditable agent access; AI-specific threat modeling	Agents enforce and evolve policies; human governs trust boundaries
Architectural Governance	No AI constraints	Basic rules in prompt files	Human-enforced boundaries; constrained solution space	Custom linters with remediation; taste invariants as code	Harnesses as org-wide templates; human defines taste
Human-Agent Interaction	N/A	Chatbot Q&A	Agent generates; human reviews every output line-by-line	Human on the loop; builds harness	Agents propose harness improvements; human directs evolution
Workflow & Process	No AI in workflow	Occasional AI for specific tasks	Daily agent use; human delegates parallel work	Always-running agents; agent-first; human sets priorities	Agents handle full delivery cycle; human steers outcomes
Reliability & Operations	Manual runbooks; engineers triage and mitigate incidents	Same ops team; agents draft queries; humans mitigate	Structured runbooks; agents analyze alerts; humans mitigate	Agents auto-triage low-severity alerts; humans handle critical incidents	Agents auto-remediate known failures; humans own novel incidents
Verification & Quality	Manual testing	Agent runs tests; human checks	Custom linters and structural tests; human reviews results	Agent-to-agent review with quality scoring; human defines criteria	Agents detect and fix regressions; human defines standards
Knowledge & Feedback Loops	Tribal knowledge; docs in wikis	README-level docs	Human structures docs in repo; retrospectives capture agent lessons	Versioned plans and quality grades in repo; agent failures feed the harness	Agents maintain docs and capture learnings; human curates strategy
Planning & Decision-Making	Manual boards; experience-driven decisions	AI helps write tickets and research options	Agent triages issues; human prioritizes; decisions validated via PoCs	Decision signals equally reachable by humans and agents; humans validate with cheap prototypes	Agents propose initiatives; humans set direction from measured results

Most teams today are at Stage 1 or 2. Stage 4 is aspirational; Stage 5 is informed speculation with no validated production examples yet.

Foundation: Context and Team

Context Engineering is how effectively agents receive the right information at inference time. The progression moves from copy-pasting snippets into a chat window to making the repository itself the single source of truth - with progressive disclosure (agents start with a map of the repo and pull in detailed documents only when a task needs them, rather than loading an encyclopedia up front). The key transition: at Stage 3 a human manually curates AGENTS.md. At Stage 4, the repository structure is the context, and documentation linters enforce that it stays current.

Team (Humans + Agents) describes the team as a single composition of humans and agents, because treating agents as second-class citizens is exactly the framing this model rejects. The row moves along two axes simultaneously: team shape (large specialist → business-oriented delivery → small generalist per initiative → generalist across multiple initiatives) and agent capability tier (none → suggestion-only → file/terminal access → ephemeral full-stack environments with browser automation and local observability → agents self-provisioning their own environments). Early pioneers built the full-stack environments from scratch - OpenAI's integration of the Chrome DevTools Protocol, which lets agents drive a real browser alongside code execution, is one example.

The human role evolves along this dimension from execution, to delivery leadership, to multi-initiative oversight. A nuance worth stating plainly: "AI provides specialist depth on demand" is only half true. Agents genuinely provide technical depth - frameworks, patterns, languages - on demand. They do not replace domain depth: the unique business rules, customer workflows, and market dynamics that give a product its edge. In complex-domain businesses, architects need to move toward close customer relationships as agents absorb execution work, not away from them. This is where the "human role evolves, never disappears" principle is most visible.

The Stage 3→4 jump is where both axes move together: teams get leaner because agent capability crosses the full-stack threshold. The trade-off is real - you pay for that leaner team with investment in the platform agents now run on (ephemeral environments, browser automation, observability) and in the governance that makes that leverage safe. Teams that try to shrink without paying either cost end up with the same number of humans and less agent trust.

Governance: Security and Architecture

Security & Trust governs how agent access, AI-generated code security, and supply chain integrity are managed. The progression follows a Zero Trust model: start with no AI-specific concerns, move through existing security tooling - SAST (Static Application Security Testing) and DAST (Dynamic Application Security Testing) applied to agent output just like human-written code - and eventually reach scoped, auditable agent access with AI-specific threat modeling. At higher stages, supply chain verification for agent tools becomes critical - treat tool descriptions and AI skills as potential attack vectors, not just convenience features.

Architectural Governance defines how constraints and design taste are enforced on agents. We use taste in the sense of refined judgment about trade-offs: which abstractions age well, which coupling is acceptable, which patterns reduce long-term risk. In an AI era where generation is cheap, taste is emerging as the new moat - an intangible, durable advantage that survives when everyone has the same models. "Taste invariants as code" means encoding that judgment into linters and structural tests so agents cannot unknowingly violate it.

Basic prompt rules give way to human-enforced boundaries, then to custom linters with remediation instructions injected directly into agent context. The OpenAI team's practice is instructive: they enforce module boundaries and data shape validation mechanically, but leave implementation choices to the agent. At Stage 5, harnesses would become organizational templates - the new golden path - and the human's central job would be defining the taste the harness enforces.

The Stage 3→4 jump is where the cost shows up: turning tribal taste into enforced linters requires senior engineers to name and codify judgment they have always held implicitly. That is slow work, and teams that skip it end up at Stage 3 forever - agents inside boundaries the humans cannot articulate, held together by constant review.

Delivery: Interaction, Workflow, Reliability & Operations

Human-Agent Interaction is the division of labor and trust boundary between humans and agents. This maps directly to Morris's loop progression: from chatbot Q&A (outside the loop) through line-by-line review (in the loop) to building the harness (on the loop) and eventually directing agents that improve the harness (flywheel). The common failure at Stage 3: teams that review every line of generated code become a bottleneck and conclude AI "doesn't save time." Remember that accountability stays with the human at every stage - the loop describes where the human engages, not whether they are on the hook.

Workflow & Process describes how humans and agents coordinate daily work. Hashimoto's journey is a practical illustration: from occasional chatbot use to "always have an agent running." The progression from "occasional AI for specific tasks" to "agent-first delivery" changes what engineering management looks like. At Stage 4, the question is not "should we use AI for this?" but "why would a human do this instead of an agent?"

Reliability & Operations covers what happens after deployment - runbooks, alerts, incident triage, mitigation, and remediation. If Verification & Quality asks "did we build it right?", this dimension asks "is it running right?"

Two axes progress together along this row. The runbook axis moves from tribal knowledge, to markdown-in-Git, to LLM-ready structured runbooks, to agent-executable playbooks - a senior SRE's ability to read dense observability queries becomes a shared capability as agents draft and simplify queries for the rest of the team. The alert-handling axis moves from manual query writing, through AI-drafted queries, to auto-analysis on alert that surfaces initial context within minutes, to severity-based auto-triage with human checkpoints on P1/P2 incidents, and finally to proactive anomaly detection that mitigates before customer impact.

The human role shifts from executing incidents, to mitigating with AI assist, to handling only critical incidents, to owning only novel failure modes. The Stage 3→4 jump here is not "agents get smarter" - it is "agents get trusted with low-severity triage end-to-end." That is a governance decision backed by better runbooks and observability, not blind delegation, and it reflects the principle that maturity is trust earned, not control removed.

Outcomes & Learning: Verification, Knowledge, Planning

Verification & Quality is how correctness is judged in agent-produced output. Early stages rely on human eyes. Later stages shift to programmatic verification - custom linters, structural tests, agent-to-agent review with quality scoring. Fowler's principle applies here: if generation gets easier, judgment must get stricter. A team at Stage 4 has encoded "what good looks like" into tooling, not tribal knowledge.

Knowledge & Feedback Loops tracks how teams capture, maintain, and evolve shared knowledge from agent interactions. The critical shift is from docs in wikis to versioned artifacts in the repository. At Stage 4, plans are first-class artifacts with progress and decision logs. Agent failures generate systematic feedback that improves the harness. Important data points - quality grades, benchmark results, regression signals - live in the repo so results from one agent iteration can be compared against previous iterations. Knowledge that lives only in Slack threads or people's heads effectively does not exist for agents.

Planning & Decision-Making shifts from heuristic-driven decisions and manual ticket boards to data-informed, continuously validated approaches. The underlying principle is equal access: every signal a human uses to make decisions - the backlog, metrics, logs, traces, fitness functions, customer feedback - must be equally reachable by agents. At Stage 4, agents propose and score items against those signals; humans validate the important decisions by building cheap prototypes and measuring real outcomes. The shift is from "we think this is right" to "we built something small and measured."

As team composition becomes more generalist, the Outcomes & Learning cluster is what prevents that generalism from becoming shallow: the measurement and feedback machinery is where taste is refined and where agents earn the trust they are given.

Brownfield Reality Check

All the source articles lean greenfield. Stages 1 through 3 apply equally to brownfield and greenfield systems - chatbot use, human review, and basic linting work regardless of codebase age. Divergence starts at Stage 4.

First, a definition. When we say "brownfield" in this article we mean the common case: a legacy codebase carrying real tech debt, with patchy documentation, weak automation, and processes shaped more by history than design. A well-maintained long-lived codebase is a different beast. A well-maintained codebase can accelerate agents - its established templates and copy-paste patterns give them more to imitate than a greenfield project where everything is built from scratch. To be clear, we believe that the blocker is not age, but rather the lack of a mature codebase (debt + disorder).

Systematic harness engineering assumes a level of architectural cleanliness that debt-laden legacy codebases rarely have. Module boundaries need to be clear. Dependency directions need to be enforceable. Test coverage needs to be sufficient for agents to verify their own work.

Harness-readiness checklist for brownfield systems. Before building a harness on top of a legacy codebase, work through these three questions:

Modularization: Can you name, in a sentence, what each module owns? If ownership is ambiguous, agents will generate code that belongs nowhere and everywhere.
Boundary enforcement: Are module dependencies explicit and directional? If not, agents will create new coupling faster than humans can untangle it.
Test coverage: Can agents verify their own changes without human intervention? Gaps in coverage become gaps in agent autonomy.

Teams that skip harness-readiness and jump straight to agentic workflows typically regress within weeks - agent-generated entropy compounds faster than manual cleanup can address it. This is why Stage 4 remains largely unvalidated for brownfield systems as of Q1 2026. What we know works: Stages 1-3 apply equally, and AI can accelerate the harness-readiness work itself. Boeckeler's analogy is apt: it is like running a static analysis tool on a codebase that has never had one. You will drown in alerts unless you tame the foundation first.

Mixed Maturity and Regression

We have seen this pattern firsthand: one team scored Stage 4 on Context Engineering - structured AGENTS.md files, repo-level documentation wired into every prompt - yet sat at Stage 2 on Verification, relying entirely on manual code review to catch agent mistakes. In practice, this looked like agents producing architecturally sound code that still broke integration tests nobody ran automatically. Once the team prioritized automated verification - wiring CI checks into the agent feedback loop - their context investment finally compounded: agents could self-correct instead of waiting for a human to spot failures. The weaker dimension was the bottleneck, not the stronger one.

Mixed maturity across dimensions is the norm, not a failure. When prioritizing, start with the dimension that creates the most pain or blocks progress in others.

Regression is also real. Last year, many teams stuck at Stage 2 abandoned AI entirely and regressed to Stage 1. This year, the most common struggle is overcoming Stage 3 - teams have justified fear of giving too much control to AI without sufficient verification and governance in place. The harness is exactly what addresses that fear: you do not hand over control, you build the system that makes delegation safe.

Where Are You? A Conversation Starter

This model is not a certification. It is a diagnostic tool - best used as a team conversation starter, not a solo assessment. Grab a whiteboard, walk through the dimensions, and see where disagreements surface. Those disagreements are the signal. A practical starting point: circulate the matrix to your senior engineers before your next staff meeting. Ask each person to score one or two dimensions independently. Bring the disagreements to the meeting - you do not need consensus, you need the conversation.

Foundation

What does your agent read before it starts a task? A single AGENTS.md? Structured documentation? The repo itself?
Is every piece of information your humans rely on equally reachable by your agents, or does some of it live in slide decks, private notes, and DM threads?
If you removed one specialist from the team, could agents cover the technical gap today? And who is close enough to the customer to cover the domain gap?

Governance

What can your AI agent access? Is that access scoped and auditable, or "whatever the developer's token allows"?
If an agent violates an architectural boundary, how long before someone notices? Minutes, days, or the next quarterly review?
Which pieces of your design taste are encoded as linters or structural tests, and which still live only in senior engineers' heads?

Delivery

How much of your day involves an agent running on a task? Zero percent? Ten? Fifty?
When an incident fires at 3am, does an agent surface initial context within minutes, or does a human start from a blank terminal?
When an agent ships something that breaks in production, is it clear that the human on call still owns the outcome?

Outcomes & Learning

Could a new team member find your verification criteria in code, or only in someone's head?
Where does your team's knowledge live - in Slack threads, wiki pages, or versioned artifacts in the repository?
Are the signals you use to decide what to build next - metrics, traces, fitness functions, customer feedback - equally available to your agents?

If you are not sure which stage your team is at for a given dimension, that ambiguity is itself a signal. It usually means you are between stages - and the uncertainty points to where investment would have the most impact.

As mentioned earlier, we typically start greenfield teams at Stage 3 and have reached some Stage 4 characteristics in select dimensions. For brownfield teams, getting to Stage 2 is usually straightforward, but moving to Stage 3 takes hard work from the whole team - and so far we have not progressed beyond that for brownfield systems. If that sounds familiar, you are in good company.

Start Where You Are

The harness is the through-line. Whether you are writing your first AGENTS.md or building custom linters with remediation instructions, you are doing harness engineering. The question is not whether to build a harness. It is how deliberately you build it - and remembering, at every stage, that the accountability for the outcome still lives with a human.

This is a Q1 2026 snapshot - a starting point that will evolve as practices mature and community feedback arrives. Pick one dimension where you feel the most pain. Identify what stage you are at and what the next stage looks like. Build one thing that moves you forward. That is harness engineering. If your team is just getting started, our guide on how to start with AI-assisted development covers the practical first steps.

Compare notes with us

We're collecting patterns across teams to inform the Q2 2026 update of the Harness Model. Whether you ran the full assessment or just found one dimension that matched your team, we'd like to compare notes - and we'll share back what we're seeing across teams in the Q2 update.

Email us at contact@handsonarchitects.com or DM either of us on LinkedIn: Maciej · Tomasz.

How to Start using AI in Software Development Today

Mon, 16 Mar 2026 00:00:00 GMT

Most engineers we talk to are not skeptical about AI-assisted development. They are interested but stuck. They are waiting for the right project, the right tooling setup, or some form of organizational buy-in before they begin. That moment rarely arrives on its own. The practical alternative is much simpler: run a small, bounded experiment today on code you already know. Not a week-long initiative. An hour.

This post gives you a mental model for working with AI, the single skill that matters most, clear guidance on when to slow down, and a concrete first task you can try before your next meeting.

Stop Waiting, Start Experimenting

Two weeks of focused, hands-on work will teach you more about where AI helps in your context than a year of reading blog posts (not this one! ;)). The trap is treating AI adoption as a big decision that requires preparation. It is not. It is a series of small experiments that build intuition. The prompts in this post work with any major AI assistant - ChatGPT, Claude, Copilot, Cursor, or whatever your organization permits. If corporate policy restricts tool access, that is a real blocker worth escalating, but it is not a reason to delay learning the underlying skill.

The key principle: start with familiar territory. Pick a codebase, a module, or a problem you already understand well. When you know the ground truth, you can evaluate AI output accurately. You will spot where it oversimplifies, where it misses domain nuance, and where it nails something that would have taken you twenty minutes to write. That calibration is the foundation for trusting AI on unfamiliar work later.

If you wait for the perfect project, you are optimizing for a scenario that does not exist. If you start with something you know, you are building judgment that transfers to everything you do next.

AI Is an Amplifier, Not a Replacement

The most useful mental model we have found: AI is a force multiplier. It does not generate quality on its own. It amplifies whatever direction you point it in - including the wrong one.

This is the part most productivity discussions get wrong. The real gains from AI-assisted development depend heavily on the experience and breadth of knowledge you bring to the session. An experienced engineer who understands the problem space, knows the codebase, and has clear intent will get dramatically more value from AI than someone who is guessing at requirements and hoping the model fills in the gaps. The difference is not a fixed number. It is directional.

It does not matter if you are the fastest runner in the world if you run in the wrong direction.

AI handles the mechanical parts of development well: boilerplate, syntax, repetitive refactoring, first-draft documentation. What it cannot do is understand the actual problem, design the right solution, or judge whether the output aligns with business goals. Those stay with you. The shift is not from engineer to spectator. It is from bricklayer to architect - you direct execution and validate results instead of doing all the mechanical work yourself.

Here is what that amplification effect looks like in practice:

Strong engineering practices combined with AI produce fast, high-quality delivery. Weak foundations combined with AI produce bugs faster and at larger scale. The AI does not judge what it amplifies. Your practices, your domain knowledge, and your engineering judgment determine the outcome - AI just makes it happen faster.

Context Engineering: The Skill That Actually Matters

If there is one skill that separates effective AI-assisted development from frustrating prompt-retry loops, it is context engineering - structuring the input you give to AI so the output is actually useful on the first pass.

Vague prompts produce generic output. Precise prompts, loaded with relevant context, produce drafts worth building on. The difference is not about cleverness or prompt tricks. It is about specificity.

Compare these two prompts for the same task:

Vague:

Write tests for the user service.

Precise:

Write unit tests for the UserService class in /src/services/user-service.ts.
It uses the Repository pattern with a PostgreSQL-backed UserRepository.
Cover the createUser and deactivateUser methods, including the case where
deactivateUser is called on an already-inactive user. Use Jest. Follow the
existing test patterns in /src/services/__tests__/.

The second prompt is not clever - it is specific. It names the file, the patterns in use, the edge cases that matter, and the conventions to follow. That specificity is what makes the output useful on the first pass rather than the third.

This is why we treat prompting as an engineering discipline. It rewards the same rigor you apply to code: clear intent, relevant context, explicit constraints, and iterative refinement. The engineers who get the most from AI are not the ones with the fanciest tools. They are the ones who invest a few extra minutes describing what they actually need.

When to Slow Down

AI-generated code can introduce subtle bugs, miss domain nuances, reference deprecated APIs, or misread business logic - all with total confidence. You are the validation layer. Human-in-the-loop is not a suggestion; it is the operating model.

Three categories consistently require tighter oversight:

Security-critical code. Authentication flows, authorization logic, input validation, secrets handling. These areas require human audit regardless of how clean the generated output looks. AI cannot assess the risk implications of what it writes.

Novel architecture and problem framing. AI is fluent in known patterns. When you face a genuinely new problem or design a system with unusual constraints, the AI will confidently anchor on the closest familiar pattern - which may not fit. Use AI to surface options, but own the design decisions yourself.

Learning fundamentals. If you are still building core skills in a language or framework, heavy reliance on code generation creates dependency rather than understanding. Use AI to explain, not to replace your thinking. A simple test: if you cannot walk through generated code line by line and explain what it does, you do not yet own it.

Beyond these categories, watch for diminishing returns. When you have revised the same prompt three or four times and the output keeps missing the mark, stop. Reframe the problem manually, or write the code yourself. Continued iteration rarely recovers a session that has lost direction. At this point we recommend taking a break and returning with fresh eyes later. There is good chance you were tackling problem from the wrong angle, and more prompting is just reinforcing that.

A practical heuristic for calibrating oversight: the harder it is to reverse a change, the stricter your review should be. A draft document is easy to discard. A refactoring committed to main and deployed is not.

Your First AI Experiment (Pick One)

The tasks below are designed for trust-building, not maximum ambition. Each one takes under an hour and uses a problem you already know. Starting with familiar territory gives you ground truth to evaluate AI output - you will catch mistakes you could not catch in unfamiliar code.

Pick the one that matches your role and try it today.

For Engineers: Explore a Codebase You Already Know

Pick a module you understand well and ask AI to explain it as if onboarding a new team member.

I am going to share a module from our codebase. You are a senior software
engineer explaining this code to a new team member who has never seen it.
Assume they are technically proficient but do not know our codebase or domain.
Please explain:
1. The overall responsibility of this module
2. The key components and how they relate to each other
3. Any design patterns in use
4. Anything that looks unusual or potentially problematic

[paste the module code here]

What to expect: A structured explanation you can check against what you know. Where does the AI get it right? Where does it oversimplify or miss context? This calibration tells you what level of trust to extend in future sessions.

For Tech Leads: Break a Feature into Backlog Items

Take a loosely defined feature your team is about to start and ask AI to decompose it.

Break down the following feature into backlog items for my team.

Feature: [describe in 2-3 sentences]
Tech stack: [list relevant technologies]
Key constraints: [deadlines, dependencies, compliance requirements]

For each item include: a clear title, short description, acceptance criteria,
dependencies on other items, and open questions.

What to expect: A structured first draft that covers the main implementation tasks and integration points. You will need to reorder, merge, or split items - the value is in surfacing the decomposition quickly, not accepting it wholesale. Once you have prompts that consistently produce useful output, save them - a shared prompt library is one of the fastest ways to scale AI-assisted practices across a team.

For Architects: Brainstorm Architectural Options

Pick a design decision you are currently weighing and ask AI to surface alternatives.

I am designing a solution for the following problem:

Problem: [describe clearly]
Current system context: [relevant services, data flows, constraints]
Key requirements: [non-negotiables]

Suggest 3-4 architectural approaches. For each: describe the approach,
list main advantages, list main disadvantages or risks, and note
assumptions it depends on.

What to expect: A set of options with trade-offs. AI is good at recalling and articulating known patterns - expect solid coverage of conventional approaches. What it will not give you is judgment about which option fits your organizational or operational constraints. Use the output as a menu of options, not a recommendation. Once you are comfortable with brainstorming, try using AI as an architect buddy for ADRs - it is a natural next step.

Pro tip: Leave your favorite approaches out of the initial prompt to avoid anchoring the model. Add them in a follow-up to compare against the options it generated independently.

Start Today, Iterate Tomorrow

The goal is not one perfect prompt. It is building a habit of AI-assisted work that consistently produces better outcomes than working without it. That habit starts with a single bounded experiment.

Here is your next step: pick one of the tasks above, set a timer for one hour, and try it on code or a problem you already know. Pay attention to where the AI is helpful and where it falls short. That observation - not any blog post - is what builds the judgment you need for everything that comes next.

The engineers who start experimenting now will have a significant advantage over those still waiting for the right moment.

The right moment is today.

AI Toolset for Software Architects (Q1 2026)

Thu, 26 Feb 2026 00:00:00 GMT

Artificial intelligence is now embedded in day-to-day architecture work. The question is no longer which AI tool to adopt - it is how to curate, evolve, and integrate a continuously updated toolset across the full software development lifecycle. Since our Q3 2025 edition, the landscape has shifted in meaningful ways: we added two categories (Planning & Project Management and Quality Assurance / Continuous Integration), one option we expected to lead (OpenAI Codex) underdelivered in practice, and our prompting approach matured into a discipline that cuts across everything we do.

This post covers the current state of our AI toolset as of February 2026, highlights what changed, and shares the lessons we picked up along the way. If you are new to the series, the Q3 2025 edition covered four categories - Searching & Learning, Brainstorming & Ideation, Prompt Engineering, and Agentic Coding & Prototyping. Here is what shifted since then.

How the Architect's Role Continues to Evolve with AI

Four principles guide how we build and maintain our toolset:

Build a complementary toolset, not one perfect tool. No single AI covers everything. Combine tools that cover research, design, planning, coding, and quality - each doing what it does best.
Treat your toolset as continuously evolving. Adopt an experimentation habit rather than searching for a stable, final configuration. What works today may be outpaced in a quarter - or sometimes within a week. For example, once Skills gained broad attention as a reusable capability standard, MCP (Model Context Protocol - a standard for connecting AI agents to external tools and data sources) usage shifted away from being a local tooling accelerator and toward a clearer role: bridging agents to remote services and live data.
Leverage AI across all lifecycle stages. From early research and design through planning, implementation, and into maintenance - there is no phase where AI cannot accelerate or augment your work.
Capture and operationalize shared knowledge. Decisions made in the morning should inform work done in the afternoon. Meeting transcription and insight extraction are key parts of this process. Maintaining up-to-date documentation is essential for AI to provide accurate, relevant assistance.

What Changed Since Q3 2025

The most significant structural change in our toolset is that Prompt Engineering is no longer its own category. It has become a metacategory - a cross-cutting discipline that shapes how we interact with every other tool. Good prompting is now an integral part of all our AI workflows, not a specialized skill in one workflow stage. (See Prompt Engineering as a Metacategory below for what this means in practice.)

Equally significant: we expanded into two new areas that were missing from the previous edition:

Planning & Project Management - we started actively using AI to support backlog grooming, task decomposition, and integration with project management tools.
Quality Assurance / Continuous Integration - we now use AI-assisted hooks, code review automation, and browser testing tools as part of our CI feedback loops.

The five-category model we use now better reflects where architects actually spend time.

AI Tool Categories - Q1 2026

1. Searching & Learning

Perplexity.ai remains our primary tool for quick search across sources, exploring new technologies, and summarizing recent research. What has changed is how we use it: increasingly via its API inside more complex, multi-step workflows rather than as a standalone chat interface.

Obsidian is where this research ultimately lands, and Markdown is now our default documentation language. That decision is mostly practical: Markdown is the native format we use to communicate with LLMs, and it handles prose, code snippets, and diagram-as-code (for example Mermaid) equally well in one place. In practice, this gives us a better AI interface than wiki-style pages such as Confluence, which are harder to keep structured, portable, and model-friendly at scale. Every research thread ends as a curated Markdown knowledge-base entry with clear explanations, practical analogies, best practices, and verified examples. We then connect this local knowledge base back into AI chats via local MCP/RAG, creating a continuously improving feedback loop between daily work and future AI sessions.

A typical pattern looks like this:

Use Perplexity API to search and identify relevant sources on a topic.
Pipe results into Firecrawl to scrape and extract full content from the most promising pages.
Summarize and synthesize using an LLM (e.g., Claude) to extract key insights from the aggregated content.
If the topic is generic enough to be reusable, create a structured Markdown entry in Obsidian with the synthesized insights, examples, and references.

This pipeline makes AI-powered research significantly more thorough and reproducible than ad-hoc searches. It is especially effective when you need to validate architectural assumptions against current documentation, vendor blogs, and community discussions simultaneously.

Over time, these entries become curated, personalized training content for the team. We loop that context back into future AI sessions so new prompts are grounded in our language, decisions, and mental models rather than generic internet summaries.

During research, we also ask AI to explain complex problems and architectures visually using Mermaid diagrams. Mermaid works especially well with Markdown-native editors because diagrams are rendered directly from source, without the extra PNG/SVG export step, which keeps them easy to update as ideas evolve. In practice, we use it for multiple diagram types: flowcharts, sequence diagrams, state diagrams, and C4 diagrams (context, container, component, and deployment views). Because these notes live in Git, diagram changes can be reviewed in normal pull requests, where team members comment on specific lines and propose improvements exactly as they do for code.

2. Brainstorming & Ideation

Chat interfaces continue to work well for early-stage thinking. We use both ChatGPT and Claude (the chat interface, not Claude Code) depending on the task - neither has displaced the other in this category. What matters more than the model is the quality of the prompt and the clarity of what you are trying to explore.

Brainstorming use cases remain largely unchanged: generating design alternatives, stress-testing assumptions, exploring edge cases before committing to a direction, and facilitating design discussions when you want a second (non-human) perspective.

What has improved is our capture-and-review loop. The difference is concrete. Before: chat logs from design discussions and meeting notes sit in Slack threads or personal notebooks, slowly going stale and disconnected from the codebase. After: AI converts those same transcripts and chat outputs into structured Markdown documentation with Mermaid diagrams that live in the repo, are version-controlled, and stay current as the design evolves. In practice, this means a 30-minute architecture brainstorm now produces a pull request with updated context diagrams and decision rationale - not a Slack message that three people will search for next week and nobody will find.

3. Planning & Project Management

This is a new category in our toolset - and one of the more nuanced additions.

For smaller projects, we prefer simple Markdown-based backlogs and task lists maintained in the repository. They are lightweight, version-controlled, and work well with AI tools that can read and update them directly.

For larger projects involving multiple team members, we use the Atlassian MCP to interact with Jira within Claude Code sessions. However, there is an important practical caveat: the Atlassian MCP is token-heavy and consumes significant context. Our recommendation is to avoid running it continuously. Instead, adopt a sync-work-sync pattern:

Sync Jira → Markdown (pull the current state into local files).
Do the planning and task management work in Markdown.
Sync Markdown → Jira (push updates back) in a separate session.

This keeps your AI sessions focused and avoids flooding the context window with Jira API overhead.

A broader observation: Jira is becoming less useful for engineering teams. In practice, it primarily serves project managers who need visibility - but it adds friction for engineers. We are actively exploring lighter alternatives, starting with Markdown-based backlogs in Git repositories, and watching Linear as a potential replacement. Some of the more technical Project Managers already started to use Claude Code to "ask" Markdown-based backlogs for status updates and to provide updates to stakeholders - this is a much more efficient workflow than navigating Jira's UI for the same purpose.

In practice, this category now consumes more time than coding in early phases of a project. We use AI to shape clearer roadmaps and produce more detailed milestones and task definitions that are implementation-ready for the team.

This up-front decomposition is essential for AI-first delivery: when milestones and tasks are explicit, scoped, and well-sequenced, execution accelerates dramatically. We intentionally invest more effort at the beginning to align on the problem and solution space before writing code.

4. Coding and Prototyping

Claude Code remains our frontier tool for agentic coding. After a brief experiment with OpenAI Codex and Augment Code, we returned to Claude Code - it remains ahead in terms of agentic behavior, context handling, and integration ecosystem.

Our current focus is on building skills and subagent libraries for specific projects and testing how those building blocks scale in larger teams (3+ engineers), with mixed but promising results so far. Rather than relying on general-purpose prompting (or off-the-shelf prompt templates and frameworks), we invest in project-specific, high-quality assets:

Skills: reusable, task-specific prompt packages (slash commands) that encode domain knowledge, constraints, and preferred workflows for a given project context. Think of them as documented expert procedures that any agent session can load. A concrete example is a Docker image validation skill focused on checking images against container best practices and verifying base-image choices. Skills like this often bundle scripts and validation logic, giving agents stronger guardrails than prompt text alone.
Subagents: specialized Claude Code agent instances scoped to a focused task - for example, a Domain-Driven Design expert that helps identify bounded contexts, a Code Reviewer tuned to a specific language or framework, or a Trade-off Analyst that evaluates architectural options against defined criteria.

The distinction matters: a skill is a single-purpose, reusable prompt package, while a subagent is a scoped execution context, that can run in scaled sessions and be orchestrated with other subagents. Skills are what subagents follow.

Once the library of skills and subagents is in place, we execute project backlogs using teams of subagents operating with those specific skills. This dramatically improves consistency and reduces the per-task prompting overhead.

We continue to use a small, curated set of MCPs:

MCP	Purpose
Serena	Codebase querying - works especially well with larger codebases
Context7	Documentation browsing during coding sessions, more here
Sequential Thinking	Structured planning and multi-step reasoning

Keeping the MCP list small and intentional is a deliberate choice. Each MCP adds context overhead and potential noise - more is not always better (we are aware of Claude's MCP auto-discovery feature, but managing context explicitly has proven more reliable).

We also maintain a few cross-project skills that multiple subagents share, such as (but not limited to):

Documentation structure: keeps commercial and open-source documentation aligned to a shared, easy-to-navigate structure, with explicit documentation-map files that make content simple for both humans and LLMs to discover and traverse.
Documentation style guide: enforces a consistent writing style inspired by the Microsoft Writing Style Guide, implemented as a practical skill split into categories with English examples.
Mermaid diagram skill: translates natural-language architecture and ADR ideas into the most appropriate Mermaid diagram type and applies consistent visual conventions (shapes, colors, and notation) across repositories.

5. Quality Assurance / Continuous Integration

This is the other new category, and it is one of the most promising areas for architects who want to influence engineering standards at scale.

Claude Code Hooks are our first "tool" here. Hooks are lifecycle event handlers in Claude Code that can execute deterministic checks at specific points in the agent workflow. A practical example: attach a script to the Stop event (or SubagentStop if you are working with subagent orchestration) that automatically runs tests and linters after each agent completes its task. This turns AI-generated code into code that is continuously validated - not just generated.

GitHub Copilot for code review has improved noticeably. AI-assisted code review now covers basic quality checks reliably, and we are increasingly comfortable using it as a first pass before human review - particularly for generated code, configuration files, or changes that are high-volume but lower-risk. We still keep humans in the loop for anything critical or architecturally significant.

Chrome DevTools MCP rounds out this category. It provides a simple but effective way to give AI visibility into web application behavior - inspecting the DOM, checking network requests, and validating rendering - without leaving the agent workflow. For web-focused projects, this closes a gap that previously required manual browser inspection.

We are exploring how to use AI to automatically validate whether code stays consistent with architectural decisions and documentation. Today we do this periodically as part of our QA process, but the goal is tighter integration - running architectural consistency checks directly in CI pipelines so that drift is caught early, not during a quarterly review. This connects directly to principle #4: if architectural decisions and documentation are kept up to date and in AI-friendly formats, automated validation becomes practical rather than aspirational. Early experiments are promising, though the challenge remains defining "consistency" precisely enough for an AI check to be actionable without generating noise.

Prompt Engineering as a Metacategory

In Q3 2025, Prompt Engineering had its own category. It no longer does - not because it matters less, but because it matters everywhere.

Good prompting is now the backbone of how we interact with every tool in the list. The principles that emerged from dedicated prompt engineering practice - clear context, well-scoped tasks, explicit constraints, iterative refinement - apply equally to Perplexity searches, Jira syncs, Claude Code subagent configurations, and QA hooks.

At the same time, this is not a rigid "ways of working" template. Prompt engineering is iterative, dynamic, and continuously adjusted as project context changes. The process is rarely linear or fully repeatable; it is a creative collaboration where human judgment remains essential.

Treating prompting as a metacategory is also a signal to teams: it is an engineering discipline, not a soft skill. It deserves the same investment as writing good tests or designing good APIs.

To Explore

The toolset never stands still. Here is what is on our radar for the coming quarter:

Linear - piloting as a lighter replacement for Jira, with a developer experience that better fits engineering-first teams.
Brave Search - evaluating as an alternative or complement to Perplexity for searching and learning, particularly for privacy-sensitive research contexts.
Playwright CLI with AI - exploring AI-assisted testing capabilities through Playwright's CLI interface.
GitHub Agentic Workflows (preview) - build GitHub Workflows with Markdown rather than YAML only. This is a promising direction to use CI in more ways.

Our Current Thoughts on Working with AI Tools

After another quarter of hands-on work, four things stand out as field notes from the trenches:

AI accelerates thinking - it does not replace it. AI is a force multiplier for engineering judgment, not a substitute. The more architectural maturity you bring to AI collaboration, the better your outcomes. Architects who lean on AI to skip the thinking tend to get mediocre results faster. (For more on the foundational skills that make this work, see our post on the Staff Engineer Toolkit.)

Prompting is an engineering discipline. Own the context. We consistently observe a significant difference between sessions that have too little context (under-informed, generic output) and sessions that have too much context (noisy, unfocused output). There is no silver bullet - but the habit of researching and documenting context close to the task before starting an AI session pays dividends every time.

AI amplifies both good and bad practices. If your processes are sloppy, AI will make them sloppier at scale. If your architectural thinking is sharp, AI will amplify it. This is perhaps the strongest argument for investing in learning and foundational understanding: in a landscape where implementation details can be delegated to AI, the competitive advantage lies in understanding the why behind those details - well enough to guide, review, and correct the AI's output.

Automated information flow and collection is key. Whether it's research, documentation, or test case generation, the quality of AI output depends critically on the information it has access to. Automated (or semi-automated) collection of up-to-date information from multiple sources (meeting transcripts/notes, Architecture Decision Records, Confluence pages, Roadmaps, changelogs, etc.) and transforming it into consistent AI-friendly formats is essential.

Closing Principles

There is still no final or perfect AI toolset for software architects. Quarter by quarter, the practice comes down to four durable commitments:

Continuous evolution: adopt, test, and retire tools as needs change. One quarter's frontier tool may be the next quarter's baseline - OpenAI Codex being a fresh example from this edition.
Balanced coverage: ensure the toolset spans research, ideation, planning, coding, and quality. Gaps in any stage create bottlenecks that slow the whole team.
Architect mindset first: tools augment judgment; they do not replace it. The most valuable thing you bring to any AI collaboration is clear thinking about what you actually need and why.
Shared knowledge as infrastructure: capture decisions, research, and context in formats that both humans and AI can use. The quality of AI output depends on the quality of information it has access to.

The architects who thrive in this environment are not those who adopt the most tools - they are the ones who build the habit of thoughtful experimentation and honest evaluation, one quarter at a time.

The following is a terminology note for readers less familiar with how "agent" and "agentic" are used in this post and in the broader industry.

Agent or Agentic

"Agent" and "agentic" are among the most overloaded terms in the AI space - right alongside "artifact" or "service." They mean very different things depending on context, and the lack of consensus creates real confusion. Here is how the ambiguity plays out in practice:

"Agent" as a runtime orchestrator - tools like Claude Code that autonomously plan, execute, and iterate on multi-step tasks. This is the strongest sense of the word: an agent that reasons about what to do next and acts on it.
"Agent" as a scoped prompt package - what many teams call a "code review agent" or "testing agent" is often just a well-crafted prompt with specific instructions, not an autonomous system. In our toolset, we call these subagents to avoid the confusion.
"Agent" as a single instruction - in some contexts, even "write tests for this code" qualifies as an "agent," stretching the term to near-meaninglessness.
"Agentic" as a behavioral property - describes a workflow or tool that exhibits autonomy, tool use, and multi-step reasoning, as opposed to a single request-response exchange. A coding session where the AI reads files, runs tests, and iterates on failures is agentic; a one-shot code generation prompt is not.

In this post, when we say subagent we mean a Claude Code agent instance scoped to a focused task and orchestrated as part of a larger workflow. When we say agentic, we mean a workflow where the AI operates with autonomy across multiple steps. We avoid using "agent" on its own precisely because of the ambiguity above.

What makes an MCP server successful?

Thu, 12 Feb 2026 00:00:00 GMT

💡 For a quick summary, jump to the TL;DR section.

The rise of MCP: A new standard for AI integration

When Anthropic introduced the Model Context Protocol (MCP) in late November 2024, they pitched it with a simple but powerful analogy: MCP is the "USB-C port of AI applications." Just as USB-C created a universal connection standard for physical devices, MCP creates a universal extension point for Large Language Models (LLMs). It lets LLMs connect to external tools and data sources - databases, search engines, ticketing systems, and countless others - in a standardized way.

Before MCP, every AI application needed custom connectors for each external system - 5 AI tools and 100 services meant 500 potential integrations. MCP turned this M×N problem into an M+N one: each AI tool implements the MCP client protocol once, each service implements the MCP server protocol once, and they all interoperate seamlessly.

What happened next surprised even MCP's creators. In November 2024, Anthropic open-sourced the protocol with comprehensive specifications, SDKs for Python, TypeScript, Java, Kotlin, and C#, and reference implementations.

Then adoption accelerated rapidly. Cursor, Windsurf, and Claude Code added support in early 2025. By April, VS Code, GitHub, and OpenAI had all adopted MCP. Cloud infrastructure providers like AWS, Google Cloud, and Microsoft Azure integrated it into their AI offerings. Even specialized tools like Figma and Stripe built their own MCP servers.

But with thousands of MCP servers now in existence, a critical question emerges: what sets the truly successful ones apart from the thousands that remain unused? To answer that, we need to look at where successful MCP servers live, what problems they solve, and how they are built.

Successful MCP servers: Where to find them

The MCP ecosystem has grown so rapidly that several dedicated directories have emerged to help developers discover MCP servers. MCP Hub, mcpmarket.com, glama.ai/mcp, and aiagentslist.com/mcp-servers all curate collections of available servers with descriptions, installation instructions, and usage statistics.

But raw counts of MCP servers tell only part of the story. The more interesting question is which servers have actually earned developer trust. When we rank MCP servers by GitHub stars, a clear picture of the ecosystem's leaders emerges:

Rank	MCP Server	GitHub Stars	Description
1	upstash/context7	44,000+	Up-to-date documentation for AI coding assistants (as of January 2026)
2	mindsdb/mindsdb	30,000+	AI integration with databases
3	1Panel-dev/1Panel	25,000+	Server management platform
4	github/github-mcp-server	20,000+	GitHub repository interaction
5	microsoft/playwright-mcp	15,000+	Browser automation for AI

Context7, developed by Upstash, stands out with a substantial lead of over 44,000 GitHub stars and over 240,000 weekly npm downloads. That kind of adoption doesn't happen by accident. Context7 has found genuine product-market fit - solving a problem that developers encounter daily and solving it well enough that adoption grows organically.

This makes Context7 an ideal case study for understanding what makes an MCP server successful. The following sections examine the problem it solves, the architecture behind it, how it implements the MCP protocol, and the key factors that drove its adoption.

Context7: What problem does it solve?

Every developer who has used an AI coding assistant has experienced the same frustration. You ask the AI to generate code for a library you are working with, and what comes back looks convincing - but doesn't compile. The method names are wrong. The API has changed since the model was trained. The code mixes patterns from two different major versions of the same framework. You end up spending more time debugging the AI's output than you would have spent writing the code yourself.

This frustration arises from four fundamental problems that affect AI coding assistants today.

Outdated training data

Large Language Models are trained on documentation snapshots that are typically 6 to 12 months behind the current releases. If you are working with Spring Boot 3.5, but the AI was trained on Spring Boot 2.7 documentation, it confidently generates code using patterns, annotations, and configuration styles that are outdated or deprecated.

Hallucinated APIs

The most frustrating problem is when the AI confidently suggests methods, classes, or annotations that don't exist. You paste the generated code into your IDE, it doesn't compile, and you waste time debugging phantom APIs that the model fabricated from patterns it learned during training. For example, a model might suggest a JUnit 5 annotation like @TestOrder that looks perfectly reasonable but was never part of the JUnit API.

Generic answers without specifics

When you ask an AI how to implement OAuth2 in your application, you often get a high-level explanation that reads more like a textbook than working code. "Use Spring Security's OAuth2 support" is technically correct but practically useless when you need the specific configuration classes, filter chain setup, and property values for your framework version.

Version conflicts

The AI frequently mixes APIs from different library versions in the same code snippet, combining deprecated methods alongside current ones. This creates code that is broken from the start - perhaps using javax.persistence annotations in a Spring Boot 3.x project that requires jakarta.persistence, or mixing Spring Security's old WebSecurityConfigurerAdapter pattern with the newer SecurityFilterChain approach.

How Context7 solves this

Context7 addresses all four problems with a straightforward approach: it gives AI coding assistants real-time access to the latest official documentation and code examples. Instead of relying on what the model memorized during training, Context7 pulls current, version-specific documentation from framework source repositories and injects it directly into the LLM's context window at the moment of prompting. The AI then generates code based on this fresh, authoritative information rather than its outdated training data.

The significance of this approach hasn't gone unnoticed by the industry. ThoughtWorks Technology Radar recently added Context7 to its "Tools" section, noting:

Context7 is an MCP server that addresses inaccuracies in AI-generated code. While LLMs rely on outdated training data, Context7 ensures they generate accurate, up-to-date and version-specific code for the libraries and frameworks used in a project. It does this by pulling the latest documentation and functional code examples directly from framework source repositories and injecting them into the LLM's context window at the moment of prompting.

Context7 benefits more than individual developers writing code. Automated code review tools produce better suggestions when they access current documentation. AI code assistants like Cursor, Windsurf, and Claude Code generate more accurate output. Even teams performing large-scale code migrations between framework versions benefit, because Context7 ensures the AI understands the specific APIs available in both the source and target versions.

Architecture behind Context7

Understanding why Context7 performs well requires looking beneath the surface at the Upstash platform that powers it. Context7 is a Software as a Service (SaaS) product, and like any successful SaaS, its architecture must balance competing concerns: scalability, resilience, performance, agility, and cost-effectiveness. Upstash's architects made design choices that reveal how they prioritized and resolved these concerns.

High-level architecture

Context7's architecture is built around a pipeline that transforms raw documentation from thousands of GitHub repositories and documentation sites into AI-optimized code snippets that can be retrieved in milliseconds.

The system comprises several interconnected components:

LLM Client - The AI application (Cursor, Windsurf, Claude Desktop, Claude Code) that needs documentation context.
Context7 MCP Server - The protocol adapter that translates MCP tool calls into Context7 API requests.
API Backend - The service layer that orchestrates library resolution and documentation retrieval.
Upstash Vector Database - Stores processed documentation as vector embeddings using the DiskANN (Disk-based Approximate Nearest Neighbor) algorithm. DiskANN keeps indexes on disk rather than RAM, enabling cost-effective semantic search across 33,000+ libraries.
Redis Caching Layer (Upstash Global Database) - A multi-region distributed cache for frequently accessed documentation, providing sub-millisecond reads from the nearest region.
Quality Assurance System - Validation and scoring infrastructure that ensures documentation quality through source reputation, benchmark scoring, and injection prevention.

Scalability and elasticity

For a system serving thousands of concurrent developers across the globe, scalability is essential. Context7 needs to handle not just steady growth in users, but also the bursty, unpredictable traffic spikes common in LLM-driven applications.

Upstash addresses this through a combination of serverless infrastructure and intelligent architectural choices. The vector database uses DiskANN and its variant FreshDiskANN (optimized for incremental index updates), which store vector indexes on disk rather than requiring everything in memory. This approach keeps memory requirements constant regardless of dataset size, allowing the system to scale to millions of documentation snippets without proportional cost increases.

Resilience and fault tolerance

Context7's caching layer uses Upstash Global Database - a multi-region Redis deployment where a primary region handles all writes while read replicas distributed globally serve read requests. If a region becomes unavailable, requests automatically route to the next-nearest region.

The caching layer uses eventual consistency, prioritizing read performance over immediate consistency. With documentation updated every 10 to 15 days, brief cache staleness during propagation is negligible.

Performance through server-side intelligence

One of Context7's most impactful architectural decisions was moving documentation filtering and ranking from the LLM client to the server side. In the original architecture, the API returned 50-100 vector search results and left it to the LLM to filter and rank them - a process that averaged 9,700 tokens, 3.95 tool calls, and 24 seconds of latency.

The current architecture performs vector search plus server-side reranking (re-ordering results by relevance) using a proprietary 5-metric scoring system. The result: an average of just 3,300 tokens, 2.96 tool calls, and 15 seconds of latency - a 65% reduction in token consumption and 38% reduction in latency. The server does more work per request, but that work uses specialized, cost-efficient models rather than expensive general-purpose LLMs.

Agility and cost-effectiveness

Context7 is composed of multiple focused components - the MCP server, the API backend, the indexing pipeline, the quality assurance system - each of which can be iterated and deployed independently. Moreover, each can be scaled independently based on demand. Combined with Upstash's serverless infrastructure, each component scales with demand without overprovisioning, and costs scale with actual usage rather than reserved capacity.

How Context7 implements the MCP protocol

This is where we move from "what" to "how." Understanding Context7's MCP implementation reveals the engineering decisions that make it responsive, reliable, and LLM-friendly.

The MCP data layer

The Model Context Protocol defines three core primitives that MCP servers can expose to LLM clients:

Tools - Functions that AI models can invoke, such as querying databases, calling APIs, or performing computations.
Resources - Data that provides context to language models, such as files, database schemas, or application-specific information.
Prompts - Predefined templates or instructions that guide the language model's responses.

Context7 exposes a minimal MCP interface:

Primitive	Implementation	Description
Tools	`resolve-library-id`	Resolves a package name to a Context7-compatible library ID
Tools	`query-docs`	Retrieves up-to-date documentation and code examples for a library
Resources	empty	No resources exposed
Prompts	empty	No prompts exposed

This minimalism is a design choice, not a limitation. By providing just two focused tools, Context7 keeps the integration surface small and predictable. LLM clients don't need to navigate a complex menu of capabilities - they resolve a library and query its documentation. That's it.

Inspecting the MCP implementation

You can examine Context7's MCP implementation yourself using the MCP Inspector, an interactive debugging tool for MCP servers. Run the following command to launch the inspector connected to Context7:

npx -y @modelcontextprotocol/inspector npx -y @upstash/context7-mcp
# Opens MCP Inspector at http://localhost:5173

This command opens a browser-based inspector at http://localhost:5173 that shows the complete MCP handshake - the initialization exchange, capability negotiation, tool definitions - and lets you invoke tools interactively to see their responses. It's an invaluable tool for understanding how any MCP server works internally.

Protocol layers: What the LLM sees vs. what the server hides

Context7's communication architecture operates across two distinct layers, and understanding this separation is key to understanding MCP's design philosophy.

Layer 1: JSON-RPC (JSON Remote Procedure Call) over stdio (standard input/output) - This is what the LLM client (Claude, Cursor, etc.) directly interacts with. The client sends JSON-RPC 2.0 messages containing tool names and parameters, and receives structured text responses. The LLM sees tool names, request parameters, tool results, and error messages - nothing else.

Layer 2: HTTPS to Context7 Backend - This is what the MCP server handles internally. It translates JSON-RPC tool calls into HTTP requests to Context7's API, manages authentication headers, handles response codes, and measures latency. None of this is visible to the LLM.

Component	Visible to LLM?	Details
Tool name (`resolve-library-id`, `query-docs`)	Yes	JSON-RPC method name
Request parameters (JSON)	Yes	`libraryName`, `query`, `libraryId`
Tool result (formatted text)	Yes	Library data or documentation snippets
HTTP URL called by MCP server	No	Abstracted by MCP server
HTTP response codes	No	Abstracted by MCP server
Request headers and authentication	No	Abstracted by MCP server
Network latency	No	Abstracted by MCP server

The MCP server handles all the complexity of communicating with the Context7 API, allowing the LLM to focus solely on generating accurate queries and interpreting results.

MCP lifecycle: Tool discovery

After initialization, the client calls tools/list to discover available tools. Context7's response reveals two tools - resolve-library-id and query-docs - with descriptions that are unusually detailed and prescriptive for an MCP server. This is intentional: the descriptions function as behavioral instructions for the LLM. Several aspects stand out:

Privacy guardrails embedded in the schema. Both tools include explicit warnings in their parameter descriptions: "IMPORTANT: Do not include any sensitive or confidential information such as API keys, passwords, credentials, or personal data in your query." This is a defensive design choice - because query text is sent to Context7's API, the tool description itself acts as a privacy firewall, instructing the LLM to strip sensitive content before making the call.

Call frequency limits. Both descriptions include: "IMPORTANT: Do not call this tool more than 3 times per question." This prevents the LLM from entering retry loops that would waste tokens and API quota while degrading user experience.

Structured selection criteria. The resolve-library-id description prescribes a ranked selection process for the LLM: name similarity (exact matches first), description relevance, documentation coverage (higher snippet counts preferred), source reputation, and benchmark score (0-100 quality indicator). It even specifies the response format - the LLM should explain why it chose a particular library and acknowledge when multiple good matches exist.

Query quality guidance. The query-docs tool provides explicit examples of good queries ("How to set up authentication with JWT in Express.js") versus bad ones ("auth", "hooks"), coaching the LLM to generate specific, context-rich searches.

MCP annotations. Both tools include "annotations": { "readOnlyHint": true } and "execution": { "taskSupport": "forbidden" }, signaling to the client that these tools only read data and should not be used in background task execution.

API endpoints

Behind the MCP protocol layer, Context7's server communicates with exactly two HTTP API endpoints. These are the real workhorses of the system.

Endpoint 1: Library resolution

This endpoint powers the resolve-library-id tool. It performs fuzzy matching and LLM-based relevance ranking to find libraries matching a developer's query.

Field	Value
URL	`https://context7.com/api/v2/libs/search`
Method	`GET`
Authentication	Optional (Bearer token with `ctx7sk_` prefix)

Query Parameters:

Parameter	Type	Required	Description
`query`	string	Yes	The user's original question, used for relevance ranking
`libraryName`	string	Yes	The library name to search for

Example Request:

GET /api/v2/libs/search?query=How+to+create+a+REST+API+in+Spring+Boot+3.5&libraryName=Spring+Boot HTTP/1.1
Host: context7.com
Authorization: Bearer ctx7sk_xxx...
X-Context7-Source: mcp-server
X-Context7-Server-Version: 1.0.23
X-Context7-Client-IDE: claude-code
X-Context7-Transport: stdio

The response returns an array of matching libraries, each with rich metadata that the LLM uses to select the best match:

{
  "results": [
    {
      "id": "/spring-projects/spring-boot",
      "title": "Spring Boot",
      "description": "Spring Boot helps you to create Spring-powered, production-grade applications...",
      "totalSnippets": 2446,
      "trustScore": 9,
      "benchmarkScore": 82.5,
      "versions": ["v3.4.1", "v3.5.3", "v3.5.9", "v4.0.0"],
      "state": "finalized"
    }
  ]
}

The trustScore field (0-10 scale) maps to a reputation label: scores of 7 or above are "High", 4-6 are "Medium", and below 4 are "Low". This scoring considers factors like organization age, repository portfolio, community size, and contributor count. It helps the LLM choose authoritative sources over potentially unreliable forks or clones.

Endpoint 2: Documentation retrieval

This endpoint powers the query-docs tool. It performs vector similarity search, applies server-side reranking, and returns formatted documentation snippets.

Field	Value
URL	`https://context7.com/api/v2/context`
Method	`GET`
Authentication	Optional (Bearer token with `ctx7sk_` prefix)
Response Format	Text (default for MCP) or JSON

Query Parameters:

Parameter	Type	Required	Description
`query`	string	Yes	The developer's question or task
`libraryId`	string	Yes	Context7-compatible library ID from `resolve-library-id`

Response Headers:

Header	Description
`x-context7-total-tokens`	Total token count in response
`x-context7-total-pages`	Total number of pages
`x-context7-has-next`	Whether more pages exist

The response contains formatted documentation snippets with code examples, each including a title, description, source URL, and language-tagged code block - ready for the LLM to incorporate into its answer.

Request headers and authentication

Every request from the MCP server to Context7's API includes a set of headers that serve both operational and security purposes:

Header	Required	Purpose
`X-Context7-Source`	Yes	Identifies the request source (`mcp-server`)
`X-Context7-Server-Version`	Yes	MCP server version for compatibility tracking
`Authorization`	No	`Bearer ctx7sk_xxx` for authenticated access
`mcp-client-ip`	No	AES-256-CBC encrypted client IP for rate limiting
`X-Context7-Client-IDE`	No	Which IDE is making the request (e.g., `claude-code`, `Cursor`)
`X-Context7-Transport`	No	Transport type (`stdio`, `http`, `sse`)

The encrypted client IP header deserves special mention. Context7 uses AES-256-CBC encryption to prevent header spoofing - a technique where unauthorized users could bypass rate limiting by sending a different IP address with each request. The encryption key is shared only between the MCP server and the API backend, so forged headers are immediately detectable.

Further reading: Rate limiting in agentic workflows introduces unique challenges - AI agents can generate bursts of requests that look nothing like human traffic. We explore cost-aware rate limiting strategies for such scenarios in Denial of Wallet - Cost-Aware Rate Limiting.

The request flow in action

Consider a complete request flow for a developer asking: "How to create a REST API in Spring Boot 3.5?"

Step 1: Library Resolution. The LLM calls resolve-library-id with libraryName: "Spring Boot" and query: "How to create a REST API in Spring Boot 3.5?". The MCP server translates this into a GET /v2/libs/search request. The API returns five matching libraries, including the official Spring Boot repository (/spring-projects/spring-boot) with a benchmark score of 82.5 and "High" source reputation.

The LLM evaluates the candidates and selects /spring-projects/spring-boot/v3.5.9 - choosing the latest 3.5.x version that matches the developer's request.

Step 2: Documentation Retrieval. The LLM calls query-docs with the selected library ID and the original query. The MCP server translates this into a GET /v2/context request. The API performs vector similarity search, applies server-side reranking, and returns five relevant code snippets demonstrating REST API creation in Spring Boot. These snippets include a complete create, read, update, delete (CRUD) example with @RestController, @RequestMapping, proper HTTP status codes, and error handling.

Step 3: Code Generation. The LLM combines the retrieved documentation with the original prompt and any existing code context to generate a response using current Spring Boot 3.5 APIs.

The entire workflow completed with just 2 MCP tool calls - the minimum possible for a query where the library name needs resolution.

Performance and quality in practice

Performance metrics and quality experiments tell us whether Context7's architecture delivers on its promises. This section presents measurements of API response times, caching behavior, scalability, and response quality across multiple programming languages.

API performance analysis

To understand how Context7 performs under real-world conditions, we conducted a series of experiments measuring API response times, caching behavior, and scalability under concurrent load. All measurements were made using direct HTTP calls to the Context7 API endpoints.

Caching behavior

We called each endpoint four times with the same query to observe caching effects:

/v2/libs/search - Library Resolution:

Call	Type	Response Time
1	Cold (first)	1.321s
2	Warm (repeat)	1.377s
3	Warm (repeat)	1.085s
4	Warm (repeat)	1.301s
Average		1.271s

/v2/context - Documentation Retrieval:

Call	Type	Response Time
1	Cold (first)	1.183s
2	Warm (repeat)	0.950s
3	Warm (repeat)	0.968s
4	Warm (repeat)	1.282s
Average		1.096s

Response times remain consistent across repeated queries (1.08 to 1.38s range for search, 0.95 to 1.28s for context). The identical response sizes across warm calls suggest server-side caching is active. However, the LLM-based relevance ranking - which carries 80% weight in the reranking algorithm - must run on every request, even for cached results, preventing dramatic cache-hit speedups.

Concurrent user simulation

To test scalability, we fired 10 parallel unique queries at each endpoint, simulating 10 active users querying different libraries simultaneously:

/v2/libs/search - 10 Parallel Queries:

Library	Response Time
Spring Boot	0.978s
Kubernetes	1.364s
Express	1.482s
Next.js	1.498s
React	1.513s
MongoDB	1.568s
PostgreSQL	1.688s
Django	1.892s
Vue.js	1.896s
Redis	2.053s
Average	1.593s

/v2/context - 10 Parallel Queries:

Library	Response Time	Response Size
Prisma	1.272s	3,716 bytes
Next.js	1.278s	4,124 bytes
React	1.369s	5,119 bytes
MongoDB	1.371s	5,224 bytes
Express	1.391s	5,840 bytes
Kubernetes	1.407s	10,595 bytes
Django	1.585s	3,122 bytes
Vue.js	1.588s	8,131 bytes
PostgreSQL	1.667s	4,695 bytes
Spring Boot	1.686s	6,325 bytes
Average	1.461s

The results show that Context7 handles concurrent queries with only a modest increase in response time - averaging around 1.5 seconds for both endpoints under 10 parallel requests. This suggests that the system's architecture effectively manages load without significant degradation in performance.

Notably, response size doesn't strongly correlate with response time - Kubernetes returned the largest response (10.6 KB) but wasn't the slowest, suggesting the bottleneck is server-side processing (e.g., 5-step ranking algorithm) rather than data transfer.

Full end-to-end workflow

A complete workflow (resolve library + fetch documentation) takes:

Workflow	Library Resolution	Documentation Retrieval	Total
Spring Boot REST API	1.174s	1.512s	2.717s
React Hooks	1.459s	0.978s	2.471s
Next.js Routing	1.095s	1.181s	2.309s
Average			2.499s

A developer can expect approximately 2.3 to 2.7 seconds for a complete Context7 query when the library name needs resolution. If the library ID is provided directly (skipping resolution), the time drops to just the documentation retrieval phase - roughly 1 second.

API key vs. anonymous access

Context7 offers both authenticated (API key) and anonymous access. Their documentation states that API keys provide "higher rate limits, priority access during peak usage, and better performance for frequent queries." We tested whether this holds true in practice.

Metric	Without API Key	With API Key	Difference
Search API (single user avg)	1.41s	1.38s	API key ~2% faster
Context API (single user avg)	1.14s	1.38s	Anonymous 17% faster
Search API (10 concurrent avg)	1.58s	1.75s	Anonymous 10% faster
Context API (10 concurrent avg)	1.40s	1.68s	Anonymous 17% faster
Full workflow (avg)	2.32s	2.81s	Anonymous 17% faster
Response time consistency	Higher variance	Lower variance	API key more consistent

The results were counterintuitive. Anonymous requests were generally faster in our tests, while API key requests showed more consistent response times with lower variance. This suggests that API key validation adds a small overhead to each request, and the "priority access" benefit likely applies during actual peak usage periods rather than the normal load conditions of our experiment.

The primary value of an API key isn't raw speed but rather higher rate limits (free plan: 1,000 API calls/month; pro plan: 5,000/seat/month) and more predictable behavior under heavy load. For production use with frequent queries, an API key is essential to avoid rate limits.

Response quality: Experiments across languages and topics

Performance metrics tell us how fast Context7 responds, but they say nothing about the quality of what it returns. To evaluate this, we conducted 12 experiments across three programming languages (Java, TypeScript, Python) and four topics (JWT Authentication, MCP Server Implementation, Database Access, Unit Testing).

Each experiment was scored on accuracy, version specificity, code quality, completeness, and relevance, yielding an overall score out of 10.

Results by experiment

The following table shows individual scores for each language-topic combination. Scores of 8.0+ indicate production-ready responses; scores below 6.0 signal that the queried library lacked sufficient coverage for the topic.

#	Language	Topic	Library Used	Score	Verdict
1	Java	JWT Authentication	Spring Security 6.5	8.2	Excellent
2	TypeScript	JWT Authentication	Express.js 5.1.0	3.5	Poor
3	Python	JWT Authentication	FastAPI	9.2	Excellent
4	Java	MCP Server	Java SDK	9.5	Excellent
5	TypeScript	MCP Server	TypeScript SDK	9.3	Excellent
6	Python	MCP Server	Python SDK	9.4	Excellent
7	Java	Database Access	Spring Data JPA	8.0	Good
8	TypeScript	Database Access	Prisma ORM	9.0	Excellent
9	Python	Database Access	SQLAlchemy 2.0	8.8	Excellent
10	Java	Unit Testing	JUnit 5	6.5	Fair
11	TypeScript	Unit Testing	Vitest	9.0	Excellent
12	Python	Unit Testing	pytest	8.5	Good

Most experiments scored in the "good to excellent" range (8.0+), with the notable exception of TypeScript JWT Authentication (3.5) - a clear outlier that highlights Context7's limitations when the queried functionality lives in separate libraries rather than the main framework.

Key quality observations

What works well: Context7 excels when a single library contains comprehensive documentation for the queried topic. MCP Server implementation scored 9.4 on average because each SDK's documentation directly covers the topic. Python consistently scored highest (8.98 average) because Python frameworks like FastAPI, SQLAlchemy, and pytest tend to have well-structured, example-rich documentation.

Where it struggles: Context7 performs best when the queried functionality is documented within a single library. The lowest score (3.5) came from asking Express.js about JWT authentication - but JWT functionality lives in separate libraries like jsonwebtoken or passport-jwt, not in Express core. Similarly, the JUnit 5 experiment (6.5) missed Mockito and Spring Boot test integration because those capabilities live in separate libraries.

The lesson: search for the specific library that implements the feature, not the framework that hosts it. Decompose cross-library queries into separate searches - for example, search for "jsonwebtoken JWT implementation" and "Express middleware integration" separately. This reflects Context7's design principle of indexing libraries, not conceptual knowledge graphs.

Overall quality: 50% of experiments scored 9.0 or above, 75% scored 8.0 or above, and only one experiment scored below 6.0. The average across all 12 experiments was 8.16 out of 10 - a strong result that positions Context7 in the "good to excellent" range, while revealing that it isn't a universal solution for cross-library integration questions.

Five key factors that make Context7 successful

Having dissected Context7's architecture, protocol implementation, and performance characteristics, we can now identify the factors that separate it from the thousands of MCP servers that never gain traction.

1. It solves a critical pain point

Every developer using an AI coding assistant encounters hallucinated APIs, outdated patterns, and version conflicts. Context7 doesn't require changing your workflow - it makes your existing tools produce better output by giving them access to current documentation. It solves one problem well and accepts the trade-off: cross-library questions require multiple searches, but single-library queries are fast and accurate.

2. Sophisticated architecture behind a simple interface

Context7's two-tool interface hides a five-stage quality pipeline (parse, enrich, vectorize, rerank, cache) detailed in the Architecture behind Context7 section. Each stage contributes to the result quality that users experience. The key architectural bets - DiskANN for cost-effective vector storage, server-side reranking to reduce token consumption, and multi-region caching for low-latency reads - allow the system to serve 33,000+ libraries economically while maintaining sub-3-second response times and an 8.16/10 quality score across our experiments.

3. Zero-setup deployment

For most developers, setup is a single line in their MCP client's configuration file (for example, .cursor/mcp.json for Cursor or claude_desktop_config.json for Claude Desktop):

{
  "mcpServers": {
    "context7": {
      "command": "npx",
      "args": ["-y", "@upstash/context7-mcp"]
    }
  }
}

This zero-friction onboarding is critical - every additional setup step is a point where potential users drop off.

4. Well-implemented MCP protocol

As the implementation analysis showed, Context7's two tools follow a logical two-phase workflow with carefully crafted descriptions, clear parameter semantics, privacy warnings, and usage constraints. The server handles errors gracefully with actionable messages for common failure scenarios (rate limiting, invalid keys, library not found).

Context7 integrates seamlessly with all major MCP-compatible editors without special configuration or workarounds - exactly what a well-implemented protocol integration should do.

5. Privacy by design

In an era of growing concern about data privacy in AI tools, Context7 takes a clear architectural stance: your code never leaves your machine. The privacy model works as follows:

The LLM client (running locally) reformulates the developer's prompt into a documentation query
Only this reformulated query - not the original code or prompt - is sent to Context7
Context7 returns documentation snippets without maintaining session history or user profiles
The LLM uses the documentation to generate code, all happening locally

This stateless, query-only design means Context7 never has access to proprietary codebases, business logic, or sensitive application details. For enterprise teams evaluating MCP servers, this is often a deciding factor.

Conclusion

The MCP ecosystem has grown from a single protocol specification to thousands of server implementations in little over a year. But as our analysis of Context7 shows, building a successful MCP server requires more than implementing the protocol correctly. It requires solving a genuine, recurring problem; building an architecture that can deliver reliable, fast results at scale; keeping the integration surface small and focused; and respecting user privacy.

For architects and technical leaders considering building their own MCP servers, Context7 offers four lessons:

Solve one problem exceptionally well. Focus is a strength, not a limitation.
Invest in the invisible pipeline. Users never see the quality infrastructure, but they experience its results.
Make adoption frictionless. Every step you eliminate between discovery and value is a step that would have cost you users.
Respect user privacy. Stateless, query-only designs build the trust that enterprise adoption requires.

As the MCP ecosystem matures, the servers that thrive will be those that combine strong engineering foundations with clear, focused value propositions. Context7 shows what that combination looks like in practice.

TL;DR

Context7, built by Upstash, tops the MCP server ecosystem with 44,000+ GitHub stars and 240,000+ weekly npm downloads. It feeds AI coding assistants with fresh, version-accurate documentation so they stop generating code based on stale training data. ThoughtWorks Technology Radar now lists it in its "Tools" section.

The engineering choices behind it tell most of the story. Shifting documentation filtering from the LLM to the server reduced token usage by 65% and latency by 38%. DiskANN stores vector indexes on disk rather than in memory, so costs stay flat as the library count grows past 33,000. Multi-region Redis caching handles reads globally with low latency and eventual consistency that fits a 10-to-15-day documentation refresh cycle.

The entire MCP surface consists of two tools: resolve a library name, then fetch its docs. Tool descriptions carry embedded behavioral guidance for the LLM, including privacy warnings, a three-call-per-question cap, and examples of effective queries. Setup takes one line of JSON config.

Our performance tests showed full queries completing in roughly 2.5 seconds. With 10 concurrent users, response times rose only 25-33%. Anonymous access was slightly faster than API-key access in raw speed, though API keys delivered steadier response times and higher rate limits.

Quality scored 8.16 out of 10 across 12 experiments spanning Java, TypeScript, and Python. Single-library topics like MCP Server implementation averaged 9.4, while cross-library tasks such as Express.js JWT authentication dropped to 3.5 because the relevant code lives in separate packages. The takeaway: target the library that owns the feature, not the framework around it.

Five factors drove adoption: it addresses a pain developers face every day, it pairs deep infrastructure with a minimal interface, it requires zero setup, its tool descriptions steer LLM behavior precisely, and user code never leaves the local machine.

Protecting Architecture with Automated Tests in Python

Wed, 28 Jan 2026 00:00:00 GMT

We started our engineering careers in the Java ecosystem, where we discovered Fitness Functions and then ArchUnit, which helped us govern our codebases more effectively. Recently, we've been exploring Python and its ecosystem, and naturally, we wanted to find similar tools for Python. Today, we'd like to share one of our findings: PyTestArch.

PyTestArch is an open source library that allows users to define architectural rules and test their code against them. It is generally inspired by ArchUnit.

There's plenty of material on why it's worth automating architectural tests, so we won't repeat it here. Instead, we'll focus on how to use PyTestArch in practice through a hands-on example.

The Example Project: Naan & Pop Restaurant

For this demonstration, we created a small project that implements a simple restaurant ordering system using layered architecture principles. The project is based on the "Naan & Pop" example from the Head First Software Architecture book by Raju Gandhi, Mark Richards, and Neal Ford.

The story is straightforward: Sangita created an Indian-inspired mom-and-pop restaurant called Naan & Pop, specializing in flatbread sandwiches and sodas. The restaurant needs a website where customers can place orders online. Since Naan & Pop is a startup with a small budget, it needs to be simple and created quickly - but also extensible enough to grow with the business.

You can find the full code on GitHub.

Understanding the Layered Architecture

The codebase is organized into three main layers, each with a specific responsibility:

Presentation Layer: FastAPI routers, endpoints, and request/response models
Workflow Layer: Business logic and use case orchestration
Persistence Layer: Entity models, value objects, and data structures

This layered architecture was chosen for Naan & Pop because it delivers one of the most important characteristics for a startup: fast time to market. Compared to more complex patterns like hexagonal or microservices architectures, it's simple to understand and implement, allowing the team to get the system running quickly without sacrificing maintainability.

But speed isn't the only benefit. One of the most powerful advantages of layered architecture is the ease of replacing entire technical layers. For example:

Switching databases: Need to move from PostgreSQL to MongoDB? You only change the persistence layer - your workflow and presentation layers remain untouched because they interact with persistence through the workflow layer, which handles data transformation between presentation DTOs and persistence entities.
Adding new interfaces: Want to add a mobile app alongside the web interface? Add a new presentation layer (e.g., a REST API or GraphQL endpoint) without touching your existing business logic.

The key architectural rules make this possible:

Allowed dependencies: Presentation → Workflow and Workflow → Persistence
Forbidden dependencies: Presentation → Persistence (layer skipping) and any reverse dependencies (lower layers depending on higher ones)

Why is layer skipping so problematic? It breaks this replaceability. If the presentation layer directly accesses the persistence layer, then changes to your database schema or data access patterns would require updates in both the workflow layer and the presentation layer. For instance, renaming a database column from order_id to id would force you to update not only your persistence models and workflow logic but also every presentation router that directly imports those models. You've lost the ability to change one technical concern in isolation. Beyond that, bypassing the workflow layer means circumventing business logic and validation rules, potentially leading to data inconsistencies.

These rules sound simple, but in practice they require constant vigilance. As the codebase grows, it's easy to accidentally introduce forbidden dependencies. A developer might be tempted to import a persistence entity directly into a presentation router "just this once" to save time, breaking the architectural integrity that makes the system maintainable. This is where automated testing becomes essential.

Protecting Architecture with PyTestArch

This is where PyTestArch comes into play. We defined a set of architectural rules using PyTestArch syntax that enforce the correct dependencies between layers.

Setting Up the Foundation

Before we dive into the test code, PyTestArch uses pytest fixtures to configure the architecture. The evaluable fixture scans your codebase (scan all python files under src for imports and build an internal representation that can later be queried), and layered_architecture defines your layer structure. These are set up once in conftest.py and reused across all tests:

# tests/architecture/conftest.py (simplified)
import pytest
from pytestarch import EvaluableArchitecture, LayeredArchitecture, get_evaluable_architecture

@pytest.fixture(scope="session")
def evaluable(src_dir: str) -> EvaluableArchitecture:
    naan_and_pop_path = os.path.join(src_dir, "naan_and_pop")
    return get_evaluable_architecture(src_dir, naan_and_pop_path)

@pytest.fixture(scope="session")
def layered_architecture() -> LayeredArchitecture:
    return (
        LayeredArchitecture()
        .layer("presentation").containing_modules(["src.naan_and_pop.presentation"])
        .layer("workflow").containing_modules(["src.naan_and_pop.workflow"])
        .layer("persistence").containing_modules(["src.naan_and_pop.persistence"])
    )

Important: When defining layer modules, ensure module paths match how PyTestArch resolves imports from your project root. In this example, we use src.naan_and_pop.presentation because our source directory is src/. Your paths may differ based on your project structure.

Writing Architectural Rules

Now let's see how these fixtures are used in a test that prevents layer skipping:

from pytestarch import EvaluableArchitecture, LayerRule, LayeredArchitecture


def test_presentation_cannot_skip_layers(
    evaluable: EvaluableArchitecture, layered_architecture: LayeredArchitecture
) -> None:
    """
    Presentation must not directly access persistence.

    All data access must go through the workflow layer to ensure business
    logic is properly encapsulated and consistently applied.

    Violations would allow bypassing business rules and validation logic.
    """
    rule = (
        LayerRule()
        .based_on(layered_architecture)
        .layers_that()
        .are_named("presentation")
        .should_not()
        .access_layers_that()
        .are_named("persistence")
    )
    rule.assert_applies(evaluable)

How It Works

PyTestArch provides a fluent API for defining architectural rules. In the example above:

LayerRule() - Creates a rule specifically for validating layer dependencies
.based_on(layered_architecture) - References our layer configuration (defined in conftest.py)
.layers_that().are_named("presentation") - Specifies which layer we're checking
.should_not().access_layers_that().are_named("persistence") - Defines the forbidden dependency
.assert_applies(evaluable) - Validates the rule against the actual codebase

The test scans all imports in the presentation layer and fails if it finds any direct imports from the persistence layer. This happens automatically during your CI pipeline, catching violations before they reach production.

Catching Violations in Action

To see PyTestArch in action, check out Pull Request #1 in our demo repository. In this PR, a developer tried to import an Order entity directly from the persistence layer into a presentation model to avoid duplicating field definitions. While this seems efficient, it creates a direct coupling between the presentation and persistence layers, breaking the architectural boundary. The architecture tests catch this violation and fail the build, preventing the problematic code from being merged.

When a violation occurs, PyTestArch provides clear error messages showing exactly which module violated the rule and what the forbidden import was. This makes it easy to understand and fix the issue quickly.

E AssertionError: "src.naan_and_pop.presentation.api.models.orders" (layer "presentation") imports "src.naan_and_pop.persistence.orders.order" (layer "persistence").

This demonstrates the real value of automated architecture tests: they act as guardrails that prevent architectural erosion over time, even as team members change and the codebase grows. The example repository includes additional tests that enforce other critical rules like preventing reverse dependencies and ensuring proper layer isolation.

Beyond Protection: Architecture as a Fitness Function

Architecture tests can do more than just prevent violations - they can actively guide refactoring efforts. In his video lesson "Fitness Function-Driven Architecture", Mark Richards discusses using ArchUnit (or similar tools in other ecosystems) as a Fitness Function that guides teams toward refactoring application architecture to a desired state.

Note: we highly recommend materials from Mark Richard's website Developer to Architect, which offers excellent resources on software architecture topics.

The approach is straightforward: write architecture tests for your target architecture before implementing it. These tests will initially fail, but they provide clear, measurable goals for your refactoring effort. As you restructure components and move code between layers, the tests gradually turn green, providing concrete feedback on your progress.

For instance, if you're restructuring your application/service codebase to support the domain better, you might write tests that enforce new boundaries between components. As you refactor, each passing test confirms you're moving in the right direction, while failing tests highlight remaining work.

This transforms architecture tests from passive guardrails into active navigation tools, helping teams systematically evolve their systems toward better architectural patterns.

Summary: Why Use PyTestArch?

PyTestArch brings the power of automated architecture testing to Python projects, offering several key benefits:

Prevent architectural drift: Catch violations early in the development process, before they reach production
Document architecture decisions: Tests serve as executable documentation that clearly expresses architectural intent
Enable confident refactoring: Know immediately if changes break architectural constraints - just like unit tests for architecture
Reduce code review burden: Automate the enforcement of architectural rules that would otherwise require manual review
Support team growth: Help new team members understand and respect architectural boundaries

Getting Started

Ready to try PyTestArch in your own project? Here's how to get started:

Install PyTestArch: pip install pytestarch
Define your layers in a pytest fixture (see our example in tests/architecture/conftest.py)
Write your first architectural rule (start with something simple like preventing layer skipping)
Run the tests and fix any violations
Add the tests to your CI pipeline

The complete example code, including all architecture tests and the full Naan & Pop implementation, is available on GitHub. Feel free to use it as a starting point for your own projects.

When to Use Architecture Tests

Architecture tests are particularly valuable for:

Codebases with multiple contributors to ensure everyone adheres to the same architectural principles
Long-lived projects that need to maintain architectural integrity over time
Teams with varying experience levels who benefit from automated guardrails
Migration projects where you're actively refactoring toward a target architecture
Open source projects where contributors may not be familiar with architectural decisions

By investing a small amount of time upfront to define your architectural rules, you gain automated enforcement that pays dividends throughout the lifetime of your project.

Denial of Wallet: Cost-Aware Rate Limiting for Generative AI Applications - Hands-On Implementation (Part 3)

Mon, 12 Jan 2026 00:00:00 GMT

This is Part 3 of the Denial of Wallet series. Part 1 introduced Denial of Wallet as a distinct failure mode in GenAI applications. Part 2 compared rate limiting algorithms and provided a practical implementation checklist. This post delivers a hands-on implementation: a working API with multi-level rate limiting, Redis-backed quota management, real-time monitoring dashboards, and load testing scenarios that demonstrate how the system behaves under different attack patterns.

Introduction

As hands-on architects, we couldn't just write theoretical articles about protecting GenAI APIs from Denial of Wallet attacks. We needed to build a working example that you can run, modify, and learn from. This post walks through a complete implementation that demonstrates:

Token Bucket Rate Limiter - Allows controlled bursts while maintaining average rate (minute-level protection)
Daily Quota - Short-term quota that prevents sustained abuse within 5-hour windows
Weekly Quota - Long-term quota that prevents extended exploitation over 7-day periods

The implementation is built for educational purposes using accelerated timeframes (5-minute "daily" and 20-minute "weekly" quotas) so you can observe the full behavior in under 30 minutes rather than waiting days. This architecture is inspired by Claude's current rate limits, which implement three levels of protection: requests per minute, 5-hour window quotas, and weekly window quotas.

The full project code is available at: https://github.com/handsonarchitects/denial-of-wallet-demo

Note on Terminology

Throughout this post, "tokens" refer to budget units for rate limiting, not LLM model tokens:

1 budget unit = $0.001 (one-tenth of a cent)

A $0.50 LLM request consumes 500 budget units

The token bucket "24 tokens" means $0.024 in budget, not 24 LLM tokens

See Part 2 for the full terminology mapping and cost calculation formulas.

Demo Enhancements for Learning

This implementation includes additional logging and observability features for educational clarity:

Explicit "cascading limit detected" messages in k6 logs

Real-time dashboard with 10-second metric snapshots

Accelerated quota windows (5 min "daily", 20 min "weekly")

In production, you'd replace these with your standard observability stack (e.g., Prometheus, Grafana, CloudWatch).

Application Architecture

The API is built with FastAPI and follows a clean architecture pattern with clear separation of concerns. Let's examine the core components and request lifecycle.

Core Components

Dependency Injection: FastAPI's built-in DI system provides RateLimiter, RedisStorage, and Settings to endpoints via app.state. This enables clean testing and makes it easy to swap implementations.

Token Bucket: Implements the continuous refill algorithm for smooth minute-level rate limiting. Unlike discrete interval-based approaches, tokens accumulate proportionally based on elapsed time, providing more granular burst control.

Quota Manager: Handles time-windowed daily and weekly quotas with automatic reset logic. Each quota tracks consumption independently and resets at configured intervals.

Metrics Recorder: A background task that periodically captures state snapshots (bucket levels, quota remaining, request success/failure rates) for historical analysis and dashboard visualization. This component is done for educational purposes; in production, you'd integrate with your existing monitoring solution.

Storage Layer: Redis-backed persistence for all state: token buckets, quotas, metrics snapshots, and request history. Redis provides atomic operations and expiration support, making it ideal for distributed rate limiting.

Request Lifecycle

The following sequence diagram shows how a request flows through the rate limiting system. In this demo, generation time (0.1-0.5s) and actual cost (1-5 budget units) are simulated randomly to mimic real LLM variability. In production you would use the LLM proviter's usage metadata (e.g., prompt_tokens + completion_tokens) to determine actual cost after generation and use SDK tools to estimate cost beforehand.

Key Design Patterns

Cost Estimation Strategy: Before calling the LLM, the system estimates cost based on input tokens and expected output length. In this demo, costs are randomly simulated (1-5 budget units) to mimic real-world variability. In production implementations, cost estimation uses:

Input token count from prompt tokenization
Maximum output tokens from the max_tokens parameter
Per-model pricing catalog (e.g., GPT-4o: $2.50/1M input tokens, $10.00/1M output tokens)

For example: a prompt with 800 input tokens and max_tokens=300 would estimate (800 × $2.50/1M) + (300 × $10.00/1M) = $0.005 = 5 budget units. See Part 2's Token Bucket section for detailed cost calculation formulas.

Optimistic Locking: The system pre-consumes the estimated cost before processing the request, then refunds the difference after determining actual generation cost. This prevents race conditions while ensuring accurate accounting.

Rollback on Failure: If any quota check fails (daily or weekly), previously consumed tokens are refunded to the minute bucket to maintain consistency. This ensures that a failed request due to quota exhaustion doesn't also consume the minute-level tokens.

Continuous Refill: The token bucket adds tokens proportionally based on elapsed time rather than discrete intervals. The bucket stores a last_refill_timestamp in Redis; on each request, the system calculates elapsed time and adds tokens accordingly (e.g., if 60 seconds elapsed at 6 tokens/minute, exactly 6 tokens are added; if 30 seconds elapsed, 3 tokens are added).

Background Metrics: An async task records snapshots every 10 seconds for historical trending without blocking requests. This enables the real-time dashboard to show rate limit state over time.

Rate Limiting Architecture

The implementation uses three complementary layers of protection, each serving a distinct purpose in preventing Denial of Wallet attacks:

1. Minute Bucket (Token Bucket)

Capacity: 24 tokens (maximum burst size)
Refill Rate: 6 tokens/minute (sustained rate limit)
Purpose: Prevents rapid successive requests that could quickly exhaust quotas

This layer allows legitimate bursts (like an agentic workflow spawning multiple LLM calls in quick succession) while maintaining a long-term average rate. The bucket can accumulate up to 24 tokens during idle periods, enabling users to burst above the steady-state 6 tokens/minute rate briefly.

2. Daily Quota (Short-term)

Total: 40 tokens per window
Reset Window: 5 minutes (demo) / e.g. 24 hours (production)
Purpose: Limits sustained usage within short periods

This quota prevents scenarios where a user stays just below the minute-level rate but maintains that usage continuously, consuming far more than intended over hours.

3. Weekly Quota (Long-term)

Total: 100 tokens per window
Reset Window: 20 minutes (demo) / 7 days (production)
Purpose: Prevents extended abuse over long periods

The weekly quota is the ultimate cost ceiling. Even if a user carefully paces requests to avoid minute and daily limits, they cannot exceed 100 tokens per week. This provides predictable budget protection.

Hierarchical Enforcement

The three layers are checked in sequence: minute bucket → daily quota → weekly quota. A request must pass all three checks to proceed. This creates a cascading protection model where exhausting the weekly quota blocks requests even when daily quota resets. We'll see this behavior demonstrated in Scenario 3.

Scenarios

We've developed three load testing scenarios using k6 to demonstrate how the rate limiting system behaves under different usage patterns. Each scenario automatically resets the system state at the start to ensure consistent results.

You are welcome to clone the repository and run these scenarios yourself. All necessary instructions are provided in the README of the denial-of-wallet-demo.

Scenario 1: Short Bursts

Duration: 6 minutes

This scenario shows how the system handles burst traffic that stays comfortably within all rate limits.

Traffic Pattern:

First burst (1 min): 10 requests at 1 request per 6 seconds
Pause (4 min): No requests - token bucket refills naturally
Second burst (1 min): 10 requests at 1 request per 6 seconds

Expected Behavior:

100% success rate (all 200 responses)
Token bucket fluctuates between bursts but never exhausts
Daily and weekly quotas decrease gradually (~40 tokens consumed total)
Dashboard shows smooth green metrics throughout

Demonstration:

The dashboard after completing Scenario 1 shows healthy quota levels across all three tiers. The minute bucket recovered fully during the 4-minute pause between bursts, demonstrating the continuous refill behavior. Daily quota dropped from 40 to 19 tokens (21 consumed), and weekly quota from 100 to 54 tokens (46 consumed).

Key observations from the k6 log output:

INFO[0000] Transitioning to stage: FIRST_BURST
INFO[0056] Success - Daily: 15, Weekly: 75, Cost: 3
INFO[0062] Transitioning to stage: WAIT
INFO[0302] Transitioning to stage: SECOND_BURST
INFO[0303] Success - Daily: 39, Weekly: 74, Cost: 1
INFO[0360] Success - Daily: 19, Weekly: 54, Cost: 1

Notice that at the second burst (second 303 -> ~5th minute), the daily quota has reset from 15 back to 39 tokens - the 5-minute daily window reset occurred during the 4-minute wait period. This demonstrates automatic quota recovery.

Key Learning: Token bucket handles burst traffic gracefully and refills during idle periods. The continuous refill algorithm ensures smooth rate limiting without artificial request delays.

Scenario 2: Daily Quota Exhaustion

Duration: 8 minutes

This scenario shows what happens when sustained traffic depletes the daily quota while the weekly quota remains healthy.

Traffic Pattern:

Exhaust phase (5 min): Steady requests at 5 requests/minute (12s intervals)
Wait phase (1 min): Continue requests to monitor exhaustion state
Recovery phase (2 min): Verify quota reset and recovery

Expected Behavior:

Initial requests succeed (~20 requests consuming ~40 tokens)
Daily quota exhausts, 429 responses begin with daily_quota_exceeded reason
At 5-minute mark: Daily quota automatically resets to 40 tokens
Requests succeed again after reset
Weekly quota remains healthy throughout (consumed less than 100 tokens throughout the scenario)

Demonstration:

After 5 minutes of sustained traffic, the daily quota is fully exhausted while the weekly quota still has capacity:

Near the end of the scenario, after the daily quota reset, requests succeed again:

Key observations from the k6 log output:

INFO[0283] Success - Daily quota remaining: 0, Weekly: 60, Cost: 2
INFO[0295] Daily quota - not enough tokens to process the request (left 0)
INFO[0307] Transitioning to stage: WAIT
INFO[0367] Transitioning to stage: RECOVERY
INFO[0367] Success - Daily quota remaining: 38, Weekly: 58, Cost: 2

At timestamp 0283 (~4:43 minute), the daily quota reaches zero while weekly quota has 60 tokens remaining. The next request (0295 -> ~4:55 minute) is rejected. After the 5-minute reset window passes, the recovery phase (starting at 0367 -> ~6:07 minute) shows the daily quota has reset to 40 tokens (38 after first request), and requests succeed again.

Key Learning: Daily quota limits provide short-term protection and automatically reset at configured intervals. The system gracefully rejects requests when quotas are exhausted and provides Retry-After headers to guide clients.

Scenario 3: Weekly Quota Exhaustion

Duration: 25 minutes

This is the most complex scenario. It shows how weekly quota enforcement persists even when daily quota resets multiple times, proving the hierarchical nature of the protection layers.

Traffic Pattern:

Exhaust phase (19 min): Steady requests at 3 requests/minute (20s intervals)
Wait phase (2 min): Continue requests to monitor weekly exhaustion
Recovery phase (4 min): Verify quota reset and recovery

Expected Behavior:

0-15 minutes: Requests succeed, daily quota resets every 5 minutes (at 5, 10, 15 min marks)
~15-17 minutes: Weekly quota exhausts (~50 requests consuming ~100 tokens)
~17-21 minutes: Requests fail with weekly_quota_exceeded even when daily quota resets
Cascading Limit Demonstration: Around the 20-minute mark, daily quota resets but requests STILL fail due to weekly exhaustion - this proves hierarchical enforcement
~21-minute mark: Weekly quota automatically resets to 100 tokens (20 minutes after test start)
21-25 minutes: Requests succeed again

Demonstration:

After 5 minutes, both quotas are healthy with daily quota showing first consumption:

At the critical moment when weekly quota is exhausted but daily quota still has capacity:

After both quotas reset and the system fully recovers:

Key observations from the k6 log output:

INFO[0305] Daily quota reset detected: 16 -> 39
INFO[0609] Daily quota reset detected: 8 -> 38
INFO[0914] Daily quota reset detected: 13 -> 39
INFO[0995] Weekly quota low: 8 tokens remaining
INFO[1056] Weekly quota low: 1 tokens remaining
INFO[1076] Request made during EXHAUST, status: 429
INFO[1076] Weekly quota exhausted (~15-17 minute mark)
INFO[1076] CASCADING LIMIT DETECTED: Weekly blocks despite 24 daily tokens available
INFO[1117] CASCADING LIMIT DETECTED: Weekly blocks despite 23 daily tokens available
INFO[1267] Daily quota reset detected: 23 -> 38
INFO[1267] Weekly quota reset detected: 98 tokens available

The log reveals several critical moments:

Multiple daily resets: At around 5, 10, and 15 minutes, the daily quota resets (16→39, 8→38 and 13→39) while requests continue to succeed
Weekly exhaustion: At 1076 (≈18 minutes), weekly quota hits zero and requests begin failing with 429 status
Cascading enforcement: At 1076 and 1117 (≈18-19 minutes), the system explicitly detects that weekly quota is blocking requests despite daily quota having 23-24 tokens available - this is the key demonstration of hierarchical enforcement
Full recovery: At 1267 (≈21 minutes), both quotas reset and requests succeed again

Key Learning: The hierarchical limit enforcement ensures that weekly quota acts as the ultimate cost ceiling. Even when daily quota resets, exhausted weekly quota continues to block requests. This prevents sophisticated attacks where users try to game the system by timing requests around daily reset windows.

Summary

This hands-on implementation demonstrates how to build rate limiting for GenAI APIs with multiple protection layers:

Architectural Takeaways:

Multi-level protection is essential: minute-level rate limiting handles bursts, daily quotas prevent sustained abuse, and weekly quotas provide ultimate cost ceilings
Hierarchical enforcement prevents gaming: all layers must approve a request, making it impossible to exploit reset timing
Optimistic locking with refunds enables accurate cost tracking even when exact costs are only known post-generation
Redis-backed state provides atomic operations and distributed consistency for production deployments

Implementation Insights:

Token bucket continuous refill is more granular than discrete intervals, improving user experience during legitimate bursts
Automatic quota resets reduce operational overhead - no manual intervention needed
Background metrics collection enables real-time observability without impacting request performance
Cost estimation with true-up accounting (provisional reservation + correction) handles unpredictable GenAI costs

The complete project code, including FastAPI implementation, Redis storage layer, k6 load tests, and dashboard visualization, is available at https://github.com/handsonarchitects/denial-of-wallet-demo.

We encourage you to clone the repository, run the scenarios, and experiment with different configurations:

Adjust bucket capacity and refill rates to see burst behavior change
Modify quota windows to match your business requirements
Add new scenarios to test edge cases specific to your application
Extend the implementation with per-user quotas, priority tiers, or cost-based pricing

Final Thoughts

Building cost-aware rate limiting is no longer optional for production GenAI applications. The unpredictable cost variability of agentic workflows, combined with the ease of programmatic access, makes Denial of Wallet attacks a real and present risk. Start with the patterns demonstrated in this series, measure your actual usage, and tune the limits to protect your budget while serving legitimate users effectively.

Denial of Wallet: Cost-Aware Rate Limiting for Generative AI Applications - Strategy (Part 2)

Wed, 10 Dec 2025 00:00:00 GMT

This is Part 2 of the Denial of Wallet series. Part 1 introduced DoW as a distinct failure mode and explained why traditional DoS protections leave your budget exposed (recap: traditional rate limiting protects your infrastructure; cost-aware rate limiting protects your budget; you need both). This post compares four rate limiting algorithms for cost-aware protection, analyzes trade-offs specific to Generative AI workloads, and provides a practical checklist for implementation. Part 3 will deliver a hands-on implementation.

Introduction

You're building an application that uses Generative AI models through an agentic architecture. Your application is exposed to end users via web or mobile interfaces. We will study scenarios where Generative AI model usage is expensive and unpredictable. Cost varies widely based on input size, model used, agent workflow complexity, and the number of tool calls executed. Compare this to traditional web applications where the cost per request is relatively stable. Serving an HTML page or handling a REST API call consumes predictable resources. Even serverless functions have relatively predictable execution time and cost within bounded input ranges.

In GenAI applications, one user request might cost $0.001 (cached response) while another costs $0.50 (multi-step agentic workflow with GPT-4o and multiple tool invocations). Both count as "one request" from a traditional rate limiting perspective. This cost unpredictability makes your application vulnerable to Denial of Wallet attacks. Malicious users (or even curious legitimate users) can exploit cost variability to drive up bills or exhaust budgets. In other words, treating all requests equally (for example, by computing average request cost) is insufficient because the standard deviation of cost per request is very high. Treating all requests equally exposes you to the risk of high bills from an attacker who figures out how to trigger the most expensive requests repeatedly. A key challenge is that exact request costs are only known after LLM responses complete, requiring provisional reservation and true-up accounting.

This post focuses on practical mitigation strategies. You'll learn how to choose the right rate limiting algorithm for your GenAI workload, how to gather the data you need before implementing controls, and how to roll out cost-aware limits safely. Think of this as an Architecture Decision Record (ADR) combined with actionable checklists. By the end, you'll know what questions to ask, what trade-offs to consider, and how to implement cost-aware rate limiting without breaking legitimate user workflows.

Rate Limiting Algorithm Options for Cost-Aware Protection

Let's compare four rate limiting algorithms in the context of cost-aware rate limiting for Generative AI applications, where preventing Denial of Wallet attacks is critical. Each algorithm offers different trade-offs between burst handling, predictability, implementation complexity, and resource overhead.

When evaluating these algorithms, remember that we're throttling cost consumption rather than just request counts. A request that triggers a complex agentic workflow with multiple LLM calls should consume more of the user's quota than a simple cached lookup. The algorithm you choose must support this cost-based accounting while still providing good user experience and operational characteristics.

Note: Terminology To avoid confusion, we use the following terms consistently throughout this post:

Budget units (or credits): The quota tracked by the rate limiter. For simplicity, we use a 1-to-1 mapping where 1 budget unit = $0.001 (one-tenth of a cent). A $0.50 request consumes 500 budget units.

Model tokens: The actual tokens processed by the LLM (input tokens + output tokens), as reported by the LLM API. For example, OpenAI's API returns prompt_tokens, completion_tokens, and total_tokens in the usage metadata.

Cost (USD): The dollar amount charged by the LLM provider, calculated from model tokens using per-model pricing (e.g., $0.15 per 1M input tokens for GPT-4o).

The conversion flow is: model tokens → cost (USD) via pricing catalog → budget units via the 1:0.001 ratio. When we say "a request consumes 500 budget units," we mean it costs $0.50 in USD, regardless of how many model tokens were used.

Token Bucket Algorithm

The Token Bucket algorithm maintains a bucket with a maximum capacity of budget units that refill at a constant rate. Incoming requests consume budget units from the bucket based on their cost (USD), and requests are rejected when insufficient budget units are available. This approach allows variable processing rates and can accommodate short-term traffic bursts (which could be triggered without user's knowledge/intention in agent workflows) while maintaining long-term rate limits.

In a cost-aware context, each request consumes budget units proportional to its cost. A $0.001 request consumes 1 budget unit, while a $0.50 request consumes 500 budget units. The bucket refills at a steady rate (e.g., 1000 budget units per hour = approximately $1/hour budget).

Note on model token consumption: For GenAI applications, you may need dual limiting: request-based (requests per second) and model token-based (model tokens per second consumed by the LLM). Since exact model token counts are often only known post-response, consider pre-request estimation based on prompt length plus hard caps via max_tokens parameter to prevent unbounded generation costs. One technique used in practice is to calculate the cost upfront based on input model tokens, then estimate output model tokens using the max_tokens parameter (provisional reservation), and adjust at the end to reflect actual output model tokens used (correction). In practice:

Provisional reservation + correction:
- at request start, calculate estimated_cost_usd = cost_usd(input_model_tokens) + cost_usd(max_tokens_estimate), then convert to budget units: reserve_budget_units = estimated_cost_usd / 0.001 - this ensures you don't exceed the budget upfront;
- once the response completes, calculate actual_cost_usd = cost_usd(input_model_tokens) + cost_usd(actual_output_model_tokens), convert to actual_budget_units = actual_cost_usd / 0.001, and refund reserve_budget_units - actual_budget_units - this ensures accurate accounting.
Per-model pricing: different models have distinct input vs output token prices. Maintain a pricing catalog keyed by model and use model-specific rates when converting model tokens to cost (USD). See also provider docs on rate limits and model-specific quotas, e.g., Anthropic rate limits.
Formula: reserve_budget_units = (cost_usd(input_model_tokens) + cost_usd(max_tokens_estimate)) / 0.001; actual_budget_units = (cost_usd(input_model_tokens) + cost_usd(actual_output_model_tokens)) / 0.001. Apply the delta to refund the difference.
Example (assuming GPT-4o pricing: $2.50/1M input tokens, $10.00/1M output tokens):

estimate:
- input_model_tokens: 800
- max_tokens: 300
- estimated_cost_usd = (800 * 2.50/1M) + (300 * 10.00/1M) = $0.002 + $0.003 = $0.005
- reserve_budget_units = 0.005 / 0.001 = 5 budget units

model returns:
- actual_output_model_tokens: 120
- actual_cost_usd = (800 * 2.50/1M) + (120 * 10.00/1M) = $0.002 + $0.0012 = $0.0032
- actual_budget_units = 0.0032 / 0.001 = 3.2 budget units
- refund = 5 - 3.2 = 1.8 budget units

Pros:

Provides granular control over cost consumption with configurable maximum capacity (budget ceiling) and refill rate (budget replenishment rate)
Allows legitimate burst traffic to be processed immediately when budget units are available, improving user experience during occasional high-cost operations
Offers flexibility to adapt to varying traffic patterns and dynamic workload requirements without rejecting expensive requests unnecessarily
Simple conceptual model that is relatively straightforward to implement and understand, making it easier to explain to stakeholders
Well-suited for scenarios where users occasionally need to exceed steady-state spending limits (e.g., triggering agentic workflow that requires multiple LLM calls)

Cons:

Requires careful tuning of bucket capacity and refill rate to prevent DoW attack vectors while still allowing legitimate usage patterns. If done poorly:
- Can be exploited by greedy users who consume all available budget units during bursts, potentially causing cost spikes if bucket capacity is set too high
- May lead to unpredictable monthly cost patterns in Generative AI contexts where accumulated budget units enable sudden resource consumption
- Does not provide strict cost guarantees as accumulated budget units enable users to spend their entire daily budget in minutes if they've been idle
Requires additional computational overhead for budget unit generation and bucket management operations, though this is typically negligible compared to AI inference costs

When to use: Token Bucket works well when your users have legitimate use cases for occasional bursts above their steady-state rate (know your agentic workflow!). For example, a user might invoke multiple agentic workflows by sending complex, multi-step requests. The burst tolerance makes the system feel responsive rather than artificially constraining legitimate workflows.

Leaky Bucket Algorithm

The Leaky Bucket algorithm queues incoming requests in a fixed-capacity bucket and processes them at a constant rate, like water leaking from a bucket (think of it as a FIFO queue with fixed capacity). Requests exceeding bucket capacity are rejected or delayed. However, for real-time applications like chat interfaces, delaying requests may not make sense, so rejection is preferred. This enforces a steady, predictable output rate regardless of input traffic variability.

In cost-aware implementations, the "leak rate" represents the steady cost budget consumption rate. This smooths cost consumption over time.

Pros:

Ensures constant, predictable cost consumption rate that directly translates to predictable monthly bills for AI applications
Smooths out bursty traffic patterns by enforcing steady processing, preventing sudden cost spikes that trigger finance alerts
Provides straightforward behavior that is easier to maintain and debug compared to more complex algorithms
Helps mitigate both Denial of Service and Denial of Wallet attacks through strict rate enforcement at the cost dimension (assuming we can estimate LLM output model tokens reliably)
Ensures fair resource distribution when implemented with per-user (or per-tenant) queues; each queue processes requests in FIFO order, preventing any single user from dominating the shared budget. Note: a single global queue can cause head-of-line blocking where one noisy user starves others
Excellent for aligning technical cost controls with financial planning cycles (e.g., $X per hour leak rate = $24X per day budget)

Cons:

Limited flexibility in adjusting to legitimate traffic pattern variations or seasonal demand without manual intervention
Cannot quickly process small bursts of legitimate requests that exceed the constant leak rate, potentially frustrating users
Requires additional computational overhead to manage the request queue and timing mechanisms
May struggle to handle very short-lived bursts that exceed bucket capacity, leading to request rejections for legitimate use cases
Strictly enforces rate limits which can negatively affect user experience during legitimate usage spikes (e.g., urgent batch processing)
Choosing optimal bucket size (queue depth) and leak rate parameters can be complex and requires understanding usage patterns

When to use: Leaky Bucket is ideal when cost predictability is paramount. For example, if your AI workflow runs in a background or asynchronous manner where strict cost controls are necessary, Leaky Bucket provides the tightest cost controls. The trade-off is reduced flexibility for users, so it does not suit scenarios where users interact with the system in real time and expect responsiveness (for example, via chat interfaces).

Fixed Window Algorithm

The Fixed Window algorithm divides time into fixed intervals (windows) and limits requests to a specific cost budget per window. When the budget is exceeded, requests are rejected until the next window begins. This approach provides simple, time-based rate limiting with minimal state tracking.

In a cost-aware implementation, each window has a cost budget (e.g., $10 per hour). Requests consume budget from the current window. When the budget is exhausted, subsequent requests are rejected until the window resets.

Pros:

Extremely simple to implement with minimal computational and memory overhead, reducing operational complexity
Provides clear, predictable cost windows that align well with billing cycles for AI APIs (hourly, daily, monthly)
Low resource consumption makes it suitable for high-scale distributed systems handling millions of users
Easy to reason about and debug due to straightforward window reset logic, simplifying troubleshooting
Good performance characteristics for stable, consistent traffic patterns with predictable user behavior
Can be implemented efficiently in distributed systems using simple counters in Redis or similar key-value stores

Cons:

Suffers from boundary burst problem where users can make 2x budget requests by timing requests at window edges (e.g., $10 at 09:59 and $10 at 10:01)
Can lead to unpredictable cost spikes when traffic concentrates at window boundaries, defeating the purpose of cost controls
Not well-suited for variable or bursty traffic patterns (common in production GenAI environments)
Window reset behavior can create sudden availability changes that impact user experience (budget available → budget unavailable at window boundary)
Ineffective at preventing DoW attacks that exploit window boundary vulnerabilities through coordinated timing

When to use: Fixed Window is appropriate when simplicity and operational efficiency are priorities and your usage patterns are relatively stable. It's a good starting point for MVPs or internal tools where the boundary burst problem is acceptable. However, once you identify real usage patterns, you may need to migrate to a more sophisticated algorithm to prevent DoW attacks effectively.

Sliding Window Algorithm

The Sliding Window algorithm maintains a continuously moving time window and tracks requests within that rolling interval, providing more accurate rate limiting than fixed windows. The classic implementation (Sliding Window Log) tracks individual request timestamps and costs. The memory-optimized variant (Sliding Window Counter) uses weighted counts from fixed windows to reduce memory overhead while approximating the sliding behavior.

In cost-aware implementations, the sliding window tracks total cost consumed within the rolling time period. Each request's cost is recorded with its timestamp. As time advances, costs outside the window are discarded.

Pros:

Sliding Window Log fully eliminates boundary burst issues by tracking exact timestamps, providing perfect cost distribution accuracy
Sliding Window Counter mostly eliminates boundary bursts through weighted approximation while using significantly less memory
Ensures fairness by continuously monitoring cost consumption rather than resetting at arbitrary intervals
Adapts well to varying traffic patterns, making it suitable for unpredictable AI workload demands
Provides more accurate rate limiting that better prevents DoW attacks compared to fixed windows
Better user experience through consistent cost enforcement without sudden window resets that surprise users

Cons:

Significantly more complex to implement, debug, and maintain compared to simpler algorithms, increasing development time
Sliding Window Log requires substantial memory (O(requests in window) per user) and CPU for timestamp management at scale
Sliding Window Counter uses approximation that allows small residual boundary bursts (often single-digit percent in practice, depending on window size and weighting scheme)
Scalability challenges in distributed systems requiring synchronization across multiple nodes to maintain consistency
Requires more sophisticated infrastructure and state management for production deployment (e.g., sorted sets in Redis for Log variant, atomic counters for Counter variant)

When to use: Sliding Window is a good choice when DoW prevention is critical, you need accurate and fair cost enforcement, and the implementation complexity is justified to protect your budget. Use the memory-optimized variant (Sliding Window Counter) for large-scale deployments.

Selection Criteria for Generative AI Applications

When choosing a rate limiting algorithm for cost-aware DoW prevention in Generative AI applications, consider these dimensions:

Cost Predictability How important is it that monthly costs stay within strict bounds? Leaky Bucket and Sliding Window provide better cost predictability than Token Bucket. Fixed Window provides predictability within each window but suffers from boundary bursts.

Burst Tolerance Do your users have legitimate use cases for occasional bursts above steady-state rates? Token Bucket allows controlled bursts. Leaky Bucket strictly enforces steady consumption. Sliding Window and Fixed Window fall in between.

Implementation Resources What engineering resources can you allocate to building and maintaining the rate limiter? Fixed Window offers the simplest implementation. Sliding Window requires the most infrastructure and engineering effort.

Attack Surface How sophisticated are the DoW attack vectors you need to defend against? Sliding Window and Leaky Bucket offer better protection than Fixed Window and Token Bucket. Fixed Window is particularly vulnerable to boundary timing exploits.

Scale Requirements How many users and requests must you support? Fixed Window and memory-optimized Sliding Window Counter scale best. Classic Sliding Window Log has higher memory requirements per user.

User Experience How important is it to maintain smooth, responsive user experience during legitimate usage spikes? Sliding Window and Token Bucket provide smoother user experience than strict enforcement algorithms. Leaky Bucket and Fixed Window can create frustrating rejection patterns.

Recommendation: If implementation complexity is a concern, start with Fixed Window to get cost controls in place quickly. Once you understand your usage patterns better, migrate to more sophisticated algorithms like Sliding Window Counter.

Trade-Offs Analysis: From Theory to Implementation

Choosing the right algorithm is only part of the solution. Before you write any code, you need to understand your system's cost characteristics, usage patterns, and business constraints. This section provides a practical 7-step checklist for implementing cost-aware rate limiting in production GenAI applications.

Think of this as the path from "we need to control costs" to "we have cost-aware rate limiting running in production." Each step builds on the previous one.

Step 1: Build Cost Observability Early

Start tracking cost metrics from day one. Even if you don't have production traffic yet, simulated workloads and exploratory tests provide valuable insights. You cannot protect costs you cannot measure.

Action items:

Instrument your application to track cost per request. Tag each LLM API call, vector search operation, and expensive computation with its cost.
Export cost metrics to your observability platform (Prometheus, Datadog, CloudWatch, etc.). Create time-series metrics for:
- request_cost_usd (histogram): Distribution of request costs, enabling p50/p95/p99 analysis and anomaly detection
- total_spend_usd (counter with labels for user/tenant): Monotonically increasing total spend; derive hourly/daily rates via recording rules or range queries
- expensive_operation_count (counter): Count of operations exceeding threshold (e.g., >$0.10)

Why this matters: Without observability, you're flying blind. You won't know if your rate limiter is working. You won't know which users to exempt from strict limits. You won't know when to adjust thresholds. Invest in observability before you invest in controls.

Common pitfall: Tracking only aggregate costs. You need per-operation granularity to identify DoW attack patterns and tune limits appropriately.

Step 2: Measure or Estimate Baseline Costs

When running Generative AI workloads, you likely won't have historical cost data yet. Start by estimating costs based on model pricing and expected usage patterns. Perform synthetic tests to validate your hypotheses. Use cost observability tools to gather real data as soon as possible.

Action items:

Do not focus on averages alone. Understand the distribution of costs: p50, p90, p95, p99. The long tail matters.
Create cost profiles for different request types:
- Cached responses: $0.0001 - $0.001
- Simple LLM calls (small prompts): $0.01 - $0.05
- Complex agentic workflows: $0.20 - $0.50
- Batch processing with large documents: $1.00 - $5.00
Identify the cost distribution across different models, input sizes, and request types. Which operations dominate your spend?
Run synthetic load tests that simulate different user behaviors. Include both "normal" users and "expensive" users to understand the cost range.
Document your findings. Create a table like this:

Operation Type	Avg Cost	p95 Cost	% of Total Requests	% of Total Cost
Cached lookup	$0.0001	$0.0001	60%	1%
Simple Q&A	$0.02	$0.05	30%	10%
Agentic workflow	$0.30	$0.80	9%	45%
Batch processing	$2.00	$8.00	1%	44%

Why this matters: You cannot set reasonable budgets without understanding baseline costs. The distribution (especially the long tail at p95/p99) determines how much burst capacity to allow. If p95 is 10x the average, Fixed Window or Leaky Bucket might reject too many legitimate requests.

Common pitfall: Optimizing for the average case. In GenAI applications, the p95 and p99 costs often drive your budget exposure. A few expensive requests can dominate total spend.

Step 3: Identify High-Cost Users and Patterns

Once you have cost observability running, invest in exploratory analysis. Query your observability stack to find high-cost users and journeys. Understanding these patterns helps you optimize before implementing strict limits.

Action items:

Sort users by total spend. Identify the top 10% of spenders. Are they legitimate power users or potential DoW attackers?
Analyze their request patterns: burst size and frequency, time-of-day and day-of-week effects, model mix (which models are used most), tool-call frequency and patterns, cache hit rates, per-session vs per-user behavior trends
For each high-cost pattern, ask: Could this be optimized?
- Can we add caching to reduce redundant LLM calls?
- Can we batch requests to improve efficiency?
- Can we use smaller/faster models for certain operations?
Estimate potential savings and effort required for each optimization. Prioritize low-effort, high-impact optimizations before implementing rate limits.

Why this matters: Rate limiting is a defensive control. Optimization is an offensive strategy. If you can reduce costs through caching, batching, or model selection, you improve both user experience and budget protection. Implement optimizations first, then add rate limiting as a safety net.

Common pitfall: Jumping straight to rate limiting without exploring optimization opportunities. You might throttle legitimate users when the real problem is an inefficient implementation.

Step 4: Define Budgets and Policies

Work with your finance team to translate business requirements into technical budgets. This is not purely an engineering decision. Budgets affect pricing, user experience, and revenue.

Action items:

Decide on budget dimensions:
- Per-user budgets: Each user gets $X per hour/day/month
- Per-session budgets: Each login session gets $Y total spend
- Per-tenant budgets: Enterprise customers get $Z per month
Map budgets to product tiers:
- Free tier: $0.50/day ($15/month)
- Pro tier: $5.00/day ($150/month)
- Enterprise tier: Custom limits negotiated per contract
Set daily and monthly caps as safeguards. Even if your algorithm allows bursts, enforce hard caps to prevent runaway costs.
Define overage policies:
- Soft limit (warning to user): "You've used 80% of your daily budget"
- Hard limit (request rejection): "Daily budget exceeded. Budget resets in 4 hours."
- Grace period for accidental overages vs. immediate cutoff

Why this matters: Technical controls must align with business policies. If you set budgets too low, you frustrate legitimate users and hurt retention. If you set budgets too high, you fail to prevent DoW attacks. Finance needs to approve these numbers because they directly impact revenue and costs.

Common pitfall: Setting budgets based only on engineering intuition. Involve finance and product teams to ensure budgets align with business model and user expectations.

Step 5: Choose Your Rate Limiting Strategy

Based on the patterns you discovered, your estimates, and your application architecture, choose the most suitable rate limiting algorithm. This is an important architectural decision. Consider writing an Architecture Decision Record (ADR) to document your choice.

Action items:

Review the algorithm comparison from the previous section (Token Bucket, Leaky Bucket, Fixed Window, Sliding Window).
Consider trade-offs specific to your context:
- Current infrastructure: Do you already have Redis? Distributed counters? Or are you starting from scratch?
- Scalability needs: Are you handling 100 users or 100,000 users? Memory requirements differ significantly.
- Implementation complexity: Do you have time and resources to build Sliding Window, or do you need Fixed Window to ship quickly?
- Maintenance burden: Who will debug rate limiter issues at 2am? Simpler algorithms are easier to troubleshoot.
If you're uncertain, start with a simpler algorithm (Fixed Window) and plan to iterate. Ship cost controls quickly, gather data, then migrate to more sophisticated algorithms if needed.
Document your decision in an ADR:
- Context: What problem are you solving? What constraints do you face?
- Options considered: List 2-3 algorithms you evaluated
- Decision: Which algorithm did you choose and why?
- Consequences: What are the trade-offs? What might you need to revisit later?
Make the implementation configurable. Use feature flags or configuration files to allow switching algorithms or tuning parameters without code changes.

Why this matters: Rate limiting algorithms have different operational characteristics. A poor choice increases maintenance burden or fails to prevent DoW attacks. Documenting your decision helps future engineers understand the rationale and constraints.

Common pitfall: Over-engineering too early. You don't need perfect rate limiting from day one. You need good-enough rate limiting that protects your budget while you learn about real usage patterns. Iterate, iterate, iterate!

Distributed deployment consideration: If you're running multiple application instances, you'll need centralized rate limiting (for example, Redis with Lua scripts for atomic operations). Local in-memory rate limiters in each instance will drift and fail to enforce global budgets correctly. Part 3 will cover distributed implementation patterns.

Step 6: Extend Observability and Alerting

Once you've chosen an algorithm and defined budgets, extend your observability to monitor rate limiter performance. You need visibility into both cost consumption and rate limiter behavior.

Action items:

Set up alerts for spending anomalies:
- Total daily spend crosses budget threshold
- Single request costs more than expected (e.g., >$1.00)
Monitor rate limiter performance metrics:
- Throttle rate: What percentage of requests are being rejected?
Create dashboards showing:
- Current spend vs. budget (per user and globally)
- Throttled requests over time
- Distribution of request costs
Integrate cost alerts into your incident response workflow. Finance should be notified of budget overruns, not just engineering.

Why this matters: Rate limiters can fail silently if you don't monitor them. You might think you're protected while costs continue to climb because the rate limiter has a bug or is misconfigured. Observability catches these issues before they become expensive.

Common pitfall: Monitoring only request rejection rates. You also need to monitor budget consumption. A rate limiter that never rejects requests is not protecting your budget.

Step 7: Optionally Start with Read-Only Mode

If you already have production traffic and breaking user experience is a concern, consider starting with a "read-only" or "shadow mode" deployment. Track and log what would be throttled without actually rejecting requests.

Action items:

Implement rate limiting logic but bypass enforcement. Log throttle decisions: "User X would be throttled for request Y (cost $Z, budget consumed: $A/$B)".
Run in shadow mode for 1-2 weeks to validate that your cost estimates and budgets are reasonable.
Analyze the shadow mode logs:
- How many requests would have been throttled?
- Are you throttling legitimate users or catching actual abuse?
- Do you need to adjust budgets or refine cost calculations?
Adjust thresholds based on real data before switching to enforcement mode.
Prepare a mechanism for switching to enforcement mode when ready:
- Feature flag controlled by configuration
- Emergency off-switch in case enforcement causes unexpected problems

Why this matters: Turning on rate limiting in production is risky. If your budgets are too low or your cost calculations are wrong, you'll throttle legitimate users and hurt retention. Shadow mode lets you validate assumptions with real traffic before committing to enforcement.

Common pitfall: Skipping shadow mode and going straight to enforcement. This works if you have very high confidence in your cost models and budgets. For most teams, shadow mode reduces risk significantly.

Summary: From Theory to Practice

You now have a framework for implementing cost-aware rate limiting:

Build observability first: You cannot protect costs you cannot measure
Understand your cost distribution: Averages can be misleading. Focus on p95 and p99
Optimize before limiting: Caching and batching reduce costs without throttling users
Align budgets with business goals: Involve finance and product teams
Choose an algorithm that fits your constraints: Perfect is the enemy of shipped
Monitor rate limiter performance: Track both budget consumption and throttle rates
Use shadow mode to de-risk rollout: Validate assumptions before enforcement

The best rate limiting strategy is the one you actually implement and maintain. Start simple. Gather data. Iterate.

Conclusion: From Theory to Production

You've learned the theory behind cost-aware rate limiting and the practical steps to implement it. Let's recap the key takeaways:

Understanding the problem: Generative AI applications introduce cost unpredictability that traditional rate limiting cannot address. One user request might cost $0.001 while another costs $0.50. Both count as "one request" from a DoS perspective, but they have wildly different budget impacts. Cost-aware rate limiting protects your budget by throttling based on cost consumption, not just request count.

Choosing the right algorithm: Four algorithms offer different trade-offs:

Token Bucket allows bursts but can lead to cost spikes
Leaky Bucket provides strict cost predictability at the expense of flexibility
Fixed Window is simple to implement but vulnerable to boundary exploits
Sliding Window offers the best DoW protection with higher implementation complexity

Generally - start with Fixed Window and iterate.

Following a practical checklist: Before you write code:

Build cost observability to measure what you want to protect
Understand your cost distribution (p95 and p99 matter more than averages)
Identify high-cost patterns and optimize before limiting
Define budgets with finance and product teams
Choose an algorithm that fits your constraints
Extend observability to monitor rate limiter performance
Use shadow mode to validate assumptions before enforcement

Gathering data before committing: The most important lesson: gather data and validate your assumptions. Run synthetic tests. Use shadow mode. Understand your users' actual behavior before you throttle them. Rate limiting is a blunt instrument. Cost observability and optimization are your precision tools.

What's Next

This post covers the decision-making framework and practical steps for implementing cost-aware rate limiting. Part 3 of this series will provide a hands-on implementation.

References

Piyush Garg. "API Rate Limiting Strategies: Token Bucket vs. Leaky Bucket." Eraser.io, April 11, 2024. https://www.eraser.io/decision-node/api-rate-limiting-strategies-token-bucket-vs-leaky-bucket
"Rate Limiting Algorithms - System Design." GeeksforGeeks. https://www.geeksforgeeks.org/system-design/rate-limiting-algorithms-system-design/
"Sliding Window Rate Limiting and its Memory-Optimized Variant." RD Blog, February 5, 2024. https://rdiachenko.com/posts/arch/rate-limiting/sliding-window-algorithm/

Agentic architecture note

Assume agentic architecture is an application that consists of multiple LLM calls orchestrated to achieve a goal, such as a multi-step reasoning process.

Denial of Wallet: Cost-Aware Rate Limiting for Generative AI Applications - Introduction (Part 1)

Fri, 21 Nov 2025 00:00:00 GMT

This post introduces a series about Denial of Wallet (DoW) as a distinct failure mode from Denial of Service (DoS). In the first part, we'll explore how to reason about cost-aware rate limiting across CDN, gateway, and application layers. After that, you'll learn when to invest in custom limits, the second part focuses on user-aware limits and how to frame cost units and design strategies. Finally, part 3 of this series will provide a hands-on implementation.

Introduction - The Cost Trap in AI-Era Systems

Six months after launching your AI-powered search feature, you get an urgent Slack message from finance: cloud costs have tripled. The operations team is confused. Your requests per second (RPS) metrics are steady. Latency is fine. No alerts fired. Everything looks healthy - except the bill.

The problem? Your uniform rate limits protect performance but ignore cost variance. A basic search query costs you fractions of a cent in Elasticsearch time. An agentic conversation with GPT-4o and multiple tool calls costs $0.20-$0.50. Both count as "one request" under your standard per-user 100 req/min limit. A single user can stay well within rate limits while running up hundreds of dollars in LLM charges.

Traditional rate limiting was built for a simpler era when most requests had similar resource costs. In today's cloud-native systems (especially those integrating LLMs, vector databases, image generation, and third-party APIs) request costs vary by orders of magnitude (different number of tokens for LLM, different data shapes for database queries). Standard RPS-based throttles protect your infrastructure but leave your budget exposed.

This post introduces Denial of Wallet (DoW) as a distinct failure mode from Denial of Service (DoS). We'll explore how to reason about cost-aware rate limiting across CDN, gateway, and application layers. You'll learn when to invest in custom, user-aware limits and how to frame cost units and design strategies.

Why Now? AI/LLM Adoption and Usage-Based Pricing

This problem has always existed with heterogeneous workloads - think image processing versus simple GETs. But LLM adoption has made it urgent. OpenAI charges $1.25 per million input tokens and $10.00 per million output tokens for GPT-4o. A single agentic workflow can easily consume 100,000 tokens, costing $0.20–$0.50 per request. Compare that to an Elasticsearch query at effectively near-zero cost per search after amortizing your cluster costs.

The cost variance is staggering. In systems where cheap and expensive operations share the same rate limit buckets, a user can unknowingly (or deliberately) shift their request mix toward expensive operations. Your performance stays fine. Your spend explodes.

To understand how to protect against this threat, we need to clearly distinguish between traditional performance-based attacks and cost-based exploitation. The key insight is that your existing DoS defenses (designed to protect availability and latency) operate on different metrics than what's needed to protect your budget. Let's examine how these two failure modes differ and why conventional rate limiting leaves you exposed to cost-based abuse.

Consider a typical scenario. You deploy a chatbot that can answer questions from cached knowledge (cheap) or invoke an LLM agent with web search capabilities (expensive). Your API gateway enforces 100 req/min per user. Everything works as designed.

Then a power user discovers they can run complex agentic queries repeatedly. They stay under 100 req/min. Your servers handle the load. But each query costs $0.40 in LLM fees. Over a month, this single user generates a $1,200 bill. Your rate limits did their job protecting availability. They just couldn't see the cost dimension.

Denial of Service vs Denial of Wallet

Denial of Service attacks aim to exhaust system resources (CPU, memory, network bandwidth) making the service unavailable to legitimate users. Your CDN, WAF, and application rate limits are designed for this threat model. They cap requests per second, enforce queue depths, and shed load when close to saturation.

Denial of Wallet attacks exploit a different vulnerability: the cost dimension. An attacker (or even a legitimate but cost-oblivious user) stays within your performance-oriented rate limits but selectively triggers expensive operations. Your infrastructure remains healthy. Your bill skyrockets.

Where DoS Controls Stop and DoW Begins

Traditional DoS defenses operate at the infrastructure layer. Your CDN throttles suspicious IPs. Your API gateway enforces global or per-API-key RPS caps. Your load balancer monitors backend health. These controls protect availability and latency. They work beautifully when all requests cost roughly the same.

But consider this scenario: your system serves both cached data lookups and live LLM inference. The cached lookup hits Redis and returns in 5ms at negligible cost. The LLM call waits 800ms for GPT-4o and costs $0.30. From a DoS perspective, the LLM call is slower but manageable - it's just one request in your queue. From a DoW perspective, it's 3,000x more expensive than the cache hit.

Your CDN and gateway can't see this cost difference. They count requests and bytes. They don't know which code path each request will take or which third-party APIs you'll invoke. Cost context lives in your application layer - in the business logic that decides whether to serve from cache, hit the database, or call an LLM.

Cost Surfaces: LLM Calls, Vector Search, Data Egress, Image Processing

Modern cloud-native systems expose multiple cost surfaces. Here are common examples where request costs vary dramatically:

LLM API calls: GPT-4o at $0.20–$0.50 per agentic conversation versus cached responses at $0.0001.
Vector search: Embedding generation ($0.0001 per query with a small model) versus brute-force similarity across millions of vectors (seconds of GPU time).
Image processing: Thumbnail generation (milliseconds, cents) versus AI upscaling or video transcoding (minutes, dollars).
Data egress: Serving a 10KB JSON response versus streaming a 500MB dataset across regions.

Each of these can coexist in a single API surface. A user might invoke /api/search with a query parameter that triggers either a cheap cache hit or an expensive LLM fallback. The endpoint looks the same. The cost is not.

Comparison: DoS vs DoW

Dimension	Denial of Service (DoS)	Denial of Wallet (DoW)
Primary goal	Protect availability/latency	Protect cost budget/spend risk
Trigger pattern	High RPS, resource exhaustion	Normal RPS, high-cost request mix
Detection signals	5xx, p95 latency, CPU saturation	Bill/usage spike, per-user spend anomalies
Controls	CDN/WAF throttles, queue backpressure	Cost-aware per-user/session quotas, budgets
Risk owner	SRE/Platform	Architecture/Product/Finance jointly

The key insight: DoS protection is necessary but insufficient. You need both performance-based and cost-based controls. DoW can occur even when all your DoS defenses pass. A single user running expensive workflows can dominate your monthly spend without ever triggering a 429 Too Many Requests response.

Real-World DoW Scenarios

To make this concrete, let's examine two scenarios that illustrate how DoW manifests in production systems.

Scenario 1: The Curious Power User

Sarah, a legitimate user on your free tier, discovers your AI-powered document analysis feature. She uploads PDFs that trigger GPT-4o summarization. Here's how it unfolds:

Day 1: Sarah makes 50 requests/hour (well within your 100 req/hour limit)
- Each request processes 20,000 tokens = $0.25 per request
- Hourly cost: $12.50/hour
- Daily cost: $300/day
Your DoS monitoring dashboard: All green
- RPS: 0.83 req/sec (far below limits)
- Latency: p95 = 2.1s (acceptable)
- Error rate: 0%
- CPU/memory: Normal
- No alerts fire
Your DoW reality: One user costs $300/day = $9,000/month
- Free tier budget allocation: $10/user/month
- Actual loss: $8,990/month on this single user
- If 100 users discover this pattern: $900,000/month burn

Scenario 2: The Malicious Actor

An attacker reverse-engineers your /api/search endpoint and discovers two code paths:

Cache hit: Instant response, negligible cost
LLM semantic search fallback: 800ms latency, $0.30 cost

The attack pattern:

Craft queries designed to miss cache (unique nonsensical phrases, random strings)
Stay at 90% of your rate limit (90 req/min from your 100 req/min limit)
Force expensive LLM fallback on every single request
Run continuously from multiple API keys to stay under per-key limits

Cost impact breakdown:

Per API key: 90 requests/min × $0.30 = $27/minute = $1,620/hour
Daily cost per key: $38,880/day
With just 10 API keys: $388,800/day
Traditional rate limiter verdict: "Everything normal, users at 90% of allowed RPS"
Your monthly bill: Catastrophic

How cost-aware rate limiting would prevent both scenarios:

In both cases, a cost-aware rate limiter would have:

Pre-execution check: Before invoking the costly LLM, estimate the request cost based on workflow type or any other heuristics
Budget check: Query the user's budget (more on this in the next part) to see how much spend remains
Enforcement decision: Decide to allow or reject based on remaining budget, not just RPS

Graceful rejection: Return 429 Too Many Requests with headers:

X-RateLimit-Cost-Remaining: 0
X-RateLimit-Cost-Reset: 1637012400
Retry-After: 3600

Cost attribution: Record actual cost post-execution for observability and billing

Sarah would have received clear feedback after hitting her $10/month limit, prompting an upgrade conversation. The attacker would have been throttled after exhausting their allocated budget, regardless of staying under RPS limits.

Anatomy of Enterprise Rate Limiting

Understanding where cost-aware limits fit requires mapping the typical layers of enterprise traffic management. Each layer sees different information and enforces different policies. Cost context becomes visible only at specific points in the stack.

Your CDN and Web Application Firewall sit at the edge. They see raw HTTP requests from IP addresses. They're excellent at detecting volumetric attacks, geographic anomalies, and malicious patterns. They enforce coarse-grained limits:

Requests per second per IP or region
Request size limits
Path-based throttling (e.g., 10 req/s for /api/*)

What they don't see:

User identity: Which authenticated user is making the request
Session or user/tenant context: Whether this is a free trial or enterprise account
Cost implications: Which backend services will be invoked
Business logic: Whether the request will hit cache or trigger expensive operations

CDN and WAF limits protect your infrastructure from brute-force abuse. They're your first line of defense against traditional DoS. But they operate on metadata-headers, paths, source IPs-not on the semantic meaning or cost of each request.

Most CDNs and WAFs cannot natively enforce cost-aware limits. In practice, most teams implement cost-aware limiting at the application layer (or integrate it into their service mesh) where cost context is naturally available, and use CDN/WAF for traditional DoS protection.

Application Layer: Tying Requests to Users, Sessions, Tenants, Plans

Cost-aware rate limiting happens in your application layer. This is where you have:

Authentication context: User ID, API key, OAuth token
Authorization context: Product plan, subscription tier, feature flags
Business logic visibility: Which code paths will execute, which external services will be called
Cost attribution: The ability to estimate or measure per-request costs

At this layer, you can answer questions like:

"How much has this user spent in the last hour?"
"Is this tenant approaching their monthly budget?"
"Should we serve from cache or allow an LLM call?"

This is where you implement custom rate limiting (e.g. using a distributed cache like Redis or in-memory state stores). You track cost-weighted tokens per user, per session, or per tenant. You enforce per-plan quotas. You integrate with observability to detect spend anomalies.

Observability: Metrics You Need Before Building Cost-Aware Limits

Before implementing custom rate limiting, instrument your application to collect baseline cost data:

Per-request cost metrics: Tag each request with estimated or actual cost. Export to Prometheus, Datadog, or CloudWatch.
Per-user/tenant aggregations: Sum costs over rolling time windows (1m, 5m, 1h, 1d).
Cost distribution: Track p50, p95, p99 costs per endpoint or workflow to understand variance.
Request mix tracking: Log which code paths executed (cache hit vs LLM call) to identify high-cost patterns.
Agentic workflow tracing: For complex multi-step operations, trace the full call graph to attribute costs accurately (some useful tooling exists already for this, e.g. LangFuse).

Without these metrics, you're flying blind. You won't know which users are driving costs, which operations are expensive, or where to set budget thresholds. Start with observability, then build controls.

Having said that, if you are working on a greenfield project with LLM integration and you don't yet have cost data, it's reasonable to start with educated estimates based on how much your business is willing to spend per user/session/tenant. You can refine your limits as real data comes in.

Cost Awareness in System Design

Designing cost-aware systems starts with modeling heterogeneous request costs. Not all operations are created equal. Even within a single endpoint, different execution paths can have wildly different price tags.

Modeling Costs per Workflow vs per Endpoint

Traditional rate limiting thinks in terms of endpoints: /api/search gets 100 req/min, /api/upload gets 10 req/min. Cost-aware rate limiting thinks in terms of workflows and resource consumption. The currency here isn’t time: LLM cost is driven by tokens (requests × tokens in/out), not latency.

Consider a search endpoint that can:

Serve from an in-memory cache (cost: $0.0001)
Query Elasticsearch (cost: $0.001)
Fall back to an LLM for semantic understanding (cost: $0.20)

The endpoint is the same. The workflow cost varies by 2,000x. A user could make 100 requests per minute, all hitting the LLM fallback, and stay within your RPS limit while burning through $20/minute or $1,200/hour.

To model this, define cost units for each workflow:

// Map operation -> cost units (normalized to $ or credits)
const COST_TABLE = {
  'search.basic': 0.0001,
  'search.es': 0.001,
  'search.llm': 0.2
};

function estimateCost(operation, params) {
  // Optionally scale by tokens, size, or retries
  const base = COST_TABLE[operation] ?? 0;
  const multiplier = params?.tokens ? params.tokens / 1000 : 1;
  return base * multiplier;
}

This simple table lets you attach a cost estimate to every request. You can refine it over time based on real billing data. The key is making cost a first-class metric alongside latency and error rate. We will cover this in more detail in following parts of this series.

Measuring and Attributing Cost to Users/Sessions/Tenants

Cost-aware rate limiting requires attributing costs to specific actors. You need counters keyed by:

User ID: Track individual user spend
Session ID: Limit costs per browsing session (useful for anonymous users)
Tenant ID: Enforce per-organization budgets in B2B SaaS
Product plan: Apply different budgets for free, pro, and enterprise tiers

Ensure these limits are clearly disclosed to customers in SLA and other customer-facing documents.

Example cost attribution flow:

Request arrives with authentication token
Extract user ID and product plan
Estimate request cost from workflow type
Check if user's current spend + estimated cost <= budget
Allow or deny request
Record actual cost post-execution for observability

This requires fast lookups. Redis is a common choice for storing per-user cost counters with TTLs matching your time windows (e.g., 1-hour sliding windows). We will cover implementation details in part 2 of this series.

When to Invest in Custom Rate Limiting

Not every system needs cost-aware rate limiting. Building and maintaining custom logic adds complexity. You need to balance the cost of implementation against the risk of budget exposure. Here are clear indicators for when to invest.

Decision Checklist (Must-Have Signals)

Invest in custom cost-aware rate limiting if you see these patterns:

Material cost variance (≥10–100x) across request types
- Quantified threshold: p99 cost / p50 cost > 100x
- Example: LLM calls ($0.30) vs cache hits ($0.0001) = 3,000x variance
- If your cheapest and most expensive operations differ by 100x or more, uniform rate limits can't protect you.
Unpredictable or spiky workloads
- Quantified threshold: Top 5% of users drive >50% of total spend, or week-over-week cost variance >30%
- Example: Power users discovering expensive workflows or seasonal usage spikes (look at current - Q4 2025 - Antropic Claude Code weekly limits ;) )
Clear budget constraints (per week/month or per tenant/plan)
- Quantified threshold: Hard monthly budget cap exists, or per-tenant contractual limits are in place
- Example: Free tier limited to $10/month per user
- Finance cares about the bill, not the RPS.

Wrapping Up Part 1

This post introduces Denial of Wallet (DoW) as a failure mode distinct from traditional Denial of Service (DoS). While DoS attacks overwhelm system performance, DoW exploits cost differences in modern applications (especially those involving LLMs, vector search, and image processing) where some requests are orders of magnitude times more expensive than others. Uniform rate limits based on requests-per-second no longer protect budgets, because even low-volume users can trigger costly operations without breaching performance limits.

Agentic conversation note

For simplicity, assume that agentic conversation is a workflow where we have a "coordinator" LLM that supervises the API request execution, calls other LLMs/tools as needed, and aggregates results. This is more expensive than a single LLM call due to multiple invocations and orchestration overhead.

LLM tokens note

For the sake of this post, think of tokens as unit of data that LLMs process. E.g. imagine API that processes text and provides a summary. Then "input tokens" is the size (e.g. number of words/characters) of the text you send to API, and "output tokens" is the size of the summary you get back. LLM pricing is often based on number of input and output tokens processed. In practice, it is hard to estimate exact token counts before making the API call (especially output tokens - but some LLM APIs let you set a maximum).

The Staff Engineer Toolkit

Thu, 06 Nov 2025 00:00:00 GMT

The Staff Engineer Toolkit

A collection of tools, techniques, and reflections from our experience working as Staff Engineers in large-scale distributed systems and reliability-focused organizations.

Who is a Staff Engineer?

A Staff Engineer is a senior individual contributor who plays a strategic technical leadership role within an organization. It’s often seen as a step up from Senior Engineer - it’s a shift from "doing more" to "thinking broader". And do not confuse it with management - Staff Engineers remain individual contributors, but they influence at a higher level.

You spend less time answering "how" and more time answering "why". You shape the technical direction of teams, align engineering initiatives with business goals, and guide others to deliver at scale.

Key characteristics of an effective Staff Engineer include:

Strong technical foundation - deep understanding of software design, architecture, and operational excellence.
Excellent communication skills - you collaborate across teams, departments, and sometimes entire organizations.
Ability to influence without authority - you lead by trust, not by title.
Time awareness - your most valuable resource is your time.
Emotional resilience - ambiguity, imposter syndrome, and unclear expectations are part of your journey.

Staff Engineer’s toolkit

The following sections summarize practical techniques we’ve found useful to grow and be effective in the Staff Engineer role.

Define and evolve your leadership style

Your leadership style is shaped by your personality, values, and experiences. It determines how you influence others and drive change. Different environments reward different styles-some value hierarchical decision-making, others thrive in collaborative, experimental cultures. Be self-aware and find what aligns with your environment.

We’ve learned that collaborative, empowering leadership works best in the long run. The organic problem-solving approach described by Gerald M. Weinberg in "Becoming a Technical Leader" focuses on enabling and empowering others to succeed - it creates an environment where people feel valued, supported, and motivated to contribute their best work (and reduces the fear of job loss common in more hierarchical organizations). Using a threat/reward or blame culture may yield short-term results, but it often leads to burnout, low morale, and fear-driven behaviors. Moreover, it prevents innovation and risk-taking (which are essential in tech).

Lead by influence, not authority - build trust and credibility through technical expertise and effective communication.
Prioritize team success over individual heroics - avoid "hero culture", celebrate team wins, and share knowledge to grow together.
Foster an environment where people feel safe to fail, learn, and share (avoid the blame game and threat/reward culture).
Apply the MOI Model (Motivation, Organization, Ideas) to lead and influence others:
- Motivation: understand what drives your team and align it with company goals.
- Organization: navigate organizational dynamics and politics to build alliances and gain support for initiatives.
- Ideas: communicate vision effectively and plant seeds of innovation.

Make sure that your leadership style aligns with your organization's culture and values. Without that alignment, even the best intentions can lead to frustration and ineffectiveness.

Recommended readings: Becoming a Technical Leader: An Organic Problem-Solving Approach by Gerald M. Weinberg; Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations by Nicole Forsgren, Jez Humble, and Gene Kim

Develop an architectural mindset

As a Staff Engineer, you are expected to make architectural decisions that balance business goals, technical trade-offs, and team capabilities. Focus on "good enough" solutions rather than perfect ones. Iterate.

Understand your stakeholders' and sponsors' needs and constraints (business drivers, technical limitations, team capabilities).
Break down complex problems and systems into smaller, manageable components - build your map to navigate complexity with your team, stakeholders, and other teams (e.g., use C4 model to visualize system architecture, define system context, containers, components, and code).
Agree on the "-ilities" (non-functional requirements) upfront to guide your architectural decisions (e.g., scalability, maintainability, performance, security) - use the Architecture Characteristics Worksheet to document and align on these requirements.
Aggregate "-ilities" into categories such as operational (e.g., performance, scalability, elasticity), structural (e.g., modularity, maintainability, testability), and cross-cutting (e.g., security, agility, deployability) to simplify decision-making and streamline discussions with stakeholders and engineering teams.
Document architectural decisions using ADRs (Architecture Decision Records) to capture context, options considered, and rationale.

Recommended readings: Fundamentals of Software Architecture: A Modern Engineering Approach by Mark Richards and Neal Ford; Software Architecture: The Hard Parts: Modern Trade-Off Analyses for Distributed Architectures by Neal Ford and Mark Richards

Writing is thinking

It clarifies ideas, preserves decisions, and enables asynchronous collaboration. In remote and distributed teams, writing becomes even more critical as a communication tool.

Capture everything that matters-designs, roadmaps, retrospectives, meeting notes.
Use consistent templates (ADR, RFC, design doc) and always include:
- TL;DR - add executive summaries to your documents to help busy stakeholders quickly grasp key points.
- Audience - define who is expected to read it, who needs to approve it, and who should be informed.
- Context - background information and relevant history.
- Problem statement - what issue are you addressing.
- Options considered - alternative solutions evaluated.
- Decision rationale - why you chose this option.
- Consequences - implications, risks, and next steps.
Timebox your writing to avoid perfectionism and overthinking. Iterate based on feedback: check comments and questions from others, then revisit and refine your writing periodically to ensure it remains relevant and accurate.
Consider using AI tools for summaries, crossing out mind short-cuts. E.g. run your notes through ChatGPT or Claude. Not to reformat, but to make it concise and easier to consume (READ IT AFTERWARDS!). Check our blog on using AI for Software Architects.

Build organizational awareness

In large organizations, technical skill alone is not enough - you must understand how the organization works.

Map key stakeholders, teams, and their relationships to understand the organizational landscape (especially important in larger organizations where you need to navigate multiple teams and departments). You can use tools like org charts, stakeholder mapping matrices, or network diagrams to visualize these relationships.
Identify influencers and decision-makers to effectively navigate organizational politics (those can be different from formal authority figures, e.g., senior engineers with strong technical credibility). Understand their motivations, goals, and pain points to build alliances and gain support for your initiatives. Start by reading their documents, adding comments, asking questions, and scheduling 1:1s to build trust over time.
If possible, meet people in person - it strengthens relationships dramatically.
Understand the company's goals, values, and culture to align your technical vision with organizational objectives (check company OKRs, your department goals, your reporting manager's goals, your team goals).
Monitor your reporting chains and ensure alignment with your manager and other leaders (e.g., regular documented 1:1s, skip-level meetings, all-hands meetings).

Learn how to learn continuously

Continuous learning is a core part of being a Staff Engineer. Think in terms of both I-shaped skills and T-shaped skills:

T-shaped skills - broad knowledge across multiple areas (architecture, DevOps, cloud, security, etc.).
- Grow as an expert generalist who can see the big picture and connect the dots between different areas.
- Focus on "stuff you don't know you don't know" to expand your knowledge and skills beyond your current expertise (e.g., Thoughtworks Technology Radar for new trends and technologies - what's new, what's coming, what to try, what to hold back on).
- Start your day with a cup of coffee/tea and 15 minutes of reading technical articles and blogs (e.g., InfoQ, DZone Refcards, HandsOnArchitects 😉).
I-shaped skills - deep expertise in a specific domain (backend, DevOps, reliability, etc.).
- Become a go-to person for specific technical challenges - write blog posts, give talks, mentor others in your area of expertise - build your personal brand.
- Always explore "other options" and alternatives to your preferred solutions to avoid tunnel vision (e.g., document pros and cons of different approaches), even if you are an expert in a specific area.
- Sharpen your problem-solving skills by practicing coding challenges, system design interviews, and architecture exercises regularly (e.g., code algorithms using different data structures in various languages; practice system design interviews with peers - see "architectural katas").
- If you lack time for coding practice, consider a dedicated weekly slot (e.g., 1–2 hours every Friday afternoon) to focus on coding practice and learning new technologies (timebox it; use the Pomodoro Technique to stay focused).

In the AI era, leverage AI tools to assist your learning (e.g., use Notebook LM from Google to summarize articles, generate code snippets, or explain complex concepts).

And finally, transform learning into a habit - build your identity as a lifelong learner - use techniques from "Atomic Habits" by James Clear (habit stacking, environment design, tracking progress).

Manage your autonomy and time

Staff Engineers have a lot of freedom - that means managing yourself well.

Set clear boundaries and expectations with your manager and stakeholders regarding your availability and workload.
Choose projects based on impact and alignment with your goals.
- "Your ‘no’ is a gift to your future self."
- Evaluate how a project affects your energy, credibility, quality of life, social capital, and skills growth.
- Every technical skill set slowly becomes obsolete over time - invest in projects that help you grow and learn new skills (learning by doing, working closely with someone who is highly skilled), otherwise you may end up learning in your free time (which is hard to maintain in the long term).
Use time blocking for "deep work", meetings, and learning (e.g., block mornings for deep work, afternoons for meetings and collaboration, and Fridays for learning and coding practice - add everything to your calendar).
Measure and track your "deep work" ratio weekly - aim for 40–50% of your time.

Ensure effective project management and execution

Staff Engineers often lead complex, cross-team initiatives. You are responsible for predictable results.

Define clear goals, success criteria, and priorities.
Use visual tools for planning and tracking progress.
Break work into manageable milestones with explicit risks and dependencies.
Use asynchronous updates - dashboards, weekly summaries, short reports.
Iterate plans based on feedback. Adapt continuously as new information emerges.

Work effectively with your Engineering Manager

Your manager can be your best ally if you clearly define their expectations.

Define success metrics together and review them regularly (1:1s, quarterly reviews), e.g., counting commits in performance reviews is not effective when you focus on non-coding activities like architecture, mentoring, or cross-team collaboration.
Clarify your role in each project (tech lead, TPM, advisor, etc.) and define responsibilities to avoid misunderstandings (like who owns communication with stakeholders, who manages timelines, etc.).
Communicate progress, risks, and dependencies regularly.
Don’t wait for instructions - shape your own path and seek feedback proactively.

If your manager doesn’t support your growth or align with your values - find an environment that does. The right manager can make or break your Staff Engineer journey.

Common traps

Even experienced Staff Engineers fall into these traps. Here are the ones we’ve faced ourselves:

Unknown unknowns
- Ambiguous problems are part of the role. Don’t rush decisions. Gather information, seek diverse perspectives, and iterate.
- When in doubt, prototype or experiment to validate assumptions.
- Leverage your network to fill knowledge gaps: talk to architects, domain experts, TPMs, and other stakeholders to gather insights. They can provide valuable context and help you avoid blind spots.
Overengineering
- Avoid elegant but impractical solutions. "Good enough now" beats "perfect later."
- Always incrementally improve - use feedback loops to refine your designs, like with TDD or iterative architecture.
Losing touch with coding
- Stay hands-on! No excuses! Pair programming, small coding tasks, or code reviews keep your credibility, social capital, and empathy with your team.
- Take part in hackathons, coding dojos, or internal tech initiatives to stay engaged with coding.
Over-specialization in proprietary (organizational) tools
- Avoid being locked into internal or vendor-specific technologies. Think in terms of transferable concepts. Think like a generalist and map your knowledge to broader concepts and technologies.
- Check alternatives and open-source solutions to avoid vendor lock-in and ensure flexibility - this will also help you to grow your skills and stay relevant in the industry.

Summary

Measure everything. Iterate often. Keep learning.

The Staff Engineer role is a rewarding path for experienced engineers who want to influence at scale without moving into management.

By cultivating architectural thinking, organizational awareness, and a strong learning habit, you can drive meaningful change across teams. The best Staff Engineers measure, iterate, and lead by example - combining technical excellence with human empathy.

Recommended resources to explore the role deeper:

The Staff Engineer's Path: A Guide for Individual Contributors Navigating Growth and Change by Tanya Reilly
Pathfinder Engineer blog by Piotr Stapp, Tomasz Pęczek, and Jakub Gutkowski

Services Architecture and Code Ownership

Mon, 20 Oct 2025 00:00:00 GMT

Moving from monolith to services architecture changes not only how systems are built, but above all the way teams work together. One of the most important principles to get right soon is ownership. Without clear ownership, services can quickly turn into hard to change, harder to scale and painful to operate distributed monoliths.

Why Ownership Matters

When a new service is created, we strongly recommend defining its clear owner from day one. Ownership is not only about maintaining code. It is about accountability for decisions, quality and the long-term health of the service but also about empowering teams to move fast.

Clearly defined ownership helps to:

Reduce unnecessary coordination between teams.
Align accountability with decision making power.
Increase autonomy and allow teams to deliver features faster.
Maintain quality and consistency over time.

The Software Architect’s Perspective

From an architect’s viewpoint, ownership means teams are empowered to make decisions without waiting for "architectural sign-off". The architect does not become a bottleneck but instead acts as a guide and supporter:

Decision-making remains with the team, especially senior developers who best understand the service internals.
The architect enables alignment, ensuring that local decisions still fit into the big picture.
Support is always available: architects should make time to coach, review, and unblock teams when needed.

Balancing these aspects is not easy but it allows teams to move fast with confidence while architects ensure consistency, long-term sustainability and system coherence.

Core Principles of Code Ownership

1. Strong Ownership vs. Collective Ownership

Collective ownership assumes any developer can work on any part of the system. While appealing in theory, it often breaks down in organizations with limited services experience. It leads to inconsistent standards, quick hacks and lack of accountability.
Strong ownership means each service has a clear owner (preferably a team, not an individual). The owning team decides on coding standards, technology choices, deployment schedules, and operational practices. This autonomy reduces dependencies and improves productivity. As the number of services grows, strong ownership prevents coordination overhead from getting out of control.

As Sam Newman and James Lewis note: services buy you options. Strong ownership ensures those options are used intentionally, not chaotically.

2. Ownership and Roadmaps

A service-owning team should control its backlog and roadmap. This helps prevent short-term fixes that block future improvements. Ownership is not only about fixing bugs. It is about deeply understanding the service from both a technical and business perspective and keeping it healthy over time (e.g., finding enough time to reduce technical debt).

3. Developer Experience as a Foundation

Ownership also means responsibility for making services easy to develop, test and run. As services multiply, running a full system locally becomes harder. Investing early in developer platforms, starter repositories, or lightweight local environments (e.g., Docker Compose stack) is crucial. Teams that neglect developer experience will struggle with productivity and morale.

Team Structures and Boundaries

Drawing on Team Topologies, ownership works best when aligned with clear team boundaries:

Stream-aligned teams should own services end-to-end, from feature design to deployment and operations.
Platform teams provide the tools and infrastructure to support ownership, ensuring consistency in CI/CD, observability and local development.

In smaller organizations, stream-aligned teams may dedicate members to platform-like responsibilities. Encourage those naturally interested in tooling and give them explicit time to improve it.

Each stream-aligned team should have full lifecycle responsibility for its services: from code to production metrics. That lets to significantly reduce feedback loops between users and developers. Clear boundaries reduce friction. When teams own services, they can focus on delivering business value without waiting for permission or external coordination.

Common Pitfalls

No Ownership at All
Services created without clear ownership quickly become abandoned, leading to quality and operational issues.
Overlapping or Shared Ownership
If multiple teams "share" ownership, accountability is diluted. Problems often fall into gaps between teams.
Neglecting Developer Experience
As systems grow beyond a few services, developer experience becomes critical. Without investment, onboarding slows, testing is painful, and productivity drops.
Platform Too Late
If you expect to grow from a handful of services to dozens, invest in a common platform early. Retrofitting standards later is far more expensive.

Recommendations for Practitioners

Assign ownership from day one for every new service. If your codebase has a shared library, make sure it has a clear owner too.
Prefer strong ownership: let teams own standards, deployments, and technology choices where possible.
Align ownership with team boundaries following Team Topologies guidance.
Invest in developer experience early, even if it means dedicating partial resources.
Ensure each team controls its roadmap, balancing short-term delivery with long-term maintainability.
Document ownership explicitly in a service catalog, repository README, GitHub Ownership file, or platform portal to avoid confusion as teams evolve.

Closing Reflection

Services promise flexibility and speed but only if paired with the right organizational practices. Clear ownership is one of the most powerful ways to align accountability, autonomy, and quality.

Strong ownership helps teams to move fast without chaos. It reduces coordination costs, improves service quality, and makes scaling both the system and the organization sustainable.

Ownership is not a process detail but rather a cornerstone of successful service architecture.

AI Toolset for Software Architects (Q3 2025)

Wed, 24 Sep 2025 00:00:00 GMT

Artificial intelligence has become a daily companion for many software architects. In 2025, it is less about “choosing the best AI tool” and more about curating and evolving a living toolset. No single platform covers all needs - at least not yet. Instead, we must constantly experiment, combine, and adapt AI capabilities to support architecture design, technical leadership, and delivery in ever-changing environments.

This post outlines how we use AI tools in our practice, along with examples of how they support the software architect role.

The Architect’s Role and AI Support

Software architects are expected to:

Shape system design by balancing business requirements, constraints, and technical trade-offs.
Guide teams in applying architectural principles and patterns across distributed, cloud-native, and DevOps contexts.
Bridge communication between engineering, leadership, and business stakeholders.

AI tools do not replace these responsibilities. Instead, they provide acceleration, augmentation, and reflection:

Acceleration: speeding up research, prototyping, and documentation.
Augmentation: extending reasoning, generating options, and exploring trade-offs.
Reflection: providing different perspectives on decisions and assumptions.

Categories of AI Tools for Architects

1. Searching & Learning

Perplexity.ai – quick search across sources, exploring new technologies, and summarizing new research.
Useful for: learning about the latest AWS/Azure/GCP services, validating assumptions against documentation, forums, and the broader web.

2. Brainstorming & Ideation

ChatGPT (GPT-5) – rapid brainstorming, generating scenarios, and “talking through” design alternatives.
Useful for: early-stage thinking, exploring edge cases, and facilitating design discussions.

3. Prompt Engineering

GitHub Copilot – helps refine prompts and supports small-scale refactoring and documentation.
OpenAI Prompt Optimizer (GPT-5) – supports improving prompt clarity, efficiency, and outcome quality.
Useful for: creating repeatable, reliable prompts that make AI collaboration more effective across tasks.

4. Agentic Coding & Prototyping

Claude Code with subagents – supports coding, proof-of-concept implementations, and technical design, including drafting Architecture Decision Records (ADRs). Example subagents:
- Domain Driven Design Expert - focuses on discovering and identifying subdomains, categorizing them as core, supporting, or generic, and drafting bounded contexts.
- Code Reviewer (specialized for specific languages or frameworks) - reviews code for best practices, potential bugs, and adherence to coding standards, may use linters and static analysis tools.
- Trade-off Analyst - evaluates different architectural options based on criteria such as performance, scalability, cost, and maintainability, providing a balanced view of pros and cons.
Useful for: almost everything from quick prototyping to detailed design documentation. The key is to leverage subagents for specialized tasks and narrow prompts context to the specific challenge.
Integrated MCPs (Model Context Protocol servers):
- Serena for querying the codebase (works especially well with bigger codebases).
- Context7 for browsing documentation.
- Sequential Thinking for structured AI planning.

Together, these tools provide architecture-aware prototyping and knowledge-driven design workflows, connecting documentation, code, and work planning into a coherent flow.

List to Explore Next

The landscape evolves quickly. Here are tools we plan to explore next:

OpenAI Codex – potential alternative to Claude Code.
Warp – positioning itself as an “Agentic Development Environment.” Worth comparing with Claude Code and Codex.
Firecrawl MCP – expanding external data integration for agent workflows.
Parallel agent workflows – experimenting with multiple agents working on subtasks simultaneously.

This exploration list serves as a reminder: an architect’s AI toolset is never static. Staying curious and testing new capabilities is part of the job.

Practical Applications in Daily Work

Here is how these categories play out in real projects:

Architecture Exploration: Use Perplexity to scan for recent practices or solutions (e.g., managed database services in AWS) or quickly validate knowledge (e.g., the TESTCONTAINERS_RYUK_DISABLED flag), and ChatGPT to brainstorm architectural options before deeper design sessions.
Prompt Quality: Refine prompts with Copilot or the GPT-5 Prompt Optimizer to achieve more consistent outcomes when generating documents or test suites.
Rapid Prototyping: Pair Claude Code with MCPs to validate feasibility through quick proofs of concept, then decide if the pattern is viable for production.
Documentation at Scale: Generate ADR drafts and architecture overviews with Claude Code (especially using dedicated subagents who are experts in trade-off analysis).
Planning as first-class work: Break down complex tasks into manageable steps, then use agents to work through them iteratively, ensuring alignment with architectural goals.
Best engineering practices matter more than ever: AI tools can introduce errors or misinterpretations. When working with AI-generated code, practices such as linting, formatting, maintaining test coverage, conducting code reviews, and enforcing CI become even more critical.

Keep the Toolset Alive

In 2025, there is no perfect or final AI toolset for software architects. Instead, the practice is about:

Continuous evolution: adopt, test, and discard tools as needs change.
Balancing categories: ensure coverage across searching, brainstorming, prompting, coding, and documentation.
Architect mindset first: tools augment judgment; they do not replace it.

The most effective architects will not only master today’s AI assistants but also develop an experimentation habit - treating AI tools as evolving collaborators in the craft of system design.

References and resources:

If you don't know how to name your technique, use "test"

Tue, 09 Sep 2025 00:00:00 GMT

In software development, naming is often harder than writing the code itself. We've all been there... Struggling to find a name that is clear, accurate and expressive enough to capture what a technique or pattern actually does.

Today we're going to roast naming conventions that use the word "test" in ways that can be misleading. It's a common shortcut that can lead to confusion about the true purpose of a technique.

It happens more often than you'd think and often, the resulting names are a bit misleading. Let's look at two well-known examples.

Test-Driven Development

Test-Driven Development (TDD) sounds like a testing methodology, but its purpose goes much deeper.

While tests are certainly written, the practice is more about incremental design. Writing a failing test first helps clarify intent, drive better structure, and guide implementation. The test is not the goal - it's the mechanism for designing code in small, verifiable steps.

In practice, TDD helps developers:

Think about edge cases early
Write modular and testable code
Keep implementation minimal and focused

So despite the name, TDD isn't centered on testing - it's about designing with feedback.

Contract Testing

Another good example is Contract Testing, particularly in the context of Consumer-Driven Contracts (CDC).

The name suggests that we're testing a contract between two services - and yes, technically, we are. But the real power of CDC is in what it enables: communication between independently deployed teams.

When a consumer's expectations change, the contract test fails - which acts as a trigger. But that failure isn't the outcome - the conversation that follows is. It leads to negotiated, versioned, and safer evolution of APIs.

In short: CDC is less about "testing contracts" and more about managing change through collaboration.

Final thought

So why does "test" keep showing up in places where the core idea isn't actually testing?

Likely because it gives the technique a sense of rigor. It suggests verification, structure, and confidence - all things teams care about.

And that's not necessarily a bad thing. But it's worth being aware of the distinction between what the name implies and what the technique actually delivers.

Partitioning of Key-Value Data in Cache Systems (Part 3)

Tue, 29 Jul 2025 00:00:00 GMT

In the first two articles (Part 1, Part 2) of this series, we looked at why it's important to split (or partition) cache data when handling large amounts of information. We also talked about common problems like hotspot issues and explained how consistent hashing can help solve them. Then, we compared different ways to connect caching systems to applications - using built-in cache libraries, sidecar proxies, and centralized proxy setups - each with their own pros and cons. These articles gave a solid theoretical background that’s helpful for Expert Generalists and software architects to understand key ideas about caching, consistent hashing, and system design choices for managing cache.

In this article, we’ll show how to put the ideas from the previous posts into practice with a real-world example. We'll use Twemproxy (also called Nutcracker) as a caching proxy for Memcached, and apply Ketama consistent hashing to spread cache keys evenly across several Memcached instances. This hands-on example will demonstrate how to build a scalable caching setup in Kubernetes, giving a more practical and specialist view of the concepts we discussed earlier.

You can find the full source code on GitHub at github.com/handsonarchitects/memcached-example.

We’ll begin by explaining what Twemproxy is and how it works as part of a caching system. Then, we’ll walk through how to set it up with Memcached and Ketama consistent hashing. Finally, we’ll show you how to deploy everything in Kubernetes and test that it works as expected.

What is Twemproxy?

Twemproxy, also called Nutcracker, is a fast and lightweight open-source tool made by Twitter. It acts as a middle layer between your app and cache systems like Memcached or Redis. Twemproxy makes it easier to manage multiple cache servers by giving your app one simple way to connect to them all.

Let’s go over the main features of Twemproxy:

Single access point for caching: Twemproxy gives your application one entry point to talk to many cache servers. This makes your code cleaner and easier to manage. It supports both Memcached and Redis protocols, so you can switch between them with only small changes in your code. Just keep in mind that Memcached and Redis protocols are not the same, so your app still needs to know which one it’s using.
Key partitioning: Twemproxy automatically spreads cache keys across multiple servers to balance the load and avoid performance issues (hotspots). It supports several consistent hashing methods like ketama (the most common), fnv1a_64, murmur3_32, and modula (a basic modulo approach). These are built into Twemproxy and easy to use, so developers can focus on their app instead of hashing logic.
Handles changing setups: If you add or remove cache servers, Twemproxy updates itself automatically - no need to change your app code. It updates its internal routing (hash ring) and reassigns keys to the right servers. This makes it easy to scale the cache or replace failing nodes without disruption.
Built-in fault handling: Twemproxy monitors the health of each cache server. If one becomes unhealthy, it removes that server from the active pool so no new requests go to it. The keys are then automatically rerouted to healthy servers using consistent hashing, helping your system stay available.
Efficient with resources: Twemproxy reuses a small set of persistent connections to each backend server instead of creating a new one for every request. This reduces connection overhead and improves performance - especially useful when Twemproxy is used as a centralized dedicated caching layer in front of many servers.
Stateless by design: Twemproxy doesn’t store any data itself. It just passes requests between your app and the cache. This makes it easy to scale and manage. Because it’s stateless, you can run it either as a sidecar container next to your app in Kubernetes or as a separate service - both options work well.

These features make Twemproxy a strong tool for managing cache clusters, especially in large distributed systems. Since it’s stateless and handles changing topologies well, you can use it in many ways - like as a sidecar in Kubernetes or as a standalone proxy layer.

In this article, we’ll focus on the sidecar deployment in Kubernetes, which is a common approach for managing caching in modern cloud-native applications.

How to configure Twemproxy as a sidecar container in Kubernetes

In the previous article, we discussed the pros and cons of running a caching proxy as a sidecar - covering deployment complexity, scalability, and operational overhead. In this post, we will focus on two key benefits of the sidecar approach: easier network setup and automatic scaling.

When we deploy Twemproxy as a sidecar container (a container running alongside the application container inside the same Kubernetes Pod), we use the local network interface to connect to the caching proxy. This lets the application talk to Twemproxy using localhost, which makes network configuration simpler and reduces latency. There are no extra network hops or DNS lookups involved.

Since the sidecar runs in the same Pod as the application, it scales together with the application. When the application Pod scales up or down, the Twemproxy sidecar scales too, keeping the caching proxy layer aligned with the application instances. This setup means each application Pod has its own Twemproxy, so multiplexing connections to the cache is limited. However, this trade-off is often worth it for the simplified networking and automatic scaling benefits.

The diagram below shows the architecture of our example application, illustrating how Twemproxy runs as a sidecar container in a Kubernetes Pod next to the application container:

Each application Pod contains two containers:

The application container, which exposes a REST API to interact with the cache.
The Twemproxy sidecar container, which acts as a caching proxy and manages all communication between the application and the cache instances.

The application talks to Twemproxy on localhost:11211, the default Memcached port. You can scale the application Pods horizontally by running multiple replicas, allowing load balancing and high availability. Each replica has its own Twemproxy sidecar, ensuring the caching proxy layer stays consistent across all application instances.

The cache instances are deployed as a StatefulSet, which gives each instance a stable network identity and persistent storage. This allows Twemproxy to use fixed hostnames for the cache servers, keeping the caching layer stable even if instances scale up or down.

In this design, Twemproxy handles all cache partitioning and provides a single cache interface to the application. The application doesn’t need to know how the cache is split, which makes maintenance and scaling easier.

Next, let’s see how to configure Twemproxy to work with Memcached and Ketama consistent hashing in Kubernetes.

High-Availability Memcached Cluster

To build a high-availability Memcached cluster in Kubernetes, we use a StatefulSet to manage Memcached instances. StatefulSets provide stable network names and persistent storage, allowing the cluster to scale while keeping data consistent. We deploy the Memcached StatefulSet with a Helm chart, which simplifies deployment and configuration.

The Helm chart is available at bitnami/charts.

In our example, we deploy two Memcached replicas using a StatefulSet. This StatefulSet uses a headless service so each Memcached instance has a stable hostname like memcached-cluster-0.memcached-cluster and memcached-cluster-1.memcached-cluster. Twemproxy uses these hostnames to configure the cache topology.

Deploy the Memcached StatefulSet with this Helm command:

helm install memcached-cluster \
  --set architecture="high-availability" \
  --set replicaCount=2 \
  oci://registry-1.docker.io/bitnamicharts/memcached

Configuring the Twemproxy sidecar container

Twemproxy needs a configuration file (usually nutcracker.conf) to define the cache topology, including cache servers, hashing algorithm, and other settings. This file is typically in YAML format.

In our example, the configuration is generated dynamically in Dockerfile's entrypoint script using environment variables passed to the Twemproxy container. This lets us change the cache setup without changing or redeploying the application. See the Dockerfile for details on how this works.

After processing environment variables, the config looks like this:

memcached:
  listen: 127.0.0.1:11211
  hash: "fnv1a_64"
  distribution: "ketama"
  auto_eject_hosts: "true"
  timeout: "2000"
  server_retry_timeout: "2000"
  server_failure_limit: "1"
  servers:
    - memcached-cluster-0.memcached-cluster:11211:1
    - memcached-cluster-1.memcached-cluster:11211:1

The servers list defines the Memcached hostnames, ports, and weights for key distribution. hash sets the hashing algorithm, distribution controls how keys are spread across instances, and auto_eject_hosts lets Twemproxy remove unhealthy servers automatically.

You can update the cache cluster by changing the SERVERS environment variable in the Twemproxy container, without touching application code or redeploying:

env:
- name: SERVERS
    value: memcached-cluster-0.memcached-cluster:11211:1,memcached-cluster-1.memcached-cluster:11211:1

Configuring the Application Container

Our application uses pymemcache (a Python Memcached client) to talk to Twemproxy. It reads the MEMCACHED_HOST environment variable to know where to connect:

client = base.Client((os.getenv("MEMCACHED_HOST", "localhost"), 11211))
value = client.get(key)

The application container is configured with:

env:
- name: MEMCACHED_HOST
  value: localhost

This points the app to the Twemproxy sidecar on the local Pod network.

Application Pod Specification

Here’s an example Kubernetes Pod spec including both containers:

spec:
    containers:
    - image: handsonarchitects/memcached-api
    ...
    env:
    - name: MEMCACHED_HOST
        value: 127.0.0.1
    ...
    - image: handsonarchitects/memcached-sidecar
    ...
    env:
    - name: SERVERS
        value: memcached-cluster-0.memcached-cluster:11211:1,memcached-cluster-1.memcached-cluster:11211:1

This spec is part of the deployment manifest k8s/memcached-api-deployment.yaml, which controls scaling and replication.

Testing and Observability

To test Twemproxy distributing cache keys across Memcached instances, we created a Kubernetes job that runs a Bash script. The script generates cache keys and uses curl to call the app’s REST API in parallel. This helps test the cache layer under load. The job definition is in k8s/tools-cache-generator-job.yaml.

After the job finishes, check app Pod logs to confirm cache keys were created and stored. Then, check cache stats on each Memcached instance using the stats command via telnet.

Use this command to connect and check stats:

kubectl run get-memcached-stats --image nginx:alpine --restart Never --rm -it --command -- curl telnet://memcached-cluster-0.memcached-cluster:11211

Replace memcached-cluster-0 with memcached-cluster-1 to check the other instance.

Inside the telnet session, run stats to see cache stats, and quit to exit.

Source code and example

The full example code, including Twemproxy sidecar setup, Memcached StatefulSet deployment, and the simple Python API, is available on GitHub: github.com/handsonarchitects/memcached-example. Clone and follow the README to deploy it in your Kubernetes cluster.

Summary

This is the third article in our caching series focused on large-scale, data-intensive applications. Here, we showed how to build a scalable caching layer using Twemproxy, Memcached, and Ketama consistent hashing in Kubernetes. We covered the benefits of running Twemproxy as a sidecar, including easier network setup, automatic scaling, and dynamic cache management. We also demonstrated how to deploy a high-availability Memcached cluster with StatefulSets and how to configure Twemproxy to work with it. Finally, we provided a simple app example and tested the caching layer under load.

We hope this practical guide, together with our previous articles (Part 1, Part 2), helps you build better caching solutions for cloud-native applications.

Partitioning of Key-Value Data in Cache Systems (Part 2)

Tue, 08 Jul 2025 00:00:00 GMT

In the previous article, we introduced cache partitioning and highlighted its advantages for scaling large applications. We also explained how separating partitioning logic from the application itself can be achieved by introducing a dedicated caching proxy layer that handles partitioning transparently. This layer can be deployed either as a sidecar container or a centralized dedicated caching proxy, depending on the system's scale and complexity. In this article, we’ll evaluate both approaches, outlining their strengths and limitations within the context of modern application architectures.

To set the stage, we’ll first explore how caching architectures have evolved-from basic client-side libraries to advanced caching layers that offer greater abstraction, scalability, and operational flexibility.

Client libraries for starting point

Caching plays a vital role in modern application architectures by enabling faster data retrieval and enhancing overall performance. As applications grow in complexity, their caching requirements also evolve, giving rise to more advanced caching strategies. In the early stages of development, caching is typically handled using client libraries that connect directly to systems like Memcached or Redis. These libraries offer a straightforward way to store and retrieve frequently accessed data, reducing latency and boosting read performance.

In distributed setups, this approach allows stateless services to share cached data across instances, improving both scalability and efficiency. Developers can choose from a wide range of client libraries tailored to specific technologies and languages - for example, spymemcached for Memcached or jedis for Redis.

The diagram below illustrates a simple application deployed with a few replicas, each connecting to a set of cache instances configured in a master-slave (also known as leader-replica) architecture. In this setup, the master (leader) node handles all write operations, while the replica nodes are used for read operations. Each application instance is typically configured with logic or client settings to direct writes to the leader and reads to the replicas.

In a Kubernetes environment, this basic architecture can be represented as follows:

For straightforward applications, using a lightweight caching library with a basic failover mechanism - such as a master-slave setup - can be an effective and practical choice. The application logic communicates with the cache directly, enabling developers to perform basic operations like get, set, and delete with minimal overhead - similar to working with in-memory structures such as dictionaries or hash maps. With just a few lines of code, it's possible to introduce caching that meaningfully boosts performance. For example, the following snippet shows how to store a key-value pair in Memcached using the spymemcached client library:

List<InetSocketAddress> addresses = Arrays.asList(
    new InetSocketAddress("cache.instance1.url", 11211),
    ...
);
MemcachedClient client = new MemcachedClient(new HashSet<>(addresses));
client.set("key", 3600, "value");

The primary advantage of this approach lies in its simplicity - it allows developers to concentrate on application logic without needing to manage caching intricacies. However, as applications grow in size and complexity, several limitations begin to surface, posing risks to performance and scalability:

Tight coupling with cache implementation: Application logic becomes closely tied to specific caching clients, making it difficult to scale or switch technologies. Most client libraries offer limited features, lacking support for advanced capabilities like consistent hashing or dynamic scaling. The situation worsens in polyglot environments where different services may require different client libraries, resulting in inconsistent behavior and increased maintenance overhead.
No abstraction layer: Since the application interacts directly with the caching protocol and data structures, it's tightly bound to a specific cache backend. Migrating from one caching system (e.g., Memcached) to another (e.g., Redis) often requires significant refactoring.
Scalability limitations: As the number of cache nodes grows, maintaining connections becomes more complex and resource-intensive. Each application instance must manage its own connections, which can lead to contention, increased memory usage, and higher latency - especially at scale. This relation is often referred to as the "N+1 connection problem," where N is the number of application instances and 1 is the number of cache nodes. As the number of application instances increases, the number of connections to the cache nodes grows linearly, leading to potential performance bottlenecks and increased resource consumption.
Rigid topology assumptions: Applications are often designed with a fixed cache topology in mind, expecting a certain number of nodes at specific locations. Any change in the cache cluster - like adding or removing nodes - requires updates to the application, reducing flexibility.
Fault tolerance challenges: Handling cache node failures and retries falls on the application, increasing logic complexity and the risk of inconsistent behavior. This can lead to increased latency and reduced performance, as the application must handle retries and fallbacks in case of cache node failures.

Decoupling caching from application

As applications become more complex and scale across environments, the demand for advanced caching architectures becomes increasingly apparent. This evolution is driven by the need to accommodate dynamic cache topologies, offer a consistent interface for all applications, and shield developers from the intricacies of underlying caching technologies. From an architect’s viewpoint, the goal is to build a resilient and adaptable caching layer capable of responding to shifting workloads and system requirements. From a developer’s standpoint, it’s about reducing complexity by abstracting cache management, allowing the application to focus on its core business logic.

To address these challenges, a key step is to decouple caching responsibilities from the application logic by introducing an intermediate caching layer that abstracts the complexity of the underlying cache system. This layer - often implemented as a proxy - can run as a sidecar container alongside the application container, offering a transparent caching interface. With this setup, the application interacts with the cache as though it were a single, unified store, regardless of how data is partitioned or distributed across cache nodes. This pattern is commonly known as the ambassador pattern, where the sidecar acts as an intermediary that manages all communication with the cache cluster. It's a simple yet powerful way to retain compatibility with existing client libraries while gaining greater flexibility and scalability.

The diagram above demonstrates how sidecar containers integrate into the overall architecture. These containers are deployed within the same POD as the main application container, allowing them to share the network namespace and communicate over localhost. Typically, adopting this model requires minimal changes to the application itself, since existing caching client libraries can often be reused. The sidecar container takes responsibility for managing cache operations - such as partitioning, scaling, and fault tolerance - freeing the application to concentrate solely on its core functionality.

That said, while the sidecar pattern offers a cleaner separation of concerns compared to directly embedding cache logic in application code, it does have some drawbacks:

Connection management: Because each sidecar container maintains its own connections to backend cache services, scaling out the application also increases the number of cache connections. This can overwhelm cache servers and degrade performance. Unlike centralized approaches, the sidecar pattern doesn’t easily allow multiplexing of multiple client requests through a shared connection pool.
Configuration complexity: Introducing a sidecar involves additional configuration in the deployment setup. For instance, when using Memcached with Ketama consistent hashing, you must configure the sidecar with the appropriate hashing algorithm and backend cache node details. As the system scales, maintaining accurate configurations becomes more challenging and error-prone.
Operational overhead: Since sidecars are co-located with application containers, they must be monitored and maintained as part of the overall system. This includes tracking their health, managing restarts, and ensuring they run correctly - adding to the system’s operational complexity.

Even with these challenges, the sidecar pattern represents a valuable step toward decoupling cache responsibilities from the main application. It’s a practical and scalable solution for most medium to large-scale systems. However, for larger systems with many application replicas, this architecture may become inefficient due to increased connection overhead and higher maintenance complexity.

Dedicated caching proxy layer

For high-traffic, large-scale applications with complex caching needs, the sidecar pattern may fall short. In these scenarios, a dedicated caching proxy layer - managed independently of the application - is often a better fit. This layer acts as a centralized service that exposes a unified interface for accessing multiple cache instances, enabling more efficient use of resources, improved performance, and simplified management of cache clusters.

Such a service can optimize connection handling by multiplexing multiple client requests through a limited set of backend connections, reducing load on cache servers. It also supports advanced capabilities like consistent hashing and can adapt to changes in cache topology without requiring any modifications to the application itself.

Along with the added flexibility and scalability, a dedicated caching layer also introduces certain trade-offs. A primary concern is the increased architectural complexity - adding a separate caching service means introducing another component that requires monitoring, maintenance, and management, which can raise operational overhead and introduce new failure points. Despite this added complexity, the trade-off is often worthwhile due to the advantages it brings: centralized cache management, enhanced performance, and more efficient use of resources. One key benefit is that the caching layer can be scaled independently of the application, enabling more precise resource tuning and performance optimization.

Another important factor to consider is the potential for increased latency, since the application must make an additional network call to reach the external caching proxy layer. While this latency can be reduced by deploying the caching service close to the application - such as within the same Kubernetes cluster - it still doesn’t offer the ultra-low latency of localhost communication available in the sidecar model.

Ultimately, deciding between the sidecar pattern and a dedicated caching proxy layer depends on the specific characteristics of the application, including its traffic load, architectural complexity, and performance expectations. For smaller or less complex systems, the sidecar approach is typically sufficient. However, as the application scales and caching demands become more sophisticated, a dedicated caching proxy layer tends to offer a more resilient and scalable solution.

A practical strategy is to start with the sidecar pattern and evolve toward a dedicated caching proxy layer when the system outgrows the initial setup. Tools like Twemproxy can help make this transition smooth, allowing you to adopt the new architecture without significant changes to the application’s code or configuration.

Summary

In this article, we explored the evolution of caching implementation strategies - starting from a basic in-app caching library, moving to the sidecar pattern, and ultimately reaching a dedicated caching proxy layer. Along the way, we discussed the core benefits and trade-offs associated with each approach, helping to clarify when and why each strategy might be appropriate depending on the application's scale and complexity.

In the final article of this series, we’ll shift from theory to practice by demonstrating a sidecar pattern implementation using Twemproxy. We’ll provide a working Memcached example that can be easily deployed in Kubernetes, along with configuration details to enable Ketama consistent hashing. This will allow you to verify how cache data is distributed across multiple nodes - giving you a clear, hands-on understanding of how the sidecar pattern works in a real-world setup.

Partitioning of Key-Value Data in Cache Systems (Part 1)

Wed, 18 Jun 2025 00:00:00 GMT

This is the first post in a blog series exploring cache partitioning strategies for distributed systems. It covers how to handle large data volumes and high traffic efficiently by dividing cache responsibilities across nodes. The post highlights why partitioning matters, compares it with replication, and explains core techniques like consistent hashing for load balancing. In upcoming articles (Part 2, Part 3), we’ll look at how to decouple the caching proxy from application logic and conclude with a hands-on example of using the sidecar pattern with Twemproxy.

When to use cache partitioning

Replicating cache data is a widely used approach to achieve high availability and fault tolerance. It involves storing identical data across multiple cache nodes (replicas), which improves read performance and ensures redundancy in case a node fails. However, for applications with large datasets or high traffic, replication alone may not be enough. That’s where cache partitioning becomes essential.

Cache partitioning involves breaking the cached data into smaller, more manageable segments (partitions) and distributing them across several cache nodes (shards). Each shard holds only a portion of the entire dataset, enabling the system to scale both read and write operations across multiple instances. Since each shard uses its own memory, disk, and CPU resources, this leads to more efficient resource usage and improved performance. The primary goal of cache partitioning is to enhance scalability.

This approach is often used in distributed systems, where multiple application instances share the cache while remaining stateless and decoupled. However, it can also be valuable in monolithic systems that require high-performance or large-capacity caching within a single application instance.

How to distribute cache keys across multiple instances

In an ideal scenario, each cache instance would manage an equal portion of the total cache data, ensuring efficient use of resources and consistent performance. Theoretically, as you add more cache instances, overall performance should scale linearly-doubling the instances should double the throughput. However, in real-world systems, achieving this perfect balance is difficult. Disparities in data distribution often lead to uneven load across cache nodes, a problem known as cache hotspots.

When cache keys aren't evenly distributed, some instances can become overloaded while others remain underutilized. This imbalance causes performance issues such as higher latency, lower throughput, and reduced system efficiency. A common partitioning method, mod-N hashing, which uses the modulo operation to assign keys - can create skewed distributions when the key space isn't uniform, concentrating traffic on certain instances.

To address this, consistent hashing offers a more effective solution. It arranges cache instances on a virtual hash ring and assigns keys in a way that balances the load more evenly, regardless of the key space. This method significantly reduces the disruption caused by adding or removing cache instances, since only a small number of keys need to be reassigned. You can read more about consistent hashing in the Consistent hashing algorithm.

Consistent hashing is a key technique used in distributed caching systems like Memcached and Redis, often implemented via tools like Ketama, to support efficient and scalable cache partitioning.

How to decouple cache partitioning from application logic

The application’s logic should remain decoupled from the cache partitioning mechanism. In other words, the application shouldn’t need to know how the cache is segmented or how keys are distributed among instances. Instead, the caching proxy layer should abstract away these details, presenting a unified interface that makes the cache appear as a single, cohesive system.

This abstraction can be achieved using a client library that supports consistent hashing - such as Ketama for Memcached - or by introducing a proxy layer between the application and the cache. The proxy-based approach provides greater flexibility and scalability, as it can dynamically adapt to changes in the cache topology (e.g., scaling nodes up or down) without requiring any modifications to the application code. It also enables switching partitioning strategies seamlessly, making it easier to evolve the system in response to shifting performance demands. We'll explore this in more detail in the next article.

Conclusion

In the next two articles of this series, we’ll dive deeper into designing a scalable caching layer in distributed systems.

The second article (Part 2) will trace the evolution of caching implementations-from a basic in-app caching library, to an externalized sidecar (ambassador) container, and finally to a dedicated proxy layer. This progression highlights how decoupling caching concerns improves scalability, flexibility, and maintainability.

The final article (Part 3) will present a hands-on example using Twemproxy (Nutcracker) as a sidecar caching proxy. We’ll show how to deploy it within a Kubernetes environment alongside your application and configure it with Memcached and Ketama hashing to enable consistent, efficient cache partitioning.

Making better architectural decisions thanks to slow thinking

Mon, 19 May 2025 00:00:00 GMT

Have you ever made a technical decision and later questioned if it was truly the best choice? As software architects, we've all experienced situations where quick, intuitive decisions seemed correct initially but later created unexpected problems. This is exactly why Architecture Decision Records (ADRs) are invaluable. ADRs encourage a deliberate and analytical approach to decision-making, known as System 2 thinking, a concept described by Daniel Kahneman in his influential book "Thinking, Fast and Slow".

ADRs

Architecture Decision Records are short, structured documents that clearly record key architectural decisions and the reasons behind them. Typically, ADRs include:

Context: Information about why a decision is necessary, including background and relevant factors.
Considered Options: Alternatives evaluated while making the decision.
Decision: The chosen solution, along with the reasons behind it and expected outcomes.

Writing decisions this way forces you to explicitly compare your options, clearly showing benefits and trade-offs.

Why context matters

Context is essential because architects often need to compare technologies that initially seem very different, like comparing apples to oranges. While these may seem impossible to compare directly, establishing clear criteria (like comparing fruits based on "skin hardness") allows for meaningful evaluation. Similarly, when choosing technologies, clearly defined context reveals exactly what attributes matter most.

For example, comparing a serverless solution to Kubernetes might initially seem impossible. However, once you define the context clearly, such as "ease of scaling under variable load," you have a specific criterion to help compare the solutions effectively.

Activating slow (System 2) thinking

According to Kahneman, our minds operate in two modes:

System 1: Intuitive, fast, and automatic. It relies on immediate impressions and quick judgment.
System 2: Analytical, slower, and deliberate. It requires careful evaluation and logical reasoning.

Architectural decisions often have long-lasting consequences, making them expensive or difficult to reverse. Relying exclusively on intuition (System 1) can be risky. Taking time to create ADRs explicitly activates System 2, encouraging architects to carefully articulate context, consider multiple alternatives, and thoughtfully evaluate their implications.

Improving context and decisions

In practice, your first attempt at defining the context and options is rarely perfect. As you explore alternatives, you’ll often discover additional factors you hadn’t initially considered. Regularly updating and refining the ADR context ensures your decisions become stronger and more complete.

Imagine deciding on a database solution. Initially, your primary context might be focused on read performance. But as you dig deeper, you realize ease of operational maintenance is equally critical. Updating the ADR context to include this criterion encourages a more balanced, thoughtful decision.

Another useful practice is clearly documenting Architecture Drivers (key architectural requirements or principles important to your project) and reviewing them with every major decision.

Joint evaluations

Kahneman highlights that joint evaluations (assessing multiple alternatives simultaneously) lead to better, more consistent decisions than evaluating options individually. Single evaluations, based mostly on intuition, tend to miss key differences because they lack a direct comparison.

To illustrate, Kahneman describes an experiment comparing two used dictionaries:

Dictionary A: Published in 1993, 10,000 entries, condition "like new".
Dictionary B: Published in 1993, 20,000 entries, slightly damaged cover but otherwise "like new".

When evaluated separately, people preferred Dictionary A due to its better condition. But when evaluated together, Dictionary B became clearly preferable, as it offered twice the number of entries - an advantage that was more obvious when compared side-by-side.

This insight directly applies to software architecture. Instead of evaluating technologies individually, use ADRs to evaluate them side-by-side. Joint evaluations help reveal key differentiators, leading to more informed and consistent decisions.

Lessons from other domains

Kahneman points out that fields like justice or public policy sometimes make mistakes by relying heavily on single, isolated evaluations, leading to inconsistent or unfair outcomes. Similarly, in software architecture, narrow or intuitive decision making can lead to costly mistakes or regrets.

Incorporating ADRs into your decision making process helps avoid these pitfalls by encouraging joint evaluations, careful comparisons, and deeper analytical thinking.

Final thoughts

As software architects and technology leaders, the decisions we make significantly impact the projects we work on. Using ADRs intentionally engages our analytical System 2 thinking, ensures clearer contexts, facilitates iterative improvements, and encourages consistent, joint evaluations of technology choices.

By adopting ADRs, you become a more thoughtful decision maker and clearer communicator, ultimately improving the quality and durability of the architectures you create.

If you frequently make important technical decisions at work, we highly recommend reading Daniel Kahneman's "Thinking, Fast and Slow". It provides invaluable insights to enhance decision-making in professional and personal contexts alike.

Who needs software surgery?

Tue, 29 Apr 2025 00:00:00 GMT

We recently watched an insightful panel discussion on YouTube featuring Daniel Terhorst-North, who compared software development to surgery. He shared a powerful thought:

"No one wants surgery. What they want is what surgery gives them, which is to be well."

He points out clearly that people don't actually want surgery itself - they want the positive outcome surgery brings: better health or relief from illness. Similarly, our customers don't want software just for the sake of having software. Instead, they want the benefits that software provides, such as solving problems, automating tedious tasks, or improving their overall experiences.

Daniel explained that, when surgery becomes necessary, people typically want:

Minimal intervention: The smallest, simplest surgery that still achieves the desired outcome.
Experienced surgeon: Someone who has performed this surgery successfully many times before - no one wants to be a learning case.
Proven procedures: Techniques that are well-known, tested, repeatable, and backed by evidence - not guesswork or experimentation.
Clear documentation: If another surgeon needs to step in later, they should easily understand exactly what was done, why it was needed, and how it was performed.

These points translate naturally into software architecture and software development:

Keep solutions simple: Deliver the smallest, simplest solution that meets your goals. Avoid unnecessary complexity.
Value experience: Teams and customers benefit from working with developers and architects who have solid, hands-on knowledge of their systems.
Use proven methods: Choose architectures, design patterns, frameworks, and cloud solutions with a strong track record. This reduces risks and improves reliability.
Document clearly: Good documentation makes maintaining software easier, helps onboard new team members, and ensures knowledge is transferred smoothly.

Who is the architect in this analogy?

If software developers are surgeons, then who is the software architect?

Following this analogy, software architects naturally play the role of experienced medical specialists or consultants. These specialists carefully plan procedures, recommend proven methods, and make sure everyone involved clearly understands what's being done, why it's necessary, and how to do it effectively.

Thinking about software architecture like surgery helps us prioritize simplicity, clarity, experience, and outcomes. It reminds us that our ultimate goal isn't building software itself - it's delivering value and benefits to users and customers.

If you're interested, we highly recommend watching the full panel discussion on YouTube.

Effective planning for Site Reliability Engineering (SRE) projects

Mon, 14 Apr 2025 00:00:00 GMT

Effective planning for Site Reliability Engineering (SRE)

Site Reliability Engineering (SRE) is a discipline that blends software engineering with operations, aiming to build and maintain scalable and reliable systems. SRE teams often operate under constant pressure from production incidents, evolving technology, and organizational priorities. That’s why having a solid, flexible, and thoughtful planning approach is essential to ensure the success of both long-term projects and day-to-day operations.

In this blog post, we share our approach to planning SRE projects, tasks, and initiatives. We focus on practical principles and proven techniques that help teams balance project execution with ongoing operational responsibilities, enabling them to deliver reliable, high-quality outcomes even in fast-paced and dynamic environments.

Context: the nature of SRE work

The scope of SRE work spans several areas, which can be broadly categorized into the following:

Project tasks, such as migrating to a new technology or platform, implementing new features like rate-limiting, improving system performance, reducing infrastructure costs, or increasing service reliability.
Operational tasks, including handling incidents, writing postmortems, maintaining runbooks, participating in on-call rotations, and setting up and managing monitoring.
Cyclical tasks, such as reviewing Service Level Objectives (SLOs) and Service Level Indicators (SLIs), preparing availability and performance reports, and conducting regular audits or reviews.

Multiple SRE projects and tasks often run in parallel, with different priorities, varying levels of urgency, and shared resources. This complexity makes planning and prioritization particularly challenging. Additionally, SRE work is frequently interrupted by unexpected incidents or outages, which can derail planned work and disrupt timelines. To make matters more complex, many tasks and projects are interdependent and require cross-team coordination and collaboration, making communication and visibility critical components of success.

Why effective planning is crucial

A thoughtful planning process is the foundation for effective SRE work. It allows teams to prioritize and manage their workload in a way that is both realistic and sustainable. Good planning ensures that technical goals are aligned with business priorities and stakeholder expectations, while also accounting for the team's operational responsibilities and potential interruptions.

Without proper planning, SRE teams may fall into cycles of constant firefighting, burnout, or lack of progress on strategic initiatives. Planning helps bring structure and clarity to the chaos, enabling the team to identify risks, allocate resources wisely, and communicate status and direction effectively.

Planning in SRE should not be rigid or bureaucratic. Instead, it should be lightweight, collaborative, and iterative, focusing on key aspects such as solution design, implementation, release, documentation, and monitoring - while remaining adaptable to change.

Best practices for planning SRE projects and tasks

Below are the key principles and practices we follow when planning SRE work. These are not rigid steps, but rather guiding patterns that we adjust based on the nature and complexity of the task or project at hand.

Start by defining clear objectives

Before any planning activity begins, it's critical to define the objectives of the project or task. This includes identifying the desired outcomes, success criteria, and key performance indicators (KPIs) that will be used to measure progress and results.

For example, an SRE project might aim to improve service reliability by reducing the number of incidents and outages by 50% over the next quarter, using metrics like mean time to recovery (MTTR) and mean time between failures (MTBF). Another objective might be to reduce the time required to roll out high-risk changes - such as operating system patches, Docker image updates, or a Java version migration - by streamlining the deployment process and redefining the rollout strategy.

Clear objectives create alignment among team members and stakeholders, and serve as a north star when prioritizing work and making trade-offs.

Use lightweight tooling to visualize the plan

Planning doesn't require heavyweight tools or complex processes. In many cases, the best results come from simple, visual approaches that help teams collaborate and stay aligned. This might include physical whiteboards and sticky notes, or digital tools such as Miro, Mural, or shared spreadsheets.

The main goal is to visualize the project as a series of interconnected tasks over time. This makes it easier to identify risks, dependencies, decision points, and resource requirements. A visual plan encourages discussion, feedback, and iterative improvements throughout the lifecycle of the project.

Embrace the shift-left approach

One of the most effective planning techniques in SRE work is to adopt a shift-left mindset. This means addressing risks, dependencies, decisions, and technical uncertainties as early as possible in the planning process, rather than waiting until they become blockers later in the project.

For example, moving rate-limiting logic from an application cotainer to a sidecar proxy service may require architectural decisions, design reviews, and cost justifications. Addressing these questions at the beginning of the project ensures that the right people are involved, and that there's enough time allocated for approvals, discussions, and iteration.

Where appropriate, teams can use Requests for Comments (RFCs), Architecture Decision Records (ADRs), or similar documentation formats to capture context, outline options, and document the reasoning behind key decisions.

Break down the work into milestones

Rather than trying to plan an entire project in one go, it's more effective to break it down into smaller, manageable chunks - each with a clear milestone and deliverables. These milestones help track progress, provide natural feedback loops, and create opportunities to validate assumptions along the way.

Each milestone can focus on a specific area such as design and architecture, implementation and testing, rollout and release, or monitoring and troubleshooting. However, it’s important to address all these areas from the start and to iterate over them continuously, rather than treating them as linear or isolated phases.

For example, a project milestone might involve completing the rollout of an internal CLI tool used by SREs, followed by a demo and collection of feedback. The next milestone could involve improving the monitoring setup or optimizing performance based on feedback and real-world usage.

Identify and focus on the critical path

The critical path refers to the sequence of tasks that determines the minimum amount of time required to complete the project. Identifying this path allows the team to focus on high-impact tasks, allocate resources strategically, and make decisions that directly influence project success.

For example, in a platform migration project, the critical path might involve migrating only the most heavily used SRE commands first, while deferring less critical ones to later milestones. This approach allows teams to deliver value early, reduce risk, and avoid bottlenecks.

Estimate work and verify team capacity

To create a realistic plan, it's important to estimate the size and complexity of individual tasks. This can be done using simple methods like t-shirt sizing (S, M, L, XL), man-day estimates (1, 2, 3, 5, 8, 13 days), or story points - depending on what the team is comfortable with.

Once estimates are in place, the next step is to verify the team’s capacity. This includes accounting for holidays, planned leave, ongoing operational responsibilities (like on-call rotations or incident response), and other project commitments. Each team member should declare their expected availability, and if the total capacity falls short of what's needed, then the plan should be adjusted accordingly - by reducing scope, extending the timeline, or increasing team size.

Communicate the plan and iterate often

Once the plan is drafted, it's important to communicate it clearly and transparently with all stakeholders. This includes sharing the objectives, timelines, milestones, risks, dependencies, and assumptions. Visual plans should include legends, clear labels, and dependency arrows to make them easy to interpret. The more visual and accessible the plan is, the easier it will be for everyone to understand and engage with it - e.g you can use four-quadrant influence/interest matrix to identify key stakeholders and their level of involvement (stakeholder mapping).

Most importantly, remember that the plan is a living document. It should be reviewed and updated regularly as the project evolves, new information becomes available, or team capacity changes. A good plan encourages collaboration, facilitates alignment, and supports continuous improvement.

A step-by-step example: visual planning in action

To make the planning process more tangible, let's walk through a practical example using a visual approach. We recommend using a digital whiteboard tool like Miro or Mural, which allows teams to collaborate asynchronously and create a shared visual plan. Below, we outline a six-step planning method with concrete tips, inspired by real-world SRE projects.

Step 1: visualize the project timeline

Start by creating a visual board to lay out the project's timeline and task structure.

Use a vertical axis to represent time - this could be weeks, months, or quarters. Mark important dates, such as deadlines, releases, holidays, or other constraints that affect team availability.
Use a horizontal axis to list tasks. Represent each task with a sticky note that includes:
- A short, descriptive title (e.g., “Add SLO dashboards”)
- A color code to distinguish task types:
  - 🔴 Red for risks
  - 🟡 Yellow for planned tasks
  - 🟢 Green for resources or capacity
  - 🔵 Blue for decisions
- Keep sticky notes small to reduce visual clutter and force concise descriptions.
Draw arrows to indicate task dependencies:
- Solid arrows for confirmed dependencies
- Dotted arrows for uncertain ones
- Ensure arrows are unidirectional and avoid loops
Add a legend to the board to explain color codes, arrow styles, note sizes, and any other symbols used.

This setup helps the team grasp the big picture and start discussions around sequencing and priorities.

Step 2: apply the shift-left mindset

Bring uncertainty, risk, and decision-making to the front of the planning process.

Identify what’s known vs unknown, and mark the unknowns clearly on the board.
Highlight risks with red sticky notes, and specify which teams or systems they affect.
Shift all risks and major decisions to the earliest possible point on the timeline, so they don’t block downstream tasks.
Estimate the time and effort needed for decisions, and add them as tasks - don’t assume decisions “just happen.”
For complex decisions, plan to write RFCs or ADRs early. Use them to explore options and involve stakeholders.
If documentation will be required (design docs, runbooks, training material, etc.), add a task and allocate time for it early in the plan.

This approach surfaces problems early, reduces surprises, and enables smoother delivery.

Step 3: break work into milestones and plan rollouts

Group tasks into meaningful milestones with clear deliverables.

Define each milestone by its outcomes - this could be a feature rollout, an internal demo, or a proof of concept.
If a milestone ends with a production rollout, plan extra buffer for coordination and risk mitigation.
Choose between time-boxed (e.g., 2 weeks) or feature-boxed (e.g., complete rate-limiting feature) milestones depending on complexity.
For projects involving multiple teams, use milestones to align on contracts or deliver mocked APIs to unblock dependent work.
Use feature flags or shadow mode deployments to gather feedback early without exposing risk to users.

This ensures that each milestone delivers value and provides a feedback loop before moving forward.

Step 4: identify the critical path

Determine the essential sequence of tasks that governs project duration.

Mark the critical path on the visual board using a different color or style (e.g., mark relevant milestones in red background).
Any risks, decisions, or dependencies on this path should receive high attention and priority.
Try to identify the critical path early so the team can optimize for speed and reduce delays.
That said, if the critical path is too risky or expensive, teams might intentionally defer parts of it - for instance, migrating only a subset of critical CLI commands first.

This step helps the team focus energy where it matters most, and creates clarity for prioritization.

Step 5: timebox tasks and check capacity

Ensure that planned work is realistic based on available team bandwidth.

Use tags or labels to estimate task size or complexity (e.g., S/M/L or 2, 3, 5 days).
Ask each team member to declare availability per week, excluding holidays, on-call time, and other commitments.
Update availability regularly to reflect changes (e.g., new projects, vacations).
If the plan doesn’t fit the available capacity, adjust by scoping down, shifting timelines, or adding contributors.

This practice protects the team from burnout and ensures that commitments are feasible.

Step 6: materialize and execute the first milestone

Once planning is complete, it’s time to operationalize the work.

Translate the first milestone into actionable tasks on your team’s sprint board (e.g., Jira, Trello).
Link sticky notes on the whiteboard to tickets in the backlog to maintain traceability.
Monitor milestone progress closely, and update the plan as the situation evolves.
Share status updates with stakeholders frequently, and be transparent about changes in scope, risk, or timelines.

By grounding your planning in both a visual workspace and a sprint tool, you maintain alignment between vision and execution.

Conclusion

Planning in the context of Site Reliability Engineering is not about creating perfect schedules - it’s about building a shared understanding of goals, constraints, and strategies. It enables teams to stay focused and effective, even in the presence of uncertainty and operational disruptions.

By using lightweight planning tools, applying shift-left practices, breaking work into milestones, identifying the critical path, and verifying capacity, SRE teams can stay resilient and adaptive. Most importantly, good planning makes work visible, promotes healthy team dynamics, and increases the chances of delivering reliable, scalable, and valuable outcomes.

If your team is struggling to balance operations and engineering work, or if your projects frequently get derailed by unexpected events, it might be time to revisit your planning practices - and hopefully, the ideas shared here will serve as a helpful starting point.

Using generative AI as an architect buddy for creating architecture decision records

Mon, 31 Mar 2025 00:00:00 GMT

As software architects, we know how valuable Architecture Decision Records (ADRs) are for documenting critical decisions clearly. However, we also understand that creating detailed ADRs can feel tedious and time-consuming. That's why we've started using generative AI as a helpful assistant to streamline the ADR creation process.

In this article, we’ll share our experiences with leveraging AI to efficiently generate high-quality Architecture Decision Records. We'll discuss practical techniques, provide examples, and outline the benefits and challenges we've encountered.

How we practically use AI in the ADR process

An effective ADR typically consists of three key sections:

Context: Explains the background and the critical factors that led to the decision. Even if not every detail is captured, a well-described context helps future readers understand the reasoning behind our choices.
Considered options: Lists possible alternatives we evaluated. Sometimes, if there was clearly only one viable option, we combine this section with the context.
Decision: Clearly defines our selected solution, along with its rationale. We typically describe the expected consequences - what will become easier, what might get harder, and necessary adjustments for the team and architecture.

To effectively integrate generative AI into ADR creation, we've adopted a structured, iterative approach that involves carefully constructed prompts.

Since generative AI currently (checked on GPT 4o, 4.5, o1, o3; Claude 3.7 Sonet; Gemini 2) struggles with independently capturing accurate context, we always start by clearly providing that context ourselves. From there, we use iterative prompting with AI:

First, we provide a detailed context and ask AI to generate a few considered options, along with their pros and cons.
After reviewing these options, we prompt the AI again, explicitly asking it to recommend a decision based on deeper analysis and project-specific factors.
Once we select an option, we explicitly ask AI to outline its consequences - what will become easier, more difficult, and what adjustments may be necessary for the team and our architecture.
Finally, we prompt AI to re-examine the decision thoroughly to make sure no better solution exists, ensuring we haven't overlooked important aspects.

This iterative approach helps us progressively refine the AI-generated content, ensuring clarity, accuracy, and practical applicability.

Example prompts

Here are some practical examples of prompts we’ve successfully used when leveraging AI for ADR creation. First prompt (<base-prompt>) to define context clearly:

As an experienced software architect who specializes in modern software development, continuous delivery, and architecture trade-off analysis you were asked to prepare an architecture decision record.
You have an analytical approach to the topics you consider. You try to objectively see the advantages and disadvantages of each option so that you can choose the one that best suits the situation.

Given the following context (inside triple quotes):
"""
<Your detailed context here>
"""

When additional architectural drivers or user stories are relevant, we add those details as follows:

<base-prompt>

Also, you consider the project in which you are working, the team, and the architecture drivers (inside triple quotes):
"""
<your archiecture drivers document>
"""

We recommend using a new session for each prompt to avoid AI using hallucinations from the previous session.

Next, we ask for considered options clearly detailed with pros and cons:

<base-prompt>
Provide a list of 2 to 5 viable options with their pros and cons.

Then we review the options and typically we have our own (human) brainstorming session to decide which option is the best. After reviewing these options, we typically ask the AI to further refine the recommendation:

<base-prompt>

The options you are considering are (inside triple quotes):
"""
1. Option 1 with pros and cons
2. Option 2 with pros and cons
3. Option 3 with pros and cons
"""
For each option consider pros and cons. Next, look at the list of options and consider the most fitting option to a given context and write the decision in the following form:
‌"""
In the context of <use case/user story u>, facing <concern c> we decided for <option o>, to achieve <system qualities/desired consequences>, accepting <downside d/undesired consequences>, because <additional rationale>.
"""

To explicitly address consequences, we ask:

<base-prompt>
We decided on <option> because (inside triple quotes):
"""
<Decision and rationale>
"""
List the consequences clearly (what becomes easier, more difficult, and necessary adjustments) in the form:
"""
Easier:
<bullet points>
More difficult:
<bullet points>
Necessary adjustments:
<bullet points>
"""

Finally, to ensure thoroughness, we perform a final check:

Look again at the context, considered options, the decision, and consequences. 
Confirm whether there's a better option (even new or combined) and state your final decision clearly, including the rationale.

The last prompt is a good way to ensure that you have considered all the options and their consequences before making the final decision. Be aware that 90% of the time AI will suggest some "hybrid" or "combination" of the previous options that needs to be ignored most of the time. But the important part of this check is the rationale behind the decision and the consequences of the decision. It sometimes shows the context aspects you forgot to mention in the context. And hence the iteration starts again.

Benefits

Generative AI has notably improved our productivity by rapidly creating initial ADR drafts, allowing us to focus more on strategic thinking rather than manual documentation, fixing grammar and spellings. It has also helped us maintain consistency and clarity in our documents, reducing misunderstandings across teams (avoiding ambiguity and mind shortcuts). Additionally, AI suggests improved wording and structure, enhancing readability and quality.

Challenges and limitations

While AI has been helpful, it’s not without challenges. The biggest limitation we've observed is its difficulty capturing context accurately without detailed guidance from us. Quality greatly depends on the clarity and completeness of the information we provide initially. We also learned to watch carefully for inaccuracies or "hallucinations," especially when AI generates pros and cons for each option. AI sometimes introduces incorrect or exaggerated details, so human validation remains essential.

Summary

Generative AI has significantly improved how we create Architecture Decision Records, making the process faster and clearer, and helping ensure consistency across documents. However, AI isn't (yet?) a replacement for architect expertise. We must still carefully provide context, validate AI-generated information, and iteratively refine outputs. When combined with human judgment, AI serves as a powerful assistant that enhances our ADRs and makes the entire decision-making process smoother and more effective.

Food truck freedom: explaining service architecture to business stakeholders

Sun, 16 Mar 2025 00:00:00 GMT

Recently, our team embarked on a major architectural transformation - moving from a monolith to a services-based architecture. As expected, the engineers on our team understood the rationale behind this shift, but the business stakeholders had concerns.

They asked: Why should we move away from a system we have already seen in action? What benefits do services bring beyond just technical improvements?

We needed an analogy that would make the differences between monolithic and service architectures clear, relatable, and memorable. So, when we were asked to present our reasoning to leadership, we came up with something that clicked immediately: comparing a large restaurant to a food truck festival.

Monolith vs. services: restaurant vs. food trucks

Think of two contrasting dining experiences. In one, you have a single large, busy restaurant with one centralized kitchen-this represents monolithic architecture. The restaurant is responsible for preparing everything: burgers, vegan dishes, desserts, and more. All the chefs share the same kitchen space, use the same equipment, and rely on a single team to serve all customers. At first glance, this setup seems efficient because everything operates under one roof. However, let’s dive deeper.

Now, imagine a large food festival with potentially thousands of customers-this represents service architecture. Each food truck in the festival is an independent service. Every truck specializes in a specific type of food: one serves burgers, another serves vegan dishes, and a third specializes in desserts. Each truck is self-contained, with its own staff, ingredients, and equipment.

Now, let’s break it down into specific areas of comparison.

Specialization and autonomy

Monolithic architecture (large restaurant): A single, large restaurant has one centralized kitchen where all food is prepared. If the kitchen becomes overwhelmed or disorganized, the entire operation slows down. The burger chef might have to wait for the dessert chef to finish using the stove, and the vegan chef might struggle to avoid cross-contamination. There’s little room for specialization because everyone depends on the same resources.

Service architecture (food trucks): Each food truck focuses on its specialty without relying on others (often they even don’t know about each other). This independence allows them to work faster and better, with no conflicts over shared staff, ingredients, or equipment.

Scalability

Monolithic architecture (large restaurant): When more customers arrive, the only way to handle increased demand is to expand the kitchen, hire more chefs, and cram more equipment into the same space. Scaling becomes expensive, disruptive, and inefficient as the kitchen becomes overcrowded (which may result with poor customer experience like cold dishes).

Service architecture (food trucks): If the festival grows, new trucks can join - just like new teams or services in software. This allows for more variety and the ability to serve more customers without disrupting existing operations.

Fault tolerance

Monolithic architecture (large restaurant): If one part of the kitchen fails - say, the stove breaks down or a chef calls in sick - the entire restaurant slows down or even stops. A single issue can ripple through the entire operation.

Service architecture (food trucks): If one truck faces issues (runs out of supplies or has a technical failure), the others continue serving customers. One truck going down doesn’t shut down the whole festival.

Coordination without overlap

Monolithic architecture (large restaurant): Because the kitchen operates as a single unit, adding a new dish (or a feature/technology in software) requires retraining all chefs, redesigning workflows, and potentially remodeling the kitchen. Miscommunication between staff can lead to delays or mistakes that impact the entire customer experience.

Service architecture (food trucks): While each truck operates independently, there’s still a festival coordinator (like a shared infrastructure team) ensuring there’s enough power, parking, and crowd management. However, each truck controls its own menu, and most changes do not require coordination with others.

Growth and flexibility

Monolithic architecture (large restaurant): Transitioning to a new cuisine or scaling up to meet growing restaurant popularity is difficult. Remodeling the kitchen, retraining staff, and introducing new workflows is time-consuming and costly. The rigid structure makes it hard to adapt to changing customer preferences or introduce new offerings.

Service architecture (food trucks): Trucks can be easily swapped, removed, or added based on customer demand. If a new food trend emerges, a new truck can join without affecting the existing ones. It’s much easier to bring in a new truck than to remodel an entire restaurant kitchen.

Resource usage efficiency

Monolithic architecture (large restaurant): The kitchen must handle all kinds of orders simultaneously, whether it’s burgers, vegan dishes, or desserts. If a large catering order comes in for desserts (akin to batch processing a large dataset in software), the kitchen must dedicate significant resources to it, slowing down service for regular diners. Customers of the restaurant have to wait until the catering order is completed.

Service architecture (food trucks): If there’s a long line at the burger truck, another identical truck can be brought in to handle the load. Other trucks (e.g., vegan or dessert) remain unaffected. If there’s a large catering order for desserts, a specialized dessert truck can park in a separate area, focus on fulfilling the batch, and leave when done - without disrupting the rest of the festival.

Summary

A well-known proverb says, "Too many cooks spoil the broth.” This perfectly describes the challenges of large teams working within a monolithic architecture.

Summarizing the benefits of moving to a services architecture:

Team autonomy: By switching to a services architecture (or the food truck approach), teams gain greater autonomy, much like how food truck staff can focus on their own recipes without interference.
Better and more granular scalability: Just like adding more food trucks can handle high demand without disrupting existing operations, services architecture allows for seamless system scaling.
Fault tolerance: If one service (or food truck) fails, others continue running, reducing overall downtime.
Limited coordination overlap: Each team (or truck) manages its operations independently, requiring minimal coordination.
Flexibility: Services architecture enables quick feature introduction or strategy shifts, just like a food truck can easily switch cuisines.
Efficient resource usage: Services architecture optimizes resource allocation, much like adding individual food trucks based on demand rather than overloading a single restaurant kitchen.

Final Thought

This analogy was meant to provide a straightforward explanation without getting lost in countless corner cases (of which we have more than a dozen for each section). While we acknowledge that both monolithic and service architectures have their strengths and weaknesses, two key factors particularly favored the services approach in our case: multiple teams developing the project independently and the unknown scale of the project.

A restaurant is often associated with quality, luxury, and a certain standard - just like a monolith, where the rules are the same for all teams, providing consistency. However, this centralized approach requires strong management and coordination, replacing autonomy with strict rules, monitoring, and validation. For example, a single tomato supplier affects all chefs - if there’s a problem with the delivery, the entire restaurant is impacted. Managing such dependencies and coordinating changes can become a significant operational cost.

On the other hand, a food festival thrives on scale and rapid product development by multiple independent teams, much like a services architecture. It allows for innovation, faster adaptation to trends, and greater resilience. However, it also requires more monitoring to ensure quality and remove underperforming food trucks.

A restaurant may not work at a festival, but in a controlled environment with a predictable number of customers, it offers distinct benefits. The real question is - can we still predict customer demand in an era driven by social media and constant change?

Hands-On with the Transactional Outbox Pattern using Python, RabbitMQ, and Elasticsearch

Fri, 28 Feb 2025 00:00:00 GMT

Introduction

The Transactional Outbox Pattern is widely discussed in theory - resources like microservices.io provide excellent overviews of the concept. However, finding practical, code-based examples can be challenging. This pattern tackles a common challenge in distributed transactions: ensuring atomicity between database operations and message publication. In distributed systems, a failure occurring between the successful storage of data and the dispatch of an event can lead to inconsistencies. In this blog post, we present a reference implementation of the pattern using Python, RabbitMQ, and Elasticsearch. Our goal is to provide a hands-on guide that bridges the gap between theory and real-world application, particularly for those building global-scale web applications and customer experience platforms.

Example

Our GitHub repository, transactional-outbox-pattern-example, contains a complete, runnable project that demonstrates the Transactional Outbox Pattern. In this example, we simulate a simple items catalog system. When a new item is added or updated, the change needs to be propagated to other parts of the system.

The example uses:

Python for the application logic,
RabbitMQ as the messaging broker,
Elasticsearch as the document database (instead of a traditional relational database).

The project is structured into three main components:

Producer: Saves data into Elasticsearch, including an outbox property.
Relay: Since Elasticsearch lacks built-in change data capture, a dedicated Relay component polls the database for unsent outbox entries and publishes them to RabbitMQ.
Consumer: Listens for messages from RabbitMQ and processes the updates accordingly.

The diagram illustrates how these components interact to ensure that every data change is eventually communicated through the message broker.

How It Works

Producer Persists Data with Outbox Metadata: The process begins when the Producer receives a request to create or update data. In a single atomic transaction, it writes the new data (document) into Elasticsearch, setting an outbox_sent flag to False.
Relay Polls for Unsent Outbox Entries: The Relay component continuously polls Elasticsearch for documents that remain unprocessed. Since Elasticsearch does not offer change data capture, this polling mechanism is essential.
Relay Publishes Messages to RabbitMQ: When the Relay finds an unprocessed document, it publishes the corresponding event to RabbitMQ.
Relay Updates the Outbox Record: After successfully sending the message (using publisher confirms), the Relay updates the outbox record to mark it as "sent". If an error occurs, the document remains in a pending state until the issue is resolved. If the database update fails, the outbox record is not marked as "sent", and the message is retried in the next polling cycle. This ensures at-least-once delivery semantics.
Consumer Processes the Message: The Consumer, subscribed to RabbitMQ, receives the published message. It processes the event accordingly - this could involve updating its local state, triggering business logic, or further propagating the change.

Ensuring Consistency in Failure Scenarios

To make this as practical as possible, our repository allows you to simulate various real-world failure scenarios that the Transactional Outbox Pattern is designed to handle.

One of the challenges in distributed systems is ensuring that once data is stored in the database, it is reliably sent to the message broker. The pattern we present today addresses this challenge effectively. In our implementation, the Relay service plays a crucial role. When data is published by the Producer, it is written to Elasticsearch along with an outbox property. If the message broker becomes unavailable, the Relay detects the issue and does not mark the outbox item as "sent". Once the broker is restored, the Relay picks up all pending items and publishes them to the Consumer via RabbitMQ. This mechanism ensures that no data is lost and that all updates eventually propagate throughout the system.

You can explore additional failure scenarios, such as Producer Failure, Relay Failure, Consumer Failure, or even a Data Store Outage, by following the instructions in the repository’s README.

Caveats and Considerations

While the Transactional Outbox Pattern is a powerful approach for maintaining consistency in distributed systems, it has its trade-offs. Here are a few considerations specific to this implementation:

Message Ordering: The pattern does not guarantee message ordering. If order is critical, additional mechanisms are required or you may want to consider alternatives to the Transactional Outbox - “Listen to Yourself” pattern.
Message Duplication: The pattern can lead to message duplication in some failure scenarios (due to at-least-once delivery). The Consumer should be idempotent to handle such cases.
Resource-Intensive Polling: The Relay component continuously polls Elasticsearch for unsent outbox entries. This can be resource-intensive and may not be suitable for high-throughput systems.
Delayed Propagation: The pattern introduces a delay between when data is written to the database and when it is processed by the Consumer. This delay can be significant depending on the polling interval, processing time, and the time it takes Elasticsearch to index the data. These factors combined may make the pattern unsuitable for real-time systems.

Summary

In this post, we explored a hands-on example of the Transactional Outbox Pattern implemented with Python, RabbitMQ, and Elasticsearch. We explained the data flow between the Producer, Relay, and Consumer, and demonstrated the pattern's behavior in a broker failure scenario. We also discussed trade-offs and considerations to keep in mind when implementing this pattern in your own systems. We encourage you to clone the repository, experiment with the code, and consider how this pattern can enhance the resilience of your own distributed systems.

Practical DevOps approach at Poznań Java User Group

Thu, 30 May 2024 00:00:00 GMT

After a short break from blogging, we are back with fresh insights from our recent presentation at the Poznań Java User Group on May 22nd. The topic? A practical approach to software lifecycle optimization for Java applications within complex systems. The session was well-received, underscoring the growing interest in integrating DevOps practices within the Java community.

Despite a late start (just after 8 p.m.) following a full working day, we were positively surprised by the audience's enthusiasm and involvement. The session concluded with an almost 20-minute Q&A, reflecting the high level of engagement. The presentation covered three critical areas: working effectively with complex systems in development environments, continuous integration, and a comprehensive approach to testing, including unit, integration, and end-to-end testing.

A key takeaway was the positive reception of DevOps principles by the Java developers in Poznań. The audience's insightful questions and active participation demonstrated a keen interest in adopting and refining these practices to enhance productivity and efficiency. During the presentation, we showcased several DevOps tools such as LensIDE, Kind, and GitHub Actions, which sparked significant interest and curiosity among attendees. Many questions revolved around the practical implementation of these tools and their integration into the software development lifecycle.

Although the presentation was not focused on the Java language itself, the feedback and interactions from this event have been incredibly encouraging. It’s clear that there is a vibrant and eager community ready to explore these topics. We are excited about the possibility of bringing this discussion to other forums and user groups, fostering a broader conversation around optimizing the software lifecycle for Java and other communities.

Stay tuned for more insights and updates as we continue to explore and share best practices in software development and DevOps. Thank you to the Poznań Java User Group for hosting such an inspiring event and to all the participants for their enthusiastic involvement. We look forward to more enlightening sessions in the future!

HandsOnArchitects celebrates its first anniversary

Tue, 12 Mar 2024 00:00:00 GMT

As HandsOnArchitects marks its first anniversary, we're excited to revisit our top-performing blog posts that have resonated with our amazing community.

As we celebrate our first anniversary, we're excited to revisit our top-performing blog posts that have resonated with our amazing community.

Simple Trick to Use Dynamic Values in Helm

Our guide on leveraging dynamic values in Helm charts has been a go-to resource for Kubernetes enthusiasts. This simple yet quite effective trick has helped readers streamline their Helm deployments, making it a valuable asset for those navigating the complexities of Kubernetes. Expect more insightful content on Kubernetes and Helm in the year ahead! Read more here.

Manage complexity for demonstration environments

Team dynamics are crucial in the tech world, and our post on building goal-oriented micro teams struck a chord with team leaders and developers alike. By diving into the principles of effective team structuring, we emphasize the importance of clear goals and streamlined communication. We will continue to explore the dynamics of high-performing teams in the next year. Read more here.

How to Manage Your Time as a Software Engineer - 5 Practical Tips and Tools

Time management is an ongoing challenge in the fast-paced realm of software development. Although it is one of the most recent posts, it has quickly become one of most popular among our readers. As we reflect on our first year, these time management strategies continue to be a beacon for developers seeking balance. There is a chance you will hear more about this topic from us, maybe in the form of presentations or workshops. Read more here.

In this blog, we touch on the concept of just-in-time knowledge sharing that we started to introduce in our teams in 2021. This post underscores the importance of timely knowledge transfer within development teams. This position resonated with our community's thirst for continuous learning. As we celebrate this milestone, we extend our gratitude to our community for making the first year of HandsOnArchitects truly exceptional. Read more here.

We look forward to another year of insightful content and meaningful connections. Thank you for being part of our journey!

Maciej & Tomek - hands-on software architects

How incorporating an integration layer in Adobe Experience Manager (AEM) project saved our clients money

Tue, 27 Feb 2024 00:00:00 GMT

Adobe Experience Manager (AEM) is more than a content management solution for building websites. It is a powerful tool that allows you to create, manage, and deliver digital experiences across different channels. However, when it comes to integrating AEM with other systems, it can be challenging to ensure seamless communication between different platforms. We learned that during work on several global brand projects in earlier stages of our careers.

The recurring challenge of integrating AEM with other systems, such as product information management (PIM), while also ensuring that experiences are delivered in a performant and scalable way to the end-users pushed us to create a dedicated integration layer that helped us to overcome these challenges. This layer was designed to ease delivering complex platforms consisting of AEM (or other CMS) and systems that were delivering content, data, or services. What is more, the concept of the integration layer is still valid today, and we believe that it can be beneficial for many AEM projects. This is why we decided to share our thoughts on this topic with you.

When you don't know what it is about - it's all about licensing

For those who work with Digital Experiences Platforms, it should be no surprise that a major concern was related to the financial aspect. AEM licenses (on-premises) come at a significant cost, and reducing the number of AEM Publish instances yields substantial savings. Decreasing the number of Publish instances from 10+ to 2 was a transformative step for both international and regional brands.

This transition was made possible by redefining the responsibilities of CMS-centric systems and introducing a dedicated integration layer. Instead of housing all business logic and integrations within a monolithic system, we shifted AEM's role to focus solely on content management (which is its core strength) and the generation of static pages, fragments, or templates. The heavy lifting of data processing, business logic, and integrations was shifted to a place that was designed to handle these tasks efficiently. This approach allowed us to reduce the number of AEM Publish instances, which resulted in significant cost savings.

Performance and scalability

Another challenge we faced was related to performance and scalability. AEM is a powerful tool, but it is not designed to handle high traffic or complex integrations. We learned it the hard way not once or twice. This was especially true during peak loads, such as marketing campaigns. Scaling AEM instances to handle these loads was not only costly but also inefficient.

The thin integration layer, a lightweight application written in a reactive paradigm, required significantly less CPU and RAM and was a much better fit for these tasks. Transferring CPU-intensive tasks, such as rendering pages with dynamic content like product details, ratings, prices, cart items, or personalized offers, to the integration layer allowed us to deliver performant and scalable experiences to the end-users. Not to mention reducing the infrastructure costs compared to traditional, CMS-centric architecture.

Feasibility and maintainability

The third money-saving factor for introducing an integration layer was related to the feasibility and maintainability of the solution. Developing and maintaining AEM projects is expensive. By moving part of the platform outside of AEM, we were able to execute projects with fewer AEM developers, who tend to be quite expensive.

After delivering the first projects it became clear that a great part of the integration patterns, such as circuit breakers, API caching techniques, API fallback strategies, sequenced API invocations, and more, can be abstracted and exposed via "no-code" configuration. This allowed our projects to address integration challenges, not only during happy-case scenarios but also in the event of failures, such as temporary unavailability of APIs. This approach made it easier to find developers with the right skill set and made it easier to maintain the solution for many years. The integration layer was also designed to be modular, which made it easier to add new features or integrations. This was crucial for brands that were constantly evolving and adding new systems to their digital experience platforms.

Composability and flexibility

There is one more cost-related reason to introduce an integration layer in AEM projects. While integrating data inside AEM is possible, it leads to a tightly coupled solution where replacing or adding new systems is difficult and costly.

The integration layer that we introduced was designed to bring composability into the digital experience platform. It enabled combining different integrations and features to create a custom solution that fits the specific needs of the brand and incorporate the tools that were already in use and people were used to. This was especially important for global brands that had different systems and requirements in different regions. The integration layer allowed us to create a custom solution for each region, while still maintaining a consistent user experience across different channels.

Additionally, the flexibility that came with the integration layer allowed developers to use the best tools for the job. To name a few examples from the projects we worked on, we integrated search engines using Apache Solr, personalization engines, Commerces, PIMs, and more. This flexibility allowed us to effectively solve specific problems, rather than being confined to Adobe's offerings.

Reusability

That brings us to the last point. Gathering experience from multiple projects, we realized that the integration layer was not only beneficial for the projects we were working on but also for other projects. This is why we decided to wrap the integration layer into a framework called Knot.x. Our solution was widely embraced across over 10 projects, including several platforms hosting more than 400 brand sites. We shared our insights and concepts at various industry events such as GeeCon, Devoxx, and numerous local tech gatherings. We open-sourced the framework and created a community around it.

Conclusion

The open-source project Knot.x acts as a pivotal integration layer that molds customer experiences across diverse channels by blending content and layout from Adobe Experience Manager with data sourced from various repositories such as Product Information Management (PIM), Commerce, or Customer Relationship Management (CRM) systems. This integration layer enabled Cognifide / Wunderman Thompson Technology to develop sophisticated personalized "dynamic" websites for global brands, thereby reshaping the architecture from AEM-centric to modular and composable. Knot.x revolutionized the creation of product listing pages, product details pages, search result displays, and the dynamic adjustment of related content components like tagged content fragments/pages, blog listings, sitemaps, and navigations. Its impact on server-side generated sites was groundbreaking from development, infrastructure, and cost perspectives. Finally, the reusability of the integration layer allowed us to deliver projects faster and with fewer resources, which resulted in significant cost savings for our clients.

Navigating Secrets Consumption in Kubernetes Pods

Tue, 13 Feb 2024 00:00:00 GMT

Kubernetes offers several avenues for consuming secrets within pods and containers, each tailored to accommodate diverse requirements. Understanding the distinctive characteristics of each option is vital for ensuring optimal functionality and resilience in your Kubernetes environment.

One crucial aspect to ponder is the adaptability of secrets to potential changes. Whether your secrets might undergo alterations, expand with new values (keys), or even vanish unexpectedly, it's imperative to evaluate how these transformations could affect your pods. This is particularly crucial when apply such mechanisms as Helm charts, which can dynamicaaly calculate secrets' hashes and update the pods with new values. Then pods should be recreated automatically to apply the new values.

The following paragraphs will explore the various secret consumption options in Kubernetes. All scenarios below are tested on a Kubernetes cluster running version 1.29.1.

Important: This article does not cover any security aspects of the secrets management in Kubernetes. It is important to remember that secrets are not encrypted by default and are stored in etcd in base64 encoded form.

Secret as Volume

We begin our exploration with the secret as a volume, a method that mounts the secret data entries as files within the pod's filesystem. This approach is particularly useful when your application requires access to the secret as a file, such as when reading a certificate or key file.

To illustrate the behavior of a pod with a secret mounted as a volume, consider the following scenario:

Create a pod with the secret mounted as a volume.

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: pod-with-secret-as-volume
  name: pod-with-secret-as-volume
spec:
  volumes:
  - name: keys
    secret:
      secretName: keys-secret
  containers:
  - image: nginx
    name: pod-with-secret-as-volume
    env:
    - name: PRIVATE_KEY
      value: /etc/keys/key.pem
    - name: PUBLIC_KEY
      value: /etc/keys/key.pub
    volumeMounts:
    - name: keys
      readOnly: true
      mountPath: "/etc/keys"

Check the status of the pod.

kubectl get pod pod-with-secret-as-volume

The pod won't be created until the secret is present.

This can be altered using the optional: true option. If an optional Secret doesn't exist, Kubernetes ignores it. (#restriction-secret-must-exist)

Create a secret with RSA keys

openssl genrsa -out key.pem 2048
openssl rsa -in key.pem -pubout -out key.pub
kubectl create secret generic keys-secret --from-file=key.pem --from-file=key.pub

Check the status of the pod.

kubectl get pod pod-with-secret-as-volume
kubectl -it exec pod-with-secret-as-volume -- ls -al /etc/keys
kubectl -it exec pod-with-secret-as-volume -- env | grep KEY

The expected result is:

pod-with-secret-as-volume   1/1     Running...
...
lrwxrwxrwx 1 root root   14 Feb 10 12:26 key.pem -> ..data/key.pem
lrwxrwxrwx 1 root root   14 Feb 10 12:26 key.pub -> ..data/key.pub
...
PRIVATE_KEY=/etc/keys/key.pem
PUBLIC_KEY=/etc/keys/key.pub

Delete the secret and observe the behavior of the pod.

kubectl delete secret keys-secret
kubectl get pod pod-with-secret-as-volume
kubectl -it exec pod-with-secret-as-volume -- env | grep KEY

The expected result is:

pod-with-secret-as-volume   1/1     Running...
...
PRIVATE_KEY=/etc/keys/key.pem
PUBLIC_KEY=/etc/keys/key.pub

When a secret is mounted as a volume, the pod retains the secret's data even if the secret is deleted. This means that the pod continues to function normally, oblivious to the absence of the secret. However, if the pod is deleted and recreated when the secret is not present, the pod will fail to start, as it will be unable to access the secret.

Check that files are still present in the pod.

kubectl -it exec pod-with-secret-as-volume -- cat /etc/keys/key.pub

The mounted files are still present in the pod.

When to Use Secret as Volume:

when your application requires access to the secret as a file
when you want to mount all the keys from the secret as files
when the new files should be automatically available in the containers after the new version of an application is deployed (POD is recreated automatically)
when the pod's container is aware of the secret's files paths

Secret as Environment Variables (envFrom)

Another method for consuming secrets in Kubernetes is by injecting them as environment variables using the envFrom field in the pod specification. This approach is suitable when your application requires access to the secret as environment variables.

Create a pod with the secret injected as environment variables.

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: pod-with-secret-env-from
  name: pod-with-secret-env-from
spec:
  containers:
  - image: nginx
    name: pod-with-secret-env-from
    envFrom:
    - secretRef:
        name: db-secret

Check the status of the pod.

kubectl get pod pod-with-secret-env-from

The pod will not be created until the secret is present.

Create a secret with a database credentials.

kubectl create secret generic db-secret --from-literal=DB_ADMIN=myadmin --from-literal=DB_PASSWORD=mypass

Check the pod's details.

kubectl get pod pod-with-secret-env-from
kubectl describe pod pod-with-secret-env-from | grep -A 2 "Environment"
kubectl exec -it pod-with-secret-env-from  -- env | grep DB_

The result should be:

pod-with-secret-env-from   1/1     Running
...
    Environment Variables from:
      db-secret   Secret  Optional: false
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lbrbb (ro)
...
DB_ADMIN=myadmin
DB_PASSWORD=mypass

Delete the secret and observe the behavior of the pod.

kubectl delete secret db-secret
kubectl get pod pod-with-secret-env-from
kubectl exec -it pod-with-secret-env-from  -- env | grep DB_

The result should be:

pod-with-secret-env-from   1/1     Running
...
DB_ADMIN=myadmin
DB_PASSWORD=mypass

As observed in this scenario, the pod is not recreated when the secret is deleted and environment variables loaded during the pod's creation still persist.

Recreate the secret with a new database password and a new key-value pair.

kubectl create secret generic db-secret \
--from-literal=DB_ADMIN=myadmin \
--from-literal=DB_PASSWORD=mynewpass \
--from-literal=DB_URL=mydb

View the environment variables in the pod.

kubectl exec -it pod-with-secret-env-from  -- env | grep DB_

The result should be:

DB_ADMIN=myadmin
DB_PASSWORD=mypass

Once the secret is created again and the pod is not recreated, then the environment variables are not refreshed.

When to Use Secret as Environment Variables (envFrom):

when your application requires access to the secret as environment variables
when we want to load all the key-value pairs from the secret as environment variables
when the new environment variables should be automatically available in the containers when pod is recreated (pod recreaction can be triggered automatically by using Helm charts, when secret's hash is changed)
when we expect the pod to be created even when the secret will not contain the required keys

Secret as Environment Variables (secretKeyRef)

The final method for consuming secrets in Kubernetes is by injecting them as environment variables using the secretKeyRef field in the pod specification. This approach is suitable when your application requires access to specific keys from the secret as environment variables.

Create a pod with the secret injected as environment variables using secretKeyRef.

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: pod-with-secret-env-secret-key-ref
  name: pod-with-secret-env-secret-key-ref
spec:
    containers:
    - image: nginx
      name: pod-with-secret-env-secret-key-ref
      env:
      - name: APP_PREFIX_DB_PASS
        valueFrom:
          secretKeyRef:
            name: db-secret
            key: DB_PASSWORD

Check the status of the pod.

kubectl get pod pod-with-secret-env-secret-key-ref

The pod will not be created until the secret is present.

Create a secret with a database credentials.

kubectl create secret generic db-secret --from-literal=DB_ADMIN=myadmin --from-literal=DB_PASSWORD=mypass

Check the status of the pod.

kubectl get pod pod-with-secret-env-secret-key-ref
kubectl exec -it pod-with-secret-env-secret-key-ref  -- env | grep APP_PREFIX_

The expected result is:

pod-with-secret-env-secret-key-ref   1/1     Running...
...
APP_PREFIX_DB_PASS=mypass

Delete the secret and observe the behavior of the pod.

kubectl delete secret db-secret
kubectl get pod pod-with-secret-env-secret-key-ref
kubectl exec -it pod-with-secret-env-secret-key-ref  -- env | grep APP_PREFIX

The expected result is:

pod-with-secret-env-secret-key-ref   1/1     Running...
...
APP_PREFIX_DB_PASS=mypass

Recreate the secret with a new database password.

kubectl create secret generic db-secret --from-literal=DB_ADMIN=myadmin --from-literal=DB_PASSWORD=mynewpass

View the environment variables in the pod.

kubectl exec -it pod-with-secret-env-secret-key-ref  -- env | grep APP_PREFIX

The result should be:

APP_PREFIX_DB_PASS=mypass

As observed in this scenario, the pod is not affected if the secret is deleted. Container environment variables are not updated when the secret is recreated.

When to Use Secret as Environment Variables (secretKeyRef):

when the secret's data key is different than the one required by the application
when your application requires access to specific keys from the secret as environment variables (more control over the keys, more secure)
when the pod should not be created if the specific key is not present in the secret during the pod's creation

The difference between the envFrom and secretKeyRef methods is that the envFrom method loads all the key-value pairs from the secret as environment variables, while the secretKeyRef method allows you to specify the keys from the secret that should be injected as environment variables. With the default settings, if the key is not present in the secret, the pod will not be created (during pod creation time).

Summary

The consumption of secrets in Kubernetes pods and containers in all scenarios above follows similar behavior patterns.

by default, the pod is not created until the secret is present
the pod keeps the secret's data even if the secret is deleted
the pod is not recreated when the secret is deleted

The key differences among the secret consumption options are:

the method of accessing the secret's data (as files or environment variables)
the level of control over the secret's keys

Summarizing, the secret as a volume is suitable when your application requires access to the secret as a file, while the secret as environment variables (envFrom and secretKeyRef) is suitable when your application requires access to the secret as environment variables. The envFrom method loads all the key-value pairs from the secret as environment variables, while the secretKeyRef method allows you to specify the keys from the secret that should be injected as environment variables.

How to manage your time as a software engineer - 5 practical tips and tools

Tue, 30 Jan 2024 00:00:00 GMT

In this post, we will share 5 practical tips and tools that help us manage time and keep our productivity high.

Spoiler alert: there will be no breakthroughs and no silver bullets in this post. You probably know or at least heard of the techniques and tools we will describe. In this post, we want to add a software engineering perspective to the time management topic and share some of the tools we use to manage our time.

1. Deep focus and Pomodoro techniques

Working as a software engineer taught us that there is no silver bullet when it comes to productivity. Some days, you have many small tasks you need to complete, and some days, you have a single big task you need to focus on.

Depending on the type of our daily plan, we use different techniques and tools to help us stay focused and productive.

When we have many small tasks that we need to complete (e.g., writing documentation, reviewing pull requests, red-green-refactor coding of already designed applications, or writing an architecture decision record), we use the Pomodoro technique. This technique uses a timer to break down work into intervals, traditionally 25 minutes in length (however, depending on the type of tasks, we make it up to 55 min per task in a row), separated by short breaks. It is a great way to stay focused and get many small things done. Pomodoro also supports pair programming sessions, helping us switch between driver and navigator roles at set intervals.

However, the Pomodoro technique is not the best way to get into the flow state (e.g., designing/coding a more demanding module, reverse engineering a system that needs refactoring, writing a blog post, preparing a presentation). These activities require "loading everything to your cache" to understand or present the problem. In these cases, we try to get into a deep focus, which is nothing else than giving all your brain power to a single but very complex task for a longer period. There is no rule on how long the deep focus session should last. It depends on the task, the person, the mood, and many other factors. However, there are two rules that we always try to follow: limit distractions and take breaks.

Tooling

For the Pomodoro technique, thanks to its simplicity, all you need is a timer. We use the following tools:
- Physical timer (our favorite one)
- Pomodoro Timer
- Be Focused (macOS app)
While the deep focus technique we tend to limit the tooling to the minimum. Read the following sections to learn more about it.

2. Turn off notifications

Limiting distractions helps not only during the deep focus but also during the Pomodoro sessions. Of course we don't recommend ignoring your teammates for the whole day. Instead, we recommend planning your day with a few hours of uninterrupted time. One of the techniques we use in the teams we work with is so-called focus hours. As a team, we agree on a few (usually 2-3) hours during the day when we don't plan meetings and do not expect quick responses to our messages. This approach helps especially with deep focus sessions. When we work most of the day with the Pomodoro technique, we plan a session for communication every 2-3 Pomodoros. This way we can focus on the task at hand and still be available for our teammates.

Tooling

macOS Do Not Disturb feature
Slack pause notifications
Calendar focus time event (various calendars support this feature, e.g. Google Calendar or Microsoft Teams)
There are also hardware solutions that can help you limit distractions and communicate your availability to your teammates. Just as an example, look at the busy light Amazon search results.

3. Take breaks and drink water

We know it might sound cliché, but we tried both approaches with or without regular breaks. We can tell you that taking breaks is a game-changer for keeping your productivity high throughout the day.

During the Pomodoro we usually take a break after every session. We use this time to stand up, stretch, change the desk's position (from sitting to standing or back), drink coffee, and talk to our teammates. This way we get back to the next session with a fresh mind, ready to switch context and start working on a new task.

During the deep focus sessions, breaks are less frequent but even more important (unless we want to end our working day as a walking zombie). We noticed that taking breaks that change the environment (e.g., going for a walk) helps us to get back to the task with a fresh mind, new ideas, and most often new perspective on the problem at hand (but this is a material for a separate blog post). Also, don't forget to have a glass of water on your desk. We know that might sound simple, but we noticed that staying hydrated helps us stay focused and productive for a longer time.

Tooling

A simple timer will do the job.

4. Context switching is expensive, avoid it and use the right tools

This section will focus on the tools and techniques, as we believe there is consensus that context switching is expensive and should be avoided.

One of the techniques that we use to avoid unwanted context-switching is to plan our day. We use the Getting Things Done (GTD) approach to plan for it (however, we modified it a bit to fit our needs). In short, GTD is a methodology for organizing tasks into actionable items (it uses of the divide-and-conquer approach). One of the greatest benefits we like in GTD is that it makes more space in our "RAM" (brain) for the actual work. We don't need to remember what we need to do. We just need to check our GTD board and pick the next task. This way, we can focus on the task at hand and not worry about forgetting something. Here's a sample board that we use for managing our tasks (and yes, this probably looks familiar to you):

Inbox	Now	Next	Waiting for	Done	Backlog/Someday

We recommend reading the GTD book to learn more about the approach, but in a nutshell, the board above works the following way:

Inbox is our "brain dump" place. Whenever we think of something we need to do, we put it in the inbox. We don't worry about the task's priority or complexity at this stage. Only two things matter: don't worry about this task any more and don't forget about it.
Now is where we put tasks that we want to complete during the next iteration (e.g., today, this week, sprint, etc.). We try to limit the number of tasks in this column so that we can focus on the most important ones.
Next is the place where we put candidates for the next iteration. Most often they need some refinement before we can move them to the Now column.
In Waiting for we put in here tasks that require some external action before we can complete them. For example, we put tasks that require a response from a customer or teammate in this column.
Done is the place where we put tasks that we completed. We use this column to track our progress and to have a sense of accomplishment. We noticed that cleaning this column gives us a lot of satisfaction ;).
Finally, Backlog/Someday is the place where we put tasks that we want to do in the future but not now. We use this column to keep our Inbox clean and to avoid forgetting about activities we want to do in the future. We review it regularly to make sure that we don't miss anything important but also to remove tasks that are not relevant anymore.

Now, some of you might ask how the GTD board complies with Jira or whatever tool your team uses to manage project. We treat GTD as a tool at a personal level. We often work on multiple projects at the same time, so we need a tool that helps us manage our time and tasks across and between projects. We use the GTD board to manage our time and tasks. We use the team's tooling to manage the project tasks. We found this approach to be the most effective for us. Also, ask yourself how many times there was an activity you had to do that was not worth putting in the team's Jira. In these cases, we have a go-to place to put these tasks.

Tooling

Trello - for managing GTD tasks. It is so far the best tool we found because of its automation capabilities.

5. When you don't know how, timebox it

The last one might also sound like a slogan (hey, we didn't promise breakthroughs, remember?), but we found it useful to keep tracking the time we spend on tasks. There are no rules on how long the timebox should be. The rule here is once the time is over, you need to decide. Either you:

know enough and finish the timebox,
you don't know enough, and you abort the task or find alternative solutions,
or you extend the timebox.

Either of the outcomes is fine. The important thing is that you don't spend too much time on a single task if you don't believe that you are making progress or it is important enough to spend more time on it. Sometimes, we decide to put the task back to Someday and get back to it later when we know more about the problem or when we have more time to spend on it.

Tooling

Yes, you guessed it right - a timer and a Trello displaying other tasks we need to complete are enough.

Conclusion

We presented five techniques and tools that help us thrive as software engineers and keep our productivity stable and at a high level. We hope you find them useful. If you have any questions or comments, please let us know using social media.

Multi-component Helm chart labeling and naming

Tue, 19 Dec 2023 00:00:00 GMT

In the world of Kubernetes, Helm charts are a powerful tool for deploying and managing applications. When you create a new chart using the helm create command, one of the artifacts generated is a set of helper functions in a file named templates/_helpers.tpl. These functions are designed with single-component charts in mind. But what happens when your application isn't so straightforward? What if your chart consists of multiple components, such as a frontend, backend, database, and cache?

The Helm documentation provides some guidance on this, suggesting the use of the app.kubernetes.io/component label for multi-component charts. However, it doesn't go into detail on how to implement these labels. In fact, a search for examples and best practices on this topic yields surprisingly few results. That's why I've decided to share my approach to this problem. In this post, I'll provide an example of a multi-component helper function for Helm charts, and explain how you can use it in your own projects.

Example helper methods

Going straight to the code, here's an example of a helper function that generates labels for a multi-component chart:

{{/*
Create a default fully qualified component name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
Usage:
{{ include "my-chart.component.fullname" (dict "componentName" "component-name" "context" $) }}
*/}}
{{- define "my-chart.component.fullname" -}}
{{- if .context.Values.fullnameOverride }}
{{- printf "%s-%s" .context.Values.fullnameOverride .componentName | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .context.Chart.Name .context.Values.nameOverride }}
{{- if contains $name .context.Release.Name }}
{{- printf "%s-%s" .context.Release.Name .componentName | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s-%s" .context.Release.Name $name .componentName | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}

{{/*
Metadata labels for chart component
Usage:
{{ include "my-chart.component.labels" (dict "componentName" "component-name" "context" $) }}
*/}}
{{- define "my-chart.component.labels" -}}
helm.sh/chart: {{ include "my-chart.chart" .context }}
{{ include "my-chart.component.selectorLabels" (dict "componentName" .componentName "context" .context) }}
{{- if .context.Chart.AppVersion }}
app.kubernetes.io/version: {{ .context.Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .context.Release.Service }}
{{- end }}

{{/*
Selector labels for chart component
Usage:
{{ include "my-chart.component.selectorLabels" (dict "componentName" "component-name" "context" $) }}
*/}}
{{- define "my-chart.component.selectorLabels" -}}
{{ include "my-chart.selectorLabels" .context }}
app.kubernetes.io/component: {{ .componentName }}
{{- end }}

Now, you can simply include this helper function in your chart's templates/_helpers.tpl file, and use it to generate labels for each component in your chart. For example, here's how you might use it to generate labels for a frontend component:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "my-chart.component.fullname" (dict "componentName" "frontend" "context" $) }}
  labels:
    {{- include "my-chart.component.labels" (dict "componentName" "frontend" "context" $) | nindent 4 }}
spec:
  selector:
    matchLabels:
      {{- include "my-chart.component.selectorLabels" (dict "componentName" "frontend" "context" $) | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "my-chart.component.labels" (dict "componentName" "frontend" "context" $) | nindent 8 }}
    spec:
      containers:
        - name: frontend
          image: nginx

Here is the output created by the helm template command:

---
# Source: my-chart/templates/frontend-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-chart-frontend
  labels:
    helm.sh/chart: my-chart-1.0.0
    app.kubernetes.io/name: my-chart
    app.kubernetes.io/instance: my-chart
    app.kubernetes.io/component: frontend
    app.kubernetes.io/version: "0.1.0"
    app.kubernetes.io/managed-by: Helm
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: my-chart
      app.kubernetes.io/instance: my-chart
      app.kubernetes.io/component: frontend
  template:
    metadata:
      labels:
        helm.sh/chart: my-chart-1.0.0
        app.kubernetes.io/name: my-chart
        app.kubernetes.io/instance: my-chart
        app.kubernetes.io/component: frontend
        app.kubernetes.io/version: "0.1.0"
        app.kubernetes.io/managed-by: Helm
    spec:
      containers:
        - name: frontend
          image: nginx

How it works

In short words, most of the work is done by the default helper function selectorLabels. Also, the default behavior of fullname function is preserved (when the Chart name is the same as the Release name, the Chart name is used as the full name). The only thing that is added is the componentName parameter, which is used to generate the app.kubernetes.io/component label and component's name postfix.

If you have any questions or suggestions, feel free to leave a comment on social media or send me an email.

Happy helming!

Manage complexity for demonstration environments

Wed, 08 Nov 2023 00:00:00 GMT

Dev/prod parity is one of the twelve-factor app principles. It clearly encourages us to “Keep development, staging, and production as similar as possible”. It is a very important rule and we try to follow it in most of the projects we work on. However, as for every rule, we can see exceptions (that of course are followed by tradeoffs).

In the past we used Vagrant to create development environments using Chef cookbooks to provision them with the same software as production. Nowaday, we use Kubernetes to create development environments and Helm charts to provision them with the same software as production (using the same Docker images). From a developer's perspective these approaches have disadvantages. They are slow to start, developer’s experience is mediocre at best (compared e.g. to live-reload some frameworks provide), and troubleshooting requires understanding the technologies that are running them (e.g. Chef or Kubernetes). What is often worse, automation often does not help these problems. It is very hard to learn higher-level automation frameworks without understanding the basics. Writing Chef cookbooks without basic knowledge of Linux and Ruby or creating Helm charts without understanding Docker and Kubernetes is like trying to learn to ride a bike for the first time by riding backwards on a rope over a cliff.

For product development teams this steep learning curve is acceptable as they will use the tools on a daily basis (even if they need to learn them) and they will benefit greatly from using them. However, for the “scouts” like architects, who want to evaluate the software without investing much time in running and maintaining the environment, running a full-blown environment might be a big obstacle. They want to try the software as soon as possible, without spending time on learning how to use all the shiny tools the development team uses.

This is where we asked ourselves: is dev/prod parity the best solution for demo environments?

What alternatives do we have? The answer to this question is not as straightforward as we would like it to be, so let’s work on an example - have a stateless web application that keeps its state in a database. Normally, (especially if this application is part of a more complex platform) we would containerize it and run with Kubernetes Deployment together with database Stateful Set. But modern cloud offers much simpler methods of running containerized applications without the need to learn Kubernetes, Chef or any Infrastructure as Code frameworks like Terraform. Services such as Google Cloud Run or AWS Beanstalk enable running containers with just a few clicks in their web consoles or single CLI command in serverless fashion.

Of course, we are left with the issue of the database, which needs persistence (which, in turn, is not a native functionality of serverless platforms). But also in this case we can rely on cloud services e.g., we can configure managed database service and use it with our container running in GCP Cloud Run (such setup is usually equally simple as using aforementioned container execution services).. In some cases, we can go a step further and completely abandon the external database (e.g. replacing it with in-memory database or local container filesystem).

As we mentioned at the beginning, this setup is not free of tradeoffs. Here are some of them:

no dev/prod parity - the production environment is not the same as the demonstration environment and sometimes requires small hacks and/or workarounds (e.g. in-memory database),
performance - the web application container resources are limited and the performance can be worse than on the production environment with e.g Kubernetes,
the data is not persistent - the web application container is serverless, so the data stored in the container is not persistent,
reliability - the number of application replicas is limited to single instance by lack of database sharing,
costs - running workloads in serverless environments such as Google Cloud Run or AWS Beanstalk is more expensive than running the same workload using similar resources in Kubernetes.

As you can see, the list of tradeoffs is not a short one. This is the cost of making things simpler and reducing cognitive load required to attend potential partners and customers for more advanced products. Enabling engineering teams with various skill sets to explore or do a quick proof of concept is a key factor, which makes these tradeoffs acceptable in a given context.

This concludes our demo/prod environments parity discussion. The twelve-factor app principles are the foundation for building web applications. However, they are not the golden rules, and as always, everything depends on the context.

Check websight.io blogpost “Introducing a streamlined approach to showcase WebSight CMS using GCP Cloud Run!“ to see a practical example of demo environments simplification.

Why content-centric architectures depend on caching

Tue, 17 Oct 2023 00:00:00 GMT

During my last presentation at AdaptTo() 2023, I mentioned reference content-centric architectures and how they leverage caching. In this blog post, I will expand on this topic.

Content-centric architectures

Content-centric architectures are a common approach to building web applications. The content is the most important part of the application, and a central repository stores it.

An example of a content-centric architecture is a platform built around the Content Management System (CMS). In such a case, the CMS is the single source of truth of all integrations work through it.

While this approach has advantages, it also has some drawbacks. In this blog, we will focus on one of them - dealing with a high volume of changing data like product information.

Content-centric caching

In the case of content-centric applications, caching usually happens between the CMS and the end user, and it is usually a read-through cache with a time-to-live (TTL) invalidation strategy.

For example, in the Adobe Experience Manager (AEM), the caching happens on the Dispatcher level (Apache Server module). The Dispatcher proxies the request to AEM, caches the responses, and serves them to the end users.

Invalidation rules for Dispatcher caching can be challenging. They depend on complex content relations such as PDP pages, listing pages, and .stat files. For more information, you can refer to the Dispatcher configuration guide.

Sample solutions

To make things more clear, let's work with some example. We will use a simple Product Details Page reference. The solution involves the CMS, which is the source of content and layouts of pages. Product Information Management (PIM) system is the source of product data presented on the product pages.

Below, you can find three examples of content-centric architectures that enable displaying product data on the Product Details Page.

Batch updates

Batch updates are one of the most popular PDP solutions. The CMS is the single source of truth responsible for rendering the PDPs. The CMS integrates with PIM via batch updates executed on a regular basis. CMS storage duplicates the data from PIM. That enables PDPs rendering with no additional callbacks to PIM (PIM is not taking part in end users' requests). The trade-off is that the information on PDP pages is outdated. That might or might not be a problem, depending on the requirement on how often PDPs should change.

This solution works well when the data in PIM changes rarely, and most of the page responses come from the cache. In such cases the batch updates can be scheduled to run once a day, or even less often. Another problem could be the volume of products. When using a cache warm-up, rendering several tens of thousands of PDPs could take up to a couple of hours. The real problem starts when the products need to be updated every time they change in PIM within a short period of time (e.g., 15 minutes) - which was the case in one of our projects in the past.

Backend integration

The concept behind this approach is straightforward: the CMS takes on the role of rendering PDPs. In this configuration, the CMS is directly linked with the Product Information Management (PIM) system, pulling data from the PIM at regular intervals, typically triggered by user requests reaching the PIM.

Directing user traffic to the PIM usually requires an additional caching layer (between CMS and PIM), introducing complexity into the platform. Furthermore, the activities of the PIM and CMS teams must be synchronized. For example, PIM updates may temporarily prevent CMS authors from publishing PDP pages because it will clear the cache. Techniques like cache warm-up and advanced cache invalidation are often employed in this context.

This method is most effective when dealing with a modest volume of products and when the PIM can effectively handle the incoming traffic.

Frontend integration

While CMS still maintains a central role, its responsibility for rendering PDPs has shifted. Instead of PDP rendering, the CMS provides templates (e.g., with ReactJS components), and the actual assembly of pages occurs within the user's browser. This transition results in the PIM system exposition to direct user traffic, which necessitates the implementation of additional caching mechanisms, typically achieved through an API Gateway.

In short words, a significant portion of product-related caching has been transferred to the API Gateway, making it a crucial part of the caching strategy.

It's important to note that assembling pages on the front end introduces trade-offs in User Experience (UX), especially when dealing with a substantial volume of products.

Summary

We have seen three different approaches to producing product detail pages with data from two different sources: CMS and PIM. Each of the solutions depends on caching.

In the context of caching, batch updates are the simplest to implement, but they might contain outdated data.
The backend integration serves the latest data but requires additional caching and synchronization between PIM and CMS.
Finally, the frontend integration with fresh data comes at the cost of an additional caching mechanism for PIM (e.g., API Gateway) and trade-offs on the UX (especially when we talk about a high volume of products).

Resources

We highly recommend D4Debugging blogpost Application Caching Strategies. It is a great read on caching strategies that nicely summarizes the topic.

For the presentation from AdaptTo() 2023, the part about referential solutions starts around ~6:40 minute.

Just-in-time knowlege sharing

Tue, 03 Oct 2023 00:00:00 GMT

In one of the previous blogs, we discussed the idea of a synchronous and non-blocking way for a code review. We pointed out knowledge sharing as an example of such a review model.

Now, we would like to elaborate on that one a little more.

How

In short words, Just-in-time (JIT) sessions are similar to meeting with your colleagues in front of a laptop and trying to introduce them to the work you have been working on lately. JIT sessions we do are almost always virtual. We started JIT sessions during the pandemic, and since then, the format has not changed thanks to its simplicity. The single session is up to 30 minutes long. The presenter is in control of who the session is for, what knowledge the attendees should have to gain the most out of the session, and the form (code sharing, brainstorming, Q&A, whiteboard drawings, etc.).

Why

When we kicked off the JITs for the first time, our motivation was very straightforward - we wanted to share our small findings that we usually talked about in the kitchen or hallway and get some feedback. We were looking for an informal meeting that does not require spending much time on preparations (in opposition to other virtual formats we had established at that moment).

In time, we found many more benefits and explored more applications of this format:

Short sessions are an alternative to more formal knowledge-sharing sessions.
The simple form encourages more developers to share and tell what things they do on a daily basis.
JIT meetings have proven to be helpful tool for gathering feedback early on your ideas.

Now, we would like to share a couple of recommendations for a quick start for the company and for presenters:

Quick-start - presenters

Time is of the essence. Don't wait for a special moment. Just give it a try. You don't need to prepare much. Talk about what you are currently doing in your project.
- Depending on the talk type, we prepare between 5-60 min. If this is a live coding session, 5 minutes is enough to have an agenda for things you want to show (it is usually good to have a note on the side to keep in timing). With ideas or summaries, when some graphics are required, we spend a bit more time (but never more than 1h).
Start small, e.g., just in your team. When you are ready, try your department or company meetings. A good time would be a 15-20 min slot right after your daily. Demo a solution you were working at to your team.
Make it natural and bring the tools you are used to work with. Do not spend much time on preparations. Treat this meeting as you would if you were to introduce your colleague to the project.
Don't bring PowerPoint presentations (we even forbid it at the beginning to distinguish this session from other more formal meetings).
Instead, prepare your IDE (make sure font will be visible to others), notes (e.g., in Markdown format), or some whiteboard tool (Miro works quite well).
Make it clear what knowledge is needed to benefit from the meeting and to whom you are addressing your speech. You may use hashtags like #docker-basics, #operational-bash, or #advanced-ReactJS.
Remember that as the presenter, you also want to benefit from the meeting. You share your knowledge, but in return, you want to get feedback and questions that will help you improve your idea.
Try to leave some space for discussion. We usually try to fit in 15 min to leave the place for conversation and questions (which sometimes leads to 1h+ meetings and vivid discussions, which help to build even better solutions).
Coupe of examples for a JIT:
- Just before starting some small project (e.g., tool/library that will help your project to do XYZ). Gather your idea, prepare a few points on what you want to achieve and what problems you want to solve.
- When you researched a new technology/library/framework. Share your insights and findings. Give the context of your project/problem you are looking for a solution for.
- After something went wrong or unexpected. Try to prepare some conclusions.

Quick-start - company

Start with a constant time window (e.g., every Tuesday at 14:00).
Do not let queue talks. Some simple (but not too many!) rules, like no slot reservation, are okay.
Create a dedicated channel (or use an existing one) for announcing the topic the morning or day before the session. The first one wins. If you find there are more people who want to present, create more slots.
Generally, one slot = one talk. Do not stack. Many JITs end up with a discussion (the prize for the presenter - they get questions and feedback there).
If you have other, more formal knowledge-sharing sessions, do not let PowerPoint presentations. Simple drawings, an IDE, a console, and a whiteboard are all what presenters need.
Recordings - this is a tricky one. As far as we observed, it is a controversial area. On the one hand, recording allows people who were unable to attend the meeting to watch the recording at any time and ask questions. On the other hand, we have observed that the recording function prevents many people from asking questions and participating in the discussion.
Generally, we do not recommend imposing the form of the meeting. Let the presenters decide which format they feel most comfortable with. A good balance could be recording sessions and taking questions/feedback after recording has stopped. Another way is to leave notes and key takeaways in some common source like Confluence or GitHub repo README.
Do not kid yourself. This format, like many others, must have several precursors to be successful. Help them organise meetings and encourage people to participate. If it succeeds, it will be like a good YT video - it will start getting thumbs-up on its own.

TL;DR

Just-in-time (JIT) knowledge-sharing sessions are low-effort meetings that are easy to adopt in the team or companywide. The essence is to share your recent (even daily) work or findings with a wider group of engineers. Don't bring any presentation. Most of the sessions are live screen sharing. There is also an additional prize for the presenters - discussion, feedback, and questions that might use the company's brainpower to create better solutions.

How to sleep well at night when your production CMS is down

Tue, 19 Sep 2023 00:00:00 GMT

The summer break is over, and it's time to get back to regular blogging. We start this season with a conference appearance.

Next week, together with Marta Cukierman, we will be speaking at AdaptTo() 2023 - "Europe's leading AEM Developer Conference".

The title of our presentation: Integrate, Grow, Repeat: How Websight Real-Time DXP Provides a Way to Break Out of the Content-Centric Model might be a little misleading compared to the title of this post. It is the main reason I have decided to write this short update, and disenchant the mysterious title.

During the session, we will showcase architectures we used to integrate with other data sources, such as Product Information Management (PIM) with CMS in the center.

We will also talk about an alternative approach to integrate with other systems - the one that actually lets you sleep well at night when your production CMS is down.

And the session will not end with just talking.

Make sure to join us on Monday, 25th September 2023 at 17:35 CEST.

Maciej

Recording

Edit: You can watch the recording here:

Zero downtime website migration using cert-manager and free certificate

Wed, 05 Jul 2023 00:00:00 GMT

In this brief note, we will share a simple way to migrate a website that uses the Let's Encrypt certificate from one Kubernetes cluster to another with no downtime.

Starting point

We start with a Kubernetes cluster that contains the current website in the USA region.

The important thing to notice is the cert-manager installed via Helm configured on the cluster with the Let's Encrypt ACME production issuer.

The issuer is triggered by the ingress-shim annotation cert-manager.io/issuer:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  # ...
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/issuer: "handsonarchitects-letsencrypt-prod"
  labels:
  # ...
spec:
  tls:
    - hosts:
        - www.handsonarchitects.com
        - handsonarchitects.com
      secretName: prod-handsonarchitects-tls
  rules:
    - host: www.handsonarchitects.com
      # ....
    - host: handsonarchitects.com
      # ....

As a consequence of cert-manager work, the CertificateRequest resource is created and finally, the Certificate and Secret are obtained and saved.

Migration

Now, we will deploy the application on the newly created cluster in Europe. All that we need to install on the fresh instance are Nginx Deployment, Service and Ingress.

To support SSL we can copy the prod-handsonarchitects-tls certificate Secret from the old cluster:

# us-east1 cluster
kubectl get secrets prod-handsonarchitects-tls -o yaml > prod-handsonarchitects-tls.yaml

and apply it in the Ingress's tls.secretName of the new cluster:

# eu-central2 cluster
kubectl apply -f prod-handsonarchitects-tls.yaml

Note that at this moment the cluster should not request certificates automatically as the DNS does not point to it. Producing requests by the issuer might quickly drain the Let's Encrypt certificate requests quota.

At this moment, the website should be working with the certificate and you may confirm it by adjusting your /etc/hosts with:

213.97.130.13 www.handsonarchitects.com

Tip: Certificate renewal

Let's encrypt certificates are usually issued for 90 days. If you need to renew the certificate before the migration there is a very nice tool for you: https://cert-manager.io/docs/reference/cmctl/#renew

It enables to manually trigger the certificate renewal by running this simple command: mctl renew prod-handsonarchitects-tls

DNS switch

Now we can safely switch the DNS to point the new cluster, as we have a working website with the certificate.

Remember, that DNS switching might take up to 24h to propagate. But this is not a problem, since you have mirrored clusters with your application.

Cert-manager installation

The last step is installing the cert-manager on EU cluser:

helm upgrade --install cert-manager cert-manager \
  --repo https://charts.jetstack.io \
  --set installCRDs=true \
  --namespace cert-manager \
  --create-namespace

At this moment, the copied Secret will match the Certificate and all should work fine.

Also, you may remove the old cluster safely.

Summary

In this short note, we migrated the static website from one Kubernetes cluster to another, while using the free Let's Encrypt SSL certificate and the cert-manager. We were able to keep the website online for the whole process of the migration.

Building goal-oriented micro teams based on WebSight

Wed, 21 Jun 2023 00:00:00 GMT

WebSight is a unique place where software engineers can play with modern architecture, edge technologies, and fully agile development techniques (experimental development approach, trunk-based development, TDD, pair programming etc.) adequate to their experiences in small, independent teams.

We believe that it is possible to create an extraordinary and unique workplace by building self-organizing small, independent teams that communicate through well-defined contracts and responsibilities.

In this article, we want to explain how we launched a completely new website using WebSight DXP & CMS in two weeks.

Goals

Here are our main goals:

launch new website www.websight.io in two weeks
prepare materials, use cases and basic principles for WebSight DXP
test a new project, the WebSite DXP event streaming platform, in a production environment on Google Cloud Platform

Here are our assumptions for all teams:

one overarching goal for all teams (both business and engineering)
self-managed and fully isolated teams define their internal plans and criteria for success
simultaneous development and content entry
teams of engineers continuously create production-ready versions and independently deploy them to production

Teams

Both business and engineering teams:

WebSight CMS team (mainly senior back-end engineers) used a simplified SDLC process to deliver a new publishing engine with limited code review, highly functional test automation (e2e)
www.websight.io CMS component development team managed by Project Manager using standard GIT flow, limited quality gates, with dedicated Release Manager role and manual testing phases
designer who created UI designs for all sites
content authors/information architects focused on site structure and content entry
infrastructure team (DevOps) using techniques such as trunk-based development, infrastructure as code, ephemeral environments, deployment automation, etc.
hands-on architects using test-driven development, pair development, and experimental development approaches to ensure platform performance and high availability
CTO who kept priorities unchanged within two weeks...

The diagram below presents how all teams worked together:

It is worth noting that there are no managerial roles at the team integration level in the above arrangement. Thanks to properly defined contracts, frequent releases, independent staging/production deployments and limited, results-oriented communication, we were able to quickly combine the results of the teams' work and move very quickly.

Summary

The ability to build small, agile teams with limited bureaucracy is crucial for projects that aspire to conquer the market. Building such open work culture is based on strong trust and high professionalism. The transformation is ongoing, but we can already see the effects.

Simple trick to use dynamic values in Helm

Mon, 19 Jun 2023 00:00:00 GMT

This time we will show you a simple trick we found useful when setting up local or ephemeral k8s environments with a valid domain that supports multiple Ingress entries.

Problem

In the current project we are working on we use Kubernetes to run developer's and production environments to have the closest possible configuration of both. Sometimes developers use local environments, in other cases (when bigger computing power is required) we use ephemeral environments (created for a short time for development) created on-demand in Google Cloud.

Our system consists of multiple services and some of them are exposed via the Ingress. To manage the application we use Helm. To have a near-to-production setup we decided to use nip.io, the "Dead simple wildcard DNS for any IP Address".

At some point, we ended with a couple of entries in the Helm's values.yaml such as:

serviceA:
  ingress:
    host: a-127-0-0-1.nip.io
# ...
serviceB:
  ingress:
    host: b-127-0-0-1.nip.io
# ...
serviceC:
  ingress:
    host: c-127-0-0-1.nip.io
# ...
serviceD:
  ingress:
    host: d-127-0-0-1.nip.io

For the local environment, it was working just fine. However, when we started to use ephemeral environments in GCP it required overriding values in a couple of places, to match the GKE cluster's external IP address.

   helm install \
      --set serviceA.ingress.host=a-34-117-173-194.nip.io \
      --set serviceB.ingress.host=b-34-117-173-194.nip.io \
      --set serviceC.ingress.host=c-34-117-173-194.nip.io \
      --set serviceD.ingress.host=d-34-117-173-194.nip.io \
      my-app . -f values.yaml

It would be much simpler to pass the ingress IP as a single value and use it in all the configurations. Until now, Helm does not support dynamic values in values.yaml with some good reasoning for that:

Solution

Fortunately, we were able to use one of Helm's tips and tricks to do the work in that case.

The tpl function allows developers to evaluate strings as templates inside a template. This is useful to pass a template string as a value to a chart or render external configuration files. Syntax: {{ tpl TEMPLATE_STRING VALUES }}

Now, the Ingress config in our project's chart could use the tpl function:

apiVersion: networking.k8s.io/v1
kind: Ingress
# ...
spec:
  rules:
  - host: {{ tpl .Values.serviceA.ingress.host . }}
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: serviceA
            port:
              number: 80

And we were able to leverage the chart's default clusterIp like:

clusterIp: 127-0-0-1
# ...
serviceA:
  ingress:
    host: a-{{ .Values.clusterIp }}.nip.io
# ...
serviceB:
  ingress:
    host: b-{{ .Values.clusterIp }}.nip.io
# ...
serviceC:
  ingress:
    host: c-{{ .Values.clusterIp }}.nip.io
# ...
serviceD:
  ingress:
    host: d-{{ .Values.clusterIp }}.nip.io

With this config, overriding the local 127.0.0.1 with the cluster's external IP is as easy as running:

   helm install \
      --set clusterIp=34-117-173-198 \
      my-app . -f values.yaml

Notice, this config will work well if you don't use dynamic values in the service.ingress.host.

Summary

In this short guide, we combined:

nip.io, the "Dead simple wildcard DNS for any IP Address",
Helm's tpl function.

The result was a simpler way to configure Kubernetes external cluster's IP in multiple Ingerss entries.

Apache Pulsar client prefetch mechanism trap

Sat, 20 May 2023 15:40:32 GMT

When adding new technologies to our portfolio, we spend some time experimenting with its core features to "eliminate guesswork". The more advanced the tool, the more time we need to discover its behaviours. Working with Apache Pulsar, a distributed streaming platform, we explore event-steaming architecture whose complexity requires us to verify empirically many aspects.

This is exactly how we approached the next challenge, which was to read only the latest messages for a given key from the topic. In our case, these are published pages, of which only the latest versions are processed. A similar situation may apply to stock exchange data, where we are only interested in the current prices of listed companies.

The mechanism that "cleans up" the old messages per key is called topic compaction. Once a topic compaction completes, all consumers assigned with active subscriptions should receive only the latest messages per key.

With such a scenario, our test should:

produce the appropriate number of messages with the same keys and send them to the topic
invoke the compacting mechanism on the topic
wait for the compaction to finish
read the next message on the consumer

Our code looks like:

Producer<String> producer = newProducer(topicName);

producer.newMessage().value("1").key("A").send();
producer.newMessage().value("2").key("A").send();
producer.newMessage().value("3").key("A").send();
producer.newMessage().value("4").key("B").send();

Consumer<String> consumer = newConsumer(topicName, "unit-test")
    .readCompacted(Boolean.TRUE)
    .subscribe();

waitUntilCompacted(topicName);

assertNextMessageValue("3", consumer);
assertNextMessageValue("4", consumer);

We expected that only the compacted data would be available to the consumer after compaction was complete. But the tests result was different:

Expecting: 3
Consumed: 190:0:-1:0 value: 1

Our hypothesis formed based on Pulsar documentation seemed wrong. We immediately remembered the Richard Feynman words:

If your guess disagrees with experiment, then it (your guess) is wrong!

We spent more time experimenting with our failing test. We quickly discovered that the Pulsar client has a prefetch mechanism that downloads events from the topic in larger batches. By default, the client downloads 1000 messages at a time, and each time receive() is called, we get a message previously downloaded from the topic. Although after compacting the topic, the messages may already be "stale".

To limit this behavior we can configure the receiverQueueSize(int receiverQueueSize) option on the consumer:

The consumer receive queue controls how many messages can be accumulated by the Consumer before the application calls Consumer.receive(). Using a higher value could potentially increase the consumer throughput at the expense of bigger memory utilization.

For the purpose of testing and learning about Pulsar, we recommend setting this value to 1. This will ensure that only the first event is "stale".

Producer<String> producer = newProducer(topicName);

producer.newMessage().value("1").key("A").send();
producer.newMessage().value("2").key("A").send();
producer.newMessage().value("3").key("A").send();
producer.newMessage().value("4").key("B").send();

Consumer<String> consumer = newConsumer(topicName, "unit-test")
    .readCompacted(Boolean.TRUE)
    .receiverQueueSize(1)
    .subscribe();

waitUntilCompacted(topicName);

assertNextMessageValue("1", consumer);
assertNextMessageValue("3", consumer);
assertNextMessageValue("4", consumer);

Our modified test case was successful. We experimentally proved our expectations regarding the compaction mechanism.

More details here.

Fixing Docker Desktop when all other methods fail

Tue, 25 Apr 2023 00:00:00 GMT

Docker Desktop is a great tool that helps to save a lot of time during local development. However, it is not perfect and can break down in ways that are hard to restore, especially when you are using an M1 Apple silicon processor...

Today we would like to share a short script that helps with uninstalling Docker Desktop from MacBooks. The tool offers several options to troubleshoot. Unfortunately, there are times, when everything fails (including purging data, resetting to factory defaults, or even uninstalling), especially after quite extensive container deployments, uninstalls, scale-ups, and especially down (don't try to scale your k8s pods to 0).

We use the following script to uninstall the Docker Desktop for Mac when the options mentioned above fail. It needs the administrator privileges to run:

#!/usr/bin/env bash
set -x

sudo rm -Rf /Applications/Docker.app
sudo rm -f /usr/local/bin/docker
sudo rm -f /usr/local/bin/docker-machine
sudo rm -f /usr/local/bin/com.docker.cli
sudo rm -f /usr/local/bin/docker-compose
sudo rm -f /usr/local/bin/docker-compose-v1
sudo rm -f /usr/local/bin/docker-credential-desktop
sudo rm -f /usr/local/bin/docker-credential-ecr-login
sudo rm -f /usr/local/bin/docker-credential-osxkeychain
sudo rm -f /usr/local/bin/hub-tool
sudo rm -f /usr/local/bin/hyperkit
sudo rm -f /usr/local/bin/kubectl.docker
sudo rm -f /usr/local/bin/vpnkit
sudo rm -Rf ~/.docker
sudo rm -Rf ~/Library/Containers/com.docker.docker
sudo rm -Rf ~/Library/Application\ Support/Docker\ Desktop
sudo rm -Rf ~/Library/Group\ Containers/group.com.docker
sudo rm -f ~/Library/HTTPStorages/com.docker.docker.binarycookies
sudo rm -f /Library/PrivilegedHelperTools/com.docker.vmnetd
sudo rm -f /Library/LaunchDaemons/com.docker.vmnetd.plist
sudo rm -Rf ~/Library/Logs/Docker\ Desktop
sudo rm -Rf /usr/local/lib/docker
sudo rm -f ~/Library/Preferences/com.docker.docker.plist
sudo rm -Rf ~/Library/Saved\ Application\ State/com.electron.docker-frontend.savedState
sudo rm -f ~/Library/Preferences/com.electron.docker-frontend.plist

sudo rm -rf ~/Library/Group\ Containers/group.com.docker/pki/
sudo rm -rf ~/.kube

It is a product of Reddit, StackOverflow and other sources research and we use it 2-3 times a month.

After removing Docker Desktop using the script above, install the new application as usually. We hope it will help some of you.

Docker Apple M1 | M2 architecture support in Apache Software Foundation projects

Sat, 22 Apr 2023 17:40:32 GMT

With new M1 / M2 MacBooks from Apple, the "new" arm64 architecture began to dominate the laptop market. In this short article, we describe how developers could help the open-source ecosystem to adopt their Docker images to the new platforms.

Developers with M1 / M2 MacBooks using Docker need support for the new arm64 architecture processors. For project maintainers, it means that instead of a single amd64 platform image, they must build several once for amd64 / arm64 platforms. It requires adopting their current CI tools to build multiple images and expose them within a single Docker manifest (under a specific tag name). This process is time-consuming and requires much effort, especially in open-source world.

With Robert Munteanu from Adobe, we adopted one of Apache Software Foundation's (ASF) projects, Apache Sling Starter, to the new multi-arch support procedure.

It required the following steps:

Create an Apache JIRA issue asking for M1 Docker images support e.g. SLING-11714
Agree with the ASF project owner to use GitHub Actions workflows that have the support for docker buildx:

I would suggest using GitHub Actions, there you can use the official suite of docker actions to build multiplatform images with little need for custom scripting [1].

source: https://lists.apache.org/thread/xd3p8nom68gkgjmjpgfylh0vyx260s5o

Provide PRs with a working solution (in our case, we use Docker Maven Plugin to manage buildx container that builds and pushes images):
The Apache project owner needs to raise an issue to Apache Software Foundation infrastructure team to enable GH actions secrets for pushing docker hub images e.g INFRA-23983.
Once all ready, PRs can be merged.

Summarizing, adding support for the arm64 platform in open-source projects is not a complex work. Satisfaction from being the ASF contributor is worth spending a few hours after your everyday routines.

How Technology Radar helps harness the tech stack at different levels

Mon, 17 Apr 2023 00:00:00 GMT

Technology Radar is a phenomenon that helps visualize the technology stack. By the technology or the technology stack, we mean not only frameworks, languages, and platforms, but also techniques, skills and everything that might be the subject of work required to deliver the software. In the Technology Radar, the particular technologies are called blips.

Let's dive into how the tool may help to harness the tech stack at various levels of business starting with some of the most popular definitions:

An opinionated guide to technology frontiers

source: https://www.thoughtworks.com/radar

A technology radar is a way of observing the market for new innovations and technologies and gather information about them in a consistent style, relate and evaluate them on behalf of the own business.

source: https://www.igi-global.com/dictionary/technology-roadmap-living-labs/29542

technical engineering documentation that works as a portfolio for you to visualize an institution's technologies and methodologies, monitor new trends, and identify which tools need to be removed, either because the technology is legacy.

source: https://dzone.com/articles/what-is-tech-radar-why-teams-need-to-have-one

Context is the king

Radars get more traction recently in the technologists community. As members of the community, we find it very useful to represent the technology stack at different levels, e.g. personal development, project or product technology stack, company-specific framework choices, or even industry trends.

Depending on the level, radar may represent different categories (quadrants) or priorities (rings). A specific situation we use the radar to visualize creates the context.

Dimensions

As mentioned above, depending on the context, different metrics could be visualized on the radar. That means, that one company can have more than one radar (which often is the case). It is simple to imagine that one radar could represent the technology portfolio of the company, while another could represent the skills the company is looking for in the new candidates and the current engineering team.

However, there are some common qualities, the radar helps to understand no matter the context:

usefulness - how useful we find the technology in the context, e.g.:
- on the personal radar (one that helps to organize what you know and what you want to learn) it may point to what we want to learn next,
- on the company radar, we may seek information on what conferences/workshops we should send employees to,
confidence - how much do we trust (but also know) the tech, e.g.:
- the project's radar shows that one technology is the default choice for some kind of platform services (e.g. we choose Quarkus over Spring when implementing backend services),
- on the personal radar, placing the blip may represent our seniority in the technology or technique (e.g. we are just getting to know Apache Kafka, but we could lead the training in the middle of the night on adopting Continous Integration),
trends (or adoption) - how widely adopted is the technology, how much did we invest in it already, is it gaining or losing popularity, e.g.:
- the digital experience industry radar may show an increasing trend in measuring their applications footprint (e.g. clients ask more and more about the "Green Cloud"),
- the personal radar may represent the technology we know well, but we don't want to invest in/work with anymore (e.g. we are experts in Maven, but it let us down so many times, that we will start new projects only using Gradle).

Summary

In this article, we showcased some scenarios of working with Technology Radars. The plural form is not a coincidence. We work with the tool for several years now. We find it extremely useful because of its simplicity and depth at the same time (the separate concern is putting it together - but that's the case for another, much longer article). We are using it for personal development and adopting it in projects and companies we are working with. It is a great tool, that does not require much introduction while it shows complex things (the technology stack) in a simple and accessible form.

Code quality dimensions

Thu, 13 Apr 2023 22:40:32 GMT

The aforementioned post Thinking About Code Review was triggered by one of the Twitter threads on CI/CD vs PRs.

The especially interesting part was "Dimensions of Variability" where two dimensions of inspecting the code were mentioned: blocking/non-blocking vs synchronous/asynchronous. The original author's conclusion was:

This categorization looks really accurate. It took us only a couple of minutes of discussion to bring the Ship / Show / Ask strategy.

When we tried to put ship/show/ask techniques into the "Dimensions of Variability" we got the following result:

The quarter worth a few words is the non-blocking and synchronous one. We treat show as a kind of knowledge sharing (with additional benefits like feedback on the solution/code). On the one hand, it does not block proceeding with taking the changes further. On the other hand, the session that is valuable for both sides (the audience and the presenter) is synchronous. It requires gathering in one space (physical or virtual) at one time to show and get feedback. Of course, the form could be asynchronous too. However, our previous knowledge-sharing sessions tell us that the greatest chance for the presenter to get feedback is during or just after the show.

Exclusive subscription messages order with Apache Pulsar

Sun, 12 Mar 2023 22:40:32 GMT

Pulsar message order

Be aware that negative acknowledgments on ordered subscription types, such as Exclusive, Failover and Key_Shared, might cause failed messages being sent to consumers out of the original order (source).

In this blog post, we try to explain the above sentence with two examples:

messages are positively acknowledged and processed in sequential order (the sequence identifier is pre-assigned by the producer)
some messages are acknowledged negatively and re-delivered with some delay.

Positive Acknowledge

The following diagram illustrates how the consumer receives and acknowledges the messages sent by the producer. Our consumer subscribes to the unit test topic with the subscription type Exclusive and the subscription initial position Earliest:

Consumer<String> consumer = client.newConsumer(Schema.STRING)
    .topic(topicName)
    .subscriptionName("unit-test")
    .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
    .subscriptionType(SubscriptionType.Exclusive)
    .subscribe();

Then it receives unacknowledged messages (the set of unacknowledged messages is called backlog) one by one and acknowledges/confirms them:

Message<String> m1 = consumer.receive();
consumer.acknowledge(m1);
Message<String> m2 = consumer.receive();
consumer.acknowledge(m2);
Message<String> m3 = consumer.receive();
consumer.acknowledge(m3);

The order of received messages reflects the initial (sequential) order: messages A, B and C:

assertEquals("A", m1.getValue());
assertEquals("B", m2.getValue());
assertEquals("C", m3.getValue());

Each receive command moves the subscription cursor to the next position (from right to left). Finally, the cursor is positioned after the C message:

All acknowledged messages are marked for deletion. Read more about conditions when messages are deleted from the topic.

Negative Acknowledge

The order of messages changes when messages are acknowledged negatively. In the following scenario, the message m1 is negatively acknowledged and rescheduled for re-delivery in the future:

producer.send("A");
producer.send("B");

Message<String> m1 = consumer.receive();
consumer.negativeAcknowledge(m1);
Message<String> m2 = consumer.receive();
consumer.acknowledge(m2);
producer.send("C");
Message<String> m3 = consumer.receive();
consumer.acknowledge(m3);
Message<String> m4 = consumer.receive();
consumer.acknowledge(m4);

The figure below shows the above scenario:

The order of received messages for our example is:

assertEquals("A", m1.getValue());
assertEquals("B", m2.getValue());
assertEquals("C", m3.getValue());
assertEquals("A", m4.getValue());
assertEquals(m1.getSequenceId(), m4.getSequenceId());

The consumer receives message "A" with some delay (second attempt). We can make sure it's the same message by checking its sequence ID.

Summary

Apache Pulsar delivers messages in sequential order as long as the consumer positively acknowledges them. The message acknowledgment mechanism allows consumers to ask Pulsar to redeliver the message. The consequence is that our processing logic cannot rely on the order of the messages. When the order of messages is critical (publish/unpublish), our algorithm needs to use additional strategies, e.g. event time / sequence id, to properly handle this situation.

References

Unit tests

Hands-on architects Blog RSS Feed

Agent Harness vs Work Harness: One Term, Two Systems, Two Owners

Two Harnesses, Nested

Inside the Work Harness: Guides and Sensors

The Steering Loop

Driving the Agent vs Running the Work Engine

Walking the Delivery Cluster

Human-Agent Interaction

Workflow & Process

Reliability & Operations

It's Engineering, Not Model Magic

Start Building Your Work Harness

TL;DR

Team Topologies in the AI Era: Where Does the Cognitive Load Go?

An AI harness is a cognitive-load lever - which is what platform teams do

Stream-aligned: the Product Engineer question

Enabling: consultants who teach calibrated trust, then leave

Complicated-subsystem: the AI-resistant boundary

The Stage 3→4 jump: load relocated, not removed

The Software House angle: the harness is the IP

What we can't settle yet

TL;DR

When the Weakest Dimension Wins: Architectural Governance at Scale

The productivity measurement trap

The Harness Model and the multi-dimensional nature of maturity

Why AI platforms amplify the governance gap

The knowledge base - making the platform model machine-readable

Building the knowledge base - the ingestion harness

The decision harness - from knowledge base to governed decisions

Context engineering at the architectural level - the Stage 3 → Stage 4 jump

15-minute exercise

Conclusion

TL;DR

E2E test harness for AI agents: better specs, not smarter models

Tests were green. The agent was testing the wrong thing.

Why E2E became the place the harness took shape

Domain aggregates: where the spec actually starts

The SCN format - and where it sits in the spec lineage

The OAuth bug, in mechanics

The artifact loop, plus what's upstream and downstream of it

The data: 15 suites, 162 steps, and the shape of the curve

The Docker image and harness build: making prod-like the default

One production-shaped image, curated overrides only

Per-suite docker-compose and the suite_containers fixture

JVM DNS cache and HTTP keep-alive

Specialized agents, skills, and commands across the SDLC

Traps when moving the team along the harness maturity stages

For Monday - write one SCN, sketch one aggregate

The Harness Model - AI Engineering Maturity Matrix, Q1 2026

The Harness Thesis

The Maturity Model

Foundation: Context and Team

Governance: Security and Architecture

Delivery: Interaction, Workflow, Reliability & Operations

Outcomes & Learning: Verification, Knowledge, Planning

Brownfield Reality Check

Mixed Maturity and Regression

Where Are You? A Conversation Starter

Foundation

Governance

Delivery

Outcomes & Learning

Start Where You Are

Compare notes with us

How to Start using AI in Software Development Today

Stop Waiting, Start Experimenting

AI Is an Amplifier, Not a Replacement

Context Engineering: The Skill That Actually Matters

When to Slow Down

Your First AI Experiment (Pick One)

For Engineers: Explore a Codebase You Already Know

For Tech Leads: Break a Feature into Backlog Items

For Architects: Brainstorm Architectural Options

Start Today, Iterate Tomorrow

AI Toolset for Software Architects (Q1 2026)

How the Architect's Role Continues to Evolve with AI

What Changed Since Q3 2025

AI Tool Categories - Q1 2026

1. Searching & Learning

2. Brainstorming & Ideation

Per-suite docker-compose and the `suite_containers` fixture